Real Numbered Analog Classification for Neural Networks

Andrew_Smith: <p>Hi everyone,</p>
<p>I am fairly new to Pytorch and I’m currently working on a project that needs to perform classification on images. However, it’s not a binary classification.</p>
<p>The outputs of the neural network are real numbers. For instance the classification I’m looking the neural network to provide is as such:</p>
<ol>
<li>Reads Image</li>
<li>says that the image has attribute A at a value of 1200 and another attribute B at a value of 8.</li>
</ol>
<p>The image data that’s fed into this neural net usually has a value range of 1200 - 1800 and another attribute range of 4 - 20. The main goal is to train the network to give an estimated analog value based on the image fed into it.</p>
<p>The classes provided by Pytorch seem to favor binary classification, such as the Dataloader and ImageFolder classes. However, I cannot provide the class folder structure that is often used since I do not have set binary classes:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/115b65e153fb5081aee92ba03896d96a8227c61e.png" alt="51%20AM" data-base62-sha1="2txW2Vc2e9SB5Zbw2F04FOHJkDk" width="445" height="243"></p>
<p>I want to discuss what possible structures of neural networks can support a problem like this. Including activation functions and what Pytorch might have to offer.</p>
<p>Are there implementations out there that I haven’t found on the web yet? What type of loss functions and optimizers in Pytorch may best serve this type of problem? Should I even be considering neural networks to perform this task in the first place?</p>
<p>To be clear I’m not asking for code or a complete solution. Just a discussion on developing a neural net that can work with this analog system.</p>
<p>Also if I need to clarify anything I will be happy to.</p>

KFrank: <p>Hi Andrew!</p>
<aside class="quote no-group">
<blockquote>
<p>I am fairly new to Pytorch and I’m currently working on a project that needs to perform classification on images. However, it’s not a binary classification.</p>
<p>The outputs of the neural network are real numbers.</p>
</blockquote>
</aside>
<p>A quick note on terminology:  To me, at least, the term<br>
“classification” has the connotation of assigning input<br>
samples to discrete classes, that is, labelling them with<br>
integer class labels.  So I might call what you’re talking<br>
about something like <em>continuous value estimation</em>, rather<br>
than “classification.”</p>
<aside class="quote no-group">
<blockquote>
<p>For instance the classification I’m looking the neural network to provide is as such:</p>
<p>Reads Image<br>
says that the image has attribute A at a value of 1200 and another attribute B at a value of 8.<br>
The image data that’s fed into this neural net usually has a value range of 1200 - 1800 and another attribute range of 4 - 20. The main goal is to train the network to give an estimated analog value based on the image fed into it.</p>
</blockquote>
</aside>
<p>This seems perfectly reasonable to attempt with a pytorch<br>
neural network (depending on the details of your problem).</p>
<aside class="quote no-group">
<blockquote>
<p>The classes provided by Pytorch seem to favor binary classification, such as the Dataloader and ImageFolder classes.</p>
</blockquote>
</aside>
<p>This is an overstatement.  Pytorch certainly supports binary<br>
classification, but it supports lots of other things, too.</p>
<aside class="quote no-group">
<blockquote>
<p>However, I cannot provide the class folder structure that is often used since I do not have set binary classes:</p>
</blockquote>
</aside>
<p>Yes, while you can certainly encode binary class labels (and<br>
multi-class labels) in a directory structure, you can’t very<br>
well encode real-number attributes that way.</p>
<p>So the straightforward approach would be to have a data file<br>
that has the real-number attributes in it.  It could be as<br>
simple as containing a sequence of (pairs of) real numbers,<br>
with the understanding that the order in which you read in<br>
your images will (somehow) match the order of the attribute<br>
values in your file.  Or your data file could have a character<br>
string for the filename of each image file, followed by two<br>
numbers for the values of the two attributes.</p>
<aside class="quote no-group">
<blockquote>
<p>I want to discuss what possible structures of neural networks can support a problem like this. Including activation functions and what Pytorch might have to offer.</p>
</blockquote>
</aside>
<p>Since you’re analyzing images your network would probably<br>
start with a couple of convolutional layers, and then switch<br>
over to a couple of fully-connected layers, with the last<br>
layer having two (floating-point number) outputs as the<br>
predictions for your two attributes.  (There are many<br>
additional techniques that could improve the performance<br>
of your network – things like <code>nn.MaxPool2d</code> or<br>
<code>nn.Dropout2d</code> – but I would recommend starting simple<br>
and adding refinements as the need arises.  Or start with<br>
a prepackaged network.)</p>
<p><em>Rectified linear units</em> (<code>nn.ReLU</code>) are commonly used as<br>
(non-linear) activation functions.</p>
<aside class="quote no-group">
<blockquote>
<p>What type of loss functions and optimizers in Pytorch may best serve this type of problem?</p>
</blockquote>
</aside>
<p>For your loss function, you would probably simply use the squared<br>
error between your predictions and actuals (<code>nn.MSELoss</code>).</p>
<p>That is:</p>
<pre><code class="lang-auto">loss = (predicted_A - actual_A)^2 + (predicted_B - actual_B)^2.
</code></pre>
<p>You might want to weight the two terms so they have comparable<br>
size lest you preferentially optimize for accuracy in A at the<br>
cost of B.</p>
<p>I always like to start simple, so I would recommend starting<br>
with plain-vanilla (stochastic) gradient descent (<code>optim.SGD</code>).<br>
You can also add <em>momentum</em> to gradient descent (supported by<br>
<code>optim.SGD</code>), and pytorch offers more sophisticated optimizers,<br>
such as the commonly-used <code>optim.Adam</code> (“adaptive moments”).</p>
<aside class="quote no-group">
<blockquote>
<p>Are there implementations out there that I haven’t found on the web yet?</p>
</blockquote>
</aside>
<p>Undoubtedly!  There are lots of (semi-) prepackaged architectures<br>
out there for image processing, but I don’t know much about them<br>
and don’t have any concrete suggestions.  But if you give us more<br>
detail about your specific problem (and post some sample images!),<br>
some of the experts here will likely have good advice.</p>
<aside class="quote no-group">
<blockquote>
<p>Should I even be considering neural networks to perform this task in the first place?</p>
</blockquote>
</aside>
<p>Likely yes, but it really depends on your problem.  (So post<br>
details and sample images!)  For example, if the two attributes<br>
you are trying to “predict” are the intensity and and saturation<br>
averaged over the pixels in a color image, you’d be much better<br>
off just calculating them directly.</p>
<p>Best.</p>
<p>K. Frank</p>

Andrew_Smith: <p>Hi K. Frank,</p>
<p>Thank you for taking the time to answer my questions! I appreciate it!</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="50657">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I might call what you’re talking<br>
about something like <em>continuous value estimation</em> , rather<br>
than “classification.”</p>
</blockquote>
</aside>
<p>I think that’s a better name for this project. I’ll use that from now on.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="50657">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>So the straightforward approach would be to have a data file<br>
that has the real-number attributes in it. It could be as<br>
simple as containing a sequence of (pairs of) real numbers,<br>
with the understanding that the order in which you read in<br>
your images will (somehow) match the order of the attribute<br>
values in your file. Or your data file could have a character<br>
string for the filename of each image file, followed by two<br>
numbers for the values of the two attributes.</p>
</blockquote>
</aside>
<p>Yes, I have this setup nicely. I can parse a png file name for the attributes and make them into labels.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="50657">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>For your loss function, you would probably simply use the squared<br>
error between your predictions and actuals ( <code>nn.MSELoss</code> ).</p>
</blockquote>
</aside>
<p>I agree, after looking through the loss functions this seems like a good choice for real value estimation.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="50657">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You might want to weight the two terms so they have comparable<br>
size lest you preferentially optimize for accuracy in A at the<br>
cost of B.</p>
</blockquote>
</aside>
<p>This is very true. Since the values ranges for A and B are much different I would need to go through this. I haven’t looked into how to weight the MSE loss function yet, but is it possible that you could give some guidance on this?</p>
<p>Lastly, I wanted to make batched sets of data. I found that Pytorch does this automatically through the Dataloader class via a parameter. I am confused on whether this Dataloader still needs the dataset parameter structured in the hierarchical fashion previously mentioned. If that’s the case would there be anything else to quickly batch data or should I code my own function?</p>
<p>Thanks again,<br>
Andrew</p>

KFrank: <p>Hello Andrew!</p>
<p>First, a general comment:</p>
<p>We will be able to give you advice that is more<br>
likely to be useful to you if you give us some<br>
concrete detail about the problem you are working<br>
on.</p>
<p>How big are your images?  How many will you be<br>
training on?  What do they look like?  What is<br>
the typical distribution of your two attributes?<br>
What is the conceptual meaning of your attributes?</p>
<aside class="quote no-group">
<blockquote>
<p>I haven’t looked into how to weight the MSE loss function yet, but is it possible that you could give some guidance on this?</p>
</blockquote>
</aside>
<p>I don’t believe that <code>nn.MSELoss</code> has a built-in<br>
way to include these relative weights.  There are<br>
a number of straightforward approaches to including<br>
such weights.</p>
<p>Myself, I would just write my own loss function,<br>
something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

# define weighted loss function

def wtSqErr (pred, targ, wts):
    return (wts * (pred - targ)**2).mean()

# construct some sample data
# use a batch size of 10
# y_targ are the actuals, y_pred are the predictions
# which, for this example, are the actuals plus noise

y_targ = torch.tensor ([1000.0, 2.0]) * torch.randn (10, 2) + torch.tensor ([2000.0, 3.0])
y_targ
y_pred = y_targ + torch.tensor ([100.0, 0.15]) * torch.randn (10, 2)
y_pred.requires_grad = True
y_pred

# set up the weights for the loss

wtA = 1.0 / 1000.0**2
wtB = 1.0 / 2.0**2

wtAB = torch.tensor ([wtA, wtB])
wtAB

# calculate loss

loss = wtSqErr (y_pred, y_targ, wtAB)
loss

# show that autograd works

print (y_pred.grad)
loss.backward()
print (y_pred.grad)

</code></pre>
<p>Here is the output of the above script:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt;
&gt;&gt;&gt; # define weighted loss function
...
&gt;&gt;&gt; def wtSqErr (pred, targ, wts):
...     return (wts * (pred - targ)**2).mean()
...
&gt;&gt;&gt; # construct some sample data
... # use a batch size of 10
... # y_targ are the actuals, y_pred are the predictions
... # which, for this example, are the actuals plus noise
...
&gt;&gt;&gt; y_targ = torch.tensor ([1000.0, 2.0]) * torch.randn (10, 2) + torch.tensor ([2000.0, 3.0])
&gt;&gt;&gt; y_targ
tensor([[2.3612e+03, 2.4401e+00],
        [2.2880e+03, 7.0144e+00],
        [1.2435e+02, 4.6300e+00],
        [3.7007e+03, 1.4845e+00],
        [1.7911e+03, 2.0490e+00],
        [2.6058e+03, 2.2381e+00],
        [6.1270e+02, 2.1648e+00],
        [6.9680e+02, 1.4656e+00],
        [1.2903e+03, 2.8559e+00],
        [1.6696e+03, 5.5197e+00]])
&gt;&gt;&gt; y_pred = y_targ + torch.tensor ([100.0, 0.15]) * torch.randn (10, 2)
&gt;&gt;&gt; y_pred.requires_grad = True
&gt;&gt;&gt; y_pred
tensor([[2.6065e+03, 2.5329e+00],
        [2.3034e+03, 7.2111e+00],
        [2.9170e+02, 4.4378e+00],
        [3.7426e+03, 1.4848e+00],
        [1.8188e+03, 2.2676e+00],
        [2.8676e+03, 2.3148e+00],
        [5.6415e+02, 2.1441e+00],
        [7.7348e+02, 1.4650e+00],
        [1.2437e+03, 2.9639e+00],
        [1.5545e+03, 5.5731e+00]], requires_grad=True)
&gt;&gt;&gt;
&gt;&gt;&gt; # set up the weights for the loss
...
&gt;&gt;&gt; wtA = 1.0 / 1000.0**2
&gt;&gt;&gt; wtB = 1.0 / 2.0**2
&gt;&gt;&gt;
&gt;&gt;&gt; wtAB = torch.tensor ([wtA, wtB])
&gt;&gt;&gt; wtAB
tensor([1.0000e-06, 2.5000e-01])
&gt;&gt;&gt;
&gt;&gt;&gt; # calculate loss
...
&gt;&gt;&gt; loss = wtSqErr (y_pred, y_targ, wtAB)
&gt;&gt;&gt; loss
tensor(0.0111, grad_fn=&lt;MeanBackward1&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; # show that autograd works
...
&gt;&gt;&gt; print (y_pred.grad)
None
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; print (y_pred.grad)
tensor([[ 2.4533e-05,  2.3200e-03],
        [ 1.5451e-06,  4.9169e-03],
        [ 1.6735e-05, -4.8059e-03],
        [ 4.1961e-06,  8.9884e-06],
        [ 2.7656e-06,  5.4653e-03],
        [ 2.6174e-05,  1.9158e-03],
        [-4.8546e-06, -5.1618e-04],
        [ 7.6680e-06, -1.5900e-05],
        [-4.6640e-06,  2.7002e-03],
        [-1.1507e-05,  1.3345e-03]])
</code></pre>
<p>Note that if you use pytorch tensors to do your<br>
calculations, autograd will work for you without<br>
your having to do anything special.</p>
<aside class="quote no-group">
<blockquote>
<p>Lastly, I wanted to make batched sets of data.</p>
</blockquote>
</aside>
<p>Pytorch naturally works with batches.  The first<br>
index of your input data, predictions, and target<br>
data tensors is the index that indexes over samples<br>
in the batch.</p>
<p>In the above example, you can understand the<br>
generated data to be a batch of 10 samples.</p>
<p>(In fact, pytorch loss functions <em>require</em> batches,<br>
even if the batch size is only 1.  Following the<br>
above example, for batch-size = 1, a “batch” of,<br>
say, predictions would then have a shape of<br>
<code>y_pred.shape = torch.Size ([1, 2]).</code>)</p>
<p>If your training data fits in memory (We don’t<br>
know – you’ve told us nothing concrete about<br>
your problem.), you can read it all into one<br>
tensor, and then use “indexing” or “slicing”<br>
to get your batches.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

all_data = torch.ones (10, 3)
first_batch_of_two = all_data[0:2]
second_batch_of_two = all_data[2:4]

</code></pre>
<p>Doing this does not create new tensors with their<br>
own storage – it just sets up a view into the<br>
existing <code>all_data</code> tensor.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Andrew_Smith: <p>Hi K. Frank,</p>
<p>Thank you so much for the detailed response! I have a prototype somewhat working now from your help!<br>
I am going to work on getting the loss down faster and get better prediction values sometime soon.</p>
<p>Thank you again,<br>
Andrew Smith</p>