How would i train my model properly?

Tom_ska: <p>Hello.<br>
I am trying too train a CNN.<br>
The CNN can be found here:</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/amadeuzou/vrn-pytorch/blob/master/vrn_unguided.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/amadeuzou/vrn-pytorch/blob/master/vrn_unguided.py" target="_blank" rel="noopener nofollow ugc">amadeuzou/vrn-pytorch/blob/master/vrn_unguided.py</a></h4>
<pre><code class="lang-py">import torch
import torch.nn as nn
import torch.legacy.nn as lnn

from functools import reduce
from torch.autograd import Variable

class LambdaBase(nn.Sequential):
    def __init__(self, fn, *args):
        super(LambdaBase, self).__init__(*args)
        self.lambda_func = fn

    def forward_prepare(self, input):
        output = []
        for module in self._modules.values():
            output.append(module(input))
        return output if output else input

class Lambda(LambdaBase):
    def forward(self, input):
</code></pre>

  This file has been truncated. <a href="https://github.com/amadeuzou/vrn-pytorch/blob/master/vrn_unguided.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
however i dont get it too really run.<br>
I want too use CrossEntropyLoss as my lossfunction. Since this is the function which is used in the model here too.<br>
The reason why i want too retrain is, that i need to mod it a little bit for a bigger Model and a bigger picture.<br>
(384 * 384 image → 384 * 384 * 384 Voxel Space)<br>
I have only 1 Dataset, this means one image and the corresponding voxel-model.<br>
I know that this never ever is going too work-out for usability but it has too be done, just too proove that it is possible. (Huge Overfitting is okay, basically i wanted too run it on 5 Datasets later and make the CNN overfit for those 5 images)</p>
<p>Now i wrote some script which should train my CNN. However, some Loss-Functions let it run but the loss never gets better or worse. And some others, like CrossEntropy dont work at all.<br>
Can someone help me and say what i am doing wrong?<br>
My Script for the CNN is following:</p>
<pre><code class="lang-auto">VRN = vrn.vrn_unguided
VRN.load_state_dict(torch.load('vrn-unguided-selfmade.pth'))
# telling the model it will be trained
VRN.train()
# lossfct to calculate error while training -&gt; sigmoid cross entropy loss same as in original vrn.
lossfct = torch.nn.modules.loss.CrossEntropyLoss()
# optimizer settings
optimizer = torch.optim.SGD(VRN.parameters(), lr=0.1)

# between the #-lines is just data-loading 
#################################################################
# file format .txt
voxel_array = np.zeros((1, 384, 384, 384), dtype=float)
file = open("sample_1.txt","r")
for line in file:
    if not "#" in line:
        x = line.partition(" ")[0]
        y = line.partition(" ")[2].partition(" ")[0]
        z = line.partition(" ")[2].partition(" ")[2].partition(" ")[0]
        voxel_array[0, int(x), int(y), int(z)] = 1
file.close()

###############
# here we load the image too train with
### get landmarks from test image
image_file = 'rsz_pic_zu_sample.jpg'
image = cv.imread(image_file)
try:
    image_height, image_width, image_depth = image.shape
except:
    print('cannot load image:', image_file)
# vrn output
inp = torch.from_numpy(image.transpose((2, 0, 1))).float().unsqueeze_(0)

###############################################################

voxel_array = torch.from_numpy(voxel_array)


# the actual training takes place here. Right now it just iterates 20 times over the same data.
for epoch in range(20):
    print(epoch)
    VRN.train()
    predicted_model = VRN(inp)[-1].data.cpu()
    vol = predicted_model.numpy()

    optimizer.zero_grad()
    vol = torch.from_numpy(vol)

    print(vol.shape)
    print(voxel_array.shape)
    loss = lossfct(vol, voxel_array)
    print(loss)
    loss.backward()
    optimizer.step()

torch.save(VRN.state_dict(), "vrn-unguided-selfmade.pth")
print("Done, saved CNN")


</code></pre>

ptrblck: <p>You are detaching the output tensor in:</p>
<pre><code class="lang-python">predicted_model = VRN(inp)[-1].data.cpu()
</code></pre>
<p>by calling the <code>.data</code> attribute (which you shouldn’t call in any way, as it might yield unwanted side effects) and also in:</p>
<pre><code class="lang-python">vol = predicted_model.numpy()
</code></pre>
<p>since you are leaving PyTorch and use numpy (Autograd won’t be able to track these operations).</p>
<p>I would assume that:</p>
<pre><code class="lang-python">    loss = lossfct(vol, voxel_array)
    loss.backward()
</code></pre>
<p>would raise an error, since you are using numpy arrays, but you might also have re-wrapped the array into a tensor to get rid of the error.</p>
<p>To fix this issue: don’t use the <code>.data</code> attribute and, if possible, use PyTorch operations instead of numpy.<br>
However, if you need to use numpy specific operations, you would have to implement the backward manually via a custom <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html"><code>autograd.Function</code></a>.</p>

Tom_ska: <p>Hello,<br>
so i changed some code in the script and get the following error:</p>
<pre><code class="lang-auto">RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4
</code></pre>
<p>however my training script is like this (the rest is unchanged):</p>
<pre><code class="lang-auto">for epoch in range(20):
    print(epoch)
    VRN.train()
    predicted_model = VRN(inp)[-1].cpu()
    optimizer.zero_grad()
    print(predicted_model.shape)
    print(voxel_array.shape)
    loss = lossfct(predicted_model, voxel_array.squeeze(1))
    print(loss)
    loss.backward()
    optimizer.step()
</code></pre>
<p>the print from the shapes says both are the same shape such as</p>
<pre><code class="lang-auto">torch.Size([1, 384, 384, 384])
</code></pre>
<p>however i understand that pytorch needs it too be [384,384,384]<br>
i change the code too</p>
<pre><code class="lang-auto"> loss = lossfct(predicted_model.squeeze(0), voxel_array.squeeze(0))
</code></pre>
<p>so that we loose the 1 dimension. However this leads too</p>
<pre><code class="lang-auto">ValueError: Expected target size (384, 384), got torch.Size([384, 384, 384])
</code></pre>
<p>which i dont understand because in any combination the function does not work.<br>
It basically should just compute the difference between the amount of the 1 in the arrays, i think.<br>
And by minimising this, the CNN gets better in predicting the real Voxel-Volume.<br>
Or am i wrong here?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects model outputs in the shape <code>[batch_size, nb_classes, *additional_dims]</code>, while the target should contain class indices in the range <code>[0, nb_classes-1]</code> and have the shape <code>[batch_size, *additional_dims]</code>. Assuming you are working on a multi-class segmentation use case with volumes, your output might be <code>[batch_size, nb_classes, depth, height, width]</code>, while the target should then be <code>[batch_size, depth, height, width]</code>. Based on the error message I assume that the target shape might be wrong.</p>

Tom_ska: <p>Since i didnt not manage too make it run with CrossEntropyLoss i simply just tryed another lossfct.<br>
With that i trained the vrn-network 5000 times with a single picture and the model.<br>
However i thought regardless of lossfct etc. i can always overfit a model too fit it.<br>
This is my script :</p>
<pre><code class="lang-auto">VRN = vrn.vrn_unguided
VRN.load_state_dict(torch.load('vrn-unguided-selfmade.pth'))
# telling the model it will be trained
VRN.train()
# lossfct to calculate error while training -&gt; sigmoid cross entropy loss  in original vrn.
lossfct = torch.nn.modules.loss.MSELoss()
# optimizer settings
optimizer = torch.optim.SGD(VRN.parameters(), lr=1)

# this is for data loading. it is checked and loading correctly
voxel_array = np.zeros((1, 384, 384, 384), dtype=float)
file = open("sample_1.txt","r")
for line in file:
    if not "#" in line:
        x = line.partition(" ")[0]
        y = line.partition(" ")[2].partition(" ")[0]
        z = line.partition(" ")[2].partition(" ")[2].partition(" ")[0]
        voxel_array[0, int(x), int(y), int(z)] = 1
file.close()
# image to model. 
image_file = 'rsz_pic_zu_sample.jpg'
image = cv.imread(image_file)
try:
    image_height, image_width, image_depth = image.shape
except:
    print('cannot load image:', image_file)
# vrn output
inp = torch.from_numpy(image.transpose((2, 0, 1))).float().unsqueeze_(0)

voxel_array = torch.from_numpy(voxel_array)
voxel_array = voxel_array.float()

for epoch in range(10):
    print(epoch)
    VRN.train()
    predicted_model = VRN(inp)[-1].cpu()
    # vol = predicted_model.numpy()
    # vol = predicted_model.numpy()

    optimizer.zero_grad()

    # voxel_array = voxel_array.squeeze(0)

    # vol = torch.from_numpy(vol)
    loss = lossfct(predicted_model.squeeze(0), voxel_array.squeeze(0))
    print(loss)
    loss.backward()
    optimizer.step()

torch.save(VRN.state_dict(), "vrn-unguided-selfmade.pth")
print("Done, saved CNN")
</code></pre>
<p>However after running this 5k times.<br>
If i test the network with the exact same picture as in training, the result is way different then the training-model.<br>
This is the model trained on<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4daaa24acf93621808a50016b91b1732bc34bb1b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4daaa24acf93621808a50016b91b1732bc34bb1b" title="model.PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4daaa24acf93621808a50016b91b1732bc34bb1b_2_411x500.jpeg" alt="model.PNG" data-base62-sha1="b54lv2A4SFlnod0J8GfgYxmTC8b" width="411" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4daaa24acf93621808a50016b91b1732bc34bb1b_2_411x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4daaa24acf93621808a50016b91b1732bc34bb1b.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4daaa24acf93621808a50016b91b1732bc34bb1b.jpeg 2x" data-dominant-color="6A6A6A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model.PNG</span><span class="informations">458×556 27.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and this is the output after 5k training on the same data<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc63b9c32841ae39eaf5ccbaaca747f8e248eca6.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc63b9c32841ae39eaf5ccbaaca747f8e248eca6" title="Bild_2021-04-14_224917"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc63b9c32841ae39eaf5ccbaaca747f8e248eca6_2_431x500.png" alt="Bild_2021-04-14_224917" data-base62-sha1="ta742sO9BnjsUrtOGM71ySUwM4K" width="431" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc63b9c32841ae39eaf5ccbaaca747f8e248eca6_2_431x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc63b9c32841ae39eaf5ccbaaca747f8e248eca6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc63b9c32841ae39eaf5ccbaaca747f8e248eca6.png 2x" data-dominant-color="666666"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Bild_2021-04-14_224917</span><span class="informations">532×616 24.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can it be, that after so many iterations, even the train data gives such a different result?</p>

Tom_ska: <p>I changed my script because i think it will run if i just use the right functions etc. However the script looks like this now :</p>
<pre><code class="lang-auto">lossfct = torch.nn.modules.loss.CrossEntropyLoss()
# optimizer settings
optimizer = torch.optim.RMSprop(VRN.parameters(), lr=0.1)

voxel_array = torch.from_numpy(voxel_array)
voxel_array = voxel_array.long()

for epoch in range(10):
    print(epoch)
    VRN.train()
    predicted_model = VRN.forward(inp)
    # vol = predicted_model.numpy()
    # vol = predicted_model.numpy()
    # voxel_array = voxel_array.squeeze(0)
    # vol = torch.from_numpy(vol)
    loss = lossfct(predicted_model[0], voxel_array)
    print(loss)
    loss.backward()
    optimizer.step()
</code></pre>
<p>but it produces the following error :</p>
<pre><code class="lang-auto">RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4
</code></pre>
<p>However if i Squeeze, i get Tensor Erros. How could i fix this?<br>
My Shape is [1, 384, 384, 384] for Target and the same for model-output. However, i dont know how too Add the C as in CrossEntropyLoss mentioned since i got like “unlimited” classes theoretically. So it simply does not make that much sense for me. But without the C i always get a Batch_size or Dimension Error</p>

ptrblck: <aside class="quote no-group" data-username="Tom_ska" data-post="7" data-topic="117804">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom_ska/48/30701_2.png" class="avatar"> Tom_ska:</div>
<blockquote>
<p>My Shape is [1, 384, 384, 384] for Target and the same for model-output.</p>
</blockquote>
</aside>
<p>This is the issue, so please take another look at my previous post, which explains the expected shapes.<br>
The issue is that the model output should contain a class dimension in <code>dim1</code>, while the target should not.</p>

Tom_ska: <p>Thanks! But shouldnt a simple unsqueeze(0) solve this?<br>
However, i got this error now:</p>
<pre><code class="lang-auto">AttributeError: 'list' object has no attribute 'log_softmax'
</code></pre>
<p>after i changed</p>
<pre><code class="lang-auto">predicted_model[0] to predicted_model
</code></pre>
<p>which should basically do the same as unsqueeze.</p>

ptrblck: <aside class="quote no-group" data-username="Tom_ska" data-post="9" data-topic="117804">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom_ska/48/30701_2.png" class="avatar"> Tom_ska:</div>
<blockquote>
<p>Thanks! But shouldnt a simple unsqueeze(0) solve this?</p>
</blockquote>
</aside>
<p>No, since this would remove the batch dimension and you would end up with the wrong shapes again.</p>
<p>Based on the second error, it seems that <code>predicted_model</code> is a <code>list</code> and not a tensor, so you won’t be able to use <code>log_softmax</code> on it.<br>
Also, note that indexing a tensor as <code>x[0]</code> is only equal to <code>squeeze(0)</code>, if the size of the dimension is 1.</p>

Tom_ska: <p>Yes silly from me.</p>
<pre><code class="lang-auto">print(predicted_model)
</code></pre>
<p>showed that is just a 1-item list of tensor, so [0] is needed.<br>
However, if i understand you right, all i need too do is add a dimension too my guessed output from my network? Since it is only sample too train with, it can be simply 1 ? But what, when i really want too train?<br>
However i changed it too</p>
<pre><code class="lang-auto">    predicted_model[0].expand(1,1,384,384,384)
    loss = lossfct(predicted_model[0], voxel_array)
</code></pre>
<p>too add 1 dimension too my predicted modell.<br>
Now i should have the shape of<br>
input = [1,1,384,384,384]<br>
and<br>
output = [1,384,384,384]</p>
<p>However, i still get a error which is:</p>
<pre><code class="lang-auto">RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4
</code></pre>
<p>and i thought i could solve this by edditing the target array the same way as the input.<br>
With</p>
<pre><code class="lang-auto">voxel_array = torch.zeros((384, 384, 384), dtype=float)
voxel_array.expand(1,384,384,384)
predicted_model[0].expand(1,1,384,384,384)
</code></pre>
<p>i encounter another error:</p>
<pre><code class="lang-auto">RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4
</code></pre>
<p>i feel like that the first approach is the right one but i dont know. <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>

ptrblck: <p>I’m not sure, what’s causing the issue, as the shapes work for me:</p>
<pre><code class="lang-python">nb_classes = 10
output = torch.randn(1, nb_classes, 384,384,384, requires_grad=True)
target = torch.randint(0, nb_classes, (1, 384, 384, 384))
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
loss.backward()
</code></pre>
<p>Could you double check the shapes of the output and target?</p>

Tom_ska: <p>The expanding of the tensors did not happen.</p>
<pre><code class="lang-auto">    print(predicted_model[0].shape)
    print(voxel_array.shape)
</code></pre>
<p>still gives</p>
<pre><code class="lang-auto">torch.Size([1, 384, 384, 384])
torch.Size([1, 384, 384, 384])
</code></pre>
<p>so i guess i’ll have too figure out why they are not transformed…</p>

Tom_ska: <p>Okay now i get</p>
<pre><code class="lang-auto">torch.Size([1, 1, 384, 384, 384])
torch.Size([1, 384, 384, 384])
</code></pre>
<p>from the prints.<br>
I forgot unsqueeze before. To expand you need too</p>
<pre><code class="lang-auto">    predicted_model[0].unsqueeze_(0)
    predicted_model[0].expand(1,1,384,384,384)
</code></pre>
<p>in case someone has the same problem.<br>
Somehow, now i run into this error:</p>
<pre><code class="lang-auto">IndexError: Target 1 is out of bounds.
</code></pre>
<p>Might this happen because i just changed the size without passing values there?</p>

ptrblck: <p>Your current output tensor contains logits for a single class (check the size of <code>dim1</code>, which is 1 in your case and which is <code>nb_classes</code> in my case), so you would have to make sure the output has the shape <code>[batch_size, nb_classes, depth, height, width]</code>.</p>

Tom_ska: <p>But shouldnt it be just 1? Because i train with my 1-unit dataset? And also, what should nb_classes be at my CNN? Since when thinking about a real-training later, there are most likely “infinite” classes. (I know its not infinite but reaaaally big). I am sorry for all my missunderstanding, i am very new too pytorch.<br>
also the cnn was already given and i am not sure how to change the output that it fits like this.</p>

ptrblck: <p>If the number of classes is set to one, you don’t really have a valid use case, since a simple model predicting zeros for all samples would achieve an accuracy of 100% (there are no other classes in the end).<br>
Also, since the target contains a class index of 1, this indicates that at least two classes are expected (target values would be at least <code>[0, 1]</code>).</p>

Tom_ska: <p>So basically for my Use-Case (Training with max. 4 samples) the CrossEntropyLoss is the wrong lossfunction?<br>
I simply have too shown that a retraining would be usefull without actually going trough the whole training because we lack in data.<br>
Can you suggest me a better lossfunction for my usecase?</p>

ptrblck: <p>No, not necessarily.<br>
The number of samples does not correspond to the number of classes.<br>
You can freely pick the former (number of samples), while the latter (number of classes) is defined by your use case.<br>
E.g. if you want to train a model to predict 10 classes, the number of classes would be 10 and the output layer would return a tensor containing the logits for all these 10 classes.</p>
<p>Even if you are using a single sample, the corresponding class might be any value between 0 and 9, so the output shape should still be <code>[batch_size, nb_classes, *]</code> and the target can contain values in <code>[0, 9]</code>.</p>