Adding extra data to standard convolution neural network in pytorch

kru2z: <p>Hello everyone. I am trying to extra data during training Resnet18. In my case it is gender information expressed as  <code>gender=Tensor([g]), where g =[-1|0|1] for [female|unknown|male]</code> , and some shape descriptor obtained in previous steps, which is of the form  <code>desc=Tensor([...]),  desc.shape = ([64,])</code> .<br>
I tried to follow the example at <a href="https://discuss.pytorch.org/t/concatenate-layer-output-with-additional-input-data/20462">this thread</a> but my loss value becomes NaN in the first epoch. Changing the learning rate does not help</p>
<p>Here is the modified resnet18 function that i use. in the comments i put the shape and magnitudes of tensors. As you can see the data that i want to add is already normalized:</p>
<blockquote>
<p>def forward(self, x,descdata,genderdata):<br>
<span class="hashtag-raw">#x</span>.shape = <span class="hashtag-raw">#torch</span>.Size([4, 3, 224, 224])<br>
# descdata.shape = torch.Size([ 4,64])<br>
# genderdata.shape = torch.Size([1,4])</p>
<pre><code>    x = self.conv1(x) #torch.Size([4, 64, 112, 112])
    x = self.bn1(x) #torch.Size([4, 64, 112, 112])
    x = self.relu(x) # torch.Size([4, 64, 112, 112])
    x = self.maxpool(x) #torch.Size([4, 64, 56, 56])

    x = self.layer1(x) #torch.Size([4, 64, 56, 56])
    x = self.layer2(x) #torch.Size([4, 128, 28, 28])
    x = self.layer3(x) #torch.Size([4, 256, 14, 14])
    x = self.layer4(x) #torch.Size([4, 512, 7, 7])

    x = self.avgpool(x) #torch.Size([4, 512, 1, 1])
    x = torch.flatten(x, 1) #torch.Size([4, 512])
    x = self.fc(x) #torch.Size([4, 35]); torch.norm(x,dim=1) == tensor([4.4604, 1.9574, 2.0123, 1.9583])


    y = genderdata.transpose(1,0) #torch.Size([4, 1]);  torch.norm(y,dim=1) == tensor([1., 1., 1., 1.])
   

    z1 = torch.cat([x,y,descdata.type(torch.float32)],dim=1) #torch.Size([4, 100]); torch.norm(z1,dim=1) == tensor([3.6243, 2.5945, 3.1520, 3.0312])
    z1 = self.relu2(self.fc2(z1))   # self.relu2 = nn.ReLU(); self.fc2  = nn.Linear(num_classes+1+64, (num_classes+1+64)*2)   defined in __init__ 
    z1 = self.fc3(z1)  #  self.fc3 = nn.Linear((num_classes+1+64)*2, num_classes)  defined in __init__ 
    return z1
</code></pre>
</blockquote>
<p>Would you know how to properly add the extra data to the layer?</p>
<p>Best</p>
<p>Chris</p>

nairbv: <p>If you just add one of the data parameters (descdata or gender) do you still see the issue? what loss function are you using? You might also want to check some of the other threads on nan loss to get some ideas:</p>
<aside class="quote" data-post="1" data-topic="11568">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vijay_dubey/48/2719_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/nan-loss-coming-after-some-time/11568">Nan Loss coming after some time</a> 
  </div>
  <blockquote>
    The loss function is a combination of Mean Sqaured error loss and cross-entropy loss. 
When i am training my model, there is a finite loss but after some time, the loss is NaN and continues to be so. 
When I am training my model just on a single batch of 10 images, the loss is finite most of the times, but sometimes that is also NaN. 
Please suggest a possible solution. 
Thanks in advance
  </blockquote>
</aside>

<aside class="quote" data-post="3" data-topic="25929">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/getting-nan-after-first-iteration-with-custom-loss/25929/3">Getting Nan after first iteration with custom loss</a> 
  </div>
  <blockquote>
    Thank you for your replay 
You are right, I did that on purpose because I am trying to mimic a paper that explained the network in this way. However, I tried to add non-linearty between them ,but unfortunately didn’t fix the NaN error. 
debugging the code, I notice the NaN appears in the weights of the model after I call the optimizer()
  </blockquote>
</aside>

<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/51033066/pytorch-loss-inf-nan">
  <header class="source">

      <a href="https://stackoverflow.com/questions/51033066/pytorch-loss-inf-nan" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/6322276/jabrams" target="_blank" rel="noopener nofollow ugc">
    <img alt="JAbrams" src="https://www.gravatar.com/avatar/769c4722a9e96507bbc20849e2ceab6a?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/51033066/pytorch-loss-inf-nan" target="_blank" rel="noopener nofollow ugc">Pytorch loss inf nan</a>
</h4>

<div class="tags">
  <strong>regression, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/6322276/jabrams" target="_blank" rel="noopener nofollow ugc">
    JAbrams
  </a>
  on <a href="https://stackoverflow.com/questions/51033066/pytorch-loss-inf-nan" target="_blank" rel="noopener nofollow ugc">11:37PM - 25 Jun 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>