Capsule Networks: Model Outputs tensor of zeroes at the middle of the training

username: pdc_87
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , would you mind checking this code in my github. It’s about Capsule Networks. It has four labels. I am facing some NaN values in the loss function or zeroes output of the model at the middle of the training. I dont know if it a problem of vanishing or exploding gradients</p>
<p>Print Statement:<br>
Validation Loss decreased (0.2539 → nan).  Saving Checkpoint Model …<br>
Training [46%]		Train Loss:0.1899  Val Loss:nan		Train Accuracy:73.73%  Val Accuracy:62.80%</p>
<h1><a name="this-is-the-architecture-1" class="anchor" href="#this-is-the-architecture-1"></a>This is the architecture:</h1>
<h1><a name="layer-typedepth-idx-output-shape-param-2" class="anchor" href="#layer-typedepth-idx-output-shape-param-2"></a>Layer (type:depth-idx)                   Output Shape              Param #</h1>
<h1><a name="capsnet-128-4-16-1-convlayer-1-1-128-256-24-24-conv2d-2-1-128-256-24-24-20992-primarycaps-1-2-128-2048-8-modulelist-2-2-conv2d-3-1-128-32-8-8-663584-conv2d-3-2-128-32-8-8-663584-conv2d-3-3-128-32-8-8-663584-conv2d-3-4-128-32-8-8-663584-conv2d-3-5-128-32-8-8-663584-conv2d-3-6-128-32-8-8-663584-conv2d-3-7-128-32-8-8-663584-conv2d-3-8-128-32-8-8-663584-digitcaps-1-3-128-4-16-1-1048576-decoder-1-4-128-1-32-32-linear-2-3-128-1024-66560-linear-2-4-128-2048-2099200-linear-2-5-128-1024-2098176-sigmoid-2-6-128-1024-3" class="anchor" href="#capsnet-128-4-16-1-convlayer-1-1-128-256-24-24-conv2d-2-1-128-256-24-24-20992-primarycaps-1-2-128-2048-8-modulelist-2-2-conv2d-3-1-128-32-8-8-663584-conv2d-3-2-128-32-8-8-663584-conv2d-3-3-128-32-8-8-663584-conv2d-3-4-128-32-8-8-663584-conv2d-3-5-128-32-8-8-663584-conv2d-3-6-128-32-8-8-663584-conv2d-3-7-128-32-8-8-663584-conv2d-3-8-128-32-8-8-663584-digitcaps-1-3-128-4-16-1-1048576-decoder-1-4-128-1-32-32-linear-2-3-128-1024-66560-linear-2-4-128-2048-2099200-linear-2-5-128-1024-2098176-sigmoid-2-6-128-1024-3"></a>CapsNet                                         [128, 4, 16, 1]           –<br>
├─ConvLayer: 1-1                         [128, 256, 24, 24]        –<br>
│    └─Conv2d: 2-1                        [128, 256, 24, 24]        20,992<br>
├─PrimaryCaps: 1-2                      [128, 2048, 8]            –<br>
│    └─ModuleList: 2-2                   –                        –<br>
│    │    └─Conv2d: 3-1                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-2                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-3                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-4                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-5                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-6                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-7                  [128, 32, 8, 8]           663,584<br>
│    │    └─Conv2d: 3-8                  [128, 32, 8, 8]           663,584<br>
├─DigitCaps: 1-3                           [128, 4, 16, 1]           1,048,576<br>
├─Decoder: 1-4                             [128, 1, 32, 32]          –<br>
│    └─Linear: 2-3                          [128, 1024]               66,560<br>
│    └─Linear: 2-4                          [128, 2048]               2,099,200<br>
│    └─Linear: 2-5                          [128, 1024]               2,098,176<br>
│    └─Sigmoid: 2-6                       [128, 1024]               –</h1>
<h1><a name="total-params-10642176-trainable-params-10642176-non-trainable-params-0-total-mult-adds-unitsgigabytes-4558-4" class="anchor" href="#total-params-10642176-trainable-params-10642176-non-trainable-params-0-total-mult-adds-unitsgigabytes-4558-4"></a>Total params: 10,642,176<br>
Trainable params: 10,642,176<br>
Non-trainable params: 0<br>
Total mult-adds (Units.GIGABYTES): 45.58</h1>
<p>Input size (MB): 0.52<br>
Forward/backward pass size (MB): 172.03<br>
Params size (MB): 42.57<br>
Estimated Total Size (MB): 215.13</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/0291fa7a68e458f321ba9e8dcfc5639d64b900ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/0291fa7a68e458f321ba9e8dcfc5639d64b900ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0291fa7a68e458f321ba9e8dcfc5639d64b900ce_2_690x473.png" alt="image" data-base62-sha1="mJIa1z5lviLCC9uDdB6eXHfNUy" width="690" height="473" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0291fa7a68e458f321ba9e8dcfc5639d64b900ce_2_690x473.png, https://discuss.pytorch.org/uploads/default/original/3X/0/2/0291fa7a68e458f321ba9e8dcfc5639d64b900ce.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/0291fa7a68e458f321ba9e8dcfc5639d64b900ce.png 2x" data-dominant-color="F5F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">910×624 46.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Mi link to github is: <a href="https://github.com/contepablod/QCapsNet" rel="noopener nofollow ugc">https://github.com/contepablod/QCapsNet</a></p>
<p>Thanks!</p>