akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Setting up Libtorch in Mac Xcode IDE

RebirthT: <p>Im new to Xcode IDE<br>
How to set the Cmake for <code>libtorch</code> in my Xcode IDE?<br>
So far I’ve set the</p>
<pre><code class="lang-auto">1.OTHER_LINKER_FLAGS
2.Header search path
3.Libraries search path
</code></pre>
<p>However it still doesn’t work</p>

bemorept: <p>To make XCode console project with libtorch :</p>
<p>$ cmake -G Xcode -DCMAKE_PREFIX_PATH=~/libtorch CMakeLists.txt …</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1142f86138f68da4278cfb4df3479fa01fe18ca2.png" data-download-href="https://discuss.pytorch.org/uploads/default/1142f86138f68da4278cfb4df3479fa01fe18ca2" title="ScrShot_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1142f86138f68da4278cfb4df3479fa01fe18ca2_2_690x451.png" alt="ScrShot_2" data-base62-sha1="2sHBcDnpwwGMyRQofLjQwyjUInM" width="690" height="451" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1142f86138f68da4278cfb4df3479fa01fe18ca2_2_690x451.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1142f86138f68da4278cfb4df3479fa01fe18ca2_2_1035x676.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/1142f86138f68da4278cfb4df3479fa01fe18ca2.png 2x" data-dominant-color="2C2C2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ScrShot_2</span><span class="informations">1142×748 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It works, well in console mode.<br>
But, I want to run in Openframeworks + XCode.<br>
Unfortunately, I have failed in Openframeworks + XCode.</p>
<p>from <span class="mention">@bemoregt</span>.</p>
Is there a bug in "torch.nn.CTCLoss"?

Farry_zhou: <p>I just train my CRNN model for Chinese text recognition.<br>
I meet this problem.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b449be7049359a973ff929e862374059329973e8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b449be7049359a973ff929e862374059329973e8" title="problem2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b449be7049359a973ff929e862374059329973e8_2_690x301.jpeg" alt="problem2" data-base62-sha1="pITWimdFVZ1uSzI4vrUDJiWSfdm" width="690" height="301" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b449be7049359a973ff929e862374059329973e8_2_690x301.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/b/b449be7049359a973ff929e862374059329973e8.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b449be7049359a973ff929e862374059329973e8.jpeg 2x" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">problem2</span><span class="informations">1022×446 73.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Maybe, it’s like a problem of “out of memory”.</p>

Farry_zhou: <p>But, the situation of GPUs is shown in the follow figure.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/89fe6be0b648a7b4bf58b7ab94502330713e02d9.jpeg" alt="problem3" data-base62-sha1="jGKvxKYaml1JniDsmxAia43L68p" width="648" height="342"><br>
When I trained my model, the PID of the process is 4585. But, while the problem occured, the process of 4585 was replaced by the process in the figure.</p>
<p>I suspect that there is a bug in the “torch.nn.CTCLoss”.<br>
Please, for help.</p>
Using the new learning rate scheduler

Bixqu: <p>I see that the new pytorch comes with a much needed learning rate scheduler. <a href="http://pytorch.org/docs/master/optim.html#how-to-adjust-learning-rate" rel="nofollow noopener">Actually, more of them.</a> I want to use the <code>torch.optim.lr_scheduler.MultiStepLR</code>, since I need it to divide my learning rate at certain milestones.</p>
<p>What I do is:</p>
<pre><code>learning_rate = 0.1
momentum = 0.9

optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=momentum, weight_decay=0.0001)
scheduler = MultiStepLR(optimizer, milestones=[60,100,150,400], gamma=0.1)

for epoch in range(number_of_training_epochs):  # loop over the dataset multiple times

    for i, data in enumerate(train_loader, 0):

        inputs, labels = data

        inputs = inputs.float()

        inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())

        optimizer.zero_grad()

        outputs = net(inputs)

        loss = criterion(outputs, labels)

        loss.backward()
        scheduler.step()
</code></pre>
<p>What I see is that now my network is not learning anything. Should I use optimizer.step also ?</p>

jxgu1016: <p>+ <code>optimizer.step()</code></p>

Bixqu: <p>And I think scheduler.step should be outside the lowest loop (i.e. one step per epoch).</p>

jxgu1016: <p>That’s right. I missed it.</p>

DonghunP: <p>When I use ‘MultiStepLR’, I also need optimizer.step() and scheduler.step(). OMG you save my time.</p>
<p>Really thanks.</p>
<p>And then… I think <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.MultiStepLR" rel="nofollow noopener">torch.optim.lr_scheduler.MultiStepLR</a><br>
need to add optimized.step() also <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> …</p>
How can I use DSNT in c++?

SamChung_Hwang: <p>I need to use DSNT (<a href="https://arxiv.org/abs/1801.07372" rel="nofollow noopener">https://arxiv.org/abs/1801.07372</a>) in C++ torch lib.<br>
DSNT can be used with python(<a href="https://github.com/anibali/dsntnn/blob/master/examples/basic_usage.md" rel="nofollow noopener">https://github.com/anibali/dsntnn/blob/master/examples/basic_usage.md</a>). But there is no DSNT module for c++.</p>
<p>Is there any way to develop with DSNT in c++ ?</p>

tom: <p>Just like the implementation you link is a third-party repository, you’d have to implement that yourself.<br>
From a cursory look, it should not be too hard to transpose the Python code into C++ equivalents.</p>
<p>Best regards</p>
<p>Thomas</p>

SamChung_Hwang: <p>Could you or anybody build the DSNT c++ code ? I am not experienced so much for pytorch now.</p>
Why model.training=True and model.train() give different behavior?

LMA: <p>I have a neural network called <code>model</code>. I expected that <code>model.training=True</code> would have the same effect as <code>model.train()</code>. However, the behaviors are <strong>different</strong>, at least for dropout. In the former, dropout seemed to be disabled (different runs model(x) give the same output), while in the latter, dropout is activated (different runs model(x) give different outputs). Thanks.</p>

ptrblck: <p><code>.training</code> is an attribute of the current module.<br>
If you call <code>model.train()</code> or <code>model.eval()</code>, this attribute will be changed for all child modules inside the parent class as can be seen <a href="https://github.com/pytorch/pytorch/blob/eb76b7a564121c5fede749ad7d0a36f2b61a0a95/torch/nn/modules/module.py#L1045" rel="nofollow noopener">here</a>.</p>
<p>Thus, if you would like to change the <code>.training</code> attribute manually, you would have to change it for each module manually.</p>
Multi-stream CNN model, how to load multiple images simutaneously

Jun_Ren: <p>Hello,</p>
<p>I want to train a multi-stream CNN model with pytorch. My data looks like this:</p>
<p>Two folder train1 and train2, under these two folders, there are 2 classes, 1 and 2, in each class there are bunch of images. The name of the pictures under train1 and train2 are same.</p>
<p>train1:<br>
----- 1:<br>
a1.jpg b1.jpg c1.jpg d1.jpg<br>
------2:<br>
a2.jpg b2.jpg c2.jpg d2.jpg</p>
<p>train2:<br>
----- 1:<br>
a1.jpg b1.jpg c1.jpg d1.jpg<br>
------2:<br>
a2.jpg b2.jpg c2.jpg d2.jpg</p>
<p>For pytorch, how can I input two images from train1 and train2 simutaneously. For example, if we set batch size as 1:<br>
train1/1/a1.jpg and train2/1/a1.jpg should be returned by the data loader at the same time.</p>
<p>How can I implement this, or any other ideas to do this any help will be appreciated<br>
Thanks</p>

ptrblck: <p>One possible approach would be to use <code>ImageFolder</code> internally in a custom <code>Dataset</code> and replace the root folder in the current path:</p>
<pre><code class="lang-python">
class MyDataset(Dataset):
    def __init__(self, train1, transform=None):
        self.dataset1 = datasets.ImageFolder(root=train1)
        self.transform = transform

    def __getitem__(self, index):
        # Get samples from train1
        x1, y = self.dataset1[index]
        # Get corresponding sample from train2
        path = self.dataset1.samples[index][0]
        path = path.replace('train1', 'train2')
        print('Loading ', path)
        x2 = self.dataset1.loader(path)
        
        if self.transform:
            x1 = self.transform(x1)
            x2 = self.transform(x2)
            
        return x1, x2, y
        
    def __len__(self):
        return len(self.dataset1)


dataset = MyDataset(train1='./data/train1')
</code></pre>

Jun_Ren: <p>Thanks very much, I will try this solution</p>

Jun_Ren: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Your solution works well. I want to know how to set batch size in your this customized Dateset class?<br>
for example: I want set batch-size as 5, so that I can get 5 images during the training process?</p>
<p>Thanks</p>

ptrblck: <p>Wrap the <code>Dataset</code> in a <code>DataLoader</code>, where you can specify the <code>batch_size</code> besides other useful features like shuffling and the usage of multiple workers. Have a look at the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">docs</a> for more information and maybe the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data loading tutorial</a> for some more information on the data loading pipeline.</p>
Larger than GPU memory tensors

lmoss: <p>Hi all,</p>
<p>Is there an elegant way to apply a network to a tensor that is larger than GPU memory?<br>
Tensorflow has tensorflow-mesh, maybe there’s something similar for pytorch?</p>
<p>I am aware of a sliding window approach, but that can lead to artifacts at edges of outputs.</p>
<p>Thanks!</p>

smth: <p>have you looked at something like <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="nofollow noopener">https://pytorch.org/docs/stable/checkpoint.html</a></p>

lmoss: <p>I think checkpoint_sequential will do the trick, thanks!</p>
Which device is expected by a Module?

aliutkus: <p>hi there,</p>
<p>for some reason, I need to know the device that is expected by a Module,<br>
is there a way to do that ?</p>

JuanFMontesinos: <p>That’s a relative question. Mistakes happen when in an operation which makes use of two tensors. Thus, from the pov of tensor 1, expected device would be tensor2’s, whereas from the pov tensor 2 it would be the other way around.</p>
<p>Don’t think about “modules” because errors are thrown by tensors inside modules, not modules themselves.</p>
<p>The closest thing you can do is, if you know which operation implies your module/tensor, to query the analogous module/tensor by calling tensor.device() or module.device().</p>

aliutkus: <p>Yes, sure, but I have to think about modules in my case, I’m handling them as black boxes</p>
<p>So I guess the answer is no and I must rely on the user to provide me with a <code>device</code> parameter</p>

JuanFMontesinos: <p>You can use a try except scheme… but does not  seem to be the best way.</p>

tom: <p>For modules with parameters, <code>next(m.parameters()).device</code>. Modules without parameters should take tensors on any device.<br>
This will fail for modules with mixed devices, but you probably don’t expect those to work, do you?</p>
<p>Best regards</p>
<p>Thomas</p>

aliutkus: <p>thanks,</p>
<p>that’s clear that I could indeed get the device of some arbitrary parameter.<br>
I guess the question becomes: is there some introspection trick to get the first parameter that will be involved in the <code>forward</code> function of a module, to then bring my tensor to that place.</p>
<p>(likewise, having the expected input shape would be cool, and is a related question)</p>
<p>anyways, for now I’m getting user instructions, at he cost of more parameters</p>
What do you think about this segmentation problem?

iArunava: <p>So, I have a lot of things going on under the hood, but everything is pretty normal, no fancy stuff going on. It should be a pretty normal looking codebase.</p>
<p>With that in mind,<br>
I have this smaaaaall model</p>
<pre><code class="lang-auto">In [178]: net
Out[178]: 
Model(
  (conv): Sequential(
    (0): Conv2d(3, 10, kernel_size=(1, 1), stride=(1, 1))
    (1): Conv2d(10, 2, kernel_size=(1, 1), stride=(1, 1))
  )
)
</code></pre>
<p>And this <strong>ONE</strong> image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/27b8cf2ba23e656a798b63a8760412672565a699.png" data-download-href="https://discuss.pytorch.org/uploads/default/27b8cf2ba23e656a798b63a8760412672565a699" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/27b8cf2ba23e656a798b63a8760412672565a699_2_500x500.png" alt="Figure_1" data-base62-sha1="5FozxZJMiWP1YnVxTNLUbCfQjJn" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/27b8cf2ba23e656a798b63a8760412672565a699_2_500x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/27b8cf2ba23e656a798b63a8760412672565a699_2_750x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/27b8cf2ba23e656a798b63a8760412672565a699.png 2x" data-dominant-color="C9C3C2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">800×800 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And I have trained for over 2000+ epochs.</p>
<p>This isn’t converging.<br>
I want to segment the sky.<br>
Actually, I am debugging. And this obviously means there is some bug somewhere.</p>
<p>What are your thoughts?</p>

ptrblck: <p>Note that you are not using any non-linearity between  the conv layers.<br>
Given that and a kernel of a single pixel, it looks like your model captures the blue-ish color successfully.</p>

iArunava: <p>Yep! You are right! I added <code>ReLU</code> to get a better convergence.</p>
<p>But, still didn’t reached <code>0</code> loss. I increased the kernel_size upto <code>5x5</code> for both the Conv Layers. I went very close, but still didn’t converged. Shouldn’t the loss go to <code>0</code>?</p>

ptrblck: <p>If the model has enough “capacity”, then the loss should converge towards zero.<br>
Just for the sake of debugging this example, you could increase the number of kernels, their spatial size, or try a huge linear layer instead.</p>
VOCDetection __getitem__() problem

vihas_makwana: <p>import torch.nn as nn<br>
import torch.nn.functional as F<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
import torchvision<br>
import torchvision.transforms as transforms</p>
<p>data2=torchvision.datasets.VOCDetection("./",download=True,transform=transforms.ToTensor(),target_transform=transforms.ToTensor())<br>
img,tar =data2[0]<br>
I get the following error:<br>
----&gt; 1 img,tar=data1[0]</p>
<p>/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py in to_tensor(pic)<br>
48     “”"<br>
49     if not(_is_pil_image(pic) or _is_numpy_image(pic)):<br>
—&gt; 50         raise TypeError(‘pic should be PIL Image or ndarray. Got {}’.format(type(pic)))<br>
51<br>
52     if isinstance(pic, np.ndarray):</p>
<p>TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘dict’&gt;</p>

ptrblck: <p>The <code>VOCDetection</code> target is a dictionary of the XML tree as stated in the <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.VOCDetection" rel="nofollow noopener">docs</a>.<br>
Since <code>ToTensor</code> works on <code>PIL.Image</code>s, you cannot use it as a <code>target_transform</code>.</p>

vihas_makwana: <p>What should I do in order to get (image,target) pair?</p>

ptrblck: <p>You would have to remove the <code>target_transform</code> and use the XML tree for your detection task.<br>
Have a look at <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">this tutorial</a>.<br>
While another dataset is used, it might be a good starter for your VOCDetection.</p>
Interpreting Weights

CRWayman: <p>Hello!</p>
<p>I’m wondering if anyone has any good recommendations for interpreting model weights for a general audience. I know this isn’t a code specific question, but I’m having trouble figuring out how I can use my weights in a meaningful way to describe the significance of my various input features in their use of predicting output.</p>
<p>I guess something like a p-value or a similar metric would be ideal. Does anyone have any thoughts on this?</p>
<p>Thanks.</p>
Horizontal Flipping Consistency and background augmentation for semantic segmentation

Bruce_Muller: <p>Hi PyTorch cummunity</p>
<p>I’m doing semantic segmentation on a facial dataset (HELEN) with classes “background”, “skin”, “hair”, “nose”, “left eye”, “right eye”, and other facial classes. See this paper for an example of work on this dataset: <a href="https://arxiv.org/abs/1906.01342" rel="nofollow noopener">https://arxiv.org/abs/1906.01342</a></p>
<p>I have a training set of 2000 images and I’d like to do random flipping as an augmentation. I have “left eye” and “right eye” but would like to keep the integers consistent when the flipping is done on left/right classes due to my research. My first question, is it okay to just duplicate the training data so I have 4000 images where half are flipped with the left/right integers swapped too?</p>
<p>My second question, is there a way to do online augmentation with PyTorch/torchvision transforms?</p>
<p>On another note, the paper above uses an augmentation where they randomly replace the background classed pixels with “non-face images or pure colors”. Does anyone know any good tools for doing something like that with pytorch?</p>
<p>Input is much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Taylor Approximation of the gradient with pytorch

LeanderK: <p>I want to correct the gradient for a custom layer using the Taylor expansion:</p>
<p><code>f'(x+delta) = f'(x) + 1/2*f''(x)*delta</code></p>
<p>where <code>x</code> is the original output of the layer and <code>f’(x)</code> the default, unmodified gradient, for simplicity this example is the univariate case and delta is something I calculate (it’s the secret sauce for the research idea, so I can’t really go too much into detail what it’s doing).</p>
<p>So I need to access the hessian (not inverse hessian) during for the custom gradient in my <code>nn.module</code>. I have never worked with Hessians and wonder what my options with PyTorch are. Can the autograd framework just compute it for me?</p>

LeanderK: <p>Ah, I’ve found a related issue that solves a similiar problem: <a href="https://discuss.pytorch.org/t/calculating-hessian-vector-product/11240" class="inline-onebox">Calculating Hessian Vector Product</a></p>
Constrain segmentation with additional loss of a pretrained model

che85: <p>Hello PyTorch Community,</p>
<p>I am seeking some help/ideas on some training I want to pursue.</p>
<p>The idea is to train a VNET for a segmentation task and use its output for two tasks:</p>
<p>a) calculating the main loss (which could be DICE or whatever)<br>
b) use it as the input to a pretrained AE model and calculating an additional loss</p>
<p>loss = mainLoss + additional loss</p>
<p>Pretrained model: AutoEncoder that was trained to learn the most prominent properties of the structure to segment.</p>
<p>Model to train: VNET</p>
<p>The idea is to have a pretrained model on a segmentation via autoencoder to preserve information about the shape of the structure to segment.</p>
<p>Do you have any tips on what to be aware of when doing that? Let me know if you have any questions.</p>
<p>Thank you,<br>
Christian</p>

ptrblck: <p>How did you pretrain the Autoencoder?<br>
Based on your explanation, it seems you’ve used the real images:</p>
<aside class="quote no-group" data-username="che85" data-post="1" data-topic="50788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/che85/48/11493_2.png" class="avatar"> che85:</div>
<blockquote>
<p>AutoEncoder that was trained to learn the most prominent properties of the structure to segment.</p>
</blockquote>
</aside>
<p>Would this approach work, if you now feed the segmentation output of your VNET to this AE?<br>
The image statistics, ranges etc. should be quite different, so I’m not sure if it’ll work out of the box or if you would need to train both models end-to-end.</p>
<p>It’s an interesting approach and reminds me of <a href="https://arxiv.org/abs/1802.07934"> Adversarial Learning for Semi-Supervised Semantic Segmentation</a>, but with an AE instead of Discriminator.</p>

che85: <p>Thanks for replying so quickly.</p>
<p>What I am trying to do is described in this paper: <a href="https://ieeexplore.ieee.org/document/8051114" rel="nofollow noopener">Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation</a></p>
<p>The AutoEncoder is pretrained with the ACNN input segmentations.</p>
<p>The idea of the paper is to use the predicted segmentation from the VNET (or whatever segmentation network you are using). The predicted segmentation and the ground-truth segmentation will be the input to 2 separate AE instances where the latent representations of both will be compared via Euclidean distance measure.</p>
<p>Does this make more sense now?</p>
<p>Thanks for your ideas/comments.</p>
<p>Christian</p>
Torch.matmul(): operate on highest dimension

Zhi_Zhang: <p>For high-dimensional tensors, the matrix multiplication can only be operated on the last two dimensions, which requires the previous dimensions to be equal. But in my research, matrix multiplication on the former dimensions also makes sense.<br>
For example, we can do this:</p>
<pre><code class="lang-auto"># Mat_A's size is (10, 20, 2, 32)
# Mat_B's size is (10, 20, 32, 3)
# torch.matmul(Mat_A, Mat_b)'s size is (10, 20, 2, 3)
torch.matmul(Mat_A, Mat_b)
</code></pre>
<p>But how can we implement this:</p>
<pre><code class="lang-auto"># Mat_a's size is (10, 20, 30)
# Mat_b's size is (5, 10)
# torch.some_operate(Mat_b, Mat_a)'s size is (5, 20, 30)
torch.some_operate(Mat_b, Mat_a)
</code></pre>
<p>I can use loops to do the calculations, but it’s not very elegant. This is a simple example:</p>
<pre><code class="lang-auto">Mat_a is:
[[a, b],
 [c, d]]
Mat_b is:
[[A, B],
 [C, D]]
then the result is:
[[aA+bC, aB+bD],
 [cA+dC, cB+dD]]
where a, b, c, d are scalars and A, B, C, D are vectors.
</code></pre>

tom: <p><code>torch.einsum('ijk,li-&gt;ljk', Mat_a, Mat_b)</code><br>
It’ll do the rearranging of dimensions for you.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>Update: <a href="https://rockt.github.io/2018/04/30/einsum" rel="nofollow noopener">See Tim Rocktäschel’s post on einsum</a> for lots of applications.</p>

Zhi_Zhang: <p>Wow, I have learned einsum but never thought about this idea. That’s awesome! TX!</p>
torch.Storage reusing memory

Andreas_Georgiou: <p>I was curious, let’s say I have a tensor <code>x=torch.ones(10)</code>. Then I do <code>x=x[:5]</code>. Now <code>x.storage()</code> is still a storage of size 10 even though I have a view of size 5. If I have no other reference pointing to that part of the storage, can another tensor use it automatically if it is needed? Or is it considered occupied?</p>

pietern: <p>It is considered occupied. If you want to release it, you’ll have to make a copy of the view, which will allocate a new tensor of the right size AFAIK.</p>
Prevent rounding numbers imported from numpy

quanguet: <p>Hello everyone,<br>
I just imported an array from numpy into pytorch:</p>
<pre><code class="lang-python">import numpy as np
import torch

a = np.array([ 1.07790701e-01,  5.27183563e-02,  1.03966855e-01])
print(a)
# [0.1077907  0.05271836 0.10396685]

b = torch.from_numpy(a)
print(b)
# tensor([0.1078, 0.0527, 0.1040], dtype=torch.float64)
</code></pre>
<p>Even though the tensor is <code>float64</code> type, every value in tensor<code>b</code> is rounded up to 2 decimals compared with the original array <code>a</code>.</p>
<p>How can I keep the values of <code>b</code> unchanged from <code>a</code>?</p>
<p>Many thanks,</p>

JuanFMontesinos: <p>It’s not a rounding error, it’s about print. PyTorch uses less precision to print.</p>
<pre><code class="lang-auto">a-b.numpy()
Out[3]: array([0., 0., 0.])
np.linalg.norm((a-b.numpy()))
Out[4]: 0.0

print(b[0])
tensor(0.1078, dtype=torch.float64)
print(b[0].item())
0.107790701
</code></pre>

ptrblck: <p>To increase the precision for print outputs, just set:</p>
<pre><code class="lang-python">torch.set_printoptions(precision=10)
</code></pre>
How to obtain a 1D output from a NN using 2D input (batches)?

Raphikowski: <p>Hi all,</p>
<p>I have the following setting: I want to learn a gaussian distribution via two NNs, i.e., one NN maps onto the mean vector and the other NN maps onto the covariance matrix. My issue is that I am feeding batches of shape (batch_dim, observation_dim) into the NN and hence obtain for the mean vector also a vector of size (batch_dim, observation_dim) and for the covariance matrix a tensor of shape (batch_dim, observation_dim * observation_dim). However, I do not want to have an individual mean vector and covariance matrix for each sample in the batch, but instead one mean vector and one covariance matrix for the whole batch. I.e., I want to get rid of the batch_dim in the output: The mean vector should have shape (observation_dim) and the covariance matrix should have shape (observation_dim, observation_dim). How can this be achieved?</p>
<p>Thank you in advance!</p>
Torch error when compiling

Yang_Yang2: <p>error message：<br>
./torch/include/ATen/core/interned_strings.h:325:1: error: expected unqualified-id before ‘do’<br>
FORALL_NS_SYMBOLS(DEFINE_SYMBOL)<br>
^<br>
./torch/include/ATen/core/interned_strings.h:325:1: error: expected unqualified-id before ‘while’</p>
What is the point of torch.no_grad in this weird place?

John_Deterious: <p>The following is from official tutorial. My question is: what’s the point of no_grad here? My understanding is that it is only useful if you want to save costs while in running the forward pass.</p>
<pre><code>with torch.no_grad():
    for param in model.parameters():
        param -= learning_rate * param.grad</code></pre>

ptrblck: <p>The <code>no_grad</code> guard disables gradient calculations, so Autograd won’t track any operations applied in this block.<br>
While it’s useful for inference, it can be also applied when you would like to manipulate some internal parameters without using the <code>.data</code> attribute.</p>
A bug when load vggface to pytorch

Qu_Yukun: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/790747f3f0690c19f95a2abda00d5cb1f3a1b25b.png" data-download-href="https://discuss.pytorch.org/uploads/default/790747f3f0690c19f95a2abda00d5cb1f3a1b25b" title="vggface_bug2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/790747f3f0690c19f95a2abda00d5cb1f3a1b25b_2_690x114.png" alt="vggface_bug2" data-base62-sha1="hgFn9S8Iphw1tH1znzVrydeLEAX" width="690" height="114" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/790747f3f0690c19f95a2abda00d5cb1f3a1b25b_2_690x114.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/790747f3f0690c19f95a2abda00d5cb1f3a1b25b_2_1035x171.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/790747f3f0690c19f95a2abda00d5cb1f3a1b25b.png 2x" data-dominant-color="33322E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">vggface_bug2</span><span class="informations">1247×207 34.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As the pictures show, I used pytorch to load vgg_face. But it appeared error.</p>
<p>I switch a package to load vggface. It can handle t7 format file. But it aslo did’t wrok.<br>
The error message is as follows:</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “C:\Users\qu199\AppData\Local\Programs\Python\Python37\lib\site-packages\torchfile.py”, line 424, in load<br>
return reader.read_obj()<br>
File “C:\Users\qu199\AppData\Local\Programs\Python\Python37\lib\site-packages\torchfile.py”, line 370, in read_obj<br>
obj._obj = self.read_obj()<br>
File “C:\Users\qu199\AppData\Local\Programs\Python\Python37\lib\site-packages\torchfile.py”, line 387, in read_obj<br>
obj[k] = v<br>
TypeError: unhashable type: ‘numpy.ndarray’</p>
<p>I download the vggface file from the website <a href="https://www.robots.ox.ac.uk/~vgg/software/vgg_face/" class="inline-onebox" rel="noopener nofollow ugc">Visual Geometry Group - University of Oxford</a> .</p>
<p>I wonder if the version of pytorch lead to the bugs. I am use the latest edition.</p>
<p>Can someone help me ?</p>

ptrblck: <p>It looks like you are trying to load a Torch7 model.<br>
Note that Torch7 is not under active development anymore and I would recommend to switch to PyTorch.<br>
There should be a few vggface-pytorch repos on GitHub. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
What's the difference between passing numpy to Dataloader and converting to tensors and passing to TensorDataset then Dataloader?

John_Deterious: <p>As above. I tried passing numpy to Dataloader directly, it worked. How does this contrast with converting numpy to tensor and passing it to TensorDataset then pass the dataset to Dataloader?</p>

ptrblck: <p>How did you pass the numpy arrays to the <code>DataLoader</code>?<br>
Did you get the expected batches for both, the data and target?</p>
<p>I would personally see it as a bit of a hack, if that’s working.<br>
This simple example won’t throw an error, but doesn’t yield the expected results:</p>
<pre><code class="lang-python">data, target = np.zeros((100, 1)), np.ones((100, 1))
loader_np = DataLoader((data, target), batch_size=1)
for x in loader_np:
    print(x.shape)
    print(type(x))
    print(x)
</code></pre>
Default Weight Initialization vs Xavier Initialization

Vigneswaran: <p>Hi, the question is very basic. PyTorch uses default weight initialization method as discussed <a href="https://discuss.pytorch.org/t/how-are-layer-weights-and-biases-initialized-by-default/13073">here</a>, but it also provides a way to initialize weights using Xavier equation. In many places <a href="https://stats.stackexchange.com/questions/326710/why-is-weight-initialized-as-1-sqrt-of-hidden-nodes-in-neural-networks" rel="nofollow noopener">1</a>, <a href="https://www.quora.com/Does-Xavier-initialization-work-well-when-the-activation-function-is-a-ReLu" rel="nofollow noopener">2</a> the default method is also referred as Xavier’s. Can anyone explain where I am going wrong? Any help is much appreciated</p>

ptrblck: <p>The post is from January 2018 and outdated by now.<br>
You can find the current weight init <a href="https://github.com/pytorch/pytorch/blob/3b1c3996e1c82ca8f43af9efa196b33e36efee37/torch/nn/modules/linear.py#L79" rel="nofollow noopener">here</a>, which is <code>init.kaiming_uniform_(self.weight, a=math.sqrt(5))</code> for the weights.</p>

Vigneswaran: <p>Great! Thanks:) One small clarification, is the method mentioned <a href="https://discuss.pytorch.org/t/how-are-layer-weights-and-biases-initialized-by-default/13073/2">here</a> is actually Xavier Initialization?</p>

ptrblck: <p>This doesn’t seem to be a <code>xavier</code> init, as only <code>fan_in</code> is used, while <a href="https://pytorch.org/docs/stable/nn.html?highlight=xavier#torch.nn.init.xavier_uniform_" rel="nofollow noopener">xavier_uniform_</a> uses a different scaling factor and the sum of the number of input and output features.</p>

Vigneswaran: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
VOC datasets's __getitem__() method not working?

vihas_makwana: <p>import torchvision.transforms as transforms</p>
<p>data=torchvision.datasets.VOCSegmentation("./",download=True,transforms=transforms.Compose([transforms.ToTensor()]))</p>
<p>img,tar=data[0]</p>
<p>I get following error:</p>
<p>/usr/local/lib/python3.6/dist-packages/torchvision/datasets/voc.py in <strong>getitem</strong>(self, index)<br>
205<br>
206         if self.transforms is not None:<br>
–&gt; 207             img, target = self.transforms(img, target)<br>
208<br>
209         return img, target</p>
<p>TypeError: <strong>call</strong>() takes 2 positional arguments but 3 were given</p>

ptrblck: <p>You are currently trying to pass a single transformation to <code>transforms</code>, which expects a <code>StandardTransform</code> object as seen <a href="https://github.com/pytorch/vision/blob/5be137e24d8eeeaefd4a2be34942fcc095012279/torchvision/datasets/vision.py#L26" rel="nofollow noopener">here</a>.<br>
Try to use <code>transform=transforms.ToTensor()</code> instead (note the missing s on the left hand side) to transform the data.<br>
If you also want to pass a transformation for the target image, use <code>target_transform</code> additionally.</p>

vihas_makwana: <p>Thanks for the help. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
No scopes name and type info when loading the trace model

jackmsye: <pre><code class="lang-python">class LinearInLinear(nn.Module):
    def __init__(self):
        super(LinearInLinear, self).__init__()
        self.l = nn.Linear(3, 5)
        self.l1 = nn.Linear(5, 5)

    def forward(self, x):
        return self.l1(self.l(x + x))


class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.l1 = LinearInLinear()
        self.l = LinearInLinear()

    def forward(self, input):
        x1 = self.l1(input)
        x2 = self.l(input)
        return x1 + x2 + x2
if __name__ == "__main__":
    dummy_input = (torch.rand(5, 3),)
    # with torch.onnx.set_training(model, False):
    trace = torch.jit.trace(model, dummy_input)
    # _optimize_trace(trace, torch._C._onnx.OperatorExportTypes.ONNX)
    trace.save("b.pt")
    print(trace.graph)
    for node in trace.graph.nodes():
        if node.kind() == "prim::Constant":
            continue
        print(list(node.outputs())[0].type().scalarType())
    print(type(trace))
    k = torch.jit.load("b.pt")
    print(type(k))
    print(k.graph)
    for node in k.graph.nodes():
        if node.kind() == "prim::Constant":
            continue
        print(node)
        output = list(node.outputs())[0]
        print(output.type().scalarType())
   the last
</code></pre>
<p>the last sentence produce the error below<br>
RuntimeError: r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch.</p>
Assignment through advanced indexing

EtienneDesticourt: <p>Hi,</p>
<p>I have a high-dimensional tensor whose values I’m trying to update along specific indices using another tensor like so:</p>
<pre><code class="lang-auto">N = 5
C = 2
K = 9
J = 4
x = torch.randn(N, C, K, J)  # Shape: (5, 2, 9, 4)
new_values = torch.randn((N, C)) # Shape: (5, 2)

index1 = torch.arange(N)
index3 = torch.randint(K, (N,))
index4 = torch.randint(J, (N,))
x[index1 , :, index3, index4] = new_values
</code></pre>
<p>This works but I’m looking for a similar operation that’d take the indices as a tuple (the indices are generated dynamically so I can’t hardcode the indexing of x like above) and would work for an arbitrary number of dimensions as in the example below:</p>
<pre><code class="lang-auto">mystery_update_operation(x, (index1, index3, index4), new_values)
</code></pre>
<p>I have looked into <code>scatter_</code>and <code>index_put_</code> but can’t figure out how to make them work in this case.</p>
<p>Cheers.</p>
Official whl file (torch=1.0.1 with CUDA 10.0) ships with wrong torch.version.cuda

PyDennis: <p>I recently downgraded from PyTorch 1.1 to 1.0.1 to get an old project running.</p>
<p>Specs/Environment:<br>
Ubuntu 18.04<br>
GPU: Nvidia 2080<br>
CUDA: V10.0.130 (from nvcc --version)</p>
<p>After downloading/installing PyTorch 1.0.1 from <a href="https://pytorch.org/get-started/previous-versions/" rel="nofollow noopener">https://pytorch.org/get-started/previous-versions/</a> via</p>
<p><code>pip install torch==1.0.1 -f https://download.pytorch.org/whl/cu100/stable</code></p>
<p>and running <code>torch.version.cuda</code>, It returns <code>9.0.176</code>.</p>
<p>I believe this is the reason why I’m getting  CUDA errors in my project. How can I get <code>torch.version.cuda</code> to return the correct CUDA version (10.0)?</p>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

PyDennis: <p>I have solved the problem by downloading/installing a different .whl file via <code>pip install https://download.pytorch.org/whl/cu100/torch-1.0.1-cp36-cp36m-linux_x86_64.whl</code>.</p>
<p>This <a href="https://github.com/pytorch/pytorch/issues/15797#issuecomment-472167754" rel="nofollow noopener">comment</a> from the official PyTorch Github has led me to the solution.</p>
Cosine Distance in Pytorch

Mohammad_Mahfujur_Ra: <p>Hi,</p>
<p>I want to define a function that will calculate the cosine distance between two normalized vectors v1 and v2 is defined as 1 - dot_product(v1, v2) in pytorch.</p>
<pre><code class="lang-auto">def     cosine_distance (f_x, f_y):
        f_xx_normalized  = F.normalize(f_x, p=2, dim=1)
        f_yy_normalized  = F.normalize(f_y, p=2, dim=1)
        f_yy_normalized_transpose = f_yy_normalized.transpose(0,1)    
        cosine_loss = 1 - torch.sum(torch.mm(f_xx_normalized, f_yy_normalized_transpose ))
        return cosine_loss
</code></pre>
<p>The expected loss would be 0 &lt; cosine_loss &lt;1, but I got the loss approximately -90.78. Is it working ? I am confused.</p>

tom: <p>Hi,</p>
<p>your inputs appear to be batches of vectors (let’s say of shape b x n).<br>
The result of <code>torch.mm(f_xx_normalized, f_yy_normalized_transpose )</code> is a b x b matrix containing the cosine of every vector in f_x with every vector in f_y, while you would likely only be interested in the diagonal. Maybe it’s easiest to express the diagonal as <code>torch.einsum('bn,bn-&gt;b', f_xx_normalized, f_yy_normalized)</code>, but you could also unsqueeze a singleton dimension and use <code>torch.matmul</code> for a batched matrix multiplication.</p>
<p>You then have a vector of length b (instead of the matrix) with a cosine of the angles, in articular values between -1 and 1. If you want to keep the structure, using <code>torch.mean</code> in place of <code>torch.sum</code> would make the result in [-1, 1], and then the <code>cosine_loss = 1-torch.mean(torch.einsum('bn,bn-&gt;b', f_xx_normalized, f_yy_normalized))</code> is between 0 and 2.</p>
<p>I might add that it is inefficient to normalize the vectors just to take the scalar product, it would probably be better to divide by the norms.</p>
<p>Best regards</p>
<p>Thomas</p>

Mohammad_Mahfujur_Ra: <p>Hi Thomas,</p>
<p>Thank you so much for your kind reply. It works now. Thanks again for your explanation.</p>
Pytorch Nightly, breaks torchvision

Diego: <p>I installed pytorch nightly for cuda10 using pip3 by running:</p>
<p>pip3 install torch_nightly -f <a href="https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html</a></p>
<p>Now whenever I try to import torchvision I get:</p>
<p>from torchvision import _C<br>
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory</p>
<p>Cuda 10 is installed on my system and is symlinked to /usr/local/cuda/lib64<br>
Here is the output of nvidia-smi</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a58ffdd9df557977c7461a81eef1d0eaa83aabf2.png" data-download-href="https://discuss.pytorch.org/uploads/default/a58ffdd9df557977c7461a81eef1d0eaa83aabf2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a58ffdd9df557977c7461a81eef1d0eaa83aabf2.png" alt="image" data-base62-sha1="nCDhyGr0yHX8v0VFPGIEFIId6My" width="690" height="317" data-dominant-color="2E2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">778×358 36.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What went wrong? Why is torchvisions looking for cuda 9.0 files?</p>

ptrblck: <p>This might be related to <a href="https://github.com/pytorch/pytorch/issues/21154#issuecomment-498679072" rel="nofollow noopener">this issue</a>.<br>
I’m not sure, if there is a better way (torchvision-nightly?), but it seems if you would like to use the PyTorch nightly binaries, you should build torchvision from source (or use a stable PyTorch binary).</p>

Diego: <p>Seems like there is a torchvision nightly as discussed <a href="https://github.com/pytorch/vision/issues/833" rel="nofollow noopener">here</a>. I should install that one with pytorch nightly, yes?</p>

ptrblck: <p>The mentioned package was maintained by <a class="mention" href="/u/stas">@stas</a> and, if I’m not mistaken, wasn’t updated anymore, so you would have to wait for the next official <code>torchvision-nightly</code> release or build from source.</p>
Local variables in forward()

Beinan_Wang: <p>I have some questions about the usage of local variables in the forward function. Say I have the following model:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(3,64,1)
        self.bn1 = nn.BatchNorm2d(64)
        self.conv2 = nn.Conv2d(64,128,1)
        self.bn2 = nn.BatchNorm2d(128)
        self.relu = nn.ReLU(inplace=True)
</code></pre>
<p>I can write the forward function in 2 ways:<br>
(1)</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.relu(self.bn1(self.conv1(x)))
    x = self.relu(self.bn2(self.conv2(x)))
    return x
</code></pre>
<p>(2)</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.relu(self.bn2(self.conv2(self.relu(self.bn1(self.conv1(x))))))
    return x
</code></pre>
<p>Does (2) use less memory than (1)?</p>
<p>Also, when relu is inplace, I get an error if I do:</p>
<pre><code class="lang-auto">x = conv1(x)
x = bn1(x)
x = relu(x)
</code></pre>
<p>However, it is fine if I do:<br>
<code>x = relu(bn1(conv1(x)))</code></p>
<p>Is it because relu is inplace I can’t assign a local variable to its result?</p>
Loading tfrecord using pytorch causes error

Zhongqi_Miao: <p>Hello. I am recently trying to load tfrecords using pytorch. However, it seems that if I load <code>tf.data.TFRecordDataset</code> in pytorch datasets and use dataloader with <code>num_workers</code> &gt; 0, the program won’t work properly. I am wondering if there is any better ways to load tfrecords or other better ways to store large scale datasets.</p>
<p>Here are the example codes:</p>
<pre><code class="lang-auto">class TestDataset(Dataset):
    def __init__(self, record_path):

        self.record_path = record_path
        self.reader = tf.data.TFRecordDataset(self.record_path).map(decoder)
        self._records_iter = self.reader.make_one_shot_iterator()
        
    def __len__(self):
        return 100

    def _parser(self, img):
        image_arr = np.frombuffer(img, dtype=np.uint8)
        sample = torch.tensor(image_arr)
        return sample

    def __getitem__(self, item):
        sample = next(self._records_iter).numpy()
        return self._parser(sample)

dataset = TestDataset(0, path)

loader = DataLoader(dataset, batch_size=1, num_workers=1)

for i in loader:
    print(i)

</code></pre>

Zhongqi_Miao: <p>OK, I fixed the problem using <code>tf.python_io.tf_record_iterator</code> although it is deprecated in tensorflow.</p>
Loss doesn’t Decrease and the output is zero

arianhf: <p>I’m trying to implement an autoencoder in pytorch but all my outputs are zero and i don’t know why <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>here is my code for the autoencoder:</p>
<pre><code class="lang-auto">
class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(686, 256),
            nn.ReLU(),
            nn.Linear(256, 64),
            nn.ReLU())
        self.decoder = nn.Sequential(
            nn.Linear(64, 256),
            nn.ReLU(),
            nn.Linear(256, 686),
            nn.ReLU())
          
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>and here is the training process:</p>
<pre><code class="lang-auto">iterations = 10
learning_rate = 0.98
criterion = nn.MSELoss()

optimizer = torch.optim.Adam(
    net.parameters(), lr=learning_rate, weight_decay=1e-5)


for epoch in range(iterations):
    runningLoss = 0.0
    for i, data in enumerate(train_dl, 0):
        inputs, labels = data
        if use_gpu:
            inputs = Variable(inputs.view(-1,686).double()).cuda()
        else:
            inputs = Variable(inputs.view(-1,686).double())
        outputs = net(inputs)
        loss = criterion(outputs, inputs)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        runningLoss += loss.data.item()
    
    print(f'at iteration: {epoch+1}/{iterations}; BC Error: {runningLoss}')
print('Finished Training')
</code></pre>

ptrblck: <p>Could you try to remove the last relu and run the code again?</p>

arianhf: <p>ok, i tried it, this time the outputs aren’t zero anymore but it seems like it doesn’t converge, and the error is very high.</p>

ptrblck: <p>That’s a good starting point to play around with some hyperparameters, e.g. lowering the learning rate etc.</p>

arianhf: <p>Thank you very much, will update here if I have more questions <img src="https://discuss.pytorch.org/images/emoji/apple/v/3.png?v=9" title=":v:t3:" class="emoji" alt=":v:t3:"></p>