akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Calculating affine transformation matrix

kekday: <p>So I am trying to learn PyTorch and as an experiment I tried to apply a specific geometric transform (rotation by 45 degrees) to an image using torch.nn.functional.affine_grid and torch.nn.functional.grid_sample. And now from this new image I try to work backwards using the MSE between the images to recover the transformation matrix. The loss does seem to come down but ultimately it doesn’t quite do the job.</p>
<p>Here’s a link to a gist with the notebook: <a href="https://gist.github.com/abnan/19782d46e8a8396267d2aee44b48a3de" rel="nofollow noopener">https://gist.github.com/abnan/19782d46e8a8396267d2aee44b48a3de</a></p>
<p>Any pointers on what I am doing wrong? Also in general what/how to debug such cases?</p>

edgarriba: <p>if you want to go more high level you can use Kornia library<br>
<a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.rotate" class="onebox" target="_blank" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.rotate</a></p>
Saving and Loading Model Error

Dafz: <p>I retained a densenet121 model for binary classification. please How do I save and reload this model to fit with the architecture and the saved parameters.<br>
Thanks in advance</p>

LeviViana: <p>You should you the model’s state_dict to save and load it again. I’d recommend you to take a look at <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">this tutorial</a>.</p>
How to remove `for loop` for this snippet?

Zhaoyi-Yan: <pre><code class="lang-python">bz=2,
h = 5
w = 5
a = torch.zeros(bz, h, w)

idx_1 = torch.tensor([[0,2,1],[2,3,0]])
idx_2 = torch.tensor([[1,2,3],[3,3,4]])

# Is it possible to remove `for loop` here ? 
for i in range(bz):
    a[i][idx_1[i], idx_2[i]] = 1

</code></pre>
<p>Expected results:</p>
<pre><code class="lang-bash">tensor([[[0,1,0,0,0],
             [0,0,0,1,0],
             [0,0,1,0,0],
             [0,0,0,0,0],
             [0,0,0,0,0]],

             [[0,0,0,0,1],
              [0,0,0,0,0],
              [0,0,0,1,0],
              [0,0,0,1,0],
              [0,0,0,0,0]]])
</code></pre>

Tony-Y: <pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; bz=2; h=5; w = 5;
&gt;&gt;&gt; a = torch.zeros(bz, h, w)
&gt;&gt;&gt; idx_1 = torch.tensor([[0,2,1],[2,3,0]])
&gt;&gt;&gt; idx_2 = torch.tensor([[1,2,3],[3,3,4]])
&gt;&gt;&gt; idx_b = torch.arange(bz).long().unsqueeze(1).expand(bz,3)
&gt;&gt;&gt; idx = idx_b * h*w + idx_1 * w + idx_2
&gt;&gt;&gt; b = a.view(bz*h*w)
&gt;&gt;&gt; b[idx] = 1
&gt;&gt;&gt; a
tensor([[[0., 1., 0., 0., 0.],
         [0., 0., 0., 1., 0.],
         [0., 0., 1., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.]],

        [[0., 0., 0., 0., 1.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 1., 0.],
         [0., 0., 0., 1., 0.],
         [0., 0., 0., 0., 0.]]])
</code></pre>
<p>This code can be made simple by using <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_put_" rel="nofollow noopener">index_put</a>.</p>
<pre><code class="lang-auto">a[(idx_b,idx_1,idx_2)] = 1
</code></pre>

Zhaoyi-Yan: <p>Thank you for your help！ It is a nice solution!</p>
WGAN Discriminator Implementation Error

Ravi_Gupta: <p>A/C to me, My code is totally correct but I got an NotImplementedError. i try different to fix it by changing some layers, reviewing my times the same code.<br>
I am not expert in Pytorch but my concept is totally right about the connection. But still i don,t know how to fix it. Anyone help me</p>
<p>code link:- <a href="https://drive.google.com/open?id=16H3oOmubok1auXVPrbiYNpuM1kZTucmK" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a1ad44f48d691e64259ff9781847d0f4d95216e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/a1ad44f48d691e64259ff9781847d0f4d95216e2" title="wgan_error"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a1ad44f48d691e64259ff9781847d0f4d95216e2.png" alt="wgan_error" data-base62-sha1="n4g6eO9va7QNkKREdDx8BKu1aTg" width="690" height="283" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">wgan_error</span><span class="informations">1243×511 19.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Ravi_Gupta: <p>please help me ,it’s very important and necessary to complete the project</p>

tom: <p>Your indentation is wrong, so the def discriminator(…) is local to <code>__init__</code> rather than in the discriminator class.<br>
Some of the code sources might not be quite au courant (like using <code>Variable</code>…).</p>
<p>Best regards</p>
<p>Thomas</p>

Ravi_Gupta: <p>thank you sir… it solve my problem effectively</p>

Ravi_Gupta: <p>sir what is the concept of weight sharing in discriminator in FMGAN</p>
How to use batchnorm1d and pack_padded_sequence at the same time?

wangyanda: <p>hello everyone, I have a question about batchnorm1d and pack_padded_sequence.<br>
I have an encoder-decoder model, and for the encoder, I use pack_padded_sequence to construct a batch input for it. Meanwhile, I also want to apply batchnorm1d to the encoder to avoid overfitting, so how should I do it.<br>
Thanks in advance.</p>
Retrain new data to pretrained lstm model

riyu94: <p>Hi I am training data on a lstm model using pytorch<br>
The model works fine when tested.<br>
But i want to improve accuracy if new sequences are detected in data.<br>
I am trying to load this pretrained model and train the model again with new data.<br>
But it doesn’t seem to detect the sequences from the pretrained model (it overwrites the pretrained model)<br>
What steps can I follow to use pretrained model as well as the new trained data for further testing.<br>
Any help appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
nn.MultiheadAttention

yngtodd: <p>Can someone explain what <code>embed_dim</code> is in <code>nn.MultiheadAttention</code>? The docs say that it is the <code>total dimension of the model</code>, but I am not quite sure what that means.</p>
<p>Would anyone be able to give a minimal example using <code>nn.MultiheadAttention</code>?</p>
<p><a href="https://pytorch.org/docs/master/nn.html#multiheadattention" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/nn.html#multiheadattention</a></p>

Ruben_Chaves: <p>The number of the dimensions the tensor have after passing trought a embedding layer. In hte paper Attention is all you need, it would be equal to d_model.</p>
I have trained a model on GPU and i want to use it's weights for inference on CPU?

Shashank_Shekhar: <ol>
<li>
<p>all my weights and biases tensors are in net.parameters with device (cuda:0) , but i want to convert those tensors to cpu so that i can use it for inferencing.<br>
prediction snippet…<br>
prediction=network_instance(test)</p>
</li>
<li>
<p>when we do optimizer.step() does it keep on updating the weights till the prediction is correct or it updates the weights for only a single backward pass.</p>
</li>
</ol>

LeviViana: <aside class="quote no-group" data-username="Shashank_Shekhar" data-post="1" data-topic="46849">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shashank_shekhar/48/12704_2.png" class="avatar"> Shashank_Shekhar:</div>
<blockquote>
<ol>
<li>all my weights and biases tensors are in net.parameters with device (cuda:0) , but i want to convert those tensors to cpu so that i can use it for inferencing.<br>
prediction snippet…<br>
prediction=network_instance(test)</li>
</ol>
</blockquote>
</aside>
<p>You should use the model’s <code>state_dict</code> for saving and loading. Take a look at <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">this tutorial</a>. You can send all of your network’s parameters to the cpu just by doing <code>model  = model.cpu()</code>.</p>
<aside class="quote no-group" data-username="Shashank_Shekhar" data-post="1" data-topic="46849">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shashank_shekhar/48/12704_2.png" class="avatar"> Shashank_Shekhar:</div>
<blockquote>
<ol start="2">
<li>when we do optimizer.step() does it keep on updating the weights till the prediction is correct or it updates the weights for only a single backward pass.</li>
</ol>
</blockquote>
</aside>
<p><code>optimizer.step()</code> only does a single backward pass.</p>
Where is c++ source code

Xin_Perry: <p>I can’t find the source code of <strong><code>torch.binary_cross_entropy_with_logits</code></strong> using <em>pycharm</em>. it only leads me to the <strong>__ init __.pyi</strong>. I guess the function is implemented by c++, but how could I find the source file ?</p>

LeviViana: <p>Here: <a href="https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L96-L112" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L96-L112</a></p>
Resnet not converging!

Diaa_elsayed_ziada: <p>When i’m using pytorch resnet model, model not converging,but when using other architecture model the training process work will.<br>
configurations :<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)</p>
<p>Resnet model</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 10)
net = model_ft

</code></pre>
<p>training steps:</p>
<pre><code class="lang-auto">[1,     4] loss: 526.469 acc: 348.250 
1393
[2,     4] loss: 529.261 acc: 346.750 
1387
[3,     4] loss: 532.855 acc: 349.250 
1397
[4,     4] loss: 530.508 acc: 348.000 
1392
[5,     4] loss: 526.442 acc: 347.000 
1388
[6,     4] loss: 531.521 acc: 348.500 
1394
[7,     4] loss: 527.574 acc: 348.250 
1393
[8,     4] loss: 527.463 acc: 349.000 
1396
[9,     4] loss: 527.096 acc: 347.250 
1389
[10,     4] loss: 529.270 acc: 348.750 
1395
[11,     4] loss: 527.993 acc: 348.500 
1394
[12,     4] loss: 532.372 acc: 349.000 
1396
[13,     4] loss: 530.069 acc: 348.250 
1393
[14,     4] loss: 531.173 acc: 348.000 
1392
[15,     4] loss: 530.445 acc: 347.250 
1389
Finished Training
type or paste code here
</code></pre>
<p>example for model that works well</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 13 * 13, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 13 * 13)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>training steps:</p>
<pre><code class="lang-auto">[1,     4] loss: 578.536 acc: 355.250 
1421
[2,     4] loss: 432.506 acc: 438.250 
1753
[3,     4] loss: 359.835 acc: 503.250 
2013
[4,     4] loss: 289.072 acc: 544.250 
2177
[5,     4] loss: 251.283 acc: 568.500 
2274
[6,     4] loss: 237.213 acc: 575.750 
2303
[7,     4] loss: 207.391 acc: 590.250 
2361
[8,     4] loss: 192.430 acc: 598.500 
2394
[9,     4] loss: 178.400 acc: 605.250 
2421
[10,     4] loss: 172.829 acc: 609.500 
2438
[11,     4] loss: 150.899 acc: 617.000 
2468
[12,     4] loss: 160.563 acc: 614.750 
2459
[13,     4] loss: 139.884 acc: 623.750 
2495
[14,     4] loss: 124.817 acc: 629.000 
2516
[15,     4] loss: 132.767 acc: 626.000 
2504
Finished Training
</code></pre>
Onnx export error with dict?

vince62s: <p>Hi,<br>
got this error:<br>
File “/usr/local/lib/python3.5/dist-packages/torch/jit/<strong>init</strong>.py”, line 201, in forward<br>
out_vars, _ = _flatten(out)</p>
<p>Only tuples, lists and Variables supported as JIT inputs, but got dict</p>
<p>But impossible to know where exactly it was triggered in the code<br>
it seems similar to:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/13040" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/olgapeled" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/34312702?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/13040" target="_blank" rel="nofollow noopener">Issue: RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got dict</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/olgapeled" target="_blank" rel="nofollow noopener">olgapeled</a>
	on <a href="https://github.com/pytorch/pytorch/issues/13040" target="_blank" rel="nofollow noopener">2018-10-24</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">Hi All,
I am trying to convert yolo2 pytorch to onnx model and I got:
"RuntimeError: Only tuples, lists and Variables supported as...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>thanks</p>

vince62s: <p>answering to myself:</p>
<p>a forward in a nn.Module cannot return a dict, which is really too bad.</p>

deepak_mangla: <p>It helped me. You found some turnaround if I need to return dictionary or tuple? Only thing I can think right now is global variable.</p>
Why must 'fillcolor' be an int? (torchvision.transforms.RandomAffine)

yepw: <p>Hi,</p>
<p>I am using <code>torchvision.transforms.RandomAffine</code> to do some data augmentation. My inputs are RGB images. However, I don’t really understand one argument, <code>fillcolor</code>, the color for the area outside the transform in the output image. It seems <code>fillcolor</code> must be an integer, but what if I want to fill the area with some RGB value? Is there any way to convert an RGB (a tuple of length 3) to an int?</p>
<p>Thank you in advance!</p>

ptrblck: <p>You can pass a RGB color code as a <code>tuple</code> to this function:</p>
<pre><code class="lang-python">trans = transforms.RandomAffine(45, fillcolor=(0, 0, 255))
img = TF.to_pil_image(torch.randn(3, 100, 100))
trans(img)
</code></pre>
<p>This will use blue to fill the pixels.<br>
Could you create an issue so that the docs might be updated, as this seems indeed to be confusing.</p>

yepw: <p>Hi ptrblck, thank you for your reply.<br>
I updated my <code>pillow</code> version and it works!</p>
What is the equivalent of np.std() in Pytorch?

Md_Mahfujur_Rahman_0: <p>To get the mean and variance in tensorflow just use tf.nn.moments.</p>
<p>mean, var = tf.nn.moments(x, axes=[1])</p>
<p>and in numpy mean, var =  np.std(x, axes=[1])</p>
<p>What is the equivalent function of getting variance in pytorch?</p>

colesbury: <p>Tensor.mean([dim]), Tensor.std([dim]), and Tensor.var([dim]) for mean, standard deviation, and variance optionally along an axis (dim).</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.Tensor([[0, 1], [2, 3]])
&gt;&gt;&gt; x.mean(dim=1)
 0.5000
 2.5000
[torch.FloatTensor of size 2]
&gt;&gt;&gt; x.std(dim=1)
 0.7071
 0.7071
[torch.FloatTensor of size 2]</code></pre>

Md_Mahfujur_Rahman_0: <p>Thanks for your reply. In pytorch, is gradients calculate for mean and standard deviation automatically?</p>

Md_Mahfujur_Rahman_0: <p>Thanks <a class="mention" href="/u/colesbury">@colesbury</a>. It works for me.</p>
<p>import torch<br>
a = torch.randn(4, 4)<br>
torch.std(a, dim=1)</p>

kuzand: <p>Btw, the std computed by numpy is different!</p>
<pre><code class="lang-auto">import numpy
&gt;&gt;&gt; x = numpy.array([[0, 1], [2, 3]])
&gt;&gt;&gt; x.std(1)
array([0.5, 0.5])
</code></pre>
<p>Which one should we use if we want e.g. to compute the std for <code>transforms.Normalize(mean, std)</code> ?</p>

JayanthRR: <p><a class="mention" href="/u/kuzand">@kuzand</a>  <code>np.std(), np.var()</code> have an additional parameter <code>ddof</code> which is the degrees of freedom. Type in <code>x.std(1,ddof=1)</code> and you’ll get the same answer. Regarding transforms, all <code>ddof</code> does is to divide by a different denominator, so I don’t think that will affect it much. Look at this thread <a href="https://stackoverflow.com/questions/41204400/what-is-the-difference-between-numpy-var-and-statistics-variance-in-python" rel="nofollow noopener">here</a>.</p>
What is the algorithm theory of torch.nn.AdaptiveAvgPool2d?

quinwu: <p>what is the algorithm theory of torch.nn.AdaptiveAvgPool2d?<br>
Is there any papers about torch.nn.AdaptiveAvgPool2d?</p>

ptrblck: <p>I’m not sure if there is a paper about it, but it is basically an average pooling layer with dynamic kernel so that the output will have the specified shape. It’s useful in case your input shapes are variable, e.g. to create a fixed number of features.</p>
Has anyone implemented multi label classification using hierarchical attention network?

Pratik_Sharma: <p>I am going through hierarchical attention network…</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.semanticscholar.org/img/favicon.png" class="site-icon" width="32" height="32">
      <a href="https://www.semanticscholar.org/paper/Hierarchical-Attention-Networks-for-Document-Yang-Yang/1967ad3ac8a598adc6929e9e6b9682734f789427" target="_blank" rel="nofollow noopener">semanticscholar.org</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://www.semanticscholar.org/img/semantic_scholar_og.jpg" class="thumbnail"></div>

<h3><a href="https://www.semanticscholar.org/paper/Hierarchical-Attention-Networks-for-Document-Yang-Yang/1967ad3ac8a598adc6929e9e6b9682734f789427" target="_blank" rel="nofollow noopener">Hierarchical Attention Networks for Document Classification - Semantic Scholar</a></h3>

<p>We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hierarchical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I found the paper fascinating. I was wondering if HAN supports multi-label classification as well? Has anyone tried it yet in pytorch? Can I get some suggestions for implementing it?</p>
Save hidden states just before prediction

slqch: <p>Hello, I’m a newbie to pytorch, I’m trying to do the following :</p>
<ul>
<li>Create a function that takes as an input my text and returns a list of the indices of my tokens.</li>
<li>Create  a function that has as an input batch size ( batch_size, seq_len) and returns hidden states : (batch_size, hidden_states)<br>
I included my model and the forward function that I belive have to make a forward_prime function that stops right before the prediction layer and save the hidden states , but I don’t know how to do it.</li>
</ul>
<pre><code class="lang-auto">class ClassificationMultiLabel(nn.Module):
    def __init__(self, op_size, n_tokens, pretrained_vectors, nl=2, bidirectional=True, emb_sz=300, n_hiddenUnits=100):
        super(ClassificationMultiLabel, self).__init__()
        self.n_hidden = n_hiddenUnits
        self.embeddings = nn.Embedding(n_tokens, emb_sz)
        self.embeddings.weight.data.copy_(pretrained_vectors)
        #         self.embeddings.weight.requires_grad = False
        self.rnn = nn.LSTM(emb_sz, n_hiddenUnits, num_layers=2, bidirectional=True, dropout=0.2)
        self.lArr = []
        if bidirectional:
            n_hiddenUnits = 2 * n_hiddenUnits
        self.bn1 = nn.BatchNorm1d(num_features=n_hiddenUnits)
        for i in range(nl):
            if i == 0:
                self.lArr.append(nn.Linear(n_hiddenUnits * 3, n_hiddenUnits))
            else:
                self.lArr.append(nn.Linear(n_hiddenUnits, n_hiddenUnits))
        self.lArr = nn.ModuleList(self.lArr)
        self.l1 = nn.Linear(n_hiddenUnits, op_size)

    def forward(self, data, lengths):
        torch.cuda.empty_cache()
        bs = data.shape[1]
        self.h = self.init_hidden(bs)
        embedded = self.embeddings(data)
        embedded = nn.Dropout()(embedded)
        #         embedded = pack_padded_sequence(embedded, torch.as_tensor(lengths))
        rnn_out, self.h = self.rnn(embedded, (self.h, self.h))
        #         rnn_out, lengths = pad_packed_sequence(rnn_out,padding_value=1)
        avg_pool = F.adaptive_avg_pool1d(rnn_out.permute(1, 2, 0), 1).view(bs, -1)
        max_pool = F.adaptive_max_pool1d(rnn_out.permute(1, 2, 0), 1).view(bs, -1)
        ipForLinearLayer = torch.cat([avg_pool, max_pool, rnn_out[-1]], dim=1)
        for linearlayer in self.lArr:
            outp = linearlayer(ipForLinearLayer)
            ipForLinearLayer = self.bn1(F.relu(outp))
            ipForLinearLayer = nn.Dropout(p=0.6)(ipForLinearLayer)
        outp = self.l1(ipForLinearLayer)
        return outp
</code></pre>
<p>full code can be found here : <a href="https://colab.research.google.com/drive/1dKvMQWukbhujgXNWJxKRJSqcfBU1IWZl" rel="nofollow noopener">https://colab.research.google.com/drive/1dKvMQWukbhujgXNWJxKRJSqcfBU1IWZl</a></p>
Torch::cuda::is_available() false (again) - WIN 10

Alex1: <p>Ok, I upgraded Pytorch from master today, built successfully <em>with</em> cuda and cudnn support - Win 10, CUDA 9.2. But my C++ app started to report that torch::cuda::is_available() is false. There was a similar issue in the past - <a href="https://github.com/pytorch/pytorch/pull/16071" rel="nofollow noopener">see here</a> Now torch.dll and torch.lib seem to be just some placeholders, so I have no idea how to link my app correctly to force torch::cuda::is_available() to return true again in a C++ app.</p>
<p>Does anybody have an idea, what’s wrong here?</p>
<p>Thanks.</p>
<p>Alex</p>

Alex1: <p>As a follow-up, I found out that linking against all the libs (c10.lib, c10_cuda.lib, caffe2.lib and caffe2_gpu.lib) makes the resulting binary depend only on c10.dll and caffe2.dll. That’s why  torch::cuda::is_available() is false. In previous versions I was linking against torch.lib which made the resulting binary depend on torch.dll which in turn depended on both caffe2.dll and caffe2_gpu.dll.</p>
<p>This is a blocking issue for all Wiin C++ users (am I the only one experiencing this?). I do not even understand why torch.lib and torch.dll are just placeholders now…</p>
<p>Thanks.</p>
<p>Alex</p>

Alex1: <p>Ok, this is a real issue that has to do with <a href="https://github.com/pytorch/pytorch/pull/20774" rel="nofollow noopener">this</a> and is also tracked <a href="https://github.com/pytorch/pytorch/pull/16071#issuecomment-497681633" rel="nofollow noopener">here</a></p>
Weight Share Concept in GAN(Specially FMGAN)

Ravi_Gupta: <p>what is the concept of weight sharing concept and their implementation in GAN?</p>
<p>Is anyone have the implementation of FMGAN which uses the concept of Weight Sharing in Discriminator?</p>

Ravi_Gupta: <p>please help me ,it’s very important and necessary to complete the project</p>
PyTorch time profiling weird behavior

jiashenC: <p>I have two neural networks (one is deeper than the other). I try to profile time in PyTorch to see how they behave differently in GPU. I have below code snippets to profile my program.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import cuda
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
from torchvision.datasets import FakeData

def profile():
    dataset = FakeData(size=10000, image_size=(3,224,224), transform=transforms.ToTensor())
    data_loader = DataLoader(dataset, num_workers=2, batch_size=60)

    net = torch.load(...).cuda()
    net.eval()

    computation_cost, communication_cost = 0, 0
    with torch.no_grad():
        for images, labels in data_loader:
            # profiling communication time
            start = cuda.Event(enable_timing=True)
            start.record()
            images = images.cuda()
            labels = labels.cuda()
            end = cuda.Event(enable_timing=True)
            end.record()
            cuda.synchronize()
            communication_cost += start.elapsed_time(end)

            # profiling computation time
            start = cuda.Event(enable_timing=True)
            start.record()
            _ = net(images)
            end = cuda.Event(enable_timing=True)
            end.record()
            cuda.synchronize()
            computation_cost += start.elapsed_time(end)
</code></pre>
<p>I got following results from this program.</p>
<blockquote>
<p>Deep network, computation: 22679 ms, communication: 1342 ms<br>
Shallow network, computation: 3180 ms, communication: 1374 ms</p>
</blockquote>
<p>Based on this result, I expect the shallow network program total runtime is also close to 15% of deep network program total runtime.</p>
<p>I used below slightly modified program to profile the total runtime.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import cuda
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
from torchvision.datasets import FakeData

def profile():
    dataset = FakeData(size=10000, image_size=(3,224,224), transform=transforms.ToTensor())
    data_loader = DataLoader(dataset, num_workers=2, batch_size=60)

    net = torch.load(...).cuda()
    net.eval()

    start = cuda.Event(enable_timing=True)
    start.record()

    with torch.no_grad():
        for images, labels in data_loader
            images = images.cuda()
            labels = labels.cuda()
            _ = net(images)

    end = cuda.Event(enable_timing=True)
    end.record()
    cuda.synchronize()

    total = start.elapsed_time(end)
</code></pre>
<p>But the result is not what I expect.</p>
<blockquote>
<p>Deep network, total: 25551 ms<br>
Shallow network, total: 20691 ms</p>
</blockquote>
<p>Do I do anything wrong in the second profiling program?</p>
Need help with setting Weight

Stefan_bunny: <p>I am new to Pytorch and need some help. I have a simple ANN:</p>
<p>n_in, n_h0, n_h1, n_h2, n_out, batch_size = 12, 0, 0, 0, 8, 32</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(n_in, n_out)<br>
def forward(self, x):<br>
x = self.fc1(x)<br>
x = nn.Tanh()(x)<br>
return x</p>
<p>model = Net()</p>
<p>I want to change some weights by myself. I try:</p>
<p>model.fc1.weight[0][0] = 0.5</p>
<p>This changes in model.fc1.weight:<br>
requires_grad=True to grad_fn=CopySlices&gt;<br>
Also if i now train the model i get the message: ValueError: can’t optimize a non-leaf Tensor.<br>
Can someone please help how to do it ?</p>

Nikronic: <p>Hi,<br>
You should access the weights by calling <code>data</code>. Here is an example:</p>
<pre><code class="lang-python">for x in model.parameters():
    print(x.size(), x.data)

p = list(disc_one.parameters())

p[1].data[0] = 3 #####
p[1]

</code></pre>
<p>Actually, here is your code:</p>
<pre><code class="lang-python">n_in, n_h0, n_h1, n_h2, n_out, batch_size = 12, 0, 0, 0, 8, 32

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(n_in, n_out)

    def forward(self, x):
        x = self.fc1(x)
        x = nn.Tanh()(x)
        return x
model = Net()

model.fc1.weight.data[0][0] = 0.5

</code></pre>
<p>By the way, you can format your code by putting “```” before and at the end of your code.</p>
<p>best</p>

Stefan_bunny: <p>It works.</p>
<p>Many Thanks</p>

Nikronic: <p>Your Welcome, Good luck.</p>
Dataloader - what should be the input type to collate_fn?

HaziqRazali: <p>What should be the <code>type</code> of the data I return through the <code>__getitem__</code> method of the dataloader class if I use the default <code>collate_fn</code>? I am confused because the dimensions I get are different depending on the type that I return.</p>
<p>If I return a list, then the length of my data is 34 instead of <code>batch_size</code>.</p>
<pre><code>def __getitem__(self, index):

    df = self.df.iloc[index]
    keypoint = df["keypoint"]
    # print(type(keypoint)) = &lt;class 'list'&gt;
    # print(np.shape(keypoint)) = (34,)                                                                                                                        
    return keypoint
</code></pre>
<p>If I return a tensor, then the length of my data is <code>batch_size</code>. What is happening in the collate_fn?</p>
<pre><code>def __getitem__(self, index):

    df = self.df.iloc[index]
    keypoint = torch.Tensor(np.array(df["keypoint"]))                                                                                                                         
    return keypoint</code></pre>
Fine tune model on new dataset with different outputs

Mdhvince: <p>Hi,</p>
<p>I have a complex problem with one of my project and I hope you can help me solve it or just give me some tips.</p>
<p>I have model fully trained that takes as input an image and outputs 16 coordinates, then compare the outputs to the ground truth <strong>16 coordinates</strong>.</p>
<p>But now, I want to fine tune my model on another dataset. The problem is that the model will output <strong>16 coordinates</strong> will the ground truth coordinates numbers of this new dataset is <strong>12 (not 16)</strong>.</p>
<p>So my question is, is there a way with PyTorch to tell my model, ok you can compute 16 coordinates, but when comparing your output with the ground truth, do not take in consideration the coordinates N. Where N is the 4 additional coordinates which are missing in the dataset.</p>
<p>I really need your help on this one.</p>
<p>Thank you,</p>
<p>Medhy</p>

Mdhvince: <p>I want to add that I do not need to know the predictions for these 4 missing coordinates.</p>

achaiah: <aside class="quote no-group" data-username="Mdhvince" data-post="1" data-topic="46672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mdhvince/48/56862_2.png" class="avatar"> Mdhvince:</div>
<blockquote>
<p>So my question is, is there a way with PyTorch to tell my model, ok you can compute 16 coordinates, but when comparing your output with the ground truth, do not take in consideration the coordinates N. Where N is the 4 additional coordinates which are missing in the dataset.</p>
</blockquote>
</aside>
<p>I don’t really think that’s what you want to do. Instead, you should just remove the last layer of your model (that used to output 16 coordinates) and replace it with a new layer just like it but with 12 outputs.</p>

Mdhvince: <p>Hmm, unfortunately, it doesn’t work for me, So I have created the four missing output by another model.</p>
Loss not converging with Adagrad

sharad: <p>Hi. I am new to Pytorch and trying to implement various Optimizer algorithms using same. While implementing Adagrad, loss function does not converge as expected. I tried to see gradients of layers on different batches and they start to become zero very early during training. I am not able to figure out the reason for network to not converge. Is there a bug in my implementation? Can you please point me in the right direction?</p>
<p>I have included the picture of implementation <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6e696dafa4ad517e5f83616e4e6f134039315299.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e696dafa4ad517e5f83616e4e6f134039315299" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6e696dafa4ad517e5f83616e4e6f134039315299.png" alt="Capture" data-base62-sha1="fKKoS9SCYQbkhdBMckkEcULJYBH" width="636" height="500" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1045×821 42 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I think a good method to check your custom implementation would be to compare the gradients and updated with the <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adagrad.html" rel="nofollow noopener">PyTorch implementation</a>.</p>

sharad: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , I figured out the issue, I was calculating square of my gradient tensor wrongly.</p>
Creating patches from image

rbunn80110: <p>If I want to split grayscale images into 14x14 patches (4 pieces total for each image in the code below) I tried the following code:</p>
<p>S = 1 # channel dim<br>
W = 28 # width<br>
H = 28 # height<br>
batch_size = 10</p>
<p>x = torch.randn(batch_size, S, W, H)</p>
<p>size = 14 # patch size<br>
stride = 14 # patch stride<br>
patches = x.unfold(2, size, stride).unfold(3, size, stride)<br>
print(patches.shape)</p>
<p>Which returns:<br>
torch.Size([10, 1, 2, 2, 14, 14])</p>
<p>I believe the shape I want is torch.Size([10, 1, 4, 14, 14])<br>
Which if you viewed each of the 4 pieces it should show each of the four quadrants of the image (in this example at least). So four sub-images each 14x14 pixels. How would I unfold the images to get this result?</p>

ptrblck: <p>You already have the results. The 4 patches are stored as <code>2x2</code> patches in dim2 and dim3, so you can just call <code>.contiguous().view(batch_size, S, -1, size, size)</code> on the result.</p>

rbunn80110: <p>It just hit me right before I saw your answer.  LOL  Not sure why I didn’t consider it before asking the question.  Thanks for the fast reply!</p>

edgarriba: <p>use this: <a href="https://kornia.readthedocs.io/en/latest/contrib.html#kornia.contrib.extract_tensor_patches" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/contrib.html#kornia.contrib.extract_tensor_patches</a></p>
C++ runs slower than python version

songziqin: <p>I have a requestion, the C++ code belows:<br>
auto start = std::chrono::system_clock::now();<br>
//sparse_segment_forward(input_batch, values, indices);<br>
auto a = torch::randn({10000, 100});<br>
auto b = torch::randn({1000,10000});<br>
auto ret = b.mm(a);<br>
auto end = std::chrono::system_clock::now();<br>
std::chrono::duration elapsed_seconds = end - start;<br>
std::cout &lt;&lt; "sparse embedding elapsed time: " &lt;&lt; elapsed_seconds.count() &lt;&lt; “s\n”;<br>
it shows the mm operation cost 0.5s.<br>
but my python version as follows:<br>
x = torch.randn(10000,100,device=device, dtype=dtype)<br>
y = torch.randn(1000, 10000, device=device, dtype=dtype)<br>
import torch<br>
device = torch.device(“cpu”)<br>
dtype = torch.float<br>
import time<br>
start = time.time()<br>
x =  y.mm(x)<br>
end = time.time()<br>
print end -start<br>
python program costs 0.02s。 Both run the same machine,so I want to know if I have not use the cpu<br>
openblas,and I want to ask how to use the openblas in C++.thanks.</p>
Computing the gradient with respect to a hidden layer's output

jerrie: <p>I’d like to compute the gradient with respect to a hidden layer’s output(pred_z in the example below). requires_grad is set to True but the gradient is still None. Why?</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

x_data = Variable(torch.Tensor([[1.0, 2.0], [2.0, 3.0], [3.0, 4.0]]))
y_data = Variable(torch.Tensor([[2.0], [4.0], [6.0]]))


class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(2, 5)
        self.l2 = torch.nn.Linear(5, 1)

    def forward(self, x):
        z = torch.nn.functional.relu(self.l1(x))
        y_pred = self.l2(z)
        return y_pred, z

# our model
our_model = Model()

criterion = torch.nn.MSELoss(size_average = False)
optimizer = torch.optim.SGD(our_model.parameters(), lr = 0.01)

for epoch in range(500):

    # Forward pass: Compute predicted y by passing
    # x to the model
    pred_y, pred_z = our_model(x_data)

    # Compute and print loss
    loss = criterion(pred_y, y_data)

    # Zero gradients, perform a backward pass,
    # and update the weights.
    optimizer.zero_grad()
    loss.backward()
    print('dL/dz', pred_z.grad)
    optimizer.step()
    print('epoch {}, loss {}'.format(epoch, loss.data[0]))

new_var = Variable(torch.Tensor([[4.0, 5.0]]))
pred_y = our_model(new_var)
print("predict (after training)", [4,5], our_model(new_var)[0].data[0])
</code></pre>
<p>I’d like to compute a gradient with of the loss w/ respect to z but I don’t actually want to apply it.</p>
Problem with batch size and torch.nn.DataParallel

ricky123: <p>I was using pytorch for my project and observed a strange phenomenon. In my DataLoader, when I was putting batch size as 16 or 32, the system was hanging. It was particularly working for the batch size 10. Even a less batch size was getting stuck. For testing purpose, I removed the torch.nn.DataParallel wrapper on my model and now it is working for all batch sizes, but it is using a single GPU. For my purpose, I need multiple GPU. So, what problem is happening with data parallel? Is there any solution available? Please reply fast because there is some urgency.</p>
Share specific weights between two defined layers

mahsa: <p>I have defined my own layer which contains two sets of weights (w1,w2). Now I want to use my layer and define a model. My model includes two layers, but I need to use the same weight for one set of weights in both layers. How can I do that? Actually, my code is as follow:</p>
<pre><code class="lang-auto">class Layer(Module):
    def __init__(self):
        super(Layer, self).__init__()
        self.weight1 = Parameter(torch.FloatTensor(F1, F3))
        self.weight2 = Parameter(torch.FloatTensor(F2, F3))

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = Layer()
        self.l2 = Layer()
</code></pre>
<p>I want to both l1 and l2 have the same weight2, but I don’t know how to do it.<br>
Thanks</p>
Network diagram interpretation

che85: <p>Hi guys,</p>
<p>I am trying to implement the segmentation network proposed in [1].</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/723854e20e4816d1e805addea27bf961951486cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/723854e20e4816d1e805addea27bf961951486cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/723854e20e4816d1e805addea27bf961951486cc_2_690x304.png" alt="image" data-base62-sha1="gir7wWjScB1pypwqaP7xnGqFD3u" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/723854e20e4816d1e805addea27bf961951486cc_2_690x304.png, https://discuss.pytorch.org/uploads/default/original/2X/7/723854e20e4816d1e805addea27bf961951486cc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/723854e20e4816d1e805addea27bf961951486cc.png 2x" data-dominant-color="EEEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">758×335 49.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have two questions regarding the interpretation of that diagram:</p>
<ol>
<li>Following the down convolution, there is always a new cube added in the back. How would you interpret this? I am guessing that it’s the</li>
</ol>
<pre><code class="lang-auto">nfilters * 1                                   nfilters * 1   
   |_ nfilters * 2                       nfilters * 2_|
      |_ nfilters * 3              nfilters * 3_|
          |_ nfilters * 4 | nfilters * 4 _|   
</code></pre>
<ol start="2">
<li>How would I execute the concatenation step? I was thinking about the following:</li>
</ol>
<pre><code class="lang-auto"> |_ nfilters * 3 (left side) ---&gt; compress(squeze to one filter) and concat  --&gt; nfilters * 2 (right side) --&gt; resulting in nfilters * 3

</code></pre>
<p>What do you guys think?</p>
<p>Thanks a lot,</p>
<p>Best<br>
Christian</p>
<p>[1] <a href="https://ieeexplore.ieee.org/document/8051114" class="inline-onebox" rel="noopener nofollow ugc">Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation | IEEE Journals &amp; Magazine | IEEE Xplore</a></p>
CPU utilization freezes after a while (GPU works great)

kishwarshafin: <p>Hello,</p>
<p>I have been trying to make a CPU accelerated inference for our model. The only thing I change from GPU to CPU is <code>torch.set_num_threads(threads)</code>. I am running the model on AWS <code>m5.24xlarge</code> with 96 CPUs. I set the num_threads to 86 and num_workers to 8. It starts pretty well, giving an ETA of ~16hours but after 40% completion, most of the pytorch processes start jumping between “S-R” (stall and running) states. It got stuck at 47%, and I had to kill it.</p>
<p>Should I do something else for a CPU-accelerated version?</p>
<p>This is on Ubuntu 18.04 (the deeplearning AMI available from AWS)<br>
Python3: 3.6.7<br>
PyTorch: 1.1</p>
<p>The model is here: <a href="https://github.com/kishwarshafin/helen/blob/master/modules/python/models/TransducerModel.py" rel="nofollow noopener">https://github.com/kishwarshafin/helen/blob/master/modules/python/models/TransducerModel.py</a></p>
<p>And the way I am doing is here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/kishwarshafin/helen/blob/master/modules/python/models/predict.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/kishwarshafin/helen/blob/master/modules/python/models/predict.py" target="_blank" rel="nofollow noopener">kishwarshafin/helen/blob/master/modules/python/models/predict.py</a></h4>
<pre><code class="lang-py">import sys
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from modules.python.models.dataloader_predict import SequenceDataset
from modules.python.TextColor import TextColor
from tqdm import tqdm
import numpy as np
from modules.python.models.ModelHander import ModelHandler
from modules.python.Options import ImageSizeOptions, TrainOptions
from modules.python.DataStore import DataStore
"""
This script implements the predict method that is used by the call consensus method.

The algorithm is described here:

  1) INPUTS:
    - directory path to the image files generated by MarginPolish
    - model path directing to a trained model
    - batch size for minibatch prediction
</code></pre>

  This file has been truncated. <a href="https://github.com/kishwarshafin/helen/blob/master/modules/python/models/predict.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Any help would be highly appreciated.</p>
Run `torch.distributed.launch` with `-m` option for the script

arc144: <p>Hi I’m using <code>DistributedDataParallel</code> to run my model across multi-GPU with sync BN. However, my script uses relative imports and is supposed to be run with <code>-m</code> option. How can I do this when launching it via <code>torch.distributed.launch</code>?</p>
<p>Example (does not work, but I’d like to do this):<br>
<code>python -m torch.distributed.launch --nproc_per_node 2 -m detector.train --arg1 --arg2</code></p>
<p>Thanks</p>

LeviViana: <p>Take a look at <a href="https://discuss.pytorch.org/t/how-to-use-multigpu-pytorch/46290/2">this snippet</a>, it could help.</p>

arc144: <p>Thanks for your reply but I think you misunderstood. The issue is not running <code>torch.distributed.launch</code> with <code>-m</code> option. The problem is that my <strong>script</strong> uses relative imports and it is supposed to be run with <code>-m</code> option. I reckon that when <code>torch.distributed.launch</code> spawns the script it uses the more natural approach <code>python detector/script.py</code>, whereas I’d like it to call like <code>python -m detector.script</code></p>

LeviViana: <p>You can create a copy of <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py" rel="nofollow noopener">this file</a> and customize it the way you want. <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py#L224" rel="nofollow noopener">Here</a> this module  will spawn parallel processes according to their rank. You could arrange your script so that the <code>cmd</code> could look like <code>cmd = python -m detector.script --local_rank --arg1 --arg2 ...</code>.</p>

arc144: <p>It is unfortunate that I have to make a copy and alter it but I guess it works! Thanks a lot =]</p>
Why the .eval() function is causing fluctuations in output loss?

xaggi: <p>I am trying to write a neural network code for which a basic sanity check I tried was to train and validate it using the same set of input data i.e. validation data = train data.</p>
<p>With .eval() active at the validation time the loss output plots like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/860f0802231ce707ee5f05d33514d3fe8b151810.png" alt="1" data-base62-sha1="j7WbuJv2bt5KZsHcj1EolNdU4fK" width="640" height="480"></p>
<p>In a similar setting when I run the same code without activating .eval() function i.e. inferring the network in training mode, the loss function seems very smooth and as expected, train and validation loss are pretty much the same:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/26ec7fb99fc9c89cf27451fc541a9cce727d9f71.png" alt="2" data-base62-sha1="5ykQ6QaYozBQoDeN7UdTsUlOsw1" width="640" height="480"></p>
<p>What might be causing this behavior in a network’s implementation?</p>
<p>My code is written in Pytorch 0.4.0</p>

al3x: <p>I think <code>torch.no_grad()</code> is the recommended way of dealing with validation. <code>torch.eval()</code> is usually reserved for testing time. What happens when you use this instead?</p>
Sub-hessian vector product (how to stop gradient for only some entries of a tensor)

Bruno1: <p>Hi,</p>
<p>I’m trying to compute the sub hessian vector product of a function efficiently. By sub hessian, I mean that I’d like to exctract a sub matrix from the regular hessian and do a vector product with it.</p>
<p>I am aware of the standard trick for computing h.v.p. efficiently and I’d like to do something similar here.</p>
<p>I think one solution could be to stop the gradient for some entries of the tensor X.  Hence I could get the “sub” hessian vector product by double differentiation.</p>
<p>Is there a way in pytorch to stop gradient for some entries of a tensor ?</p>
<p>Thanks for your help</p>