akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Decentralized optimization and multiple dimension optimization

JACK_Q: <p>I am doing an optimization problem.</p>
<p><strong>Background</strong>:</p>
<p>Agents talk with each other in a graph. The initial states of each agent can be described as distributions x. Then they will maximize their (intimacy - privacy loss) with adding noise u.</p>
<p>Let’s say we have 3 nodes or 3 agents. Each of them is trying to maximize their objective funtion obj_1, obj_2 and obj_3 respectively.</p>
<p>The way they do the optimization is to find a u (adding noise) for themselves.<br>
Previously, what I did is as follows. That means I am trying to find a u to max the mean of (obj_1 + obj_2 + obj_3). That works.<br>
(the shape of u: (80, 3, 50). 80 means they talk with each other 80 times, 3 means 3 nodes, 50 means how many bins in the initial information distribution)</p>
<blockquote>
<p>net.u.requires_grad = True</p>
<p>optimizer = torch.optim.Adam([net.u], lr=1e-3)</p>
<p>optimizer.zero_grad()</p>
<p>for i in range(0, iteration):</p>
<pre><code>obj = - net.obj(x, T=T).mean()

obj.backward()

optimizer.step()

optimizer.zero_grad()
</code></pre>
</blockquote>
<p>However, currently, the target changes. Each of the agent should play their own strategy to optimize their objective function separately.  But as before, they will still exchange information. So I have to find u for each of them one by one not together. So I wrote the code as follows to try to find an optimized u for agent 1.</p>
<h1><a name="ushape-t-node-nbins-80-3-50-1" class="anchor" href="#ushape-t-node-nbins-80-3-50-1"></a>u,shape = (T, node, nbins) = (80, 3, 50)</h1>
<blockquote>
<p>net.u.requires_grad = True</p>
<p>obj_0 = -net.obj(x, T = T)[0]</p>
<p>grad = torch.autograd.grad(obj_0, net.u)[0]</p>
<p>net.u[:,0,:].grad = grad[:,0,:]</p>
<p>optimizer = torch.optim.Adam</p>
<p>optimizer = torch.optim.Adam([net.u[:,0,:]], lr=1e-3)</p>
<p>optimizer.step()</p>
<p>optimizer.zero_grad()</p>
</blockquote>
<p>However, there some errors as follows.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5bf1edfcc360b3d5adca9f60883e374fa0a6f35.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5bf1edfcc360b3d5adca9f60883e374fa0a6f35" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5bf1edfcc360b3d5adca9f60883e374fa0a6f35.png" alt="image" data-base62-sha1="nEgfUtdiwJKihbsKbXhEIAJYo1D" width="556" height="500" data-dominant-color="252928"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">613×551 20.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can I ask how solve this? Or any suggestions to do such optimization with Pytorch?</p>
<p>Thank you so much!</p>
GPU-efficient pytorch code

gabinkbl: <p>I am using pytorch to implement an algorithm which is not related to neural networks or ML. Speed is paramount, so I intend to run it on a GPU.<br>
Are there are any general guidelines for writing time-efficient code with pytroch that will best utilize the GPU’s processing power and parallelism?</p>

albanD: <p>Hi,</p>
<p>If you’re working only in python. The main idea is to perform as few ops as possible (making them as big as possible). The GPU is amazing to speeding up a given op but is very bad at switching between op and very very bad to execute small ops.<br>
So make sure everything is stored in Tensors as large as possible and always work with the full Tensor.</p>
<p>Do you have particular code in mind?</p>

Atul_Kumar: <p>What I learned with some experience of mine are (including above advice)</p>
<ol>
<li>Try to map each operation to matrix multiplication as much as possible</li>
<li>avoid gather and select</li>
<li>don’t let GPU wait for CPU batch preparation, make sure that batch data is processed beforehand or done in background process. Keep GPU busy all the time. “nvidia-smi” is your friend to check GPU usage.</li>
</ol>
When is retain_graph=False and create_graph=True useful?

pinocchio: <p>I thought such a combination should throw an error. I can’t imagine why anyone would want to compute higher derivates but then forget the graph. Wouldn’t the screw up the computation of higher derivatives by freeing the graph to early?</p>
<p><strong>Why doesn’t that throw an error?</strong></p>
<p><a href="https://pytorch.org/docs/stable/autograd.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html</a></p>
<p>Docs:</p>
<blockquote>
<ul>
<li><strong>retain_graph</strong>  (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"> <em>bool</em> </a> <em>,</em>  <em>optional</em> ) – If  <code>False</code> , the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to  <code>True</code>  is not needed and often can be worked around in a much more efficient way. Defaults to the value of  <code>create_graph</code> .</li>
<li><strong>create_graph</strong>  (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"> <em>bool</em> </a> <em>,</em>  <em>optional</em> ) – If  <code>True</code> , graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults to  <code>False</code> .</li>
</ul>
</blockquote>

albanD: <p>Hi,</p>
<p>I don’t think there is any practical reasons except debugging where you want to inspect the graph but don’t need to actually run backward on it.</p>

pinocchio: <p>yea that combination disallows to compute even higher order info since the graph is freed up…right? If we kept calling <code>torch.autograd.grad</code> for some reason</p>

pinocchio: <p>what is a debugging example?</p>

albanD: <p>The one I have in mind is if you use things like <code>torchviz</code> that need the graph to exist but don’t need to backward through it.<br>
But yes it is a weird use case. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Append Tensor to Libtorch TensorList

Hiperdyne19012: <p>How to append Torch::Tensor to Torch::TensorList?</p>

ups: <p>A TensorList is just an ArrayRef. As ArrayRef does not own the storage  it is impossible to add a Tensor to the TensorList. Maybe create a vector of Tensors (owns the storage) and then create a new TensorList based on the vector?</p>
Models.video input size

Federico_Vasile: <p>Hi everyone,<br>
the documentation about torchvision.models in the video classification paragraph says that the expected input size of the frames should be 112x112. Why is there this constraint?<br>
I used a Resnet(2+1)D model as a fixed feature extractor with 180x320 input frames, and it seems to work without any problems, should I resize frames to 112x112 or not? Why?</p>
<p>Thanks.<br>
PS: I think it works anyway thanks to the AdaptivePool at the end of the network, is it right?</p>

Nikronic: <p>Hi,</p>
<p>The reason that documentation says it <em>expects</em> 112x112 is that the official paper of Kinetic400 which demonstrated benchmarks for 3DConvNets, have used random crop to 112x112 as a data augmentation process. <a href="https://arxiv.org/pdf/1705.06950.pdf" rel="nofollow noopener">Section 4.3 and 4.4</a> of have explained the procedure.</p>
<p>And as you have mentioned, in models with adaptive pooling in the last layer, you can pass inputs with arbitrary size but a minimum as the kernel sizes and strides will make input size zero.</p>
<p>Another instance is all image classification pretrained models that use adaptive pool and expect at least 224x224 input size.</p>
<p>Bests</p>
Backproping alternatives for long tensors?

kirk86: <p>I’m trying to backprop through a long tensor but autograd complains. Changing it to float still throws an error.</p>
<p>Here’s a MWE, could some point out any alternative solutions?</p>
<pre><code class="lang-auto">x = torch.randn(100, 3)
y = torch.randint(3, (x.size(0),))
w = torch.nn.Parameter(torch.randn(3, 3))
optimizer = torch.optim.Adam([w])
for i in range(300):
  pred = x @ w
  y_hat = torch.softmax(pred, dim=1).argmax(dim=1)
  mask = y_hat.eq(y).bitwise_not()
  loss = F.cross_entropy(pred, y) + (y[mask].float() - y_hat[mask].float()).sum()
  print(loss.item())
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()
</code></pre>
<p>Error Message:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fccfa7d66cea4d89c4ce03d757e3bf826e69bca3.png" data-download-href="https://discuss.pytorch.org/uploads/default/fccfa7d66cea4d89c4ce03d757e3bf826e69bca3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fccfa7d66cea4d89c4ce03d757e3bf826e69bca3_2_690x261.png" alt="image" data-base62-sha1="A4tdNTA6DFpBrg05NRTQBV49hSP" width="690" height="261" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fccfa7d66cea4d89c4ce03d757e3bf826e69bca3_2_690x261.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fccfa7d66cea4d89c4ce03d757e3bf826e69bca3_2_1035x391.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fccfa7d66cea4d89c4ce03d757e3bf826e69bca3_2_1380x522.png 2x" data-dominant-color="EFEFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2342×886 301 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>can you update to the latest 1.5.1 release? This should be fixed <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to pass python object to custom c++ operators

ayl: <p>I follow the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">tutorial</a> to add a custom c++ operator. In the middle of the operator I want to allreduce some tensor in all process. At beginning of the python code, I use the <code>dist.init_process_group</code> before doing <code>torch.nn.parallel.DistributedDataParallel</code> training and other communication. So how can I pass the process group related information to my custom c++ operator?  Only After getting this information, I can call the nccl function with the right commnicatior and its corresponding stream.</p>
Libtorch Tensor usage

Hiperdyne19012: <p>What is the C/C++ version of this python code?</p>
<p>myTensor[:, :, :, i]</p>
<p>Is is this? 	<br>
myTensor.index({ “…”,"…","…",i });</p>
<p>or this?</p>
<p>myTensor.index({ Slice(),Slice(),Slice(),i });</p>

ptrblck: <p><a class="mention" href="/u/yf225">@yf225</a> posted the translation between the Python and C++ API in <a href="https://discuss.pytorch.org/t/indexing-in-c/66929/5">this post</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hiperdyne19012: <p>Thanks, just what I was looking for.</p>

yf225: <p>We have this <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="nofollow noopener">https://pytorch.org/cppdocs/notes/tensor_indexing.html</a> in the docs as well <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
AdaptiveMaxPool2d and MaxUnpool2d

RonMcKay: <p>Hi everyone,</p>
<p>recently I was wondering how AdaptiveMaxPool2d and MaxUnpooling2d could be working together if the inputs are of variyng sizes. In the documentation it says that AdaptiveMaxPool2d can also return indices for later unpooling but how do one determines the kernel sizes and strides for the MaxUnpool2d to work? Does one has to calculate the values from the input dynamically and if yes how would you do it?</p>
<p>I would love to hear some thoughts on that.<br>
Thank you very much.</p>
Bag of Words embeddings problem

serdarrader: <p>I am trying to implement neural network using term-document matrix (and using sign() so it is like one hot encoded matrix). My term document matrix has 25k row and 800000 column. It is in the scipy csr matrix format. I created dictionary called res_dict that holds non-zero values’ indexes in each document like:</p>
<pre><code class="lang-auto">res_dict = {key: [] for key in x.nonzero()[0]}

for row, col in zip(*x.nonzero()):
    res_dict[row].append(col)
</code></pre>
<p>My network will be like this:</p>
<p>(w + w_adjusted) * r</p>
<p>where w is the learnable parameter, w_adjusted is constant number, and r is the constant vector that has the form (1,800000). r is coming from Naive Bayes(). I will give w_adjusted 0.4. I am trying to give credit to naive bayes’s result so I am adding w_adjusted to ensure regularization will not make weights 0 but w_adjusted.</p>
<p>w has the form (1,800000), weights for every words in dictionary, r (1, 800000). I will give res_dict[idx] to the network and do lookup and calculate the ouput. But  the problem is that I don’t know how to do effective lookup for batch, I can do it for single entry but writng for loop for creating corresponding w etc does not seem effective to me.</p>
<p>Here is my network so far:</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):
    def __init__(self, nf, r, b, w_adj = 0.4):
        super().__init__()
        self.w_adj = w_adj
        self.w = nn.Embedding(nf, 1) # (1, 800000)
        self.w.weight.data.uniform_(-0.1,0.1)
        self.b = nn.Parameter(torch.tensor(b)) # (1)
        self.r = r
        
    def forward(self, x):
        
        x =  x * (self.w + self.w_adj) # It will not be like this we should do lookup here
        x = x @ self.r.T + b
        return x
</code></pre>
<p>r is:</p>
<pre><code class="lang-auto">x = trn_term_doc
y = trn_y

p = x[y==1].sum(0) + 1
q = x[y==0].sum(0) + 1

r = np.log((p/p.sum())/(q/q.sum()))

b = np.log(len(p)/len(q))
</code></pre>
Model evaluation for multi class image classification

rizal_alfarizi: <p>Hi, i want to ask what metric can be used to evalutate my CNN model for multi class, i have 3 classes for now and i’m just using accuracy and confussion matrix also plot the loss of model, is there any metric can be used to evaluate my model performance?</p>

SrJ: <p>You can use ROC-AUC curve to test your model performance.It is a really good way to test your model’s performance.</p>

rizal_alfarizi: <p>is ROC-AUC can be implemented for multiclass? as far i know it’s just for binary classification</p>
Torchvision v0.6.1 is out on pypi but no github tag exist?

tastyminerals: <p>I wonder why <strong>torchvision v0.6.1</strong> is out on pypi when on github I cannot find even a v0.6.1 tag or anything related to v0.6.1 release notes? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
How to compute Pytorch C++ Tensor mean?

cruzas: <p>Hi everyone,</p>
<p>I’ve taken a look around but haven’t been able to find an answer.<br>
Is there an equivalent to Python’s <code>torch.mean()</code> in C++?</p>
<p>Thank you for your help.</p>

albanD: <p>Note that if you have a pytorch Tensor on your cpp side, <code>at::mean(t, dim)</code> will work .<br>
You can find these in the cpp API doc: <a href="https://pytorch.org/cppdocs/search.html?q=mean&amp;check_keywords=yes&amp;area=default">https://pytorch.org/cppdocs/search.html?q=mean&amp;check_keywords=yes&amp;area=default</a></p>

cruzas: <p>Hey <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Thank you very much!<br>
It works just as you stated <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Creating a TorchScript model with loss and optimizer included

Sujoy_Saraswati: <p>Hi,</p>
<p>Is there a recommended way to create a TorchScript model with torch.jit.trace/script that can include the loss and optimizer methods? I see the usual way is to perform a torch.jit.trace/script over the model, and the loss and optimizer methods are kept outside. My intention is to create a graph that can encapsulate as much as possible, so that each op in the graph can go via the graph optimizations and can be executed by the same runtime executor.</p>
<p>If there isn’t any possible way to do this, then the follow-up question is - what is the impact on the training time due to the fact that these methods become separate ops outside the purview of the TorchScript model.</p>
<p>Regards,<br>
Sujoy</p>
Synchronize a specific cuda stream in python

ayl: <p>In PyTorch, we can use <code>torch.cuda.synchronize</code> to synchronize all streams on a CUDA device. Is there any way to synchronize a specific stream in python like <code> cudaStreamSynchronize</code> from C++ runtime API?</p>

ptrblck: <p>You could use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Stream.synchronize"><code>torch.cuda.Stream.synchronize()</code></a> or <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Stream.wait_stream"><code>torch.cuda.Stream.wait_stream(stream)</code></a> to synchronize with the passed CUDA stream.</p>
How can I predict my own Handwritten digit?

Mohammad_Islam: <p>when I trained and test my ANN with MNIST it worked but when I am trying to predict my own handwritten digit(captured by cellphone) it’s predicting always 3 where I’m giving image of 1,0,8.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6fb656aeb9c2057135ddb70d9281fa961fc853c4.png" alt="ss" data-base62-sha1="fWfEMSmaVr8v7m30TWUvie6eqxK" width="556" height="420"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe8bd2b3da5ba4deeb4d7dd4d9c1f88f3c1dd2b9.png" alt="ss" data-base62-sha1="AjOQvUC0l9Sv2fMWi0dC2r4HJgt" width="525" height="378"></p>

ptrblck: <p>The posted image might not be recognized properly, as it seems to have some background noise.<br>
You could try to binarize your input image (or at least try to remove the background noise using a threshold).<br>
Also, you could first try to classify “fake” written images created in paint or any other editor.<br>
This would make sure to have a perfect image without any artifacts. If that’s working, you might need to add some preprocessing to your cellphone pictures.</p>
Impede a layer which requires grad to be impacted by a given loss

NHH: <p>Consider the following sequential model:<br>
Net A -&gt; Net B --&gt; Net C</p>
<p>Also consider Loss B, computed with the output of Net B, and Loss C, with the output of Net C.</p>
<p>I would like Loss C to impact the weights of Net A, B and C, and Loss B to only impact Net B.</p>
<p>If I add the two losses and run loss.backward(), Loss B will impact Net A. If I set .requires_grad= False for Net A, then it won’t be impacted by Loss C.</p>
<p>Could anyone help me achieve the behavior I want?</p>

ptrblck: <p>You could create two different forward passes through <code>NetB</code> with the output of <code>NetA</code> and with the detached output of <code>NetA</code>:</p>
<pre><code class="lang-python">outA = NetA(input)
outB = NetB(outA)
outB_det = NetB(outA.detach())
outC = NetC(outB)

loss = criterion(outC, target)
loss.backward() # will calculate the grads for A, B, C

lossB = criterion(outB_det, taret)
lossB.backward() # will add grads to B
</code></pre>
<p>Let me know, if this would work for you.</p>
Predict on central slice, with adjacent slices provided as additional channels in input tensor?

darteaga: <p>I am looking to implement a model using a ‘pseudo-3D approach’, similar to this:<br>
<a href="https://www.groundai.com/project/evaluation-of-multi-slice-inputs-to-convolutional-neural-networks-for-medical-image-segmentation/1" rel="nofollow noopener">https://www.groundai.com/project/evaluation-of-multi-slice-inputs-to-convolutional-neural-networks-for-medical-image-segmentation/1</a></p>
<p>Specifically, I am working with medical data and want to add adjacent imaging slices (above and below the center slice) as contextual information in order to improve 2D segmentations on the center slice, i.e. a pseudo-3D approach (as the paper calls it). The paper in question adds these neighboring slices as additional channels in the input tensor. My question is - how do I organize this data in the input tensor such that actual segmentations are only being performed on the center slice? This approach appears to be quite different than a normal multi-channel input image, which may just have different color channels.</p>
<p>My thought was that if my input is CxDxHxW, that the first channel would be my center slice (image of interest that I want a predicted segmentation for), and that any additional channels would be the adjacent slices to provide contextual data. Does this sound correct?</p>

ptrblck: <aside class="quote no-group" data-username="darteaga" data-post="1" data-topic="85957">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/darteaga/48/25746_2.png" class="avatar"> darteaga:</div>
<blockquote>
<p>My thought was that if my input is CxDxHxW, that the first channel would be my center slice (image of interest that I want a predicted segmentation for), and that any additional channels would be the adjacent slices to provide contextual data. Does this sound correct?</p>
</blockquote>
</aside>
<p>That sounds correct and a reordering of the channels should not make a difference, as the conv kernels will use all input channels anyway (they will not slide in this dimension).<br>
You should just make sure to keep the same order of slices.</p>
<p>The target would of course only be the segmentation map of the “center slice”.</p>
<p>Let me know, if you have more questions.</p>
Export onnx on matlab from pytorch model

Peter1: <p>Hello,</p>
<p>I am trying to export a torchvision.model.</p>
<p>the model is r3d-18 (resnet3D)</p>
<pre><code class="lang-auto">import torch
import torch.onnx
import torchvision

dummy_input = torch.randn(1, 3, 4, 112, 112)
model = torchvision.models.video.r3d_18(pretrained=True)
torch.onnx.export(model, x, "ResNet3D-18.onnx")
</code></pre>
<p>I tried to load the onnx file on matlab.</p>
<p>but matlab say ‘mismatch size error, It is different with expected size’.</p>
<p>I found the input shape at <a href="https://github.com/pytorch/pytorch/pull/30121/files" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/30121/files</a></p>

ptrblck: <p>You are passing <code>x</code> to <code>torch.onnx.export</code>, while it seems <code>dummy_input</code> should have been passed.<br>
Could you check it or is it a copy-paste error?</p>
Propagate manually computed gradients

soshishimada: <p>I want to backpropagate manually computed Jacobian J which consists of gradients.<br>
Let’s say, my pipeline is composed of neural networks A, and its output is sent to B which is the differentiable model but does not have any learning parameters. Finally, compute the loss using the output from B:</p>
<p><strong>input -&gt; A -&gt; B -&gt; get output and compute loss L</strong></p>
<p><strong>A: composed of  neural networks</strong><br>
<strong>B: a differentiable model with no learning parameters</strong><br>
<strong>J: Jacobian consists of gradients of the loss L w.r.t. the input to B (which is equivalent to the output from A)</strong></p>
<p>Here, I compute J manually. Then I want to propagate the gradients from the very last layer of A to the input layer of A.<br>
In this case, is “<strong>output_from_A.backward(J)</strong>” what I have to do?</p>
<p>Thanks in advance!</p>

ptrblck: <p>I’m not familiar with your use case, but given your description I think your suggestion sounds right.<br>
<code>output_from_A.backward()</code> can take any gradients in the shape of <code>output_from_A</code>, so as long as this condition is met you could pass the manually gradients to this method.</p>
Equi probable selection from Tensor List

Jaideep_Valani: <p>Is there any pytorch function that can help select every time equal no of times from Tensor list<br>
say tensor([0, 1])<br>
If i make sample(tensor[0,1],1)   20 times i should nearly 50 pct of times 0 ,50 pct of time 1</p>

ptrblck: <p>You could use <code>torch.multinomial</code> to sample using weights:</p>
<pre><code class="lang-python">x = torch.tensor([0, 1])
weights = torch.tensor([0.5, 0.5])
idx = torch.multinomial(weights, num_samples=100, replacement=True)
out = x[idx]
</code></pre>
<p>Note that the indexing is not really necessary here, since the indices are already the target output.</p>
FasterRCNN transfer learning strategy

loicdtx: <p>Hi everyone,</p>
<p>I’m trying to train a torchvision faster RCNN object detection model using transfer learning.<br>
My dataset consists of tree species from overhead imagery; therefore it is widely different from the coco dataset on which the pretrained models are trained.</p>
<p>I am looking here for recommendations/advices about the transfer learning strategy to adopt.<br>
For my first attempts (on a relatively small dataset) I tried freezing the backbone and training only head + rpn, but the results are not very good.</p>
<p>Should I instead make all layers trainable from start, or train different components separately (e.g.: first only the head, then only the rpn and finally head + rpn + backbone with fpn).</p>
<p>Thanks for any advice,<br>
Loïc</p>

ptrblck: <p>Generally you could try to retrain more parameters the more data you have and especially if it comes from another domain.<br>
If you are dealing with a small dataset, training too many parameters might yield to overfitting.<br>
Unfortunately, my best advice is that you would have to try it out and see what is working best for your use case.</p>

maldeer: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
How to retrain more parameters? this might be a simple thing to do, but please forgive me as I am new in this field.<br>
In fact I have the same problem. I have a very small data set (100 recurrence plots) and I used this nice tutorial (<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a>) to do transfer learning. Using mobilenet (for example) and with 100 epochs and dropout of 0.t I get 100 as training Acc. and 70% Val. Acc.</p>
<p>Any other suggestions to avoid overfitting?<br>
Thank you</p>

ptrblck: <p>If you want to train more parameters, you should freeze less parameters from the complete model.<br>
I assume you might be currently freezing all parameters besides the last classifier?</p>
<p>However, since you are dealing with a small dataset, training more parameters might even result in more overfitting.<br>
You could try to apply more aggressive data augmentation or in the best case try to collect more samples.</p>
Question about torch.nn.DataParallel module in backward pass

zarzen: <p>Hi there,<br>
I would like to use <code>torch.nn.DataParallel</code> for intermediate layers in my model. the structure looks like following:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        ... &lt;some other layers&gt;
        self.linear = nn.DataParallel(nn.Linear(100,100))
        self.linear2 = nn.DataParallel(nn.Linear(50,500))

    def forward(self, x):
        x.to("cuda:0") # assume moved to gpu-0
        ... &lt;some other operation on inputs&gt;
        x = self.linear(x)
        x = self.linear2(x)
        ...
        return x
</code></pre>
<p>While I have several concerns:</p>
<ol>
<li>the output of the <code>nn.DataParallel</code> by default is the gpu-0, will it cause extract overhead? because of the memory copy among different GPUs, assume we have 8 GPUs for Data parallel.</li>
<li>When we calculate the loss, if the loss value is calculated on GPU-0, how does intermediate <code>nn.DataParallel</code> layer do the right backward pass?</li>
<li>does pytorch allow cross device backward pass? (e.g. I have a tensor <code>A</code> at <code>cuda:0</code>, later I copied <code>A</code> and moved it to <code>cuda:1</code>, <code>B = A; A = A.to('cuda:1')</code>, later when I do backward on variable <code>A</code>, will the gradient computation propagate to <code>B</code> at <code>cuda:0</code>?</li>
</ol>
ValueError: Expected input batch_size (100) to match target batch_size (64)

KIJ: <p>Hello,<br>
I get this error when i run the model below on MNIST dataset. I use <a href="https://github.com/meryemJanatiIdrissi/federated-learning-1/blob/master/data_utils.py" rel="nofollow noopener">this script</a> to get the data loaders:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4 * 4 * 50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = f.relu(self.conv1(x))
        x = f.max_pool2d(x, 2, 2)
        x = f.relu(self.conv2(x))
        x = f.max_pool2d(x, 2, 2)
        print("before view",x.shape)
        x = x.view(-1, 4 * 4 * 50)
        print("after view", x.shape)
        x = f.relu(self.fc1(x))
        x = self.fc2(x)
        return f.log_softmax(x, dim=1)
</code></pre>
<p>I suspect that this the reshape in this function might be the cause. Am i right? any idea please?</p>
<pre><code class="lang-auto">def get_mnist():
  '''Return MNIST train/test data and labels as numpy arrays'''
  data_train = torchvision.datasets.MNIST(root=os.path.join(DATA_PATH, "MNIST"), train=True, download=True) 
  data_test = torchvision.datasets.MNIST(root=os.path.join(DATA_PATH, "MNIST"), train=False, download=True) 
  
  x_train, y_train = data_train.train_data.numpy().reshape(-1,1,28,28)/255, np.array(data_train.train_labels)
  x_test, y_test = data_test.test_data.numpy().reshape(-1,1,28,28)/255, np.array(data_test.test_labels)
  
  return x_train, y_train, x_test, y_test

</code></pre>
<p>I checked other conversations about the same issue and i tried the proposed solutions yet the same error.</p>

KIJ: <p>Found the problem. I was resizing the images to 32 while I should keep them 28. thank you</p>
How to get the feature vectors of the region of interest？

JasonChenhx: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/3987507ff3db7c2b92aa7aa7a5d1d8eb39f0d97e.png" alt="图片" data-base62-sha1="8cV7Na7hbhvOA31dcZwQs8bY1no" width="531" height="410"><br>
This image is a segmentation result obtained by a segmentation algorithm. And the original image is put into a neural network to get the feature vectors.<br>
So my question is that I want to get the feature vectors of yellow and green regions.<br>
Thanks!</p>

Nikronic: <p>Hi,</p>
<p>Can you share a sample code?</p>
<p>But generally, in segmentation models, the outputs’ shape would be <code>[batch, classes, height, width]</code> which based on your image, it is <code>[batch, 3, h, w]</code>. So, you can get a feature map of for each class just by using an index for channel dimension such as <code>[batch, 0, h, w]</code> for possibly black background.</p>
<p>Bests</p>

JasonChenhx: <p>Firstly, thanks your reply.<br>
But the segmentation result is already owned. I just want to obtain the feature vectors of the interest of regions, which means that I want to obtain the feature vectors for the yellow wand green regions to do other tasks.<br>
And My current idea is that the original image can be operated through the convolutional neural network to obtain feature maps, and then the feature vectors can be obtained through the position one-to-one correspondence of the segmentation result map. What do you think of this idea?</p>
Parallel on multiple process but only do optimization step on process 0

stclaireva: <p>I am trying to implement parallelization as follows but not sure if it is possible.<br>
For example, train data with multiple processes (CPU cores). Have each process deal with independent batches. Instead of taking the optimization step independently for each batch, I want to gather loss and gradient from all processes and only take optimization step on Process 0.<br>
Is that possible to do that with torch.distributed package?</p>
<p>Currently, I followed instructions from <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/dist_tuto.html</a>. And have it work.</p>

mrshenli: <p>Hey <a class="mention" href="/u/stclaireva">@stclaireva</a></p>
<blockquote>
<p>Is that possible to do that with torch.distributed package?</p>
</blockquote>
<p>This is possible. You can</p>
<ol>
<li>Run forward-backward locally.</li>
<li>Use <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather" rel="noopener nofollow ugc"><code>all_gather</code></a> or <a href="https://github.com/pytorch/pytorch/blob/d14d47b9b5c0b860e0e285c91a1c7e74163a5bca/torch/distributed/distributed_c10d.py#L1194" rel="noopener nofollow ugc"><code>all_gather_coalesced</code></a> to collect all gradients into rank 0.</li>
<li>Manually add/average those gradients into <code>param.grad</code> on rank 0.</li>
<li>Run <code>optimizer.step()</code> on rank 0 to update parameters.</li>
<li>Let rank 0 <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast" rel="noopener nofollow ugc">broadcast</a> the updated parameters to other ranks.</li>
<li>go to step 1 to start a new iteration</li>
</ol>
<p>Curious, why do you need the above algorithm instead of letting DistributedDataParallel handle it for you?</p>
Is it possible to get pyobject* from a torch tensor?

rahul003: <p>I want to develop an asynchronous op which takes a tensor as argument. Right now I have to check if op is finished and clear this tensor reference. I would like for the CPP op implementation’s callback clear this memory bu using Py_INCREF and Py_DECREF methods. But these methods take a PyObject*. Is it possible to get the PyObject* from a torch::Tensor</p>

albanD: <p>The way we do it internally is with PyObject * THPVariable_Wrap(at::Tensor t)` from torch/autograd/python_variable.h<br>
Not sure if it is the cpp API way to do it though.</p>
Different input and output size in autoencoder

mhnazeri: <p>Hello everyone.<br>
I have an autoencoder where the input size is 15, and at the decoder, I only want the first 10 numbers. If I use slicing the autograd will capture that as a grad_fn, I wanted to know will this affect the network behavior? I made a simple example for this and the grads for the sliced out elements were zero.</p>
<pre><code class="lang-auto">a = torch.tensor([1,2,3], requires_grad=True, dtype=torch.float)
b = 2 * a
c = 2 * a + b
d = c[:1] # like the slicing in the last layer of decoder before calculating the loss
d.backward()
print(a.grad) # prints: tensor([4., 0., 0.])
# if I would do this for c, I would get this
# c.backward()
# print(a.grad) # prins: tensor([4., 4., 4.])
</code></pre>
<p>But I want the effect of these 5 elements in the latent space (does this capture that?). Making their gradient zero means that their weights won’t change. Is this correct? What is the best way to do this?</p>

albanD: <p>Hi,</p>
<p>Since these elements don’t contribute to the final output, their gradient is 0.<br>
Note that in a neural net, usually the weights are used to get values for all the outputs or you use a softmax layer that couples all the outputs together. And that will make all the weights of your net to be updated.</p>

mhnazeri: <p>Thanks for the reply. It is correct that they don’t contribute to the final output which in this case is the output of the decoder. But they contribute to the output of encoder i.e latent space.</p>

albanD: <p>If they contribute to this then they will get gradients yes.</p>
Docs for torch.utils.hooks missing in Pytorch

Sathvik_Joel: <p>I am new to PyTorch so apologies if my question is very basic<br>
I am trying to implement some visualization techniques from different papers and I observed that hooks really play a very important role in this.</p>
<p>But when I wanted to find the documentation for torch.utils.hooks, I could not find it anywhere.<br>
Is it because it’s not a completely developed feature of PyTorch or they ignored it since it is not a very important topic to be in the docs?</p>
<p>Thanks in advance…</p>

albanD: <p>Hi,</p>
<p><code>torch.utils.hooks</code> are just internal utility functions that we use to implement hooks but that’s it.<br>
You can find Tensor hooks by looking for <code>register_hook()</code> on Tensor.<br>
You can find nn.Module hook by looking for <code>register_forward_hook()</code>, <code>register_pre_forward_hook()</code> on nn.Module.</p>
Bug in Quantization example?

xieydd: <p>When i follow tutorial from <a href="https://github.com/pytorch/vision/blob/39021408587eb252ebb842a54195d840d6d76095/references/classification/train_quantization.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/39021408587eb252ebb842a54195d840d6d76095/references/classification/train_quantization.py</a>,  when i reload pretrained qat model, it will cause an error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_quantization.py", line 461, in &lt;module&gt;
    main(args)
  File "train_quantization.py", line 311, in main
    args.print_freq, priors)
  File "train_quantization.py", line 67, in train_one_epoch
    optimizer.step()
  File "/opt/conda/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/optim/sgd.py", line 106, in step
    buf.mul_(momentum).add_(d_p, alpha=1 - dampening)
RuntimeError: expected device cpu but got device cuda:0
</code></pre>
<p>I notice the optimizer reload error, when i use a new optimizer it will be ok</p>
<pre><code class="lang-auto">if args.resume:
        checkpoint = torch.load(args.resume, map_location='cpu')
        model_without_ddp.load_state_dict(checkpoint['model'])
        #optimizer.load_state_dict(checkpoint['optimizer'])
        lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
        args.start_epoch = checkpoint['epoch'] + 1
    optimizer = torch.optim.SGD(
        model.parameters(), lr=args.lr, momentum=args.momentum,
        weight_decay=args.weight_decay)
model.to(device)
</code></pre>
<p>I use the newest pytorch docker image form dockerhub.</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/xieydd">@xieydd</a>, there have a been two bug fixes for these issues, both landed in master.  One of the fixes updated device affinity for enabling distributed mode, and another updated device affinity for converting a model to QAT.  Would you mind trying with the nightly build?  All of the fixes will be a part of the upcoming v1.6 release as well.</p>
<p>If it’s hard to get the nightly, for a quick fix, inserting <code>model.to(device)</code> after line 71 should work.</p>

xieydd: <p>The error is after when i set  <code>model.to(device)</code>, i will clear it.</p>

xieydd: <p>The description of mine is not clear?</p>

Vasiliy_Kuznetsov: <p>would you mind letting us know:</p>
<ul>
<li>the pytorch version you are using</li>
<li>the exact command you are using to call the tutorial script?</li>
</ul>
<p>that would help get a local repro</p>
Regarding optimizer.zero_grad

jayanth: <p>Hi everyone, I am new to PyTorch. I wanted to know where optimizer.zero_grad should be used. I am not sure whether to use them after every batch or I should use them after every epoch. Please let me know. Thank you</p>

albanD: <p>Hi,</p>
<p>there is not hard rule of when you use it. You should just make sure that the gradients accumulated when you call optimizer.step() is the one you want.<br>
In general, you want to zero_grad() just before the backward.<br>
For more general problems, you can check this thread that discuss this at length: <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903" class="inline-onebox">Why do we need to set the gradients manually to zero in pytorch?</a></p>
What does the * signify when creating Sequential?

arnabsinha: <pre><code class="lang-auto">self.fcVar = nn.Sequential(*[nn.Linear(output_ndf, output_nc)])
</code></pre>
<p>What does the * signify in this line of code when creating a network?<br>
Thanks for the help in advance.</p>

albanD: <p>Hi,</p>
<p>This is a python construct.<br>
It can be used to unpack the elements of a list or tuple into positional arguments for the function:</p>
<pre><code class="lang-auto">def foo(a, b):
    return a + b

data = [1, 2]
foo(data) # Error as a single argument is given
foo(*data) # Works</code></pre>

arnabsinha: <p>Thanks for the reply mate! Have you by any chance read the BicycleGAN paper? It would be great if you could clear some issues of mine.</p>

albanD: <p>I haven’t sorry.<br>
But you can try to open a new topic and ask your question. Maybe someone more knowledgeable will <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

arnabsinha: <p>Sure mate! Thanks a lot.</p>
Strange problem with torch reproducibility

Lintianqianjin: <p>I have already set random seed like beblow:</p>
<pre><code class="lang-python">import torch
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

import numpy as np
np.random.seed(0)

import random
random.seed(0)
</code></pre>
<p>And, in the first two epoch, it seems that the model can produce same result. However, then the result becomes different with the other. Does anyone know why?</p>
<pre><code class="lang-auto">Epoch 00000 | Loss 2.046489238739 | Accuracy 0.455997971088 
Epoch 00001 | Loss 4.619217872620 | Accuracy 0.129850367740 
Epoch 00002 | Loss 10.045362472534 | Accuracy 0.448135937104
Epoch 00003 | Loss 5.151265144348 | Accuracy 0.441795587116 
Epoch 00004 | Loss 3.293608665466 | Accuracy 0.432919097134 
Epoch 00005 | Loss 3.348810672760 | Accuracy 0.501648490997 
Epoch 00006 | Loss 2.631152153015 | Accuracy 0.417448643165 
Epoch 00007 | Loss 2.960071325302 | Accuracy 0.429368501141 
Epoch 00008 | Loss 2.069313049316 | Accuracy 0.421760081156 
Epoch 00009 | Loss 2.420869827271 | Accuracy 0.429114887142 
Epoch 00010 | Loss 2.132256269455 | Accuracy 0.406036013188 
Epoch 00011 | Loss 2.006080150604 | Accuracy 0.406036013188 
Epoch 00012 | Loss 2.006951093674 | Accuracy 0.441541973117 
Epoch 00013 | Loss 1.845007181168 | Accuracy 0.533603854933 
Epoch 00014 | Loss 1.875343561172 | Accuracy 0.518133400964 
</code></pre>
<pre><code class="lang-auto">Epoch 00000 | Loss 2.046489238739 | Accuracy 0.455997971088 
Epoch 00001 | Loss 4.619217872620 | Accuracy 0.129850367740 
Epoch 00002 | Loss 10.045364379883 | Accuracy 0.448135937104
Epoch 00003 | Loss 5.151239871979 | Accuracy 0.441795587116 
Epoch 00004 | Loss 3.293608665466 | Accuracy 0.432919097134 
Epoch 00005 | Loss 3.348819971085 | Accuracy 0.501648490997 
Epoch 00006 | Loss 2.631155490875 | Accuracy 0.417448643165 
Epoch 00007 | Loss 2.960067749023 | Accuracy 0.429368501141 
Epoch 00008 | Loss 2.069319725037 | Accuracy 0.421760081156 
Epoch 00009 | Loss 2.420872926712 | Accuracy 0.429114887142 
Epoch 00010 | Loss 2.132259845734 | Accuracy 0.406036013188 
Epoch 00011 | Loss 2.006079673767 | Accuracy 0.406036013188 
Epoch 00012 | Loss 2.006967306137 | Accuracy 0.441541973117 
Epoch 00013 | Loss 1.844992518425 | Accuracy 0.533857468932 
Epoch 00014 | Loss 1.875328063965 | Accuracy 0.518133400964 
</code></pre>

albanD: <p>Hi,</p>
<p>Have you read the note on reproducibility? <a href="https://pytorch.org/docs/stable/notes/randomness.html">https://pytorch.org/docs/stable/notes/randomness.html</a><br>
Do you use any function that is not deterministic?</p>
Autograd customized gradient requires_grad as True?

fsh: <p>Hi,</p>
<p>After defining a custom layer with customized gradient computation, do I need to have the gradients with requires_grad=True? Or is it enough just to have the requires_grad as true for the input and the output of that layer?</p>
<p>Does it make a problem if unnecessarily I put requires_grad as True for some variables?</p>
<p>Thanks in advance!</p>

albanD: <p>Hi,</p>
<p>It might create the the autograd state for Tensors that don’t need it. But it won’t cause any problem no.<br>
If you define a custom autograd.Function, you can use it as any other function: The inputs should have requires_grad=True during the forward if you want to be able to backprop. The grad_output given to your custom backward will have requires_grad=True if you asked for <code>create_graph=True</code> when calling backward and Tensors require gradients.</p>

fsh: <p>Thanks a lot! Where should I exactly use <code>create_graph=True</code>? Do I really need it while I am not using <code>autograd.grad</code>?</p>
<p>Here is my code, where it is defining some new custom loss class with all the computations inside <code>LossAndGradients</code>:</p>
<pre><code class="lang-auto">class CustomLoss(Function): # Inherit from Function
    
    @staticmethod
    def forward(ctx, X_input, imgD):        
        Loss, Gradients = LossAndGradients(X_input, imgD)
        ctx.save_for_backward(Gradients)
        return Loss
    
    @staticmethod
    def backward(ctx, grad_output):
        Gradients, = ctx.saved_tensors
        grad_imgD = None
        grad_input = Gradients
        return grad_input, grad_imgD
</code></pre>
<p>In the above function I have set  <code>requires_grad=True</code> for <em>Loss</em> and <em>X_input</em>, and <code>requires_grad=False</code> for Gradients. And I employ it as below in the main code:</p>
<pre><code class="lang-auto">criterion = CustomLoss.apply
</code></pre>
<p>I am not really sure if it is back-propagating the gradients in a correct way during training.</p>

albanD: <p>You should use <code>create_graph=True</code> when you want higher order derivatives (like hessian or gradient penalty) otherwise you don’t need it.</p>
<p>In the custom Function, the <code>forward</code> always run in a no_grad mode. So you don’t need to check the <code>requires_grad</code> field in there. The history will properly be attached to the Tensor you return on exit.<br>
For the <code>backward</code>, it run with grad mode enabled only if you set <code>create_graph=True</code>. If you don’t, then it will also not run in grad mode and you can ignore the requires_grad fields in there.</p>