akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Understand PyTorch Install Command

xinqi: <p>When I select:<br>
PyTorch Build: Stable (1.3)<br>
Your OS: Linux<br>
Package: Conda<br>
Language: Python 3.7<br>
CUDA: 9.2</p>
<p>It gives me this command:<br>
conda install pytorch torchvision cudatoolkit=9.2 -c pytorch</p>
<p>My Question is, what does this mean?<br>
(Let me give a starting but not complete answer to my question so you may understand me better. )<br>
Answer (initial): The command means, you will use ‘conda’ package management software to install pytorch (1.3?) and torchvision (0.4?) which is pre-compiled with cuda library version 9.2. (Please correct the first part if it is incorrect or not complete) In addition, what is the ‘-c pytorch’ in the end?</p>

tom: <p><code>-c pytorch</code> means from the Pytorch anaconda channel, i.e. packages prepared by the PyTorch team (sometimes I think Soumith himself does that).</p>
<p>The <code>pytorch</code> and <code>torchvision</code> selects the latest packages available on that channel (where the stable releases are put, so currently 1.3 / 0.4, as you say).</p>
<p>The cuda library is provided in a separate package cudatoolkit of which you explicitly select version 9.2 on the command line.)</p>
<p>The end result is as you describe it</p>
<p>Best regards</p>
<p>Thomas</p>

xinqi: <p>Thanks for your reply.</p>
<p>According to your answer, there is a misunderstanding of cuda installation in my prior description. I thought <code>cudatoolkit=9.2</code> means the installed pytorch is pre-compiled with cudatoolkit=9.2. However, it should be that it will install <code>cudatoolkit 9.2</code>. Base on this <a href="https://discuss.pytorch.org/t/will-conda-install-pytorch-torchvision-c-pytorch-also-install-cuda-and-cudnn/19717">PyTorch Forum Discussion</a>, it also means it will not only install a cudatoolkit, but also install a corresponding version of CuDNN.</p>
<p>So, my question is that,<br>
Is this <code>cudatoolkit</code> and <code>cuDNN</code> installation in our <code>conda environment solely</code> or it is <code>system-wide</code>?</p>
PackedTensorAccessor is slow?

flyerjia: <p>The PackedTensorAccessor32 is faster than PackedTensorAccessor64, and is it slower than the original pointer?</p>

tom: <p>While 64 vs 32 matters on the GPU (because 64 bit arithmetic for indexing is much slower), I’ve not seen a visible impact of using accessors vs raw pointer arithmetic. (Let’s say at least for regular fp32 things, I don’t really know about fp16, where you want to load end bloc even more.)<br>
On CPU, using accessors most likely means not using vectorization (AVX2 and similar), which makes it slow.</p>
<p>Best regards</p>
<p>Thomas</p>
How to smooth a curve plotted from TFBoard

OBouldjedri: <p>I am plotting some accuracy curves in python  that I got using tensorboard and exported as CSV but all of them look the same as if you set  the smooting parameter in TF board to 0,  is there any way to tune this parameter in matplotlib or plotly ?</p>

JuanFMontesinos: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/42281844/what-is-the-mathematics-behind-the-smoothing-parameter-in-tensorboards-scalar" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/49530/willian-mitsuda" target="_blank" rel="nofollow noopener">
    <img alt="Willian Mitsuda" src="https://www.gravatar.com/avatar/45584aced38bad6c1250569623ad1909?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/42281844/what-is-the-mathematics-behind-the-smoothing-parameter-in-tensorboards-scalar" target="_blank" rel="nofollow noopener">What is the mathematics behind the "smoothing" parameter in TensorBoard's scalar graphs?</a>
</h4>

<div class="tags">
  <strong>tensorflow, tensorboard</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/49530/willian-mitsuda" target="_blank" rel="nofollow noopener">
    Willian Mitsuda
  </a>
  on <a href="https://stackoverflow.com/questions/42281844/what-is-the-mathematics-behind-the-smoothing-parameter-in-tensorboards-scalar" target="_blank" rel="nofollow noopener">06:24PM - 16 Feb 17 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How could I compute the local mean and variance

coincheung: <p>I am trying to use sliding window method the compute the local mean and variance withing the sliding window. Does pytorch provide the associated function to do that please?</p>

JuanFMontesinos: <p>You can use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Fold" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Fold</a> to perform sliding window.</p>
Feed data which has multiple patterns into regression model?

pahome: <p>I try to feed data into one deep regression model, but always can’t catch patterns.</p>
<p>When I use random forest, it works find and be able to catch patterns<br>
But it failed when I want to use deep networks.</p>
<p>What I thought is using relu and linear to catch patterns, is there anything wrong?</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as FUNC
class Reg(torch.nn.Module):
	def __init__(self, n_feature, n_hidden, n_output):
		super(Reg, self).__init__()
		self.name = self.__class__.__name__
		self.hidden = torch.nn.Linear(n_feature, n_hidden) # hidden layer
		self.F = FUNC.relu
		self.out = torch.nn.Linear(n_hidden, n_output)

	def forward(self, features):
		x = self.hidden(x)
		x = self.F(x)
		x = self.out(x)
		return x
</code></pre>
<p>optimizer=ADAM, criterion=torch.nn.MSELoss()<br>
n_feature=2, n_hidden=256, n_output=1.</p>

ptrblck: <p>Did you normalize your data? Normalization helps especially in training neural networks. If I’m not mistaken, RandomForest classifiers are not really sensitive to different value ranges.</p>
<p>Also, try to overfit a small data sample, e.g. just 10 samples. Once your model predicts these samples (almost) perfectly, try to scale out the training.</p>

pahome: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Yes, I normalize all data to 0~1.<br>
Or maybe I need to adjust hidden layer? e.g. add more relu or dense layers?</p>
Conv2d & FullyConnect Output Size

ysii: <p>Hello,</p>
<p>I am wondering why the output of my fc2 would be 8 * 5.<br>
The input X is of size 128 * 3 * 32 * 32.</p>
<p>I have tried the following but it is not working…</p>
<pre><code class="lang-auto">x = x.view((128, 512))
        print("0: ", x.shape)
        x = x.view(-1, 512)
        print("1: ", x.shape)
</code></pre>
<p>Any suggestion would be appreciated. Thank you!</p>
<pre><code>def __init__(self):
    super().__init__()

    # TODO: define each layer
    self.conv1 = nn.Conv2d(3, 16, (5,5), stride = (2,2))

    self.conv2 = nn.Conv2d(16, 64, (5,5), stride = (2,2))

    self.conv3 = nn.Conv2d(64, 32, (5,5), stride = (2,2))

    self.fc1 = nn.Linear(512, 64)

    self.fc2 = nn.Linear(64, 32)

    self.fc3 = nn.Linear(32, 5)
    #

    self.init_weights()

def forward(self, x):
    N, C, H, W = x.shape

    # TODO: forward pass
    print("x.shape")
    print(x.shape)
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = x.view(-1, 512)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    z = self.fc3(x)
    print(z)
    #

    return z</code></pre>

ptrblck: <p>The output of your <code>fc2</code> layer is defined as 32, so I’m not sure why you assume it should be 40?</p>
<p>Anyway, your model architecture won’t currently work, since the output activation of <code>conv3</code> will have the shape <code>[batch_size, 32, 1, 1]</code>, so the number of input features to <code>fc1</code> should be <code>32</code> instead of <code>512</code>.</p>

ysii: <p>I figured out by changing the padding to the right size. thanks.</p>
<p>Now the output of my conv3 is torch.Size([128, 32, 4, 4])</p>
<p>is view(-1,512) the right way of “reshaping” the tensor as the input of fc1?</p>

ptrblck: <p>The output of <code>conv3</code> is would be <code>[batch_size, 32, 1, 1]</code>, and I would recommend to use <code>x = x.view(x.size(0), -1)</code> after this layer so keep the batch size equal.</p>
How can i save immediate tensors to a file which I wish to use out of the code?

Nilesh_Pandey: <p>I have been getting device asserted error, and I wish to inspect the variables before it throws error. So, is there any good way to store tensors and reuse with numpy.</p>
<p>Thanks</p>

richard: <p>Try <code>torch.save</code>: <a href="https://pytorch.org/docs/stable/torch.html?highlight=torch%20save#torch.save">https://pytorch.org/docs/stable/torch.html?highlight=torch%20save#torch.save</a></p>

Nilesh_Pandey: <p>isn’t that to save model and dictionary? I just wish to save 2 variable tensors, should be able to get by numpy as well.</p>

richard: <p>You can use <code>torch.save(tensor)</code> on a tensor as well.</p>

sbelharbi: <p>this may help.</p>
<pre><code class="lang-python">import datetime as dt
import sys

import torch
import pickle as pkl


print(10 * "=")
print("Python version: {}".format(sys.version))
print("Pytorch version: {}".format(torch.__version__))
print("Pickle version: {}".format(pkl.format_version))
print(10 * "=")

torch.manual_seed(0)

d = 100
tensor = torch.rand((d, d))
print("Tensor size: {}".format(tensor.shape))

# Store
t0 = dt.datetime.now()
torch.save(tensor, "tensor-pytorch.pt")
print("Torch save-time: {}".format(dt.datetime.now() - t0))

t0 = dt.datetime.now()
with open("tensor-pickle.pkl", "wb") as fout:
    pkl.dump(tensor, fout, protocol=pkl.HIGHEST_PROTOCOL)
print("Pickle save-time: {}".format(dt.datetime.now() - t0))

# Read
t0 = dt.datetime.now()
tensorx = torch.load("tensor-pytorch.pt")
print("Torch load-time: {}".format(dt.datetime.now() - t0))

t0 = dt.datetime.now()
with open("tensor-pickle.pkl", "rb") as fout:
    tensor_pkl = pkl.load(fout)
print("Pickle load-time: {}".format(dt.datetime.now() - t0))
</code></pre>
<h1>OUTPUT:</h1>
<pre><code class="lang-auto">==========
Python version: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 20:15:55) 
[GCC 7.3.0]
Pytorch version: 1.3.0+cu100
Pickle version: 4.0
==========
</code></pre>
<h1>small tensor: (100, 100)</h1>
<pre><code class="lang-auto">Tensor size: torch.Size([100, 100])
Torch save-time: 0:00:00.002519
Pickle save-time: 0:00:00.002136
Torch load-time: 0:00:00.000512
Pickle load-time: 0:00:00.000347
</code></pre>
<h1>larg tensor: (10^4, 10^4)</h1>
<pre><code class="lang-auto">Tensor size: torch.Size([10000, 10000])
Torch save-time: 0:00:04.070195
Pickle save-time: 0:00:04.018530
Torch load-time: 0:00:03.638956
Pickle load-time: 0:00:05.501440
</code></pre>
Multiplying tensor in place

Carsten_Ditzel: <p>With two tensors</p>
<pre><code class="lang-auto">a = torch.ones([256, 512, 32])
b = torch.ones([32, 2])
</code></pre>
<p>what is the most efficient way to broadcast b onto every associated entry in a, producing a result with shape</p>
<p>[256, 512, 32, 2] ?</p>
<p>Is there an inplace variant maybe? I want to avoid making copies as much as possible…</p>

dhpollack: <pre><code class="lang-auto">a.unsqueeze(-1).mul(b.unsqueeze(0).unsqueeze(0))
</code></pre>
<p>But it’s sort of hard to see in your example because you are multiplying everything by ones.  You can open a REPL.  I also think with pytorch 1.3 you could use named tensors and just make sure that <code>a</code>'s dimensions are named something like <code>N</code>, <code>C</code>, <code>H</code> and <code>b</code>'s dimensions are <code>H</code>, <code>W</code>… basically just ensure that the dimensions align correctly.</p>

Carsten_Ditzel: <p>thank you, but do I really need those last two unsqueezes?</p>
<pre><code class="lang-auto">
a = torch.ones([256, 512, 32]).unsqueeze(-1)
b = torch.ones([32, 2])
a * b
</code></pre>
<p>also, wouldnt be mul_ be more efficient here?</p>

albanD: <p>The output is not the same size as either of the inputs. So you can’t really do it inplace.</p>
Batch size suddenly changes during the iterations!

Sara_Wasl: <p>Hi all,</p>
<p>I’m working with GAN model on my dataset, everything going well but the problem is when I tried to print the input size before and after each layer; suddenly at some point (after many successful iterations) the batch size changes (while printing I notice that) and the running stops because it will a problem of dimensions.</p>
<p>So, originally I chose <code>batch_size = 36</code> and when it changes, it becomes <code>24</code>!</p>
<p>And as I said it works well after many iterations but then this happens!</p>
<p>This is an example of what I did:</p>
<pre><code class="lang-auto">#the x of size [36,1,32,32]
x = F.relu(self.conv1(x))
print(x.shape) #here x will be [36,128,16,16]
</code></pre>
<p>And after a number of iterations the <code>x</code> after <code>x = F.relu(self.conv1(x))</code> become <code>[24,128,16,16]</code></p>
<p>Is there an explanation for this strange thing, please?</p>

spanev: <p>Hi <a class="mention" href="/u/sara_wasl">@Sara_Wasl</a>,</p>
<p>Does it always happen at the same iteration? Maybe the last iteration of the first epoch?</p>
<p>If your <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">dataloader</a> has <code>drop_last </code> set to <code>False</code> (which is the default value), and if your dataset is not a multiple/aligned on your batch size, the last iteration an epoch will be equal to <code>epoch_size % batch_size</code>.</p>

Sara_Wasl: <p>Thanks for your reply <a class="mention" href="/u/spanev">@spanev</a>,</p>
<p>Yes that’s true, happened in the last iteration of the first epoch.</p>
<p>No, I didn’t add <code>drop_last</code></p>
<p>The last <code>iteration number =1667</code>, <code>number of epochs = 50</code> and the <code>batch_size = 36</code>, do you mean I should chose appropriate batch_size? if yes, could you please tell me how?</p>

spanev: <p>So <code>drop_last </code> was set to <code>False</code>, by default.<br>
I think that setting it to <code>True</code> is the best thing to do here.</p>

Sara_Wasl: <p><a class="mention" href="/u/spanev">@spanev</a> I tried it just now (set it to <code>True</code>), first epoch works well now but it stops in the first iteration of the second epoch and the <code>batch_size</code> changed to <code>25</code>!</p>
Assignment causes grad attribute to disappear?

Goldname: <pre><code class="lang-auto">    image2 = image + epsilon*sign_data_grad
</code></pre>
<p>In the above line of code, image.grad exists, but afterwards image2.grad no longer exists. How do I keep it there?</p>

tom: <p>Yes, the name <code>image2</code> gets assigned a new tensor (the result of the computation on the rhs).<br>
If you want to keep the gradient, you would need to use an inplace operation (with all that entails) à la</p>
<pre><code class="lang-python">with torch.no_grad():
  image2.copy_(image + epsilon*sign_data_grad)
</code></pre>
<p>(or <code>add_</code> or <code>+=</code> if you actually want to only add to <code>image2</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

Goldname: <p>Thanks for the response. I tried that, but I got: RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.</p>
<p>Do you know why this is a problem?</p>
Could someone please explain me. Why the shape of hidden in GRU or LSTM is not same output shape? When we set batch_first is True

Pisit_Bee: <p>I am so confusing. I try to put my data to GRU/LSTM model with batch_first is true.<br>
I reshape my data to be of shape (batch,seq, features). I expect the model will give the output have the same shape. Yes, the output of the model has the same shape (batch,seq, features), but I have no idea. Why last hidden output still to be of shape (batch,seq, features). Could you please someone explain to me?<br>
This is my code.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b8a38766261e5299f853c218625e120454335ff.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9b8a38766261e5299f853c218625e120454335ff" title="2019-11-03_16-19-37"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b8a38766261e5299f853c218625e120454335ff_2_509x500.jpeg" alt="2019-11-03_16-19-37" data-base62-sha1="mbY8VU5PWuQJfNonYYKh0vnh41F" width="509" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b8a38766261e5299f853c218625e120454335ff_2_509x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b8a38766261e5299f853c218625e120454335ff_2_763x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b8a38766261e5299f853c218625e120454335ff.jpeg 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-11-03_16-19-37</span><span class="informations">790×775 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <p>The first dimension isn’t sequence length but num layers * num directions (2 for bidirectional else 1). The <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.GRU" rel="nofollow noopener">documentation for <code>nn.GRU</code></a> is a bit long and the output and shape subsections which both describe the output shape are towards the end.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" rel="nofollow noopener">Don’t use Variable, sources that still use that have not been updated for a year</a>.<br>
P.P.S.: If you use triple backticks (```) before and after your code, it formats nicely as text, too. That makes it much easier to read than images are.</p>
Loss between tensors of different shape

sharjeel: <p>I made 2 models with only convolutional layer to extract the feature maps of the images for siamese networks. I wanted to ask if there is a loss function or a way to calculate the loss between two feature maps of two different shapes for eg: [25,32,67,67] and [25,32,30,30].</p>

spanev: <p>Hi <a class="mention" href="/u/sharjeel">@sharjeel</a></p>
<p>The loss is a metric showing the difference between two distributions.<br>
How would you compare a scalar to an array of size 10? You would have to compare the scalar against all the values of the array, or reduce the array to scalar.</p>
<p>Similarly, you will have to project the features into same shape tensors before being able to compare them with a loss function. In your use case, mean or average pooling should be the best.</p>
Error in input.contiguous()

PyExtreme: <p>I am getting this error<br>
<code>error: passing ‘const at::Tensor’ as ‘this’ argument discards qualifiers</code><br>
while running the following code</p>
<pre><code class="lang-auto">input = input.contiguous()
</code></pre>
<p>where input is a torch tensor defined as const Tensor&amp; input</p>
<p>Is it because of const?</p>

yf225: <p>Yes we can’t do <code>input = input.contiguous()</code> if <code>input</code> is <code>const Tensor&amp;</code>. Changing it to <code>Tensor</code> should work <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

tom: <p>One might add that if input is an input to your function, the correct way is to leave it <code>const Tensor&amp;</code> and declare a new local variable as<br>
<code>auto cont_input = input.contiguous();</code><br>
This is assuming that your function won’t actually try to change input (which it should not when you call <code>contiguous()</code>…</p>
<p>Best regards</p>
<p>Thomas</p>
Data augmentation

copyrightly: <p>I am using the following code to do data augmentation of MNIST:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
        datasets.MNIST('../data', train=True, download=True,
                       transform=transforms.Compose([
                           transforms.RandomHorizontalFlip(),
                           transforms.RandomResizedCrop(28),
                           transforms.ToTensor(),
                           transforms.Normalize((0.1307,), (0.3081,)),
                       ])),
        batch_size=args.batch_size, shuffle=True, **kwargs)
</code></pre>
<p>I have a question about the line <code>transforms.Normalize((0.1307,), (0.3081,)),</code> 0.1307 and 0.3801 are mean and standard deviation of the original MNIST dataset. They should have been changed after those augmentation. So should I use the new mean and deviation to do normalization? Another question: should I do the same augmentation on test set? If not, training with augmentation and test would be from different distribution, right?</p>

paganpasta: <p>The idea of augmenting data is generating ``similar" samples from the data generating distribution possibly to avoid overfitting. The values of mean and std_dev will change however, I believe the difference wouldn’t be significant. You can go ahead and test this hypothesis by computing mean and std_dev after transforms. Also, you apply the same transformation on the test set. Your reasoning is correct in this regard.</p>
Gradient becomes None while using Autograd

mukh94: <p>I am trying to implement Newton’s method using the following code:</p>
<pre><code class="lang-auto"># initial guess
guess = torch.tensor([1,1], dtype=torch.float64, requires_grad = True) 

# function to optimize
def my_func(x):
    alpha=Variable(x[0], requires_grad=True)
    beta=Variable(x[1], requires_grad=True)
    K=(torch.tensor([[0., beta, 0.], 
                        [beta, alpha,beta], 
                        [0., beta, 0]], requires_grad=True))
    c=torch.ones((3,1),dtype=float)
    return torch.mm(torch.mm(torch.t(c),K),c) # random function to optimize
def gradient_hessian(J, params):
        d = torch.autograd.grad(J, params, create_graph=True)
        d2 = [torch.autograd.grad(f, params, retain_graph=(i &lt; len(d)-1)) for i,f in enumerate(d)]
        return torch.tensor(d), torch.tensor(d2)

def newton(func, guess, runs=10): 
    for _ in range(runs): 
        gamma=1
        # evaluate our function with current value of `guess`
        value = Variable(my_func(guess),requires_grad=True)
        d,d2=gradient_hessian(value,guess)
        guess.data-=gamma*torch.mm(torch.inverse(d2),d)
        print(guess.data)
        
    return guess.data # return our final `guess` after 5 updates

# call starts
result = newton(my_func,guess)
print(result)
</code></pre>
<p>This produces the following error:<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
<p>When I try to make allow_unused = True, the gradient becomes none. Where am I losing the gradient? Thanks!</p>

ptrblck: <p>By wrapping tensors (which already require gradients) in a new tensor, you are detaching the new tensor from the computation graph.</p>
<p>E.g. <code>my_func</code> should return a valid tensor, which shouldn’t be wrapped again.</p>
<p>Also, <code>Variables</code> are deprecated, so you can just use tensors and set their <code>requires_grad</code> attribute to <code>True</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

mukh94: <p>Thanks much for the reply! I understand that forming the tensor K might have caused the issue. However my cost function involves matrix operations (such as the one I have mentioned, ie., (c_transpose * K * c), where K is composed of the 2 element parameter I want to optimize). Any suggestion how to circumvent that? TIA!</p>

ptrblck: <p>You could most likely create <code>K</code> by using <code>torch.cat</code> and <code>torch.stack</code> calls to avoid creating a completely new tensor.<br>
Let me know, if you get stuck. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Question about the "output_padding" in nn.COnvTrasnpose2d

huanhuanxuxu: <p>Hello everyone,</p>
<p>I’ve got one question about the “output_padding” parameter in nn.ConvTranspose2d,<br>
while what does this parameter means? or where could I find the source realization file,<br>
like c/c++, or anything else so I could figure out the operation.</p>
<p>Thank you very very much ~ ~</p>

royboy: <p>Have you looked at the documentation for this module? In particular, the second note under nn.ConvTranspose2d has a lot of information on output_padding.</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.ConvTranspose2d" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.ConvTranspose2d</a></p>

huanhuanxuxu: <p>There’s lots of information about everything. I do not  think it explained the parameter clearly. So what does it mean by “it controls the amount of implicit zero-paddings on both sides of the output for output_padding number of points for each dimension.” “Note that output_padding is only used to find output shape, but does not actually add zero-padding to output.”</p>

SimonW: <aside class="quote no-group" data-username="huanhuanxuxu" data-post="3" data-topic="19740">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/9e8a1a/48.png" class="avatar"> huanhuanxuxu:</div>
<blockquote>
<p>“it controls the amount of implicit zero-paddings on both sides of the output for output_padding number of points for each dimension.</p>
</blockquote>
</aside>
<p>Ignore this. It is inaccurate. See the note and the description below that.</p>

Mata_Fu: <p>I still can’t understand this “Note that output_padding is only used to find output shape, but does not actually add zero-padding to output.” even with the note. Could you explain it? Thx!</p>
Typing issue with LambdaLR scheduler

arubio: <p>I’m trying to create my own scheduler class that inherits from <code>torch.optim.lr_scheduler.LambdaLR</code>. My code looks like this:</p>
<pre><code class="lang-python">class DetectionWarmupScheduler(LambdaLR):
    r"""Warm-up learning rate scheduler for detection.

    For iteration x, the learning rate variation is given by:

      /
     |  =&gt; (warmup_factor * (1 - alpha) + alpha) if x &lt; warmup_iters
    &lt;
     |  =&gt; 1 otherwise
      \

        with alpha = x / warmup_iters.

        For this particular scheduler, warmup_factor = 1/warmup_iters, and
        warmup_iters can't be greater than the dataloader length.

    Args:
        optimizer (torch optimizer): optimizer object.
        warmup_iters (int): number of warmup iterations.
        ndataloader (int): length of the dataloader.

    """

    def __init__(
        self,
        optimizer: torch.optim.Optimizer,
        warmup_iters: int,
        ndataloader: int,
    ) -&gt; None:
        warmup_factor: float = 1. / warmup_iters
        min_warmup_iters: int = min(warmup_iters, ndataloader - 1)

        def f(x: int) -&gt; float:
            if x &gt;= min_warmup_iters:
                return 1
            alpha: float = float(x) / min_warmup_iters
            return warmup_factor * (1 - alpha) + alpha

        super().__init__(optimizer, lr_lambda=f)
</code></pre>
<p>The code works as expected, but when I try to run a <a href="http://mypy-lang.org/" rel="nofollow noopener">mypy</a> test , it fails giving the following error:</p>
<pre><code class="lang-auto">44: error: Argument "lr_lambda" to "__init__" of "LambdaLR" has incompatible type "Callable[[int], float]"; expected "float"
</code></pre>
<p>Reading the docs (and taking a look at the code), argument <code>lr_lambda</code> to <code>LambdaLR</code> can be a function or a list of functions, but it is <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L92" rel="nofollow noopener">not explicitly typed in the code</a>. I don’t know why it says that the expected type is float.</p>
<p>However, if I copy the definition of the class <code>LambdaLR</code> to that same file and inherit from there, the problem disappears.</p>
<p>Any clue on what could be the problem?</p>
<p>Thanks</p>

Tony-Y: <p>If you are interested in LR warmup schedulers, try <a href="https://github.com/Tony-Y/pytorch_warmup" rel="nofollow noopener">pytorch_warmup</a>.</p>
<p><a href="https://colab.research.google.com/drive/1gWGM6jQLmPZABaYwW9O-A4TtFuoG4dh8" rel="nofollow noopener">Colab example</a> | <a href="https://pypi.org/project/pytorch-warmup/" rel="nofollow noopener">PyPI</a></p>
How to use weighted SmoothL1Loss?

Ir1d: <p>I noticed that this was implemented here: <a href="https://github.com/pytorch/pytorch/blob/master/modules/detectron/smooth_l1_loss_op.cc" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/modules/detectron/smooth_l1_loss_op.cc</a></p>
<p>However, I couldn’t find docs about calling this detectron module.<br>
How should we call this function in PyTorch?<br>
In <a href="https://pytorch.org/docs/master/nn.html#torch.nn.SmoothL1Loss" rel="nofollow noopener">https://pytorch.org/docs/master/nn.html#torch.nn.SmoothL1Loss</a> , we couldn’t pass the weights arguments as <a href="https://github.com/pytorch/pytorch/blob/master/modules/detectron/smooth_l1_loss_op.cc#L41" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/modules/detectron/smooth_l1_loss_op.cc#L41</a></p>
<p>Thank you in advance.</p>

Ir1d: <p>It seems this can be implemented with simple lines:</p>
<pre><code class="lang-python">def weighted_smooth_l1_loss(input, target, weights):
    # type: (Tensor, Tensor, Tensor) -&gt; Tensor
    t = torch.abs(input - target)
    return weights * torch.where(t &lt; 1, 0.5 * t ** 2, t - 0.5)
</code></pre>
<p>Then apply reduction such as <code>torch.mean</code> subsequently.</p>
Accessing tensors present on different GPUs

nbansal90: <p>Hey Folks,</p>
<p>I am running my model on multiple gpus.And I have a tensor, which will be present on each Gpus, which I want to access. Now, I am looking to get hold of all these tensor, on all the gpus, and do some operation, in a synchronous fashion, and then broadcast the result on all the gpus, to be used in the next step.</p>
<p>For Example: the tensor we are talking is <code>T</code>, with <code>T</code> being present in all cuda 0-3(for 4 gpus). Now, I need to get hold of this <code>T</code> tensor(which has different values at different gpu) and get some stat out of this, and then send back this stat to all gpus.</p>
<p>Please suggest me how this can be achieved.</p>

albanD: <p>Hi,</p>
<p>The simple way I see to do this is the following:<br>
You will have to first send all these Tensors to a common GPU. agregate the restults and compute your update. Then send the result back to each GPU.</p>

nbansal90: <p>Thanks Alan! I was wondering how do I send all the tensors to one gpu, and perform operations, which I want to perform, before the value of any of the tensor changes in any of their respective gpus.</p>
<p>Meaning, I don’t want the values of these tensor present on different gpus to change, before I complete my operation and send them back to respective gpu.</p>

rvarm1: <p>Hi Nitin,</p>
<p>To send all tensors to one GPU, you’d want to use <code>dist.gather</code>, which will gather all of the tensors onto one gpu (this is assuming you have one process running per gpu). If your tensor is <code>t</code>, then your call would look like:</p>
<pre><code class="lang-auto">t = your_tensor(size)
if rank == 0:
# rank 0 is the node all tensors will be gathered on
    gathered_tensors = [torch.zeros(size) for _ in range(WORLD_SIZE)]
    dist.gather(t, gathered_tensors, dst=0)
else:
    dist.gather(t, dst=0)
    
</code></pre>
<p>Then, you can compute what you want and send the result back wrapped in a tensor via a <code>dist.scatter</code>. To make sure the tensors don’t change on the gpu before they are gathered, ensure all nodes call into <code>gather</code> at the same time when all nodes have the desired value. You could also use <code>torch.distributed.barrier</code> if you need additional synchronization. Check out the docs at <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html</a></p>

nbansal90: <p>Thank you Rohan for exhaustive explanation!</p>
Including Optional Head parameters in optimizer/correct use of submodules?

Faolian: <p>So I have been working using a shared network architecture where there are multiple “network heads” that can be active depending on which task should be active. I implemented this as a list which gets created modules for these heads appended to it, and the head to be used is taken as an input parameter, which denotes which head to put the output of the shared layers through.</p>
<p>I realised in my work that my system wasn’t learning anything useful and was converging very oddly. I confirmed gradients are going as correctly, looked through all sorts of things and finally realised that the weights in the heads were not updating. The weights in the bodies were.</p>
<p>So i realised that if i take my optimiser call, which has parameters from self.actor.parameters(), this call does not actually include the head parameters - in essence, everything seems to be working, it’s just not being optimized.</p>
<p>So, how do i include the heads parameters when done in this way in the optimizer? Or is there an overall implementation change i should make here? Bare in mind that ideally i would like to simply take in num_tasks as an input</p>

ptrblck: <p>Your debugging path sounds quite good.<br>
Most likely you are using a Python <code>list</code> to store the different heads, which won’t properly register the submodules in the parent module (and thus won’t return the head parameters in <code>model.parameters()</code>).<br>
Could you use an <code>nn.ModuleList</code> as a drop-in replacement and check your code again?</p>

Faolian: <p>Sorry forgot to respond to this, yes this solved the issue.</p>
Periodic boundary condition

reangdeba: <p>Suppose I have</p>
<pre><code class="lang-auto">tensor([[[-1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [-1., 1., 1., 1.],
         [1., 1., 1., 1.]]])
</code></pre>
<p>How can I get</p>
<pre><code class="lang-auto">tensor([[[-1., 1., 1., 1., -1],
         [1., 1., 1., 1., 1.],
         [-1., 1., 1., 1., -1],
         [1., 1., 1., 1., 1],
         [-1., 1., 1., 1., -1]]])
</code></pre>
<p>In particular, for a N x N matrix, I want another N+1 x N+1 matrix such that the (N+1)th entry is the same as the 0th entry.</p>
<p>Also, I have multiple matrices. For instance, I have 50 4 x 4 matrixes. I want 50 new 5 x 5 matrices after the above operation.</p>

albanD: <p>Hi,</p>
<p>I think you will have to use <code>cat</code> for that:</p>
<pre><code class="lang-python">import torch

a = torch.tensor([[[-1., 1., 1., 1.],
          [1., 1., 1., 1.],
          [-1., 1., 1., 1.],
          [1., 1., 1., 1.]]])

out = torch.cat([a, a.narrow(-1, 0, 1)], -1)
bottom = torch.cat([a.narrow(-2, 0, 1), a.narrow(-2, 0, 1).narrow(-1, 0, 1)], -1)
res = torch.cat([out, bottom], -2)

print(res)
&gt;&gt;&gt; tensor([[[-1.,  1.,  1.,  1., -1.],
         [ 1.,  1.,  1.,  1.,  1.],
         [-1.,  1.,  1.,  1., -1.],
         [ 1.,  1.,  1.,  1.,  1.],
         [-1.,  1.,  1.,  1., -1.]]])
</code></pre>

reangdeba: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"><br>
That was what I needed precisely.</p>
Seeing improvement only with deterministic settings

amirhf: <p>Hello,</p>
<p>I’m developing a new method in PyTorch and supposedly it should improve my baseline. However, I can only see this improvement when I run my code in deterministic mode (no mater what seed I use with 4 workers), namely when I use the following line:</p>
<pre><code class="lang-auto">if deterministic:
        random.seed(seed)
        torch.manual_seed(seed)
        cudnn.deterministic = True
        cudnn.benchmark = False
        np.random.seed(seed)
</code></pre>
<p>As soon as I turn off the deterministic mode, the results are only comparable with my baseline. And I mean I see improvement in terms of better loss minimization and in my application better f1 and AUCPR scores.</p>

albanD: <p>This happens whatever the value of the seed?<br>
Can you try by just setting the cudnn flags and not the seeds?</p>
Using PackedSequence efficiently in forecasting seq2seq model

HappyPants: <p>I was wondering what would be the recommended approach to deal with PackedSequence objects in seq2seq models when inferring future values over a variable amount of step. Without, packed sequence and fixed amount of step I’d usually write the following in the forward function of my model:</p>
<pre><code class="lang-auto">def forward(self, sequence: torch.Tensor, steps: int, hx: torch.Tensor = None):
    hidden_states, hx = self.encoder(sequence, hx)
    #...
    h_t = hidden_states[:, -1, :].unsqueeze(1)
    h_t_inf = [] 
    for _ in range(steps):
        h_t, hx = self.decoder(h_t, hx)
        h_t_inf.append(self.activation(h_t))
    h_t_inf = torch.cat(h_t_inf, axis=1)
    forecast = self.output(h_t_inf)
    return forecast
</code></pre>
<p>However, how would that work if the <code>sequence</code> object would be a <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence" rel="nofollow noopener">PackedSequence</a> and steps would be a variable number of steps like in a tensor? I’ve been able to deal with PackedSequence using the solution provided with <a href="https://discuss.pytorch.org/t/getting-output-after-unpacking-padded-sequence/36730">this explanation</a> however, I am not sure how to deal with variable length steps efficiently.</p>
<pre><code class="lang-auto">def forward(self, sequence: PackedSequence steps: torch.Tensor, hx: torch.Tensor = None):
   #how would you deal with for _ in range(steps) where steps is variable
</code></pre>
Average of tensors

Shubham_Patel: <p>I am using pretrained VGG16 model to get features of an image. Image dimension is 256X340 however, VGG16 takes image of dimension 244X244, so I made 5 images out of original image, each of dimension 244X244 (topleft, topright, bottomleft, bottomright, center). Now I am passing these 5 images to model, dimension [5, 3, 244, 244] and getting output of [5, 4096] (modified last layer of VGG16 to get 4096 feats). Now I want to take the average of these 5 outputs so that I can get a final set of feature for original image.  Any suggestion on how to do this ?</p>

Shubham_Patel: <p>Does doing mean on axis = 0 can be a solution here.  ??</p>

albanD: <p>Hi,</p>
<p>Yes it is <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
It will give you one set of features for the set of images.</p>

Jordan_Howell: <p>Hello <a class="mention" href="/u/alband">@albanD</a>.  I have something similar.  My data class returns the image, label and ID.  How would I average over the ID.  each ID could have 1:N pictures in the models.  Code so far:</p>
<blockquote>
<p>class SuperEncoder(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MyModel, self).<strong>init</strong>()<br>
self.roofEncoder = nn.Sequential(<br>
nn.Conv2d(3, 6, 3, 1, 1),<br>
nn.ReLU(),<br>
nn.MaxPool2d(2),<br>
nn.Conv2d(6, 12, 3, 1, 1),<br>
nn.ReLU(),<br>
nn.MaxPool2d(2)<br>
)</p>
<pre><code>    self.dwellingEconder = nn.Sequential(
        nn.Conv2d(1, 6, 3, 1, 1),
        nn.ReLU(),
        nn.MaxPool2d(2),
        nn.Conv2d(6, 12, 3, 1, 1),
        nn.ReLU(),
        nn.MaxPool2d(2)
    )
    
    self.fc1 = nn.Linear(54*54*16, 1000)
    
    self.fc2 = nn.Linear(54*54*16, 1000)
    
    
    
    self.fc_out(x)
        
    
def forward(self, x1, x2):
    x1 = self.roofEncoder(x1)
    x1 = x1.view(x1.size(0), -1)
    x1 = F.relu(self.fc1(x1))
    
    
    x2 = self.dwellingEncoder(x2)
    x2 = x2.view(x2.size(0), -1)
    x2 = F.relu(self.fc2(x2))
</code></pre>
</blockquote>
<p>For more context see: <a href="https://discuss.pytorch.org/t/how-do-i-average-photo-feature-outputs-for-later-concatenation/59792" class="inline-onebox">How do I average photo feature outputs for later concatenation?</a></p>
Distributed Sampler in C++ frontend

soumyadipghosh: <p>Hello,<br>
I am trying to modify <a href="https://github.com/goldsborough/examples/blob/cpp/cpp/mnist/mnist.cpp" rel="nofollow noopener">this example C++ code</a> to incorporate distributed learning across multiple processors. For distributed sampling of the dataset, <code>torch.utils.data.distributed.DistributedSampler</code> is usually used in Python. However, I am not sure what is its equivalent in the C++ frontend. I believe I found it in the docs <a href="https://pytorch.org/cppdocs/api/classtorch_1_1data_1_1samplers_1_1_distributed_sampler.html#exhale-class-classtorch-1-1data-1-1samplers-1-1-distributed-sampler" rel="nofollow noopener">here</a> but I am getting errors in the following code snippet:</p>
<pre><code class="lang-auto">auto trainset = torch::data::datasets::MNIST("./data")
                  .map(torch::data::transforms::Normalize&lt;&gt;(0.1307, 0.3081))
                  .map(torch::data::transforms::Stack&lt;&gt;());
auto train_sampler = torch::data::samplers::DistributedSampler(trainset.size().value(), numranks, rank, false);
auto train_loader = torch::data::make_data_loader&lt;torch::data::samplers::DistributedSampler&gt;(std::move(trainset), batch_size);
</code></pre>
<p>Can anyone point out what is the right way to use it ?<br>
Thanks.</p>
[Quantization] (Error): No function is registered for schema aten

jiacheng1gujiaxin: <pre><code class="lang-auto">I tried to quantizate my own shuffle model through 'STATIC QUANTIZATION WITH EAGER MODE IN PYTORCH'.
But when it comes to forward propagation, the time to assess model losses, I failed
</code></pre>
<p>RuntimeError: No function is registered for schema aten::native_batch_norm(Tensor input, Tensor? weight, Tensor? bias, Tensor? running_mean, Tensor? running_var, bool training, float momentum, float eps) -&gt; (Tensor, Tensor, Tensor) on tensor type QuantizedCPUTensorId; available functions are CPUTensorId, CUDATensorId, MkldnnCPUTensorId, VariableTensorId<br>
The above operation failed in interpreter, with the following stack trace:<br>
at code/ <strong>torch</strong> /shuff_slim/___torch_mangle_297.py:316:13</p>

dskhudia: <p>Please fold batchnorms with preceding convs manually. Look for the APIs to do this for you in the Resnext-101 tutorial.</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/dskhudia">@dskhudia</a> could you share the link for the Resnext 101 tutorial?</p>
<p>Or could you provide an example ? That would be very helpful</p>

jerryzh168: <p>You can find tutorial for post training static quantization here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a></p>
Training independent networks in parallel with reproducibility

pgf: <p>I would like to train, say, 10 independent neural networks on 5 GPUs in parallel (by training two on each GPU assuming there are no memory constraints). Also, I would like the code to be reproducible. Therefore, I have been training each network by re-running the same Python script, changing only the GPU device. I specify <code>torch.manual_seed</code> for reproducibility.</p>
<p>Is there anyway to do this in a single Python script where PyTorch does the work of distributing the networks to the GPUs <strong>while maintaining reproducibility</strong>? I am aware of threading as one way to do this (something similar to <a href="https://github.com/waitwaitforget/modelparallel_pytorch" rel="nofollow noopener">ModelParallel</a>), but I am worried about reproducibility of my code when using threading.</p>
<p>Thank you for the help!</p>

albanD: <p>Hi,</p>
<p>Even with the same code, if you use different hardware/software versions, we don’t guarantee reproducibility. See the doc about this <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">here</a>.</p>
<p>If you want to ensure that the scripts are independent. I would recommend using a simple batch script that launch all the jobs <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pgf: <p>Thanks for the reply! I have seen the documentation on reproducibility.</p>
<p>In this case, to begin with, I am only looking for reproducibility for my particular set-up (hardware and software). I am indeed using a shell script to launch the jobs, but I thought it would be neater if I could do this all within one script with PyTorch responsible for the distribution.</p>

albanD: <p>The level of “independence” between the run would be much smaller if you run the whole think in a single process. They will share the same python interpreter, the same cuda allocator, same memory space.</p>
<p>The ModelParallel tool that you linked seems interesting. But I don’t know of any other work along those lines…</p>
Combining Segmentation Predictions from Multiple Networks

maaft1: <p>For Defect Segmentation I have trained a distinct segmentation network for each defect. I.e. 20 classes -&gt; 20 models where the output of each model is a sigmoid activated segmentation-map of shape Bx1xHxW.</p>
<p>Now I want to fuse all predictions in a single segmentation map of shape Bx20xHxW.</p>
<p>Here the main issue is, that the network predictions are always close to 1 (0.97-0.999), even for false positives which makes a pixel-wise class decision impossible.</p>
<p>I already tried to use negative examples from other classes during training to lower the confidence score in case of false positives but this didn’t change much.</p>
<p>Any Ideas how I can improve my training procedure such that the network outputs are more evenly distributed between 0-1? The single-network-per-class unfortunately is a requirement.</p>

ptrblck: <p>I assume you’ve trained each model in a binary fashion, i.e. each model only predicts its corresponding class as the positive class and all other classes as the negative class?<br>
If so, how is the performance of each model (confusion matrix)?</p>

maaft1: <p>Exactly.</p>
<p>Until now I only evaluated the dice-score which floats around 0.8 for most defects but can be lower or higher.</p>
<p>What additional information do you think can the confusion matrix tell me here? I am fully aware that false positives happen, I just want their confidence to be lower than the correct predictions.</p>