akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Version of Imagnet Usend on Torchvision

username: nicolalandro
cooked: <p>Someone know what version of ImageNet are used to train the torchvision pretrained models? I do not find it into documentation <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">here</a>.</p>

username: ptrblck
cooked: <p>They were trained on <a href="https://discuss.pytorch.org/t/how-to-get-the-imagenet-dataset-on-which-pytorch-models-are-trained-on/84538/5">ILSVRC2012</a> and were not retrained, if I’m not mistaken.</p>
RuntimeError: cublas runtime error : the GPU program failed to execute at pytorch/aten/src/THC/THCBlas.cu:249

username: liyanan
cooked: <p>The above error occurred when I used fairseq to train the language model. I don’t know what to do. I used torch0.4.1</p>

username: ptrblck
cooked: <p>Could you update PyTorch to the latest stable version (<code>1.6</code>) and rerun the code again, please?</p>
Make values consistent across versions

username: chinmay5
cooked: <p>I have a model which was trained on version 0.4, and I wanted to use the weights directly in the latest  version (1.6). From what I read, this should not be an issue since the stored model weights are backward compatible. However, I see a difference in the values across the versions with the older version giving slightly higher value. I think this is due to the nn.Upsample() layer that is used by the model. Is there some way by which I can make the values consistent across the two versions?</p>

username: ptrblck
cooked: <p>How did you isolate the difference to <code>Upsample</code> and could you post an example tensor, which would recreate the different outputs between these versions?<br>
Note that you might want to have a loop at the <code>align_corners</code> or the <code>mode</code> argument.<br>
I don’t know if the default values have changed, but it should be a quick check.</p>

username: chinmay5
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the reply. From the <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">link</a> and looking at my model, I realized that at least in the forward pass, just the nn.Upsample() layer seems to be the reason why values would be different across the two versions. I explicitly used <code>mode='trilinear', align_corners=True</code>  while training the model in PyTorch 0.4 . Are there other elements I should keep a check on since the value difference is a bit strange.</p>

username: ptrblck
cooked: <p>I don’t have any specific layers in mind, as <code>0.4</code> is quite old by now and I don’t remember exactly what has changed since then. However, it seems that you’ve already isolated it to <code>nn.Upsample</code> or are you not sure where the difference is coming from?</p>

username: chinmay5
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I am not sure where the difference is coming from. I just took a guess based on the attached link.</p>

username: chinmay5
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> if you can suggest certain items that might be causing this issue and I should take a look at, it would be really great.</p>

username: ptrblck
cooked: <p>I think a full comparison of the forward pass would be useful.<br>
To do so, you could use forward hooks (as explained <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>) to save all intermediate output tensors.<br>
Once the hooks are working, use a constant tensor (e.g. all ones) for both models, store all activations, and compare them.</p>
How to avoid `can't multiply sequence by non-int of type 'float'`

username: HaFred
cooked: <p>Hi,<br>
I got two dict <code>self.theta</code> &amp; <code>self.beta</code> and a fp value <code>eta</code>, when I tried to modify it as below:</p>
<pre><code class="lang-auto">self.theta[0]=self.theta[0]-eta*self.beta[0]
</code></pre>
<p>It returned <code>TypeError: can't multiply sequence by non-int of type 'float'</code>.<br>
If I put it into a for loop of the dict, <strong>it worked</strong>, but I cannot separately manipulate with a single key such as [0] above, have to finish the modification in the for loop in this case.</p>
<pre><code class="lang-auto">for key in self.theta.items(): 
      self.theta[key[0]]=self.theta[key[0]]-eta*self.dE_dTheta[key[0]]
</code></pre>
<p>I wonder is it possible to directly modify the <code>self.theta[0]</code> without getting the error? The numpy array seems like a workaround but I need to change the code a lot so I am thinking about solely torch methods.</p>

username: ptrblck
cooked: <p>This doesn’t seem to be a PyTorch-specific issue, but seems to be raised by Python directly.<br>
Could you post the content of both <code>dicts</code> and explain the use case a bit?</p>

username: HaFred
cooked: <p>Thank you for your reply, you are totally right and I solve it with some help from stackoverflow. Specifically, I messed up the index of <code>self.theta</code> which makes me do <code>eta * []</code> so it raised the error.<br>
Sorry for the ‘unrelated’ post.<br>
(<a href="https://stackoverflow.com/questions/63302668/why-does-items-work-for-the-multiplication-between-sequence-and-float#" rel="nofollow noopener">https://stackoverflow.com/questions/63302668/why-does-items-work-for-the-multiplication-between-sequence-and-float#</a>)</p>
Pytorch installation

username: Sowrirajan_Nandhagop
cooked: <p>ERROR: Could not find a version that satisfies the requirement torch===1.6.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch===1.6.0</p>
<p>How to solve tbe above issue while installing pytorch</p>

username: ptrblck
cooked: <p>Could you try to update your <code>conda</code> base environment? Based on the error it seems that <code>conda</code> isn’t able to lookup the new packages.</p>
Access to input / output names in a model (torch::jit::script::Module)

username: dawnleave
cooked: <p>Hi there,</p>
<p>I am working on a wrapper layer on top of PyTorch, in which the following methods need to be implemented:</p>
<pre><code class="lang-cpp">std::vector&lt;std::string&gt; GetInputNames const;
std::vector&lt;std::string&gt; GetOutputNames const;
void SetInput(std::string, at::Tensor);
at::Tensor GetOutput(std::string);
</code></pre>
<p>It seems straight-forward to implement those in TensorFlow’s context. See an example:</p>
<pre><code class="lang-cpp">  tensor_dict feed_dict = {
      {"input", data},
  };

  std::vector&lt;tensorflow::Tensor&gt; outputs;
  TF_CHECK_OK(sess-&gt;Run(feed_dict, {"output", "dense/kernel:0", "dense/bias:0"},
                        {}, &amp;outputs));
</code></pre>
<p>where I can save “input” as input key, data as input value, “output” as output key etc.</p>
<p>I wonder is there a built-in way to implement the counterpart for PyTorch?<br>
I read through <a href="https://github.com/pytorch/pytorch/blob/master/torch/onnx/utils.py#L382" rel="nofollow noopener">the post to convert PyTorch Torch Script model to ONNX</a>, it seems to be doable, but I am not sure if it is the correct direction.</p>
<p>Thanks in advance!<br>
Dong</p>

username: nicklhy
cooked: <p>Any updates here? Got the same problem recently.</p>

username: PascalIversen
cooked: <p>I got the same issue…</p>
Convert the tensor to one hot vector

username: anil_sarode
cooked: <p>I am learning PyTorch and currently working on DCGAN. I am getting my image data along with its age from the <code>dataloader</code>. Now while training the DCGAN I want to give this age values as a condition one-hot vector to make it cDCGAN.</p>
<p>The size of the tensor data1[1] which has age values is batch size=64. following is the python code (step by step ) I am using to get the one-hot vector.</p>
<pre><code>data1[1].size()


# torch.Size([64])

data1[1]

# tensor([18,  6, 20, 65, 41, 18, 15, 63, 17, 16, 45, 21, 62, 19, 13, 32, 41, 24,
    24, 18, 81, 55, 21, 20, 34, 27, 31, 32, 21, 45, 45, 31, 32, 24, 24, 13,
    44, 37, 30, 24,  8, 24, 51, 16, 46, 32, 32, 64, 34, 26, 31, 54, 19, 27,
    17, 31, 16, 21, 27, 29, 30, 35, 24, 20])

samples=data1[1].tolist()

# age categories 
limits = sorted([18, 29, 39, 49, 59, 100])

onehot = []  # this is where we will store our one-hot encodings

for sample in samples:
    row = [0]*len(limits)  # preallocating a list
    for i, limit in enumerate(limits):
          if sample &lt;= limit:
              row[i] = 1
             break

   # storing that sample's onehot into a onehot list of lists
    onehot.append(row)
</code></pre>
<p>As you can see I am converting the tensor to list and then converting to the one-hot vector.<br>
My question is it possible  to the same using  <code>torch.nn.functional.one_hot()</code> ?</p>
Some of weight/gradient/input tensors are located on different GPUs

username: sankara68
cooked: <p>I get the following error, while using transformers<br>
Some of weight/gradient/input tensors are located on different GPUs. Please move them to a single one. at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:24<br>
Any ideas on how to fix it?<br>
Thanks,<br>
Sankar</p>

username: ptrblck
cooked: <p>As the error explains, some tensors are not on the same device and the some operations thus cannot be executed.<br>
Feel free to post some code so that we can help any further. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Will there be Python 4?

username: narendra_nareshit
cooked: <p>I just want to know Will there be <strong>Python</strong> 4? and what is the latest version in python?</p>

username: ptrblck
cooked: <p>You’ve landed in the PyTorch discussion board, so general Python question might be better answered in their mailing lists / discussion boards / or StackOverflow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Combine train and test set (torchtext) but ConcatDataset object has no attribute "get_vocab"

username: EmreTokyuez
cooked: <p>I want to use the examples in the test set of the IMDB Sentiment Analysis Dataset for training, as I have built my own benchmark with which I will compare the performance of various Models (my Matura Thesis)</p>
<p>So after trying, I got the appending working and also managed ot split it, so that I have a validation set as well. The code is the following:</p>
<pre><code class="lang-auto">from torchtext.experimental.datasets import IMDB

from torchtext.data.utils import get_tokenizer

tokenizer = get_tokenizer("spacy")

train_data, test_data = IMDB(tokenizer = tokenizer)
train_data, valid_data = torch.utils.data.random_split(train_data, [17500,7500])

Number of training examples: 17500
Number of validation examples: 7500
Number of testing examples: 25000
</code></pre>
<p>Now I want to merge the train and the test set and leave the validation set alone, so I do this:</p>
<p>from torch.utils.data import ConcatDataset</p>
<p>train_data = ConcatDataset([train_data, test_data])</p>
<p>print(f’Number of training examples: {len(train_data)}’)</p>
<p>Number of training examples: 42500</p>
<p>But when I try to get the vocabulary, I get the following error:</p>
<pre><code class="lang-auto">vocab = train_dataset.get_vocab()
**AttributeError** Traceback (most recent call last) in **----&gt; 1** vocab **=** train_dataset **.** get_vocab **(** **)** **AttributeError** : 'ConcatDataset' object has no attribute 'get_vocab'
</code></pre>
<p>How can I solve this?</p>

username: ptrblck
cooked: <p><code>random_split</code> will return <code>Subset</code>s, which wrap the dataset.<br>
To access the underlying dataset, you could use <code>train_dataset.dataset.get_vocab()</code>.<br>
However, this would call the <code>get_vocab()</code> methon of course on the complete dataset.<br>
I’m not familiar with this method, but if it creates the vocabulary based on the used dataset, it would contain the words from the training and validation dataset.</p>
Perform operations on a given index

username: m4nu
cooked: <p>Hi,</p>
<p>I have a tenor of a shape <code>(n,m,2)</code>, I would like to perform an operation (a rotation) on the last dimension.<br>
That operation depends on a leaf variable that should be backpropagated. I am using the below code but the <code>theta</code> is getting a gradient of <code>None</code>.</p>
<pre><code class="lang-auto">        left_rotation = torch.tensor([torch.cos(self.theta), -torch.sin(self.theta)])
        left = (aligned_grid * left_rotation).sum(dim=2).pow(2) / self.alpha.pow(2)
        # Do more things with the "left" variable
</code></pre>
<p>The intent here is to broadcast the value of <code>torch.cos(self.theta), -torch.sin(self.theta)</code> on the last dimension.<br>
What would be the proper way to go about this?</p>
<p>Thanks a lot for helping</p>

username: Kushaj
cooked: <p>When you do <code>torch.tensor([torch.cos(self.theta), ...])</code>, you are creating a new leaf variable which dosen’t connect the graph to theta. So now in your graph there is no way to reach theta, thus no gradient.</p>
<p>Use a list to hold those two values and then do matrix multiply using python <code>@</code>.</p>

username: m4nu
cooked: <p>Thanks, Kushajveer. This was not working either, I got an operand type not matching error. Besides, that would not broadcast properly if I were to do that right?<br>
I’ve ended up refactoring this function by using <code>.clone()</code> when needed to keep new tensor in the graph.</p>

username: Kushaj
cooked: <p>Can you give an example of how you did that?</p>

username: m4nu
cooked: <p>Something like this:</p>
<pre><code class="lang-auto">        left = self.meshgrid.clone()
        right = self.meshgrid.clone()
        
        left[:,:,0] -= self.k
        left[:,:,1] -= self.k
        right[:,:,0] -= self.h
        right[:,:,1] -= self.h
        
        left[:,:,1] *= -torch.sin(self.theta)
        left[:,:,0] *= torch.cos(self.theta)
        right[:,:,0] *= torch.sin(self.theta)
        right[:,:,1] *= torch.cos(self.theta)
</code></pre>
I can't install pytorch in python

username: Sergio_Torre
cooked: <p>Hello , I was testing install pytorch when pip3 give me a error unknow and I can’t install, I don’t know because its wants install pytorch I am dissapointed for this package,I don’t know what to do to install pytorch, I had my intentions, if anyone could help me install it, I tried but refused.</p>
<p>Thanks you for help me.<br>
bye</p>

username: qap
cooked: <p>Which set of installation instructions are you following?</p>
How could I mix two tensor at variable place and size?

username: coincheung
cooked: <p>Hi,</p>
<p>I am trying to do this:</p>
<pre><code class="lang-auto">a = torch.randn(4, 3, 10, 10)
b = torch.randn(4, 3, 10, 10)
x1 = torch.randint(0, 5, (4,))
y1 = torch.randint(0, 5, (4,))
x2 = torch.randint(6, 10, (4,))
y2 = torch.randint(6, 10, (4,))

b[:, :, x1:x2, y1:y2] = a[:, :, x1:x2, y1:y2]
</code></pre>
<p>This gives the error message of</p>
<blockquote>
<p>TypeError: only integer tensors of a single element can be converted to an index</p>
</blockquote>
<p>How could I make it work please ?</p>

username: JuanFMontesinos
cooked: <p>What are you trying to do?<br>
You are  using indexing over a tuple…</p>

username: coincheung
cooked: <p>Hi,</p>
<p>My tensors of a and b have size of (N, C, H, W) and I want to crop (h, w) crops from tensor a and patch it to the associated positions to tensor b. The problem is the I defined N different (h, w) patches which has different positions and sizes. That means I need to copy a[0, :, h0_0:h0_1, w0_0, w0_1] to b[0, :, h0_0:h0_1, w0_0, w0_1], and copy a[1, :, h1_0, h1_1, w1_0:w1_1] to b[1, :, h1_0, h1_1, w1_0:w1_1], and a[2, :, h2_0:h2_1, w2_0:w2_1] to b[2, :, h2_0:h2_1, w2_0:w2_1], and so on.</p>
<p>Would you please tell me how can I do this with pytorch ?</p>

username: JuanFMontesinos
cooked: <p>Hmmm I don’t really know if it’s possible to do it in a single line.</p>
<p>Typically (here and in numpy) when you pass a list for indexing it means “gather elements in those indices”</p>
<p>For example:</p>
<pre><code class="lang-auto">a[[0,2,3],1:5,3:8]
</code></pre>
<p>would get [1:5,3:8] for elements 0,2,3.</p>
<p>In your case however it’s more like doing a for loop together with indexing.</p>
<p>Soo you could do something like</p>
<pre><code class="lang-auto">import numpy as np 
x = np.array([[ 0,  1,  2],[ 3,  4,  5],[ 6,  7,  8],[ 9, 10, 11]]) 
   
print 'Our array is:' 
print x 
print '\n' 

rows = np.array([[0,0],[3,3]])
cols = np.array([[0,2],[0,2]]) 
y = x[rows,cols] 
   
print 'The corner elements of this array are:' 
print y
</code></pre>
<pre><code class="lang-auto">Our array is:                                                                 
[[ 0  1  2]                                                                   
 [ 3  4  5]                                                                   
 [ 6  7  8]                                                                   
 [ 9 10 11]]
 
The corner elements of this array are:                                        
[[ 0  2]                                                                      
 [ 9 11]] 
</code></pre>
<p>(From numpy’s examples)<br>
And passing a matrix which get indices element-wise but it would be a bit messy compared to a for loop</p>
<pre><code class="lang-auto">import torch

N = 4
a = torch.randn(N, 3, 10, 10)
b = torch.randn(N, 3, 10, 10)
X1 = torch.randint(0, 5, (N,))
Y1 = torch.randint(0, 5, (N,))
X2 = torch.randint(6, 9, (N,))
Y2 = torch.randint(6, 9, (N,))

for i,(x1,y1,x2,y2) in enumerate(zip(X1,Y1,X2,Y2)):


    b[i, :, x1:x2, y1:y2] = a[i, :, x1:x2, y1:y2]
</code></pre>
Saving PyTorch models with new zipfile-based format

username: tominator
cooked: <p>In the PyTorch 1.6 release note I saw that the old pickle-based serialization format was replaced by a new zipfile-based format.</p>
<p>I want to save the entire PyTorch model and NOT the state_dict for deployment reasons. “The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself.” (<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model" rel="nofollow noopener">Link</a>). My question: does the new format eliminate this disadvantage? Is it saving the model class or not?</p>
Pytorch 1.6.0: Wrong true_divide on cuda

username: thongnt
cooked: <p>Hello,</p>
<p>I just used new <code>true_divide</code> method on  pytorch 1.6.0.<br>
It possibly has a bug which took me quite a lot of time to trace back.</p>
<p>This code <code>torch.tensor([435]).true_divide(15).ceil().long()</code> gives <strong>29</strong> (correct result), while its cuda version <code>torch.tensor([435]).to("cuda:1").true_divide(15).ceil().long()</code> gives <strong>30</strong>.</p>
<p>It might be something related to precision of math operators on cuda.<br>
Do you guys know how to fix it?<br>
Thank you.</p>

username: ptrblck
cooked: <p>The error is not created by the <code>true_divide</code> operation, but by the <code>ceil</code> operation, since the CUDA result is not exactly <code>29</code>, but contains a small <code>eps</code> value of <code>~1e-6</code>. This <code>eps</code> might be created, if the division is executed as a multiplication using the reciprocal.</p>
ReduceLROnPlateau reduces LR without need

username: vojtooo
cooked: <p>Hi,</p>
<p>I have a problem with ReduceLROnPlateau. I initiated it using:<br>
<code>scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optim, verbose=True, patience=5)</code></p>
<p>and then, after each epoch (validation), I perform<br>
<code>scheduler.step(loss_meter_validation.mean)</code></p>
<p>The problem is, that the scheduler decreases LR even though the loss does not plateau.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38c7fcafcacd31fb2751dc485516f36f74d4bb93.png" alt="train_loss" data-base62-sha1="86jcW0DdZ5zDrpfFceEtG1a9wMX" width="640" height="480"></p>
<p>Can somebody please help me out?</p>
<p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Could you print the values of <code>loss_meter_validation.mean</code> for the first steps, which seem to trigger the decrease in the learning rate, please?</p>
How does PyTorch handle incomplete batches?

username: bfeeny
cooked: <p>I know that when training, if you are using <code>BatchNorm</code> and have only 1 sample in your batch, then it can error out, which makes sense.  I also know that you can use <code>drop_last=True</code> in your dataloader for those situations.  But <code>BatchNorm</code> aside, how does PyTorch handle incomplete batches in training and inference?  Is it able to deal with the different batch sizes or does it somehow pad the batch?</p>
<p>And the second part, similar question, does it matter at all if the size of your dataset is divisible by batch size when doing inference (validation/test)?</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/bfeeny">@bfeeny</a>, considering the last mini_batch is not the same size, as the rest. I guess one can implement a batch loader with variable size. Also the pytorch model need (batch_size,width,height,channel) where batch size can vary.</p>

username: ptrblck
cooked: <p>There is no padding involved and is not needed, since you never specify the batch size as a model attribute. As you’ve already explained, a batch size of 1 (potentially the last batch returned by the <code>DataLoader</code>) could create an error, if specific layers (such as batchnorm) cannot calculate some statistics using a single sample. The workaround would be to drop this batch as you’ve mentioned.</p>
<p>As <a class="mention" href="/u/usama_hasan">@Usama_Hasan</a> said, the input tensor should contain the batch dimension (usually in dim0) and your model will be able to deal with arbitrary batch sizes (assuming your system doesn’t run out of memory).</p>
<p>Usually you would call <code>model.eval()</code> while using the validation dataset or executing the inference, which switches the behavior of some modules. E.g. batchnorm layers will use their running stats and will thus not need to calculate the stats from the input batch and will not raise any errors for a single sample in the batch.</p>
Efficient way to use torchtext TabularDataset on large dataset

username: mathematics
cooked: <p>I found problem when loading torchtext TabularDataset when using large dataset,<br>
It takes too much ram.</p>
<p>I tried to load as given which is too slow to run,</p>
<pre><code class="lang-auto">tokenize = lambda x:x.split()
TEXT = data.Field(sequential=True, tokenize=tokenize)
LABEL = data.LabelField()
fields=[('customer_review',  TEXT),('polarity', LABEL)]
train,test = data.TabularDataset.splits(path='.',format='csv', train="/content/drive/My Drive/cleaned_train.csv", test="/content/drive/My Drive/cleaned_test.csv", fields=fields)
</code></pre>
<p>Is there any efficient way to load.</p>

username: vdw
cooked: <p>If your dataset is indeed too large to fit into memory, you need to split it and train using the different “sub-datasets” one after another. In this case, I would actually prepocess the text data completely so that the file already contained the sequences of indexes to speed up the training. Otherwise, you would have to do the preprocessing potentially in each epoch.</p>
<p>As a side comment, when using <code>tokenize = lambda x:x.split()</code> I hope your input text documents have whitespaces before punctuation marks – and after them, which is not a given in user-generated data.</p>

username: mathematics
cooked: <p>Thanks but it is alot of messy doing this.<br>
However , when using through spacy It is running  smoothly without taking too much ram.</p>
<p>To run on pytorch I had to split training csv to 5 pieces.</p>
Are torch.transform.MelSpectrogram same as torch.compliance.kaldi.fbank

username: victkid
cooked: <p>By reading this <a href="https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html" rel="nofollow noopener">artical</a>, it looks like the fbank is just the mel scaled spectrogram. Could anyone confirm that?</p>
<p>I understand the results from transform.MelSpectrogram and from compliance.fbank might not end up the same even with the same parameter settings. Are the general concepts between the two functions match?</p>
Pysyft torch.mm function

username: SandPhoenix
cooked: <p>Hello everyone.<br>
I am using pysyft with pytorch. The output of a model is then passed to a permutation function that does just the multiplication with permutation Matrix P.<br>
The code is working just fine without the integration of pysyft but for some reason I am getting this error: RuntimeError: matrices expected, got 1D, 2D tensors at /pytorch/aten/src/TH/generic/THTensorMath.cpp:131 when I try to do the exactly the same with pysyft.<br>
Any help?? I would appreciate</p>

username: ptrblck
cooked: <p>Could you check your code for any usage of <code>squeeze()</code>?<br>
If you don’t pass a <code>dim</code> argument to this method, all dimensions with a size of 1 will be removed.<br>
I don’t know how you are using <code>pysift</code>, but could the batch size be reduced to 1, which would then remove this dimension via <code>squeeze()</code>?</p>

username: SandPhoenix
cooked: <p>I don’t seem to understand what you said exactly can u explain more ? But for sure I am not using the <code>squeeze</code> function</p>

username: ptrblck
cooked: <p>Thanks for the information. Could you post a code snippet, which would create this issue, so that we can debug further?</p>
Import error when using torch.utils.bottleneck

username: 111357
cooked: <p>I use <code>torch.utils.bottleneck</code> to profile the code.<br>
But I got some import error as follow:</p>
<pre><code class="lang-auto">  File "./benchmark/csm/gnn.py", line 17, in &lt;module&gt;
    from ...data import sc as snc
ImportError: attempted relative import with no known parent package
</code></pre>
<p>How can I use ‘torch.utils.bottleneck’ to profile my code without modifying my project code? Because there are a lot of <code>from ...xx import xx as xx</code> in my project.</p>
Autocast for DataParallel nn.Sequential?

username: Martin_cb
cooked: <p>I am trying to get automatic mixed precision’s autocast to work for an nn.Sequential model. The tutorial here:<br>
<a href="https://pytorch.org/docs/master/notes/amp_examples.html#dataparallel-in-a-single-process" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/notes/amp_examples.html#dataparallel-in-a-single-process</a></p>
<ul>
<li>suggests to overwrite the forward() when using multiple GPUs via DataParallel.<br>
What is the best way to do this for the nn.Sequential case?</li>
</ul>
<p>here is how I define my model currently:</p>
<blockquote>
<p>layers = [ &lt;bunch of layers&gt;]<br>
model = nn.Sequential(*layers)<br>
model = nn.DataParallel(model)<br>
model.to(device)</p>
</blockquote>

username: Martin_cb
cooked: <p>I came up with this solution, extending the Sequential class and overriding its forward like this:</p>
<pre><code class="lang-auto">class AutocastSequential(nn.Sequential):
    @autocast()
    def forward(self, input):
        for module in self:
            input = module(input)
        return input
</code></pre>
<p>It executes, although I am unsure how to check if mixed precision is actually enabled for all GPUs</p>
Checkpoint_sequential for DataParallel

username: Martin_cb
cooked: <p>I’ve been trying to use checkpoint_sequential for DataParallel, here is my solution:</p>
<pre><code class="lang-auto">out= checkpoint_sequential(model.module, num_checkpoints, batch_data)

</code></pre>
<ul>
<li>I basically obtain the nn.Sequential object inside the DataParallel, and apply the checkpoints to that.  It is also one of those things that is difficult to check if it actually works and applies checkpoints onto all of the GPUs, would appreciate some feedback <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
</ul>
Does Batchnorm work for gradient accumulation?

username: Martin_cb
cooked: <p>A popular technique that claims to reduce model RAM requirements is gradient accumulation. However, from what I could gather, it seems that it cannot be used in real world applications as it is incompatible with batchnorm.<br>
See the discussion here:<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://forums.fast.ai/uploads/default/optimized/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5_2_32x32.png" class="site-icon" width="16" height="16">
      <a href="https://forums.fast.ai/t/accumulating-gradients/33219" target="_blank" rel="nofollow noopener" title="02:04AM - 22 December 2018">Deep Learning Course Forums – 22 Dec 18</a>
  </header>
  <article class="onebox-body">
    <img src="https://forums.fast.ai/uploads/default/original/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://forums.fast.ai/t/accumulating-gradients/33219" target="_blank" rel="nofollow noopener">Accumulating Gradients</a></h3>

<p>I have a custom batch sampler that I am training with but I also want to accumulate the gradients and do weight updates only at the end of each epoch but not each batch. What would be the most fastai way to do this with minimal effort. Thanks !</p>

  <p><span class="label1">Reading time: 20 mins 🕑</span>
    <span class="label2">Likes: 132 ❤</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>My question is, are there are any recent proven workarounds or fixes that have been used in production environments that would let one use gradient accumulation?</p>

username: Kushaj
cooked: <p>You can change batchnorm to RunningBatchNorm which was proposed by Jeremy Howard in fastai course. It removes the dependence on batch size.</p>

username: Martin_cb
cooked: <p>What about GroupNorm?</p>

username: Kushaj
cooked: <p>That is also one option.</p>
AUTOGRAD for the argmax function

username: Massivaa
cooked: <p>Hello,<br>
I have a question about the argmax function when I put :<br>
<code>torch.argmax (output, 1) </code>I get a result: <code>grad_fn = NotImplemented </code>and when I tried:</p>
<pre><code class="lang-auto">torch.argmax (output, 1) .float (). requires_grad_ (True)
</code></pre>
<p>it shows me</p>
<pre><code class="lang-auto"> grad_fn = CopyBackwards.
</code></pre>
<p>so can you explain to me what exactly does that mean<br>
Thank you in advance</p>

username: smth
cooked: <p>argmax is not mathematically differentiable without relaxations</p>

username: Massivaa
cooked: <p>thanks for your reply<br>
I didn’t understand your explanation, can you explain more please.</p>

username: KFrank
cooked: <p>Hello Massivaa!</p>
<aside class="quote no-group" data-username="Massivaa" data-post="3" data-topic="92102" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/54ee81/48.png" class="avatar"> Massivaa:</div>
<blockquote>
<p>I didn’t understand your explanation, can you explain more please.</p>
</blockquote>
</aside>
<p>Another way to say it is that <code>argmax()</code> is not <em>usefully</em> differentiable.</p>
<p>Consider <code>torch.argmax (torch.FloatTensor ([x, 1.0]))</code>.<br>
<code>argmax()</code> will be <code>1</code> for <code>x &lt; 1.0</code> and <code>0</code> for <code>x &gt; 1.0</code>.  In both cases<br>
its derivative (gradient) with respect to <code>x</code> will be zero, and it won’t<br>
be mathematically differentiable right at <code>x = 1.0</code>.</p>
<p>Having zero gradient almost everywhere isn’t useful for gradient<br>
descent optimization, so pytorch doesn’t bother to implement<br>
autograd (<code>grad_fn</code>) for <code>argmax()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Massivaa
cooked: <p>thank you for your explanation K. Frank <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Upgrade torch for windows 10 and CUDA 11

username: liran
cooked: <p>Hi,<br>
My OS is Windows 10 and I have also CUDA 11 installed. currently, I`m working on torch 1.0.0 with CUDA 10.0 installation.</p>
<ol>
<li>Can I upgrade my torch version and CUDA version with Pip? Or may I need to remove torch 1.0.0 first?</li>
<li>I saw that there is no installation for CUDA 11, so install for CUDA 10.2 is ok?</li>
</ol>
<p>Thank you</p>

username: ptrblck
cooked: <ol>
<li>If you are using virtual environments (e.g. via <code>conda</code>) you can have multiple installations on your system (one per environment)</li>
<li>Since the CUDA11 nightly binaries are not ready yet, you could install the latest PyTorch version with CUDA10.2 from the binaries or try to build from source using your local CUDA11 installation.</li>
</ol>

username: liran
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your answer. I am working on pycharm IDE, so I need to delete torch 1.0.0 before upgrade to the newest version?<br>
Thank you!</p>

username: Nikronic
cooked: <p>Hi, virtual environments are not part of the IDEs so you can create them using PyCharm. But PyCharm has implemented an GUI for it, so you can do it easily.</p>

username: liran
cooked: <p>Hi <a class="mention" href="/u/nikronic">@Nikronic</a><br>
Im using Pycharm without Anaconda, so I need to delete the old version?<br>
Thank you</p>

username: Nikronic
cooked: <p>No, use this doc.<br>
<a href="https://docs.python.org/3/library/venv.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.python.org/3/library/venv.html</a></p>
Graph convolutional network with various sizes of input

username: thecho7
cooked: <p>Hi there,</p>
<p>I’m wondering whether it is possible to forward various size of input to the graph convolutional networks (GCNs) or not.<br>
For example, I have 3 different inputs as below, (please ignore the values of tensors(0))</p>
<pre><code>in_a = torch.zeros(50, 100)
in_b = torch.zeros(80, 100)
in_c = torch.zeros(150, 100)
</code></pre>
<p>Will GCNs which have 100x100 nodes be trained well with those inputs?<br>
Please reply any comments whenever you need further explanations.<br>
Thanks</p>
Obtaining the image name after doing random split

username: karen-gishyan
cooked: <p>Hi everyone,</p>
<p>I was wondering if it is possible to obtain the original  name or the full path of the image after using torch.utils.data.random_split(). Achieving this would allow to save train and test split images with their original names to local directories. Thanks.</p>

username: ptrblck
cooked: <p><code>random_split</code> wraps the <code>Dataset</code> into <code>Subsets</code> as seen <a href="https://github.com/pytorch/pytorch/blob/0444bac940695ccfa43854f9241839f759ba5f28/torch/utils/data/dataset.py#L278-L296">here</a>. The used indices can be access via <code>subset_dataset.indices</code> and could be used to get the image paths.<br>
It depends what kind of <code>Dataset</code> you are using, but e.g. <code>ImageFolder</code> should contains the paths in its <code>dataset.samples</code> attribute.</p>

username: karen-gishyan
cooked: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a>  for the reply.</p>
Hyperparameter Grid Search Pytorch

username: Megh_Bhalerao
cooked: <p>Hi everyone,</p>
<p>I was wondering if there is a simple way of performing grid search for hyper-parameters in pytorch? For example, assuming I have 3 possible values for parameter <code>a</code>, 3 for param<code> b</code> and 4 for param <code>c</code>, I have a total of <code>3 * 3 * 4 = 36</code> different combinations of hyper-parameters. Is there any way to do this efficiently? Or any external library which is integrated with PyTorch to perform this search.</p>
<p><em>My motivation</em>: Hyperparameters can drastically change the performance of a model. This seems like a  very scary prospect since you can never know if your model is actually performing well or it is failing due to some hyper parameter combinations…<br>
Please let me know if anyone has a solution for easy hyperparameter search.<br>
Thank you!</p>

username: mariosasko
cooked: <p>Check out this thread:</p><aside class="quote" data-post="1" data-topic="19943">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhiqiu_lin/48/4239_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-perform-hyper-parameter-search-in-pytorch/19943">What is the best way to perform hyper parameter search in PyTorch?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    ^^What is the best way to perform hyper parameter search in PyTorch? Are there frameworks that can ease this process?
  </blockquote>
</aside>


username: Megh_Bhalerao
cooked: <p>Thank you, I will take a look at it.</p>
How to Perform 2d indexing, gather In Pytorch?

username: textangel
cooked: <p>Use case: I am writing an NLP program which requires a 2d indexing at some point.<br>
The setup is:</p>
<ul>
<li>
<code>k_next_words</code> is shape (batch_size, d1, d2) = (4,5,6)</li>
<li>
<code>ix</code> is shape (batch_size, k_num_to_select, ix_dim=2) = (4,7,2),<br>
where the 4 dim is batch size, the 7 dim is number of samples to look up, and the 2 dim is a <code>2d</code> index corresponding to <code>d1</code> and <code>d2</code> in <code>k_next_words</code> (such that the first entry in the index is between 0 and 4 and the second entry is between 0 and 5).</li>
</ul>
<p>Problem: We want to, over all (5x6) datapoints in the batch, look up the 7 entries corresponding to the entries of <code>k_next_words</code> at the 2d index.</p>
<p>This is probably a use case for <code>gather</code>, but I haven’t gotten it to work so far!</p>
Pytorch with cuda9.2 10.1 10.2, what's the best in most situations?

username: darknight
cooked: <p>If we have the latest Nvidia driver, we can install pytorch with cuda in 3 ways:<br>
conda install pytorch cudatoolkit=9.2 -c pytorch<br>
conda install pytorch cudatoolkit=10.1 -c pytorch<br>
conda install pytorch cudatoolkit=10.2 -c pytorch</p>
<p>All three methods work. So what is the difference among installed pytorchs with 3 types of cuda? Such as running efficiency or features?</p>
<p>Which one should we choose in most situations?</p>

username: smth
cooked: <p>use the latest, i.e. 10.2</p>

username: darknight
cooked: <p>Could you briefly describe the benefits of using cuda10.2?</p>

username: Shisho_Sama
cooked: <p>in a nutshell, you get more optimizations, fixes, etc.<br>
sometimes newer versions interduce some bugs, in which case, you check and if that affects you, you use the former (bug-free) version.<br>
in any case, as a rule of thumb try to use the latest version unless you have a good reason not to (compatibility issues, etc in your environment or pipeline for example)</p>
<p>Also you can always check the changelog and see whats new/different compared to the former version.</p>
Why am I getting mkldnn_stream_destroy could not be located in the dynamic link library caffe_2_detectron_ops.dll?

username: Shisho_Sama
cooked: <p>I tried to use a C++ DLL that calls a python module in my C++ frontend which uses torchlib  and upon trying to import the mentioned module, it produces this error :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91eb8b97a9cbdfd78a7db2872c4deb021e595409.png" alt="image" data-base62-sha1="kORSYj2L4Z1Z8mk1uksGHtIl6YN" width="413" height="173"></p>
<pre><code class="lang-auto">The procedure entry point mkldnn_stream_destroy could not be located in 
the dynamic link library ..\torch\lib\caffe_2_detectron_ops.dll
</code></pre>
<p>The issue doesnt exist, when I call the dll in a standalone project (that does not use torchlib).<br>
what is the problem here?</p>

username: Shisho_Sama
cooked: <p><a class="mention" href="/u/peterjc123">@peterjc123</a> Can you kindly give me a hand in  this?<br>
I’m really confused about the casue of this error and how to solve it</p>

username: Shisho_Sama
cooked: <p>OK. it seems it was caused becasue of a mistmatched versions between the installed torch (1.5.1) and using the latest libtorch (nightly build).<br>
I had previously reverted back to 1.5.1 and forgot that the libtorch was from the latest nightly build. so this was casued.<br>
removing and reinstalling both of them got rid of this issue (</p>
Alternative for keras Tokenizer in pytorch

username: mathematics
cooked: <p>I wanted to have text to sequences so that i can feed to network,<br>
so are there any Pytorch alternative  for doing this</p>
<pre><code class="lang-auto">tokenizer = Tokenizer()
tokenizer.fit_on_texts(x_train)
x_train = tokenizer.texts_to_sequences(x_train)
#[[1,3,32],[2,34, 1].......
</code></pre>
Torch::from_blob gives access violation in debug mode but is fine in release mode!

username: Shisho_Sama
cooked: <p>The following snippet of code which converts an image into a tensor, results in access violation exception when run in debug mode only.</p>
<pre><code class="lang-cpp">auto ToTensor(cv::Mat img, bool show_output = false, bool unsqueeze = false, int unsqueeze_dim = 0)
{
    try
    {
        if (show_output)
            std::cout &lt;&lt; "image shape: " &lt;&lt; img.size() &lt;&lt; " channels: " &lt;&lt; img.channels() &lt;&lt; " depth: " &lt;&lt; img.depth() &lt;&lt; " type: " &lt;&lt; img.type() &lt;&lt; std::endl;
        at::Tensor tensor_image = torch::from_blob(img.data, { img.rows, img.cols, img.channels() }, at::kByte);

        if (unsqueeze)
        {
            tensor_image.unsqueeze_(unsqueeze_dim);
            std::cout &lt;&lt; "tensors new shape: " &lt;&lt; tensor_image.sizes() &lt;&lt; std::endl;
        }

        if (show_output)
        {
            std::cout &lt;&lt; tensor_image.slice(2, 0, 1) &lt;&lt; std::endl;
        }
        
        if (show_output)
            std::cout &lt;&lt; "tenor shape: " &lt;&lt; tensor_image.sizes() &lt;&lt; std::endl;
        
        return tensor_image;
    }
    catch (const c10::Error&amp; e)
    {
        std::cout &lt;&lt; "error occured: " &lt;&lt; e.msg() &lt;&lt; e.what() &lt;&lt; std::endl;
        return torch::tensor({ 0.229, 0.224, 0.225 });
    }
}
And here is  the full exception message :  
</code></pre>
<p>exception message :</p>
<pre><code class="lang-auto">Exception thrown at 0x00007FF70568DB3A in test.exe: 0xC0000005: Access violation writing location 0x0000000000000003.
</code></pre>
<p>Whats wrong here?</p>

username: Shisho_Sama
cooked: <p>OK, it was caused becasue of the release libs were in the PATH and it would read the libs and crash.<br>
It’d be great to have different names for debug and release codes so the headaches regarding these would be minimized.</p>
problems occurs when i try to use loss.backward() with my diceloss function

username: Sunrise_95
cooked: <p>I copy Diceloss function code online. the function need  the input and its correspoding label to be one-hot encoded, so  I need to transform the inputs to one-hot format at first, and feed them into the loss function.<br>
I have experimented a more simple situation to find the problem, and the problem are exactly same as above.<br>
my codes  are below:<br>
x = torch.tensor([[1,2],[3,4]]).float()<br>
w = torch.tensor([1]).float()<br>
w.requires_grad = True<br>
x_ = x*w<br>
x_o_h = get_one_hot(x_, 5)  <span class="hashtag">#转换为one-hot编码</span> size = (5,2,2)<br>
res = sum(x_o_h)<br>
res.backward()<br>
print(w.grad)</p>
<p>function to get one-hot format:<br>
def get_one_hot(label, N): <span class="hashtag">#输入是一个二维的label</span>!!!且label是long类型的数据,例如(3,3)</p>
<pre><code>size = list(label.size())
label = label.view(-1)  # reshape 为向量
ones = torch.eye(N)
ones = (ones.index_select(0, label.long()))# 用上面的办法转为换one hot
size.append(N)  # 把类别输目添到size的尾后，准备reshape回原来的尺寸（3，3，9）
res = ones.view(size).permute((2,0,1))   ##size = (9, 3, 3)
return res
</code></pre>
<p>the error is below:<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>

username: ptrblck
cooked: <p>If I’m not mistaken, the target tensor would have to be one-hot encoded, not the prediction tensor, which seems to break the computation graph.<br>
To do so, you could use <code>F.one_hot</code>. Also, <a href="https://github.com/pytorch/pytorch/issues/1249#issuecomment-305088398">this post</a> provides an implementation in case you get stuck.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Running Pytorch on GPU

username: justinyue1643
cooked: <p>Hello, I know that this is a question probably asked quite often on this forum, but after going through the forum posts, I still am perplexed and could use some clarification. I am running Pytorch 1.6.0 on Ubuntu 20.04 with Cuda Toolkit 11.0 installed (hopefully, that’s all I needed to get everything running smoothly), and I am running some models on my local GPU.</p>
<p>Some models are as small as a few dense layers while I have also done transfer learning with the densenet121 model. I can tell that my GPU is being used via Nvidia X Server and by printing my Tensors to see that they have been loaded onto my GPU. However, I can also see that my CPU usage has increased significantly (to a much more significant degree on the densenet121 model of course) even when the GPU is being used. Is this normal?</p>

username: smth
cooked: <p>it’s normal. the GPU is driven by CPU code, and it’s normal for the CPU to spin while driving the GPU</p>
Not apperaing Tensorboard in browser

username: 887574002
cooked: <p>Hi, I installed tesnorboard, but when I add something using SummaryWriter I can’t see tensorborad on my browser.</p>

username: hal9000
cooked: <p>I also struggled with this before.<br>
“To launch TensorBoard, we need to run the tensorboard command at our terminal”<br>
“tensorboard --logdir=runs”<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://deeplizard.com/images/deeplizard-1.png" class="site-icon" width="16" height="16">
      <a href="https://deeplizard.com/learn/video/pSexXMdruFM" target="_blank" rel="nofollow noopener">deeplizard.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:128/72;"><img src="https://i.ytimg.com/vi/pSexXMdruFM/maxresdefault.jpg" class="thumbnail" width="128" height="72"></div>

<h3><a href="https://deeplizard.com/learn/video/pSexXMdruFM" target="_blank" rel="nofollow noopener">TensorBoard with PyTorch - Visualize Deep Learning Metrics</a></h3>

<p>Welcome to this neural network programming series. In this episode, we will learn how to use TensorBoard to visualize metrics of our PyTorch CNN during training process.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>