akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Differentiable Weight quantization

username: Santanu_Rathod
cooked: <p>Hello all,</p>
<p>I want to ask in this case:<br>
//<br>
vgg16_qu.state_dict().update(quant_dict)<br>
vgg16_qu.load_state_dict(quant_dict)<br>
//<br>
quant_dict is dictionary containing W= f(w). (W are the weights I use instead of w in my network), (weights of my original vgg16 network)=w.<br>
Now I want gradients wrt my original parameters(w), even though the weights I use for forward pass are a differentiable representation(W= f(w)) of my earlier weights.</p>
<p>The above piece of code replaces the weights by W and also calculates the gradients wrt W instead of w, which is what I want.</p>
<p>How do I do this?<br>
Can anyone help?</p>
<p>Thanks in advance.</p>
Generating indices for repeating values in a tensor

username: melonb
cooked: <p>Hi,<br>
I have a very simple question.<br>
If i have an tensor array which is like [1,1,1,1,2,2,3,3,3,2,2]<br>
is there any function or method to count the repetition and generate another tensor which is: [0,1,2,3,0,1,0,1,2,0,1]</p>

username: ptrblck
cooked: <p>The simple way would be to use a window of two values in a loop:</p>
<pre><code class="lang-python">x = [1,1,1,1,2,2,3,3,3,2,2]
res = [0]
for idx, (a, b) in enumerate(zip(x[:-1], x[1:])):
    if a == b:
        tmp = res[idx] + 1
        res.append(tmp)
    else:
        res.append(0)
</code></pre>
<p>I’m not sure, if any PyTorch/numpy method might speed it up, as e.g. <code>unique</code> seems to be fruitless here, since you are repeating values.</p>
Aclass(torch.nn.Module) pass the derivations to the inputs

username: cherepanovic
cooked: <p>My purpose is to create a class which inherits all methods from torch.nn.Module.</p>
<p>Inside the fit function it would be a specific calculation, I wonder how to deal with derivations? I have a analytical derivation for this function but I don’t know how to extend the class.</p>
<p>Then the parameters of that function are always the same, that means that the training would not effect affect them, that class need to pass the gradient to the inputs during training.</p>
<p>Need your advices guys (examples were great)</p>
<p>Thank you!</p>
dataset.ImageFolder outputs different length of images

username: Hassan_Saeed
cooked: <p>When I imported the data in dataset.ImageFolder, I tried to print the length of the data, the length is supposed to be 54000 image but when I run print(len(data)) it outputs different number and when I run it again it output another totally different number, does anyone know why it doesn’t shows the exact length of the dataset? does it have limit for numbers of images??</p>

username: ptrblck
cooked: <p>Could you post the code you are using to create your <code>Dataset</code>?<br>
Also some information about the folder structure would be interesting to see.</p>

username: Hassan_Saeed
cooked: <blockquote>
<p>import torchvision.transforms as transforms<br>
from torchvision import datasets</p>
</blockquote>
<blockquote>
<p>transforms = transforms.Compose([transforms.RandomRotation(30),<br>
transforms.Resize(30),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])</p>
</blockquote>
<blockquote>
<p>Arabic_train_data = datasets.ImageFolder(Arabic_train_dataPath, transform = transforms)<br>
Arabic_test_data = datasets.ImageFolder(Arabic_test_dataPath, transform = transforms)</p>
</blockquote>
<p>here is the code I’ve used, the data path has 10 folders of arabic handwritten numbers each folder contain 6000 bmp image file</p>

username: Hassan_Saeed
cooked: <p>every time I run</p>
<blockquote>
<p>print(len(Arabic_train_data))</p>
</blockquote>
<p>it outputs wrong length of the dataset and it gets increasing everytime I run it</p>
<p>although the test data length is 10000 as it should be</p>

username: ptrblck
cooked: <p>So if you run these command sequentially, you’ll get different results?</p>
<pre><code class="lang-python">Arabic_train_data = datasets.ImageFolder(Arabic_train_dataPath, transform = transforms)
print(len(Arabic_train_data))
print(len(Arabic_train_data))
print(len(Arabic_train_data))
print(len(Arabic_train_data))
</code></pre>

username: Hassan_Saeed
cooked: <p>at the same time no, but if there were a delay maybe about 2 mins the value increases, as if it keeps loading the data in the ImageFolder by the time</p>

username: Hassan_Saeed
cooked: <p>and another weird thing, the length now exceeded the number of images in the dataset, it reached 56k and it supposed to be 54k</p>

username: ptrblck
cooked: <p>Are you working on a shared drive or are you moving data around?<br>
This seems really weird, as the samples and targets are created in the <code>__init__</code> call.<br>
So after initializing the <code>ImageFolder</code>, the length should be constant even if you add new files to the folders.<br>
Or are you reinitializing the dataset also?</p>

username: Hassan_Saeed
cooked: <p>I am working on  google colab and connecting it to my google drive account</p>

username: ptrblck
cooked: <p>Still the lendth shouldn’t be changed once the dataset is initialized, so are you reinitializing it somewhere in your code?</p>
Pytorch cuda training speed comparison between data on Nvme vs Sata ssd

username: Rafi_zunaed
cooked: <p>I know this is a hardware related question but i did not find any article regarding their speed improvements in data science field, only gaming or os benchmark. I want to upgrade my storage system but can not decide between nvme or sata ssd.<br>
My gpu is gtx 1070, cpu i7 7700k.might upgrade later. I am using cuda 10 version. I want to know your experiences about training on gpu with data on nvme vs sata. My datasets often includes huge number of small size individual files, often i convert them into numpy files which takes total size of more than 100gb(combined).<br>
So guys, should i go with nvme or sata ssd? In the same price range i can get more storage with sata ssds.</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this great post</a> by <a class="mention" href="/u/rwightman">@rwightman</a>.</p>
<p>Based on his experiments he recommends buying NVME drives:</p>
<blockquote>
<ul>
<li>Quick fix: buy an NVME SSD drive, or two.</li>
<li>SATA SSD is not necessarily going to cut it, you can saturate them with small to medium image files + default loader/dataset setups feeding multiple GPUs.</li>
</ul>
</blockquote>
One of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 1]], which is output 0 of torch::autograd

username: 4104db8a3e583e5e705e
cooked: <p>lr = 0.1<br>
num_epochs = 3<br>
net = linreg<br>
loss = squared_loss<br>
num_inputs = 2<br>
num_examples = 1000<br>
features = torch.randn(size = (num_examples,num_inputs))</p>
<p>w = Variable(tdist.Normal(torch.tensor([0.0]),torch.tensor([0.01])).sample((num_inputs,)),requires_grad=True)<br>
b = Variable(torch.zeros(size = (1,1)),requires_grad=True)<br>
l =torch.empty(10,1, requires_grad=False)</p>
<p>for epoch in range(num_epochs):<br>
i = 0<br>
for X, y in data_iter(batch_size,features,labels):<br>
i += 1<br>
l[:] = loss(net(X,w,b),y)</p>
<pre><code>if i ==1 :
  l.backward(torch.ones(l.size()))
sgd([w],lr,batch_size)
</code></pre>
<p>train_loss = loss(net(features,w,b),labels)<br>
print(‘epoch %d, loss %f’ % (epoch + 1, train_loss.mean()))</p>
<p>i want solve problem what should do … please save me<br>
i think Variable w has something wrong</p>
Element-wise multiply Nd matrix with 1D gate along specific dimension

username: AIJoris
cooked: <p>Hi,</p>
<p>I’m trying to elementwise-multiply a 4D matrix <em>M</em> of size <em>[100,8,1,400]</em> with a tensor <em>gates</em> of size <em>[8]</em> along a specific dimension.<br>
I can’t get it to work by reshaping, bmm, or elementwise products. What I want is this, but then vectorized:</p>
<pre><code class="lang-auto">for i,gate in enumerate(gates):
    M[:,i,:,:] *= gate
</code></pre>
<p>It doesn’t seem like this should be hard, but I can’t seem to figure it out. Any help is appreciated!</p>

username: AIJoris
cooked: <p>I got it to work. It’s not pretty, but gets the job done without a for loop:</p>
<pre><code class="lang-auto">gate = gate.view(8,1,1,1).expand_as(M.transpose(1,0))
M = (gate*M.transpose(1,0)).transpose(1,0)
</code></pre>

username: ngimel
cooked: <p><code>M*gate.unsqueeze(-1).unsqueeze(-1)</code></p>
Contributing to torchvision

username: jmaronas
cooked: <p>Hello.</p>
<p>I have finished creating three datasets that are not available in torchvision, and that I think it might be interesting to have them available. I have take a look on how to contribute and it says I should first post about it in order to discuss the implementation details.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Could you post your current working branch?<br>
CC <a class="mention" href="/u/fmassa">@fmassa</a></p>

username: jmaronas
cooked: <p>Yes</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/jmaronas/pytorch_datasets" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/15247250?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/jmaronas/pytorch_datasets" target="_blank" rel="nofollow noopener">jmaronas/pytorch_datasets</a></h3>

<p>Some unavailable pytorch datasets. Contribute to jmaronas/pytorch_datasets development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>For the moment I have only uploaded tinyImagenet (still preparing the others to be as closed as possible to the other datasets in torchvision). Files are selfcontent and prepared at least for torchvision 0.2.0. What I have done is just copy the new functions from 0.3.0 that do not appear in 0.2.2.</p>

username: jmaronas
cooked: <p>I have finished uploading the three  datasets, though I will add some comments we can discuss about them. Btw, there are design choices to be done. For the moment two of the three datasets has been split into two datasets (one for classification and one for object detection). This is because images have different shapes and for classification a preprocess is done to avoid heavy computations during training. We can discuss about this, I have just made it as an initial approximation</p>
Error : No module named lr_scheduler

username: Srikanth_Lukka
cooked: <p>I got the following error during execution of code<br>
import torch.optim.lr_scheduler<br>
ImportError: No module named lr_scheduler<br>
how to resolve it</p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>I have pytorch version 1.1.0. It works perfectly fine in my side. Please check your pytorch version if it is a previous version of 1.0. If it is so, please update it. It would work.</p>
<p>Thanks</p>

username: Srikanth_Lukka
cooked: <p>thank you   for sharing information .I will try</p>
torchvision.transforms.ToTensor() can't work

username: waa8128
cooked: <p>import sys<br>
print(sys.version)<br>
import torchvision<br>
print(torchvision.<strong>version</strong>)<br>
import numpy as np</p>
<p>a = np.array([[[255,255,255]]])<br>
b = torchvision.transforms.ToTensor()(a)<br>
b</p>
<p>output:<br>
3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]<br>
0.3.0</p>
<p>tensor([[[255]],</p>
<pre><code>    [[255]],

    [[255]]], dtype=torch.int32)</code></pre>

username: ptrblck
cooked: <p>Try to pass an <code>np.uint8</code> array, if you would like to get a normalized output tensor.</p>

username: waa8128
cooked: <p>It works, thank you.</p>
Questions about retain_graph = True

username: njuww
cooked: <p>suppose i have a dict named ‘admm’,contains admm[name] = [U,Z]<br>
try to update weigths by:</p>
<pre><code>for name, param in net.named_parameters():
    if param.requires_grad and 'weigth' in name:
            wei = param.clone()
            U,Z = admm_dict[name]
            admm_loss += rho / 2 * torch.norm(wei - Z + U, 2)
return admm_loss
loss = admm_loss
loss.backward()
</code></pre>
<p>it said i should use loss.backward(retain_graph = True),but then it will become"CUDA OUT OF MEMORY"，how can i do loss.backward()  without  retain_graph = True?</p>
Help with save and load code design

username: kaszperro
cooked: <p>Hi,</p>
<p>I’m building big machine learning project. In this project I have many <strong>different</strong> models, with different inputs, outputs, and serving different purposes. I like to keep code transparent, simple and flexible, so I use various design patterns (loads of Strategy design patterns, Dependency Injections and so one).</p>
<p>But now comes my question: Do you guys have idea how to design save, load system? For example I have one model:</p>
<pre><code class="lang-auto">class AutoEncoder(nn.Module):
    def __init__(self, input_size, encoded_size, ac_function='relu'):
        super().__init__()

        self.ac_function = ac_function
        self.encoded_size = encoded_size
        self.input_size = input_size
...
</code></pre>
<p>And I want to save it and keep information about input_size, encoded_size and so one…</p>
<p>Also each model uses some sort of Normalizations, Standardizations to transform dataset data, like:</p>
<pre><code class="lang-auto">class Standardization(StatisticTransformation):
    def __init__(self, mean, std):
        self.mean = mean
        self.std = std

    def _apply_key_val(self, key, val, indexes=None):
</code></pre>
<p>So with each model I’d like to keep information about type of Transformation and for example means and stds.</p>
<p>I’d love to discuss with someone about it.<br>
Thanks</p>
Model.train() vs model.eval() on pretrained model

username: Ashwin_Raju
cooked: <p>while using a pretrained model given by the author I evaluated with my code and I am able to get the same accuracy as they prescribe, but now when I train the model and then calculate the accuracy by setting it to <code>model.train()</code> and using the same evaluation dataset, I get less accuracy but not too worse . The only difference i made is setting to <code>model.train()</code> and commenting <code>with torch.no_grad()</code>.<br>
Note: I did not do <code>optimizer.step()</code> in train method. I just ran the same code as in evaluation script in my train script and just changed those two lines.</p>

username: TheShadow29
cooked: <p>Even if you don’t do <code>optimizer.step</code>, if the pretrained model has a batch norm layer, those values will get updated when you are evaluating. If the pre-training was done on a large data, best not to change. Also if the model has a dropout layer, the predictions will be a bit random. Hence it is better to evaluate only after <code>model.eval()</code></p>

username: Ashwin_Raju
cooked: <p>Sounds reasonable. Yes the pretrained model has batch norm layer . This raises me a new question . If I want to train a new dataset with these pretrained model weights then do i need to freeze the batch norm layers after loading the pretrained model weights into my model ?</p>

username: TheShadow29
cooked: <p>Depends on the new dataset. If the new dataset has quite a lot of data, then unfreezing batch norm is helpful, else not. In practice, I haven’t noticed too large of a difference though.</p>

username: Blankit
cooked: <p>What’s the difference between model.train(False) and model.eval()?</p>

username: ptrblck
cooked: <p>There is no difference and <code>.eval()</code> will internally call <code>.train(False)</code> as shown in <a href="https://github.com/pytorch/pytorch/blob/042a2fd8100b78e7937b0355491423e19251791d/torch/nn/modules/module.py#L1063" rel="nofollow noopener">this line of code</a>.</p>
Optimizer in PyTorch

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>I have created the following optimizer.</p>
<p><code>optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum)</code></p>
<p>Based on a condition, I am updating the weights for the new inputs. Whenever I update the weight, I am running through the following code body.</p>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9) # doubt here
optimizer.zero_grad()
inputs_new, _ = data_list[max_index]
outputs_new = model(inputs_new)
loss = criterion(outputs_new, torch.tensor([y]))
loss.backward()
optimizer.step()
</code></pre>
<p>Is it correct to instantiate optimizer again and again (the line with the comment)? Is it a wrong practice? Is it logically wrong?</p>
<p>Thanks in advance.</p>

username: sai_tharun
cooked: <p>i dont see anything wrong here,it should work fine.<br>
R u facing any problems mate?</p>

username: Oli
cooked: <p>I guess the momentum wouldn’t work very well. Never seen anyone doing that before. Most people don’t reinitialise it</p>

username: Pranavan_Theivendira
cooked: <p>Hi ,<br>
The accuracy has gone down by nearly 20 % in the first case. (With one initializer)</p>
<p>Thanks</p>

username: Pranavan_Theivendira
cooked: <p>Thanks for the input. Actually, the momentum works well in that case.</p>

username: sai_tharun
cooked: <p>accuracy depends on other parameters also, i dont think re-instantiate optimizer again and again is a good practice but you then there is nothing in doing that.</p>
<p>can you post the full code and tell what exactly you r doing?</p>

username: shubhvachher
cooked: <p>This should help. I had a similar question recently…</p>
<aside class="quote" data-post="1" data-topic="48997">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shubhvachher/48/29925_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/clarification-on-re-initializing-optimizer-in-every-epoch/48997/">Clarification on re-initializing optimizer in every epoch</a> 
  </div>
  <blockquote>
    You shouldn’t be doing that, right? fmi as well as <a href="https://github.com/psyec1/Lipreading-PyTorch/issues/7" rel="noopener nofollow ugc">my issue here</a>
  </blockquote>
</aside>
Why does nn.GRU accept inputs of shape (1, x, input_size)

username: shapeare
cooked: <p>Why is nn.GRU designed to only accept inputs of shape (1, x, input_size)? What does the second dimension mean?</p>
<p>If an input variable of dimension (1, input_size) together with a hidden state of dimension (1, hidden_size) are fed into nn.GRU, the following error will occur:</p>
<blockquote>
<p>RuntimeError: matrices expected, got 1D, 2D tensors at /py/conda-bld/pytorch_1493680494901/work/torch/lib/TH/generic/THTensorMath.c:1232</p>
</blockquote>
<p>Only after adding another dimension in the middle will the code runs.</p>

username: jmrf
cooked: <p>I’m very new to PyTorch but from the documentation I see that when initializing nn.GRU takes (input_size, hidden_size, num_layers).</p>
<p>And regarding the inputs the dimensions seem to correspond to: (batch_size, seq_len, input_size)</p>
<p>Adapting a bit from the examples:</p>
<pre><code>batch_size = 5
input_size = 10
num_layers = 2
hidden_size = 20
seq_len = 3
rnn = nn.GRU(input_size, hidden_size, num_layers)
inp = Variable(torch.randn(batch_size, seq_len, input_size))
h0 = Variable(torch.randn(num_layers, seq_len, hidden_size))
output, hn = rnn(inp, h0)
</code></pre>
<p>So the second dimension is the sequence length. This is the behaviour you can see in almost all recurrent modules / layers of every DL library.</p>

username: Anuj_Chopra
cooked: <p>If it takes a tensor of shape (batch_size, seq_len, input_size), then it is fine. But when I open its code, it seems to accept the tensor of shape (seq_len, batch_size, input_size) which is a big problem. Can anyone confirm if I am right?</p>

username: tom
cooked: <p>The default is <code>(seq_len, batch_size, input_size)</code> by default, but you can specify <code>batch_first=True</code> and use <code>(batch_size, seq_len, input_size)</code>, so it is not a big problem unless you forget the parameter.<br>
The reason for the default is that the RNN will iterate over the <code>seq</code> dimension, so it is efficient to access individual timestep, batched data, which is contiguous if you pass in contiguous tensors of form <code>(seq_len, batch_size, input_size)</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
Loading data using torchtext

username: Cat12
cooked: <p>Does torchtext support loading large files of 2GB?<br>
I am unable to load 2.7GB json file using torch text.</p>

username: ptrblck
cooked: <p>Do you get a specific error message or does your code just hang?<br>
Could you post a code snippet reproducing this error?</p>

username: Cat12
cooked: <p>neither the code hangs nor i get any error message. i am trying in notebook and the cell is just executing from a long time (hours)<br>
Code :<br>
train_data , test_data = data.Tabulardataset.splits(path = path, train = “train.json” , test = “test.json” , format = “json”, fields = fields")</p>

username: ptrblck
cooked: <p>Is the <code>train.json</code> somewhere available?<br>
If not, could you just post a few sample rows so that I could create a dummy and try it on my machine?</p>

username: Cat12
cooked: <p>i tired using a small dataset in the same format. i tried it using just two samples and it loaded in 1 minute. I understand i have a lot of samples in my original file but still its loading from 3-4 hours.</p>
<p>the sample is like a normal json :<br>
{key 1: value, key2:value}<br>
{}<br>
{}…</p>

username: Yingqiang_Gao
cooked: <p>I have the same problem here. I was trying to load a file that is 1.4 GB, minutes after the process got killed</p>
<pre><code class="lang-auto">&gt;&gt;&gt; REF = data.Field(lower=True, tokenize=tokenize_char, init_token='&lt;sos&gt;',eos_token='&lt;eos&gt;')
&gt;&gt;&gt; SRC = data.Field(lower=True, tokenize=tokenize_char)
&gt;&gt;&gt; train = data.TabularDataset('./train.csv', format='csv', fields=[('src', SRC), ('ref', REF)])
Killed
</code></pre>
<p>and this works fine for a smaller dataset.</p>
<p>Anyone can tell me why this is happening? Thanks!</p>
How to convert labels into one-hot encoded tensors

username: AmardeepGanguly
cooked: <p>I wrote the following code to get the name of the label of every image from a csv file which has the corresponding label for every image In this case the number of labels are three how can i modify my CustomDataset to return one-hot-encoded tensors instead of text  labels<br>
This my code ,i have three labels namely “YOUNG”,“MIDDLE”,“OLD”.</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
  def __init__(self, labels_df, img_path, transform=None):
    self.labels_df=labels_df
    self.img_path=img_path
    self.transform=transform
    
  def __len__(self):
    return  len(self.labels_df)
  
  def __getitem__(self,idx):
    image_name=os.path.join(self.img_path, self.labels_df.ID[idx])
    #img=io.imread(image_name)
    img=Image.open(image_name).convert('RGB')
    label=self.labels_df.Class[idx]
    
    
    
    if self.transform:
      img=self.transform(img)
    return [img, label]  
</code></pre>
<p>This is how my csv file looks like<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3a4b923db6bbae5c3cb8cdde0c09a08b52382de2.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a4b923db6bbae5c3cb8cdde0c09a08b52382de2" title="csv"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3a4b923db6bbae5c3cb8cdde0c09a08b52382de2.png" alt="csv" data-base62-sha1="8jHBrAwLTHzyS9xUQCRfD8aIbKi" width="113" height="500" data-dominant-color="DDDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">csv</span><span class="informations">119×524 4.25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>This topic has been discussed previously. Have a look at this <a href="https://discuss.pytorch.org/t/convert-int-into-one-hot-format/507/3">thread</a>.</p>
How to synchronize the image size when using DistributedDataParallel?

username: CgYoung
cooked: <p>Since I want to adopt multi-scale training for object detection, the image sizes would be changed per fixed frequency. When using dataparallel, the image sizes trained on each GPU can be synchronized.</p>
<p>To speed up the training phase, I want to use DistributedDataParallel, but I don’t know how to synchronize the input image size. Any suggestions, please?</p>

username: Oli
cooked: <p>Hi!</p>
<p>One way to do it is to resize your batches in a custom collate function that you send to your dataloader. This collate function would make sure that all the images within one batch have the same dimension.</p>
<p>I would try to get this working before you start with the DistributedDataParallel. Actually, depending on your multi-scale schedule (fixed-frequency), the DistributedDataParallel might not offer any difficulties once you got the collate function up and running.</p>
<p>Google + search at these forums on how to implement a custom collate function for the dataloader and give me a poke if you want to talk something over.</p>
<p>Good luck <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Edit: Since it’s object detection you also need to transform your bounding boxes. I recommend using <a href="https://imgaug.readthedocs.io/en/latest/source/examples_bounding_boxes.html" rel="nofollow noopener">imgaug</a> for this, so much easier</p>
Update cudnn failed in PyTorch

username: huangbiubiu
cooked: <p>I want to update the cudnn version from 7.5 to 7.6. However, after I update the cudnn, the pytorch still use the older version cudnn.</p>
<p>Here’s my updating steps:</p>
<ol>
<li>Download the <code>libcudnn7_7.6.1.34-1+cuda10.0_amd64.deb</code> from NVIDIA, and install it with <code>sudo dpkg -i</code>.</li>
<li>I run <code>cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2</code>, I get:</li>
</ol>
<pre><code class="lang-auto">#define CUDNN_MAJOR 7
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 1
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"
</code></pre>
<p>It looks like I have cudnn 7.6.1.</p>
<p>HOWEVER, when I print the cudnn version in pytorch with <code>torch.backends.cudnn.version()</code>, I got <code>7402</code>. What’s going on?</p>

username: Deepali
cooked: <p>PyTorch is statically linked with <code>cuDNN</code>, so updating ‘cuDNN’ does not help here.</p>
<p>Building PyTorch from source is however, an option.</p>
Indexing a higher dimension tensor with a lower dimension tensor, i.e. (B,S,D) and (B,S) shape tensors

username: neonrights
cooked: <p>Suppose I have a tensor <strong>A</strong> of shape (B, S, D) and an indexing tensor <strong>B</strong> of shape (B, S).  I want to use the indexing tensor to select D dimensional vectors from tensor <strong>A</strong> resulting in an output tensor of shape (B, S, D).</p>
<p>For example suppose</p>
<pre><code class="lang-auto">A = [[[ 1,  2,  3],
      [ 4,  5,  6]],
     [[ 7,  8,  9],
      [10, 11, 12]]]
</code></pre>
<p>a (2,2,3) matrix and</p>
<pre><code class="lang-auto">B = [[0, 0],
     [1, 0]]
</code></pre>
<p>Then the result would be</p>
<pre><code class="lang-auto">[[[ 1,  2,  3],
  [ 1,  2,  3]],
 [[10, 11, 12],
  [ 7,  8,  9]]]
</code></pre>
<p>This can be accomplished using a for loop like so:</p>
<pre><code class="lang-python">def function(A, indices):
    C = torch.zeros_like(A)
    for i in range(A.size(0)):
        C[i] = A[i,indices[i]]
    return C
</code></pre>
<p>Is there a way to do this faster and without a for loop?</p>

username: ngimel
cooked: <aside class="quote no-group" data-username="neonrights" data-post="1" data-topic="49171">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neonrights/48/9053_2.png" class="avatar"> neonrights:</div>
<blockquote>
<p>def function(A, indices): C = torch.zeros_like(A) for i in range(A.size(0)): C[i] = A[i,indices[i]] return C</p>
</blockquote>
</aside>
<p><code>A[torch.arange(B.size(0)).unsqueeze(1),B]</code></p>
Using Dataset to iterate over lines of large files

username: byd
cooked: <p>Hi,</p>
<p>I have a folder of 500 text files, each contains about 200 million lines, which is not wise to load them into memory all at once.</p>
<p>I want to batch 64 lines every iteration just by their orders in the files, but haven’t found any efficient way to do so in pytorch.</p>
<p>It seems to me that in order to support random sampling, <code>data.Dataset</code> demands implementing <code>__getitem__</code> and <code>__len__</code> , these two methods is unnecessary in my case and holds me back from creating my customized Dataset.</p>
<p>I come up with one solution, which is splitting these files in small ones, create a Dataset for each of them and use <code>ConcatDataset</code> to iterate, but is doesn’t seem graceful enough.</p>
<p>Any idea on how to do that just like in Tensorflow?</p>

username: bhushans23
cooked: <p>I would recommend dataset as follows</p>
<blockquote>
<ol>
<li>
<p>in <strong>init</strong> process and compute full data length. This is required for iterator<br>
Don’t confuse this with actual data size.<br>
I will go ahead and store in chunk more than 64 to minimize disk/file access. e.g. read 4096 lines everytime and store in buffer. Actual datasize will be overall length and we maintain a pointer modulo to 4096 which will iterate till last chunk.</p>
</li>
<li>
<p><strong>getitem</strong>: If still in current chunk, directly return what you already have and increment local pointer<br>
If local index equals to 64 (64 8 64 = 4096) then you are at the end of buffer. Now load next 4096 lines.</p>
</li>
</ol>
</blockquote>
<p>Now, control 4096 (number of lines to cache) depending on how much RAM you want to spend in buffer (it’s a tradeoff)  You will have to maintain a list of file name to iterate over</p>

username: thiagocrepaldi
cooked: <p>Refer to <a href="https://discuss.pytorch.org/t/how-to-use-dataset-larger-than-memory/37785/9" class="inline-onebox">How to use dataset larger than memory?</a>. The ChunkDataset API might help you</p>
What is the proper way to use optim.ASGD?

username: remilepriol
cooked: <p>When using the optimizer ASGD, the optimizer stores the average parameter values in <code>optimizer.state_dict()['state'][...]['ax']</code>. Is there a natural way to perform evaluation with this model ? At the moment I am using</p>
<pre><code class="lang-auto">tmp = {}
for p in model.parameters():
    tmp[p] = p.data
    p.data = optimizer.state[p]['ax']

evaluate(model)

for p in model.parameters():
    p.data = tmp[p]
</code></pre>
<p>But I feel weird cloning my model at every evaluation and I am wondering wether this could be done with some pretty context manager.</p>
One of the differentiated Tensors appears to not have been used in the graph

username: maralm
cooked: <p>I am running a one layer linear network. I want to use autograd.grad to calculate the gradients of output with respect to input. However I get the “<strong>One of the differentiated Tensors appears to not have been used in the graph</strong>” error when running as follows:</p>
<blockquote></blockquote>
<p>input = input_list[i] # input coming from cpu<br>
input_cuda = Variable(input.cuda(“cuda:1”), requires_grad=True)<br>
output = model(input_cuda)<br>
grad = torch.autograd.grad(output, input_cuda, torch.ones_like(output), create_graph=True)</p>
<blockquote></blockquote>
<p>I’m wondering what is wrong with this code.<br>
Thanks.</p>
PyTorch Startup Memory Occupation

username: maralm
cooked: <p>Hi,</p>
<p>I have a simple linear network class in pytorch with 6000 hidden units.<br>
When I call the following lines, it occupies 893MB memory.<br>
device = torch.device(“cuda”)<br>
model = DeepNN().to(device)</p>
<p>where DeepNN is the class name. This initial memory will increase by increasing the number of hidden units.<br>
Wondering what this means and is there any way to fix it?</p>
<p>Update: To give some more information, with a network with nn.Linear(6000,6000) , I expect the initial memory to be 144MB but it is currently 600MB. When I increase the size of hidden units to nn.Linear(30000,30000), I expect the memory to be around 3.5GB but it preoccupies 5GB.</p>

username: JuanFMontesinos
cooked: <p>Basically the bigger your net is the more memory it requires.</p>

username: maralm
cooked: <p>I understand. My problem is that why it preoccupies the memory before running the actual computations? Is that normal?<br>
Even with a one layer network, it constantly occupies this amount of memory.</p>

username: JuanFMontesinos
cooked: <p>Cuda requires some memory to initialize libraries. It depends on gpu model. Around 600 Mb</p>

username: maralm
cooked: <p>Thanks for the reply.<br>
What I see is that the memory preoccupied keeps increasing by increasing the number of hidden units. With 20000 hidden units, it occupies around 3GB memory.<br>
Is there any way to avoid it? I am basically loosing memory required for computation.</p>

username: ptrblck
cooked: <p>The parameters have to be stored on the GPU so that the computation itself can be performed on the device.<br>
As <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> said, the CUDA context will use some memory besides that.</p>

username: maralm
cooked: <p>If it is cuda context memory, why it is related to the number of hidden units? If it is for parameters, the numbers I am getting is not what calculations say. I edited my question with some examples.</p>

username: ptrblck
cooked: <p>Thanks for the additional information!<br>
Could you run the following code and check the memory usage:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

torch.cuda.synchronize()  # Create CUDA context (947MB in my case)

model = nn.Linear(6000, 6000).to('cuda')
expected_mem = (model.in_features * model.out_features + model.out_features) * 4 / 1024**2
print('Max allocated {:.3f}MB, expected {:.3f}MB'.format(
    torch.cuda.max_memory_allocated() / 1024**2,
    expected_mem))

model = nn.Linear(30000, 30000).to('cuda')
expected_mem = (model.in_features * model.out_features + model.out_features) * 4 / 1024**2
print('Max allocated {:.3f}MB, expected {:.3f}MB'.format(
    torch.cuda.max_memory_allocated() / 1024**2,
    expected_mem))

&gt; Max allocated 138.023MB, expected 137.352MB
&gt; Max allocated 3572.138MB, expected 3433.342MB
</code></pre>
<p>On my system, the CUDA context uses a constant memory of ~950MB, while the model parameters fill up additional memory.</p>

username: maralm
cooked: <p>Thanks. Problem solved. There was a bug in model definition.</p>
Should I reset the hidden and cell states of my LSTMs each time I use the generator and discriminator in my GAN?

username: andelane
cooked: <p>Hi,</p>
<p>I’m trying to build a GAN with both the generator and the discriminator composed of LSTM networks. I’ve created a function to initialise the hidden and cell states of both networks. However, I am confused as to when and how often these hidden and cell states need to be initialised during the training iterations.</p>
<p>For example, I first train my discriminator with fake data. To do this, I use the generator to generate fake data and the discriminator to classify it as real or fake. To use both of these, I initialise the hidden and cell states of both the disxriminator and the generator.</p>
<p>Next, I need to train the discriminator with real data. Here, I use real data and feed it to the discriminator for classification. Do I need to reset the hidden state of the discriminator now as they have been trained for the fake data in the first step?</p>
<p>Finally, I train my generator. Once again, I generate noise data and feed it into the generator. As the generator has already been used, do the hidden and cell states need to be reset at this point? Similarly for the discriminator internal states as it has been used twice now?</p>
<p>Any help on this would be greatly appreciated!</p>
<p>Thanks,</p>
<p>Anne Marie</p>
Masked loss on a batch of images

username: ResByte
cooked: <p>I would like to implement a masked MSE loss function for training a model.  I am trying to get hadamard product between mask image and original image. Is there any torch function for this?</p>

username: ResByte
cooked: <p>Got it.</p>
<pre><code class="lang-auto">class MaskedMSE(nn.Module):
    def __init__(self):
        super(MaskedMSE, self).__init__()
        self.criterion = nn.MSELoss()

    def forward(self, input, target, mask_a, mask_b):
        self.loss = self.criterion(input*mask_a, target*mask_b)
        return self.loss
</code></pre>

username: Daniel_Ortega
cooked: <p>This work for you in batches?</p>
Training of FCRN for depth prediction on my own dataset

username: abdur_rahman
cooked: <p>Hello everyone, I am using pytorch implementation of Deeper Depth Prediction by laina.<br>
Actually I have prepared my own data set of indoor scene in my environment and want to train model on that. I am freezing all other layers except for the up projection blocks initializing with pretrained weights and the result is not so good. There is fixed pattern of squares on depth predicted. Even I trained it on as small data set as 600 images and achieved 82 percent accuracy but the results were not good visually.<br>
I do not know the reason of that maybe you can suggest me something.<br>
Additionally the accuracy is not improving beyond 82 percent, Can you give me any suggestions how to improve. I have to use the depth for SLAM which requires greater accuracy. I want it to perform good on my data set of around 25000 images.<br>
And the images I want to train are approximately 6k.<br>
The pretrained weights with NYU are even performing better.<br>
batch_size = 32<br>
learning_rate = 1.0e-3<br>
monentum = 0.9<br>
weight_decay = 0.0005<br>
num_epochs = 70<br>
BerHu Loss<br>
optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=learning_rate,<br>
momentum=monentum, weight_decay=weight_decay)<br>
and lr is halved after 10 epochs.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/46248215/60284606-5fb44480-9925-11e9-9032-a6e4f90f9062.png" title="60284606-5fb44480-9925-11e9-9032-a6e4f90f9062.png" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/46248215/60284606-5fb44480-9925-11e9-9032-a6e4f90f9062.png" alt="MS_LAB_269_unfilled" width="690" height="388"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">60284606-5fb44480-9925-11e9-9032-a6e4f90f9062.png</span><span class="informations">1920×1080</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
<p>Validation depth image<br>
<img src="https://user-images.githubusercontent.com/46248215/60284080-19121a80-9924-11e9-9c73-2689e3ed9831.png" alt="Screenshot from 2019-06-27 21-37-28" width="500" height="500"></p>
<p>rgb image<br>
<img src="https://user-images.githubusercontent.com/46248215/60284489-1e239980-9925-11e9-9481-e6e043afafb7.png" alt="Screenshot from 2019-06-27 21-44-59" width="500" height="443"></p>
Casting to FloatTensor moves the tensor to cpu?

username: Even_Oldridge
cooked: <p>I’m curious why the .cuda is necessary in this line of code when targs is on the gpu?</p>
<blockquote>
<p>ts = targs[::self.ns].type(torch.FloatTensor).cuda()-1.</p>
</blockquote>
<p>without the .cuda I get an error about a cpu backend.  Does casting really move a tensor, and what’s the reasoning behind that?</p>
<p>Also I’m hoping behind the scenes this is being kept on gpu?</p>

username: Even_Oldridge
cooked: <p>I’m guessing I should be specifying it as a torch.cuda.FloatTensor?</p>

username: ptrblck
cooked: <p><code>.type(torch.FloatTensor)</code> should move the tensor to the CPU.<br>
I assume you would like to change the <code>dtype</code> of the tensor without changing the device?<br>
If so, you could use<br>
<code>targs[::self.ns].float()</code><br>
or<br>
<code>targs[::self.ns].to(torch.float)</code>.</p>

username: Even_Oldridge
cooked: <p>That’s perfect.  I was looking for a solution that would let me run on both cpu (for debugging) and gpu without having to change it.  Thanks!</p>
Same model uses much more memory in C++ than Python

username: TimothyDelton
cooked: <p>Consider the following purely illustrative C++ code:</p>
<pre><code class="lang-auto">/*Create PyTorch Model and Send to GPU*/
Foo model();
model.to(torch::kCUDA);

/*Make Optimizer*/
torch::optim::Adam adam_opt(model.parameters(), torch::optim::AdamOptions(1e-3));

/*Use Training Mode*/
model.train();

/*Random Training/Label Batch*/
torch::Tensor input_batch = torch::rand({ BATCH_SIZE, 1, HEIGHT, WIDTH}).to(torch::kCUDA);
torch::Tensor label_batch= torch::rand({ BATCH_SIZE}).to(torch::kCUDA);

/*Train for 100 Iterations*/
for (int i = 0; i &lt; 100; i++) {
	/*Zero the gradient buffers*/
	adam_opt.zero_grad();

	/*Forward Through Model*/
	torch::Tensor output = model.forward(input_batch );

	/*Calculate Loss*/
	torch::Tensor loss = torch::binary_cross_entropy(output, label_batch);

	/*Backprop then update using optimizer*/
	loss.backward();
	adam_opt.step();
}
</code></pre>
<p>The subsequent code is the Python equivalent to the above C++:</p>
<pre><code class="lang-auto">#Create PyTorch Model and Send to GPU
model = Foo()
model = model.to(torch.device("cuda:0"))

#Loss Function
loss_fn = torch.nn.BCELoss().to(torch.device("cuda:0"))

#Make Optimizer
optimizer = torch.optim.Adam(model.parameters())

#Use Training Mode
model.train()

#Random Training/Label Batch
input_batch = torch.rand(BATCH_SIZE, 1, HEIGHT, WIDTH).to(torch.device("cuda:0"))
label_batch = torch.rand(BATCH_SIZE).to(torch.device("cuda:0"))

#Train for 100 Iterations
for i in range(100):
	#Zero the gradient buffers
	optimizer.zero_grad()

	#Forward Through Model
	output = model(input_batch)

	#Calculate Loss
	loss = loss_fn(output, label_batch)

	#Backprop then update using optimizer
	loss.backward()
	optimizer.step()
</code></pre>
<p>If I comment out the loss.backward() function (and subsequently the optimizer.step() function) in the Python code, then the amounts of GPU memory used by the Python and C++ codes are identical. However, if I run the code as originally presented with the loss.backward() function (and optimizer.step() function) present in both the C++ and Python codes, then the Python code uses significantly less memory.</p>
<p>An explanation for why backward() reduces memory usage was given here: <a href="https://discuss.pytorch.org/t/calling-loss-backward-reduce-memory-usage/2735" class="inline-onebox">Calling loss.backward() reduce memory usage?</a></p>
<p>However, this does not explain why the equivalent C++ code doesn’t also have similar memory reduction. Is there a way to reduce the C++ code’s memory usage to match the Python one? This seems like an important issue for those of us trying to port our code over.</p>
LSTM CTC model not learning in PyTorch

username: dnnagy
cooked: <p>Hi! I was working on an ASR model in tensorflow keras, but now I want to swich to pytorch. I’m trying to reimplement keras model in pytorch, but I think, I did a mistake, because the same model on  the same data does not learn in pytorch.</p>
<p>Here is a full jupyter notebook of my problem: <a href="https://github.com/dnnagy/asr_tf_pytorch/blob/master/asr_tf_pytorch.ipynb" rel="nofollow noopener">notebook on github</a></p>
<p>As You can see, the TF model overfits the random data, as expected, but the pytorch model does not learn anything.</p>
<p>I’m using pytorch 1.1.0 with CUDA, and Tensorflow 2.0.0-beta1</p>

username: smth
cooked: <p>initialization of weights will matter here.</p>
<p>the <code>nn.Linear</code> layers that you created, initialize their weights to something other than the default initialization and see if it makes a difference. You can use <a href="https://pytorch.org/docs/stable/nn.html#torch-nn-init" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch-nn-init</a> for convenience to try different initializations.</p>

username: dnnagy
cooked: <p>Could You post a simple example on how to properly initialize weights?<br>
Do I have to re-initialize LSTM weights after each epoch or sample?</p>

username: dnnagy
cooked: <p>Ok, I initialized all my Linear weights based on <a href="https://discuss.pytorch.org/t/weight-initilzation/157/15">this comment</a>, but pytorch is still not learning:</p>
<pre><code class="lang-auto">def weight_init(m): 
  if isinstance(m, nn.Linear):
    size = m.weight.size()
    fan_out = size[0] # number of rows
    fan_in = size[1] # number of columns
    variance = np.sqrt(2.0/(fan_in + fan_out))
    m.weight.data.normal_(0.0, variance)
    
baseline_model = FCBaseline(SEGMENT_WIDTH, SEGMENT_HEIGHT, SEGMENT_CHANNELS, num_classes)
baseline_model.apply(weight_init)
</code></pre>

username: a3VonG
cooked: <p>Could be that I missed it but it seems like a possible reason is that you forgot to zero the gradients before/after running a batch. You only seem to do it at the start. Try adding the following INSIDE your training loop:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
</code></pre>
<p>Does this solve your issue?</p>
<p>See <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">here</a> for an example or <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903">here for the reason why this is needed.</a></p>

username: dnnagy
cooked: <p>Thank You, this solved my issue. I forgot to zero out gradients after each minbach. Now works fine.</p>
<pre><code class="lang-auto"># Optimizer needs the gradients of this minibatch only, so zero out prev grads.
optimizer.zero_grad()
loss.backward() # Calculates derivatives with autograd
optimizer.step() # Update weights
</code></pre>
Utility of padded_seq_batch

username: JohnCwok
cooked: <p>I understand why we use <code>padded_seq_batch</code> when training RNNs with variable lengths input, but I can’t help but wonder :<br>
Why is it so complicated ? Couldn’t we just have made a list of all the input sequences of the batch and then iterate over it ?</p>
<p>I suppose this is probably a performance  issue, but I don’t really understand why.</p>
<p>Can somebody explain it in simple terms ?</p>

username: shubhvachher
cooked: <p>This is exactly what pack padded sequence does. Check this out</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pack_padded_sequence" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pack_padded_sequence</a></p>