akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Issues with batch norm

smu226: <p>Hello! I am having some issues on using batch norm. I am in the beginning of building my NN so for now I am using 100 samples for training and I want to overfit to it, just to make sure that the network can learn. The input and output are bout 1500 each. Here is my network:</p>
<pre><code class="lang-auto">class GW_NN(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(n_inp, 2000)
        self.linear2 = nn.Linear(2000, 2000)
        self.linear3 = nn.Linear(2000, 2000)
        self.linear4 = nn.Linear(2000, 2000)
        self.linear5 = nn.Linear(2000,n_out)
        self.bn = nn.BatchNorm1d(2000)
            
    def forward(self, x):
        x = F.softplus(self.bn(self.linear1(x)))
        x = F.softplus(self.bn(self.linear2(x)))
        x = F.softplus(self.bn(self.linear3(x)))
        x = F.softplus(self.bn(self.linear4(x)))
        x = self.linear5(x) 
        return x

model_gw = GW_NN().cuda()

lrs = 1e-2
optimizer_gw = optim.Adam(model_gw.parameters(), lr = lrs)

for epoch in range(10001):
    model_gw.train()
    for i, dtt in enumerate(my_dataloader):
        optimizer_gw.zero_grad()

        inp = dtt[0].float().cuda()
        output = dtt[1].float().cuda()

        loss = F.mse_loss(model_gw(inp),output)

        loss.backward()
        optimizer_gw.step()

    if epoch%100==0:
        print(loss.data.cpu().numpy())
</code></pre>
<p>The loss goes down okish:</p>
<pre><code class="lang-auto">29418.57
20.279129
11.549426
8.563468
8.235117
8.161551
9.561671
7.5749683
7.60303
7.609553
7.265949
7.824227
10.810941
7.803124
7.6215243
7.977992
7.9355087
7.574047
7.326716
</code></pre>
<p>But when I want to try the trained NN (on the same data used for training) it fails:</p>
<pre><code class="lang-auto">idx = 10
y_real = output_data_phi[idx].data.cpu().numpy()
model_gw.eval()
y_pred = model_gw(input_data)[idx].data.cpu().numpy()

print(((y_pred-y_real)**2).mean())
</code></pre>
<p>I am getting <code>169114.33</code>. I assume that the problem is with using batch norm in eval mode, but I am not sure what to do. Can someone help me? Thank you!</p>

ptrblck: <p>You are currently reusing the batchnorm layer in each step, which might throw the running stats as well as the trainable parameters off.<br>
Could you try to use a separate layer and run the code again?</p>

smu226: <p>Thank you for your reply. Do you mean instead of repeating <code>self.bn</code> 4 times to use <code>self.bn1, self.bn2, self.bn3 self.bn4,</code> or something like this (different for each layer)?</p>

ptrblck: <p>Yes, I meant the creation of 4 different layers as <code>self.bn1</code>, <code>self.bn2</code> etc. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

smu226: <p>Awesome, it’s working! Thank you!</p>
Assumptions around autograd and Python multi-threading

Yaroslav_Bulatov: <p>I’m wondering if it’s safe to assume that users will not call my gradient library from multiple Python threads simultaneously</p>
<p>IE, are there reasons to support this use-case?</p>
<p>It seems current PyTorch doesn’t fully support this use case either, thread-global storage for .grad means that two different Python threads can interfere with each other, example <a href="https://gist.github.com/yaroslavvb/e53c83c40c8385cd90cdc15c7c61fa63" rel="nofollow noopener">https://gist.github.com/yaroslavvb/e53c83c40c8385cd90cdc15c7c61fa63</a></p>

albanD: <p>I don’t think we took any particular attention to this case appart from: “It should not deadlock”, “It should compute correct gradients”.<br>
For the absence of thread-local storage, my feeling as a python user was that, unless you redeclare something in your new thread, everything is global to all threads.<br>
In particular if you want to do learning in multiple threads (not that it is a useful thing to do), you should have two different models to do that.<br>
You think we should document / change this behavior?</p>

Yaroslav_Bulatov: <p>Documenting this would be useful. In particular the fact that there’s global autograd engine shared among Python threads, and that a call into <code>backward</code> will block until all concurrent <code>.backward</code> calls complete. Perhaps here? <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward</a></p>
<p>Disallowing concurrent backward calls in a process seems fine since Python support for concurrency is not great.</p>

creigelde: <p>The Python Global Interpreter Lock or GIL, in simple words, is a mutex (or a lock) that allows only one thread to hold the control of the Python interpreter. All the GIL does is make sure only one thread is executing Python code at a time; control still switches between threads. What the GIL prevents then, is making use of more than one CPU core or separate CPUs to run threads in parallel.</p>
<p>Python threading is great for creating a responsive GUI, or for handling multiple short web requests where I/O is the bottleneck more than the Python code. It is not suitable for parallelizing computationally intensive Python code, stick to the multiprocessing module for such tasks or delegate to a dedicated external library. For actual parallelization in Python, you should use the multiprocessing module to fork multiple processes that execute in parallel (due to the global interpreter lock, Python threads provide interleaving, but they are in fact executed serially, not in parallel, and are only useful when interleaving I/O operations).  However, <a href="http://net-informations.com/python/pro/threading.htm" rel="nofollow noopener">threading</a> is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.</p>
How could I fine tune torchvision model zoo with different learning rate?

coincheung: <p>I would like to fine tune the models in the torchvision model zoo on my own dataset. I need to set different learning rate on the original layers and the modified classifier layer. I modified the resnet model like this:</p>
<pre><code class="lang-auto">    model = torchvision.models.resnet101(pretrained = False)
    model.fc = nn.Linear(in_features = 2048, out_features = 10)
</code></pre>
<p>how could I design the optimizer?<br>
This seems not work</p>
<pre><code class="lang-auto">    optimizer = torch.optim.SGD(
            {'params': model.parameters()[:-1], 'lr': 1e-4, 'momentum': 0.9, 'weight_decay': 1e-4},
            {'params': model.parameters()[-1], 'lr': 5e-3, 'momentum': 0.9, 'weight_decay': 1e-4},)
</code></pre>

ptrblck: <p>I think there are some minor errors (missing bracket and the slice op on a generator).<br>
This should work:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD([
    {'params': list(model.parameters())[:-1], 'lr': 1e-4, 'momentum': 0.9, 'weight_decay': 1e-4},
    {'params': list(model.parameters())[-1], 'lr': 5e-3, 'momentum': 0.9, 'weight_decay': 1e-4}
])
</code></pre>

coincheung: <p>Got it, but consider a weird case of Resnet 101</p>
<pre><code class="lang-auto">ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential( ... )
  (layer2): Sequential( ... )
  (layer3): Sequential( ... )
  (layer4): Sequential( ... )
  (avgpool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (fc): Linear(in_features=2048, out_features=10, bias=True)
</code></pre>
<p>If I need to  fine tune the parameters of layer2 with lr = 1e-3 while finetune the other parameters with lr = 1e-4. How could I write them to optimizer then?</p>

ptrblck: <p>In this case, I would set the learning rate for <code>layer2</code> and use the default for all others as shown in the <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">docs</a>.</p>
<p><a href="https://discuss.pytorch.org/t/change-learning-of-only-one-layer/22299/2">Here</a> I’ve created a small example, how to filter out special layers.</p>

coincheung: <p>Thank you so much, that is very helpful!</p>

Roshan_Santhosh: <p>For the resnet model, I believe this method might not be sufficient. The final Linear layer would have 2 parameters, the weight and the bias. By using model.parameters()[-1], you are using a different LR just for the bias term of the final Linear layer.</p>
Concatenate Tensors without Loop

Stefaanhess: <p>I have a tensor t that I would like to transform. Therefore I made up a for loop that yields the desired result:</p>
<pre><code class="lang-auto">t = torch.tensor([[ 1,  2,  3], [ 4,  5,  6], [ 7,  8,  9], [10, 11, 12]])
N, d = t.shape
new = torch.zeros((N, N, 2*d))
for i in range(N):
    for j in range(N):
        new[i, j, :d] = t[i]
        new[i, j, d:] = t[j]
new

&gt;&gt;&gt;
tensor([[[ 1.,  2.,  3.,  1.,  2.,  3.],
         [ 1.,  2.,  3.,  4.,  5.,  6.],
         [ 1.,  2.,  3.,  7.,  8.,  9.],
         [ 1.,  2.,  3., 10., 11., 12.]],

        [[ 4.,  5.,  6.,  1.,  2.,  3.],
         [ 4.,  5.,  6.,  4.,  5.,  6.],
         [ 4.,  5.,  6.,  7.,  8.,  9.],
         [ 4.,  5.,  6., 10., 11., 12.]],

        [[ 7.,  8.,  9.,  1.,  2.,  3.],
         [ 7.,  8.,  9.,  4.,  5.,  6.],
         [ 7.,  8.,  9.,  7.,  8.,  9.],
         [ 7.,  8.,  9., 10., 11., 12.]],

        [[10., 11., 12.,  1.,  2.,  3.],
         [10., 11., 12.,  4.,  5.,  6.],
         [10., 11., 12.,  7.,  8.,  9.],
         [10., 11., 12., 10., 11., 12.]]])
</code></pre>
<p>Do the for loops cause any errors with the gradient calculations? Is there any other way that concatenates the tensors in the same way without any for loops?</p>

ptrblck: <p>You could repeat the tensor <code>t</code> and use <code>torch.cat</code> to create the new tensor:</p>
<pre><code class="lang-python">t_new = torch.cat((t.repeat(4, 1, 1).transpose(0, 1), t.repeat(4, 1, 1)), 2)
</code></pre>

saba: <p>Hi Ptrblck</p>
<p>I get my result but i need to concatenated predicted and target for 10-folds I use torch.stack but can not concatenate 3 tensors in a loop. I need concatenate them for each loop vertically to use them.Indeed at the end ROCTotal should be (xx,2), which ROCTotal[:,0] should be all predicted value and ROCTotal[:,1] all targeted value. It works for two tensors but not for concatenating in a loop. the size of Predictedvalue and Target  is (106,1) which 106 can be different for each loop</p>
<p>ROCTotal=tensot.Torch()</p>
<p>for ii in range(1,11,1)</p>
<p>ROCTotalFolds=torch.stack([ROCTotalFolds,torch.from_numpy(AveragePredicted),torch.from_numpy(Target1)], dim=0)<br>
ROCTotalFolds=torch.cat([ROCTotalFolds,torch.from_numpy(AveragePredicted),torch.from_numpy(Target1)], 1)</p>

ptrblck: <p>Could you post an example code using random tensors with the right shapes, so that we can have a look at it, as I’m currently unsure how all tensors are defined.</p>

saba: <p>Dear ptrblck</p>
<p>I solved the problem</p>
Do we initialize hidden state of LSTM during validation or testing or use the weights learned during training?

bkuriach: <p>I am confused with hidden state initialization during validation/testing for LSTM.<br>
During training, we initialize hidden states and cell states as zero and start the training, but what needs to be done during validation or testing?</p>
<ul>
<li>Should we reinitialize hidden states and cell states as zero during validation? - In that case, all the learning would be lost right?</li>
<li>Or can we reuse the hidden states learned during training?</li>
</ul>
<h1>Validation</h1>
<pre><code>    with torch.no_grad():
        val_h = net.init_hidden(batch_size) **# initialize to zero --&gt; is this the right way?**
        net.eval()
        for inputs, labels in valid_loader:
            val_h = tuple([each.data for each in val_h])
            inputs, labels = inputs.cuda(), labels.cuda()
            log_ps, val_h = net(inputs, val_h)
            valid_loss += criterion(log_ps, labels.long())</code></pre>
SVM after Feature Extraction in PyTorch

pedro: <p>In a specific image classification task, I found a paper stating that better results are obtained if the classifier is a SVM instead of the fully-connected layers.</p>
<p>How would I be able to compute this using PyTorch for the feature extraction part (and maybe sklearn for the SVM?) ? Is there any prior examples online?</p>
<p>Thanks in advance!</p>
How to correctly mirror / inverse an encoder for the decoder architecture?

Esteban_Lanter: <p>Hi all,</p>
<p>For the purpose of autoencoders, It seems to be common practice to build the decoder architecture in a way so that it mirrors the encoder architecture. However, there seem to be 2 approaches of doing this. Following example aims to create embedding vectors of length 1024 for time series inputs, using CNNs and an LSTM pooling layer (note we use the last hidden layer as the output). Which one of these is the correct “mirror” for the encoder?</p>
<p>(For convenience, I’m also counting the number of total and trainable parameters)</p>
<p><span class="hashtag">#Encoder</span><br>
a = torch.nn.Conv1d(2, 128, 3, stride=1)<br>
b = torch.nn.LSTM(input_size=128,hidden_size=512,num_layers=1)<br>
c = torch.nn.Linear(512,1024)<br>
model_1 = torch.nn.Sequential(a,b,c)<br>
num_all_1 = sum(p.numel() for p in model_1.parameters())<br>
num_train_1 = sum(p.numel() for p in model_1.parameters() if p.requires_grad)<br>
print("%s and %s" % (num_all_1, num_train_1))</p>
<p><span class="hashtag">#Decoder</span> version 1<br>
a = torch.nn.LSTM(input_size=1024,hidden_size=512,num_layers=1)<br>
b = torch.nn.ConvTranspose1d(512, 128, 3, stride=1)<br>
c = torch.nn.Linear(128,2)<br>
model_2 = torch.nn.Sequential(a,b,c)<br>
num_all_2 = sum(p.numel() for p in model_2.parameters())<br>
num_train_2 = sum(p.numel() for p in model_2.parameters() if p.requires_grad)<br>
print("%s and %s" % (num_all_2, num_train_2))</p>
<p><span class="hashtag">#Decoder</span> version 2<br>
a = torch.nn.Linear(1024,512)<br>
b = torch.nn.LSTM(input_size=512,hidden_size=128,num_layers=1)<br>
c = torch.nn.ConvTranspose1d(128, 2, 3, stride=1)<br>
model_3 = torch.nn.Sequential(a,b,c)<br>
num_all_3 = sum(p.numel() for p in model_3.parameters())<br>
num_train_3 = sum(p.numel() for p in model_3.parameters() if p.requires_grad)<br>
print("%s and %s" % (num_all_3, num_train_3))</p>
Naming convention for PyTorch-based packages?

bdusell: <p>Is there a preferred naming convention for Python packages that are based on PyTorch and published on PyPI? For example, let’s say I’ve implemented the <code>foo</code> algorithm in PyTorch. Since my implementation is not compatible with frameworks other than PyTorch, I’d prefer not to name it <code>foo</code>, but something like <code>torch_foo</code> or <code>torchcontrib_foo</code>.</p>

Kushaj: <p>I don’t think there is one. But doing torch_foo would atleast give the idea that the foo repo is based on pytorch.</p>
Splitting Pre-Trained Model by its Parameters

uertenli: <p>Hi,</p>
<p>I want to use a pretrained DenseNet-121 model which I load directly from Torchvision. However, I want to be able to split the model from one of the transition layers, say, transition2, and use the values up to that point.</p>
<p>There has been some discussion going on: <a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113">How to load part of pre trained model?</a>, but it got me confused.</p>
<p>How can I split the parameters until a certain point. to clarify myself further, I want to have the values of the keys in such a way:</p>
<pre><code class="lang-auto">for key,value in densenet.state_dict().items():
   print(key)
   if key == "features.transition2.conv.weight":
		break
</code></pre>

Kushaj: <p>Say you have defined a model that follows the same structure as densenet121 till the transition2. Now you want to load the original weights of densenet in your new model. Every layer in your model will have a name and corresponding weights. All the weights of your model are stored as a dictionary, where the key is the layer name and value is the weights of that layer.</p>
<p>To get a part of the model, you can thus only select those (key, value) pairs that you want which in your case are till transition2.</p>

uertenli: <p>Is there a way to get around this without having to write all these layers manually?</p>
<p>Thanks!</p>

Kushaj: <p>You can select the children layers as required. You can select the first 10 layers as</p>
<pre><code class="lang-python">new_model = nn.Sequential(*(list(model.children())[:10]))
</code></pre>
<p>For more complicated models, you will have to write source code again. But the above method will work in your case.</p>

uertenli: <p>Because of how the pretrained DenseNet model is defined, it has only two children. The entire model up to classifier and the classifier. It seems as though I will have to write the source code myself.</p>
<p>Thanks.</p>
High face image resolution or low face image for training for face recognition

hunar: <p>high face image resolution or low face image resolution  for training , in order to get better accuracy for face recognition in a pre-trained model?<br>
thanks</p>

Kushaj: <p>Do progressive training. Say you start with 64x64 image. Train you model and then change the resolution to 128x128 and then train again. In this way you provided your model with 4x times more data and thus more regularization.</p>
Why some code is using "torch.autograd.Variable(inputs)"?

BarCodeReader: <p>I see some pytorch training code using something like this:</p>
<pre><code class="lang-auto">for batch_idx, (inputs, targets) in enumerate(trainloader):
    inputs, targets = torch.autograd.Variable(inputs), torch.autograd.Variable(targets)
    ...
    pred = model(inputs)
    ...
</code></pre>
<p>why? becuase I think  (inputs, targets) are already tensors? thus we can directly do like:</p>
<pre><code class="lang-auto">for batch_idx, (inputs, targets) in enumerate(trainloader):
    pred = model(inputs)
    ...
</code></pre>
<p>Is it because I misunderstand something or the first usage is just some old version of pytorch?</p>
<p>really thanks</p>

JuanFMontesinos: <p>Yes, from pytorch 0.4 onwards it’s no longer needed.</p>
RuntimeError: cuda runtime error (4) : unspecified launch failure

Neda: <p>I faced this error today when pushing the model to the gpu.</p>
<pre><code class="lang-auto">
  File "C:\Users\Neda\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 377, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)

RuntimeError: cuda runtime error (4) : unspecified launch failure at c:\programdata\miniconda3\conda-bld\pytorch_1533100020551\work\aten\src\thc\generic/THCTensorCopy.cpp:20
</code></pre>
<p>pytorch version: 0.4.1<br>
I installed it using anaconda on windows.<br>
my cuda version is 9.2.148<br>
gpu model is Quadro M5000 and <code>print(torch.cuda.get_device_capability(0))</code> is (5, 2)<br>
I tried on another gpu which is <code>GeForce GTX 1080 Ti (6, 1)</code> but gives another error <code>RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</code>.</p>
<p>I did a lot of experiments before with the same gpu and was fine.<br>
I also restart my computer as I read in this forum that the gpu doesn’t work too well with computers sleeping, but didn’t resolve the issue.</p>

ArturoDeza: <p>I just encountered the same issue as you today. It happened randomly as I was running some of my experiments. Maybe the batch size you are using is too big? This happened to me when I was using a batch size of 128 for images that were 256x256 of size. Still haven’t fixed this issue though!</p>

ntelo007: <p>I am having the same error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_mtl.py", line 472, in &lt;module&gt;
    main()
  File "train_mtl.py", line 463, in main
    train(epoch)
  File "train_mtl.py", line 206, in train
    inputsBGR = Variable(inputsBGR.float().cuda())
RuntimeError: cuda runtime error (4) : unspecified launch failure at c:\new-builder_3\win-wheel\pytorch\aten\src\thc\generic/THCTensorCopy.cpp:20
</code></pre>
<p>I am running PyTorch 0.4.1 on Windows 10 with NVIDIA Quadro P3200 with Max-Q Design.</p>
<p>The algorithm that I am trying to train is retrieved from the following repository:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/anilbatra2185/road_connectivity" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/7984207?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/anilbatra2185/road_connectivity" target="_blank" rel="nofollow noopener">anilbatra2185/road_connectivity</a></h3>

<p>Improved Road Connectivity by Joint Learning of Orientation and Segmentation (CVPR2019) - anilbatra2185/road_connectivity</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>My batch contains about 2000 images.</p>
Keras2Pytorch model conversion tool

shivsd: <p>I have model trained in keras with tensorflow backend, I want to use this trained model in pytorch is there any tool or code which can convert hd5 to .pth.</p>

Eta_C: <p>Maybe <a href="https://github.com/microsoft/MMdnn" rel="nofollow noopener">MMdnn</a>? I am not sure.</p>
Who has The Glass defect image dataset

Foolish: <p>Recently, I want to make a case on my own in deep learning.help!!!</p>

ptrblck: <p>Could you clarify your question a bit?<br>
Where have you seen a mention of this dataset and have you contacted the authors, if you cannot find it publicly hosted?</p>

Foolish: <p>I just want to find the glass defect image data set, but their data set is not public. After searching for a long time on the Internet, no result</p>

ptrblck: <p>Did you see it mentioned somewhere (paper, blog post etc.)?<br>
If so, I would recommend to contact the authors directly.<br>
Maybe they could give you some information about this dataset.</p>

Foolish: <p>The key is that I can’t get in touch. I sent an email to his mentor, but I don’t know if I will get a reply, so I ask for help here.</p>

Eta_C: <p>I think we should not make public the dataset without the consent of its author.</p>
One of the variables needed for gradient has been modified by an inplace

skyline: <p>When I set on the detection I get the trace I seen this sentence went wrong</p>
<pre><code class="lang-auto">alphas[t] = torch.logsumexp((potential[t]*alphas[t-1].unsqueeze(dim = 1)),dim = 0)
</code></pre>
<p>How to fix it,does this sentence use inplace?<br>
One of the variables needed for gradient has been modified by an inplace</p>

ptrblck: <p><code>alphas[t]</code> modifies <code>alphas</code> inplace.<br>
I’m not sure what your code is exactly doing, but maybe storing the results in a temporal <code>list</code> and creating a tensor with <code>torch.cat</code> would work for you.</p>

skyline: <p>My code is much like RNN<br>
that is :</p>
<pre><code class="lang-auto">for t in length_sequence:
  alphas[t] = torch.logsumexp((potential[t]*alphas[t-1].unsqueeze(dim = 1)),dim = 0)
</code></pre>
<p>Where potential is weight of  RNN cell t (which require grad) and <code>alpha[t]</code> is time <code>t</code> output.<br>
After that I compute loss on alpha and backward to change the weight of RNN</p>
<p>You mean store alpha in a list after that convert it to tensor?</p>

skyline: <p>After cat it works ,but gradient for potential is None,I want gradient for potential since I need to optimize this parameter</p>

ptrblck: <p><code>torch.cat</code> will not block the gradient flow:</p>
<pre><code class="lang-python">alphas = [torch.ones(1)]
potential = torch.randn(5, requires_grad=True)
for t in range(5):
  alphas.append(
      torch.logsumexp(
          (potential[t]*alphas[t-1].unsqueeze(dim = 1)),
          dim = 0))

alphas = torch.cat(alphas)
alphas.mean().backward()

print(potential.grad)
&gt; tensor([ 0.5746,  0.2980, -0.0386,  0.0045, -0.0703])
</code></pre>

skyline: <p>It works So thanks !!!<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"><br>
By the way do you have some way to systematically learn pytorch.I have read the recently published tutorial and I want to learn more.</p>

ptrblck: <p>Good to hear it’s working!<br>
Which tutorial are you referring to?<br>
I would recommend to skim through all tutorials on our website and then pick a use case to get your hands dirty. If you get stuck or don’t know how to use specific methods, try to search for similar questions here, and please feel free to ask here in case you cannot find a good reference. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

skyline: <p>Deep-Learning-with-PyTorch this book</p>
Do a batch in multiple iteration

farhad-bat: <p>I have a deep network for person re-identification. My batch size is 256, when I run my code, I gives an error due to lack of gpu memory. However, I don’t want to reduce my batch size. Is there any way that I break my batch into several parts and after processing each batch I update the weights?</p>

ptrblck: <p>Yes, you could delay the <code>optimizer.step()</code> using different approaches as described in <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">this post</a>.<br>
Note that these approaches might yield the desired gradients, but e.g. the running stats in batch norm layers might suffer from the smaller batch sizes, so you might need to adjust the <code>momentum</code>.</p>
<p>Alternatively, you could use <code>torch.utils.checkpointing</code> to trade compute for memory and keep the high batch sizes.</p>
Add_ doesn't use all the GPUs and throw CUDA out of memory

jpainam: <p>Hi, I’m using <code>nn.DataParallel(model)</code>. The training is working well. But After some epochs; i do some computation which seems to be happened on one GPU and thus a <code>CUDA out of memory</code> is thrown. This is the part of the code that gives the error</p>
<pre><code class="lang-python">  File "/home/xxx/xx/xxx/trainers.py", line 115, in compute_distance
    dists.add_(torch.tril(100000 * torch.ones(len(features), len(features)).cuda()))
RuntimeError: CUDA out of memory. Tried to allocate 3.96 GiB (GPU 0; 11.17 GiB total capacity; 9.46 GiB already allocated; 1.39 GiB free; 17.78 MiB cached)
</code></pre>
<p>Is there a way to make <code>add_</code> op works. I don’t want to reduce the batch-size as this will increase the training time.</p>
<p>Thank you</p>

ptrblck: <p>I assume this method is called in your <code>forward</code> method?<br>
If so, use a device from a known parameter (e.g. <code>.to(my_param.device)</code>) instead of the default device using <code>.cuda()</code>.</p>

jpainam: <p>Thank you,<br>
unfortunately,  this is not inside the <code>forward</code> method. I compute the pairwise distance of the extracted feature. Since i have a reference to the model, Can I use <code>.to(model.device)</code>? or <code>.to(model.my_param.device)</code>?</p>
<p>Thanks.</p>

ptrblck: <p>If this is not inside the <code>forward</code> method, <code>nn.DataParallel</code> won’t be able to replicate it on each device.<br>
Since you are running it outside, you could try to not use the default device, as it’s memory usage will usually be bigger than from the other devices.</p>

jpainam: <p>Thank you., I managed to make it work up to some point. Now, the data in cuda:0 need to be added to the data in cuda:1. If i moved the data in cuda0 to cuda1 before the addition, it throw CUDA out of memory.<br>
Let say, if i have</p>
<pre><code class="lang-python">a0 = torch.randn(4, 4) # in cuda 0
a1 = torch.randn(4, 4) # in cuda 1
</code></pre>
<p>To compute <code>a0 += a1</code>, i first need to move <code>a1</code> to cuda0 or <code>a0</code> to cuda 1. Which throws cuda out of memory. However, the result of <code>a0 += a1</code> if computed can hold in cuda0 or cuda1.<br>
Is there a way to solve this problem. I’m using our University GPU , a Tesla K80<br>
Thank you.</p>

ptrblck: <p>I don’t think there is a peer2peer add function, so one device would have to store both tensors before the addition.</p>

nairbv: <blockquote>
<p>i first need to move  <code>a1</code>  to cuda0 or  <code>a0</code>  to cuda 1. Which throws cuda out of memory. However, the result of  <code>a0 += a1</code>  if computed can hold in cuda0 or cuda1.</p>
</blockquote>
<p>If a0 and a1 don’t both fit on a single GPU, you could try putting part of each tensor each GPU, add each chunk, and then concatenate the results with <code>torch.cat</code>. i.e. split each tensor in half, put the first half of each tensor on ‘cuda:0’, add them, then concatenate the result of doing the same with the second half of the tensor.</p>

jpainam: <p>Thank you. It working this way.</p>
Gtx 1060 or 780 for GPT2 Text generation

Tom1: <p>I’m trying to decide which would be better for text generation using GPT2, The 1060 has a higher clock speed but the 780 has more cuda cores, which should I go with? Or should I look at something else around $100?</p>

travis-harper: <p>These cards do not possess the memory capacity to fine-tune GPT-2. For other models that require less, the 6GB version of the RTX 1060 would be an affordable card to practice with. Tim Dettmers at least has a bench mark for it:<br>
                    <a href="https://i1.wp.com/timdettmers.com/wp-content/uploads/2019/04/performance_RTX.png?resize=1024%2C598&amp;ssl=1" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i1.wp.com/timdettmers.com/wp-content/uploads/2019/04/performance_RTX.png?resize=1024%2C598&amp;ssl=1" width="690" height="402">
          </a>

<br>
If this is out of your budget, you likely will be better off settling for free cloud computing services like Google Colab.</p>
Guiding the JITer to properly fusing non-contiguous map-reduce operations

Enamex: <p>Example case:</p>
<pre><code class="lang-auto">ONES = ones(1).to(DEVICE)
@jit.script
def foo(feats, mask):
    # mask : [batch features]
    # feats : [features features]
    return where( # ((1))
        mask,
        feats.unsqueeze(0),
        # shape: [_ features features]
        ONES
    # shape before prod: [batch features features]
    ).prod(1) # ((2))
    # shape after prod: [batch features]
</code></pre>
<p>((1)) and ((2)) should be fusible by directly storing the reduction in the out tensor while iterating over the mask, the data, and the reduction operation.</p>
<p>This… would probably be hell for the vectorizer. But at this point I’d take the hit in exchange for being able to run the thing in memory.</p>
<p>BUT… if any guru around here can show me the logical way I’ve been missing for doing this memory efficiently with existing primitives, please do! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
How to construct an attention

111137: <p>Hi,</p>
<p>I would like to use an attention like this;</p>
<pre><code class="lang-auto">attention = torch.relu(self.w_fc1(x))
self.x[0] = self.x[0] * torch.softmax(self.w_fc2(attention), dim=0)
</code></pre>
<p>where x is input vector.<br>
<span class="hashtag">#I</span> removed avg_pool of 1st stage in attention.<br>
This is in forward path description.<br>
Model() includes;</p>
<pre><code class="lang-auto">self.register_buffer('x', torch.stack([torch.zeros(NUM_INPUT) for _ in range(NUM_HIDDEN)]))
</code></pre>
<pre><code class="lang-auto">self.w_fc1 = nn.Linear(NUM_INPUT, NUM_INPUT)
self.w_fc2 = nn.Linear(NUM_INPUT, NUM_INPUT)
</code></pre>
<p>Then I meet an error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [28]] is at version 56; expected version 28 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>What does mean <code>version 56</code> ?<br>
When I try to do like this;</p>
<pre><code class="lang-auto">attention = torch.relu(self.w_fc1(x))
self.x[0] = attention
</code></pre>
<p>Same error is happen.<br>
I have no idea to solve and where I made a wrong.<br>
Can anyone point out, or suggest to solve?</p>
<p>Best,<br>
S.Takano</p>

Kushaj: <p>The error is due to <code>self.x = attention</code>. The exact reason depends on the complete code. The main idea is as follows: Let ‘a’ depend on ‘x’ and ‘y’ for its gradient computation. Now when you do self.x = , then it is changing the value of x that a depend on, thus you have no way to compute the gradients for ‘a’.</p>

111137: <p>Hi Singh-san,</p>
<p>Where is “a” and “y” in your explanation?</p>
<p>S.Takano</p>

Kushaj: <p>Those are madeup variables as it not possible to tell the exact problem with partial code. So I was trying to make up some dummy scenario in which case this error would occur.</p>

111137: <p>OK, I have modified my explanation, please replying.</p>

Kushaj: <p>Doing <code>self.x[0] = something</code> is an in-place operation i.e. no copy of the object is being made. So are modifying the original object but that original object is also needed for backprop computation.</p>

111137: <p>I think <strong>NO</strong>. I think you do not understand python, sorry.</p>

111137: <p>I checked behavior with print insertion. Then <strong>second</strong> loss.backward() fails to do in a mini-batch.</p>
How to run the calling of F(x1), F(x2), F(x3), F(x4) at a time, and speed up it?

li2515: <p>There are 4 submodels x1, x2, x3 and x4 in my training, which are trained separately. In my script, I need to call F(x1,y), F(x2,y), F(x3,y) and F(x4,y) (F is the same function for x1, x2, x3 and x4, and y is a common variable) one by one every time to do some computation and update their weights. But the speed is slow. I wander if there is a way to call F(x1,a), F(x2,a), F(x3,a) and F(x4,a) in a way like multiprocessing for speed-up, since they use the same function F.</p>

albanD: <p>If every operation they do is big enough, it will already be parallelized at low level. So you won’t get much benefit of forcing them to run in parallel.</p>

Kushaj: <p>Use this code for reference for parallelism on CPU.</p>
<pre><code class="lang-python">from multiprocessing import Process, Manager

# Common function
def my_func(x, output_dict):
    out = x**2
    output_dict[x] = out

input = [1, 2, 3, 4]

# Create manager for shared variable (output_dict)
manager = Manager()
output_dict = manager.dict()

# Instantiate process with arguments
procs = []
for i in input:
    proc = Process(target=my_func, args=(i, output_dict))
    procs.append(proc)
    proc.start()
    
# Complete the processes
for proc in procs:
    proc.join()

print(output_dict.items())
# [(1, 1), (2, 4), (3, 9), (4, 16)]
</code></pre>
Beginner question about astype("float32") and float()

Upgrade_Yourself: <p>In the customized dataset file, in a multi-label context,<br>
1/Is there a reason for the use of float() in addition to .astype(“float32”) in this code?<br>
2/And why cast the labels to float and not leave it as a numpy array of integers(one-hot encoded)?</p>
<p>labels = torch.from_numpy(<br>
item.iloc[1:].values.astype(“float32”)).float()</p>
<p>Thanks in advance</p>

Kushaj: <p>I don’t think there is need to do so. But you check the value of <code>labels.dtpe</code> with and without <code>float()</code>.</p>
<p>Why convert to float? I don’t think there is any important reason, maybe the method they implement under the hood requires these values to be in float. In PyTorch 1.3 type promotion was updated so I think we can leave this step also.</p>
Call by object reference, tensor object

Jonathan_Weissman: <pre><code class="lang-auto">def flatten(t):
    t = t.reshape(1, -1)
    t = t.squeeze()
    t[5] = 5
    return t

t = torch.ones(4, 3)
flatten(t)
print(t)
</code></pre>
<p>I couldn’t understand why the result is tensor([[1., 1., 1.],<br>
[1., 1., 5.],<br>
[1., 1., 1.],<br>
[1., 1., 1.]]). I expected it to stay the original matrix of 1’s, because in the function, t is just a copy to the reference to the object. And after the first command: “t = t.reshape(1, -1)”, t is not referencing to a new object. (another object). the same happens to the next line. and when we try to change t (t[5] = 5), we changed a <em>different</em> object (because as said t is now referencing a new, another object).<br>
so how come t changed?</p>

Kushaj: <p>Tensors are stored in memory as a single contiguous block of memory. It may be a straight line. Every tensor has two other components <strong>dimension</strong> and <strong>stride</strong>. Dim tells the size of the tensor in each dimension and stride tells the number of bytes you need to step in each dimension when traversing the array.</p>
<p>Reshape function will return a view i.e. in your case it would just change the dimension with the same underlying data (<strong>no change in bytes in memory and the same bytes are returned</strong>). Same is true for squeeze. Now when you run <code>t[5]=5</code>, it is changing the same underlying data that ‘t’ originally pointed to. For clarity the below code, gives the same output as yours.</p>
<pre><code class="lang-python">def flatten(t):
    t = t.reshape(1, -1)
    t = t.squeeze()
    t[5] = 5
    return t

x = torch.ones(4,3)
flatten(x)
print(x)
</code></pre>
How to define a function to construct a specific type of tensor with given shape

songqsh: <p>A tensor is called i-type if its entries are equal to the sum of index squared.<br>
I want to have a function to construct an i-type tensor with arbitrarily given shape. This can be done with numpy as follows:</p>
<pre><code class="lang-auto">import numpy as np
def i_tensor(shape = [2,2,2]):
    t = np.zeros(shape)
    it = np.nditer(t, flags=['multi_index'])
    while not it.finished:
        ind = it.multi_index
        t[ind] = np.sum(np.array(ind)**2)
        it.iternext()
    return t

</code></pre>
<p>Test 1:</p>
<pre><code class="lang-auto">#test the function
&gt; i_tensor([4,4])

&gt; array([[ 0.,  1.,  4.,  9.],
        [ 1.,  2.,  5., 10.],
        [ 4.,  5.,  8., 13.],
        [ 9., 10., 13., 18.]])
</code></pre>
<p>Test2:</p>
<pre><code class="lang-auto">&gt; i_tensor([2,2,3])
&gt; array([[[0., 1., 4.],
        [1., 2., 5.]],

       [[1., 2., 5.],
        [2., 3., 6.]]])
</code></pre>
<p>Can I have the same function for torch tensor? I do not see tensor replacement of nditer.</p>

albanD: <p>The following should work (and be much faster as it only loops over the number of dimensions, not the whole output:</p>
<pre><code class="lang-python">
def i_tensor(shape = [2,2,2]):
    ndim = len(shape)
    view_shape = [1,] * ndim
    res = 0.
    for d in range(ndim):
        dim_size = shape[d]
        t = torch.arange(dim_size)
        view_shape[d] = dim_size
        t = t.view(*view_shape).expand(*shape)
        view_shape[d] = 1
        res += t

    return res
</code></pre>

songqsh: <p>Great answer, and I am trying to digest. I may be back here to ask more if I do not understand. Thanks albanD</p>

albanD: <p>Sure !<br>
The trick is to create a Tensor that contains all the indices of one dimension, then add them element-wise.<br>
The view / expand trick is used to reduce memory usage. By only having one of these Tensors to be the full size and all the other ones only the size of the given dimension.</p>

songqsh: <p>First, it is my apology to have a mistake in my original numpy code. Each entry of tensor shall be sum of index , but I did not square it. And your code may be needed a slight change, and I am trying to figure out how to make a square before taking the sum. I am very new to torch …<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> But I learned a lot tricks from this short code given by you.</p>

songqsh: <p>Just got it, change one line will work<br>
<code>t = torch.arange(dim_size)**2</code></p>
When does Pytorch Dataset or Dataloader class convert numpy array to Tensor

Xuyang_Bai: <p>Previously I directly save my data in numpy array when defining the dataset using <code>data.Dataset</code>, and use <code>data.Dataloader</code> to get a dataloader, then when I trying to use this dataloader, it will give me a tensor. However, this time my data is a little bit complex, so I save it as a dict, the value of each item is still numpy, I find the <code>data.Dataset</code> or <code>data.DataLoader</code> doesn’t convert it into Tensor automatically. I wonder when and how does pytorch  <code>data.Dataset</code> or <code>data.DataLoader</code> do the convertion from numpy array to Tensor?</p>
<p>Thanks a lot !</p>

ptrblck: <p>The <code>collate_fn</code> and in particular <a href="https://github.com/pytorch/pytorch/blob/36d17f4105f48d264fe085ce3dc0e4e825ebfb42/torch/utils/data/_utils/collate.py#L56">default_collate</a> would transform the bumpy arrays to tensors.</p>

Xuyang_Bai: <p>Thanks a lot for your reply!</p>
Generic static quantization

Joe_Heyward: <p>Hi,</p>
<p>I’m trying to quantise pre-trained models from torchvision, and I’ve hit an obstacle I 'm struggling to get past. I’ve been able to fuse layers and replace relus as needed, I’ve then used Quantwrapper to get the quant and dequant around the forward function and then I can prepare and convert using the quantisation tools.</p>
<p>The problem I’m at now is that I get an error every time I try to run a model with an operation that needs to be quantised. For example inceptionV3 uses a some operations during the forward pass and errors will be like:</p>
<p>'RuntimeError: Didn’t find kernel to dispatch for operator ‘aten::mul’. Tried to look up kernel for dispatch key ‘QuantizedCPUTensorID.’</p>
<p>I have bee able to run these networks by editing the model to replace operations with nn.quantised.FloatFunctional() operations, is there any way I can get past this without editing each network individually?</p>

dskhudia: <p>Hi <a class="mention" href="/u/joe_heyward">@Joe_Heyward</a>,</p>
<p>Currently, replacing these non-parametric operations with FloatFunctional is the only way. More details are here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture</a></p>
<p>The reason behind the need for replacement is that to quantize these operations we need to observe input/output values.</p>

raghuramank100: <p>We are also working on automating this with quantization after scripting (graph mode quantization), but this feature is still in progress.</p>

Joe_Heyward: <p>Ok, thanks! What’s the best way to keep up to date on this?</p>
<p>Ps I thought I would be able to get past this if I could get the quant and dequant tight enough around the network forward pass, so I wrote a function similar to: <a href="https://pytorch.org/docs/stable/_modules/torch/quantization/quantize.html#add_quant_dequant" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/quantization/quantize.html#add_quant_dequant</a>, where a quantwrapper is added around every submodule to a certain level but this gives me a separate error:</p>
<p>RuntimeError: size mistmatch, m1: [20160 x 224] , m2: [1024 x 100] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:197</p>

dskhudia: <p>Keeping an eye on release notes here (<a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases</a>) is one way to keep up to date on this. Another more involved way is checking quantization related diffs.</p>
<p>It’s hard to say what’s wrong here without looking at the code.</p>
Loss.backward() affect other variables?

taleslimaf: <pre><code class="lang-auto">class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.linear1 = nn.Linear(in_features=40, out_features=50)
        self.bn1 = nn.BatchNorm1d(num_features=50)
        self.linear2 = nn.Linear(in_features=50, out_features=25)
        self.bn2 = nn.BatchNorm1d(num_features=25)
        self.linear3 = nn.Linear(in_features=25, out_features=1)

    def forward(self, input):  # Input is a 1D tensor
        out_list = []
        y = F.relu(self.bn1(self.linear1(input)))
        out_list.append(y)
        y = F.relu(self.bn2(self.linear2(y)))
        out_list.append(y)
        y = self.linear3(y)
        out_list.append(y)
        return out_list
    
model = network()
x = torch.randn(10, 40)
output_list = model(x)
</code></pre>
<p>Let’s say that I have the target for each one of the outputs in the output_list.<br>
Each one of the output will have a loss associated. Can I run loss.backward() for each one without update variables that didn’t contribute to the loss? Like, the loss of the first output of the list shouldn’t change the weights of the last linear.</p>
<p>If I sum all the loss and run total_loss.backward(), the loss of the first linear will affect the update of the last linear? I want to total_loss.backward() doesn’t affect other variables without contribution to ‘loss’.</p>

ptrblck: <p>Yes, this should be the case as shown by the following code snippet using your example code:</p>
<pre><code class="lang-python">model = network()
x = torch.randn(10, 40)
output_list = model(x)

output_list[0].mean().backward(retain_graph=True)
for name, param in model.named_parameters():
    print(name, param.grad)

output_list[1].mean().backward(retain_graph=True)
for name, param in model.named_parameters():
    print(name, param.grad)

output_list[2].mean().backward()
for name, param in model.named_parameters():
    print(name, param.grad)
</code></pre>
Efficient way to calculate kronecker product in pytorch

gebrahimi: <p>Hi,</p>
<p>I have tensors of size [785, 785] and [301, 301]. What is the most efficient way to calculate the kronecker product for them? Whatever I used to do that resulted in kerned dying.</p>
<p>Thanks</p>

richard: <p>If I’m understanding this correctly, the kronecker product will have size [301 * 785, 301 * 785]. Assuming you are using float tensors, the total number of bytes we need for the tensor is 785 * 785 * 301 * 301 * 4, which is roughly 208gb. Do you have that much RAM?</p>