akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.load() is behaving strangely

Shipali_Singh: <pre><code class="lang-auto">require 'torch'   -- torch
require 'optim'
require 'nn'      -- provides a normalization operator

function string:split(sep)
  local sep, fields = sep, {}
  local pattern = string.format("([^%s]+)", sep)
  self:gsub(pattern, function(substr) fields[#fields + 1] = substr end)
  return fields
end

local f_file = io.open(arg[1], 'r')
local p_file = io.open(arg[2], 'w')
local data = torch.Tensor(1, 351)
local name = ''
for line in f_file:lines('*l') do
local l = line:split(',')
first = true
	for key, val in ipairs(l) do
	  if first == false then
	  data[1][key] = val
	  else data[1][key] = 0
	  first = false
	  name = val
	  end
	end
end

local X = data[{{},{2,-1}}]



 model = torch.load('estimation_model.dat')

 myPrediction = model:forward(X)

p_file:write('NAME,F1,F2,F3,F4\n')
p_file:write(name..','..tostring(1000*myPrediction[1][1])..','..tostring(1000*myPrediction[1][2])..','..tostring(1000*myPrediction[1][3])..','..tostring(1000*myPrediction[1][4])..'\n')

```my model is not being loaded,I think its being in .dat format might be the problem .
Here is the erro
th&gt; model=torch.load(tracking_model.dat)
[string "model=torch.load(tracking_model.dat)"]:1: attempt to index global 'tracking_model' (a nil value)
stack traceback:
	/home/sunny/torch/install/share/lua/5.2/trepl/init.lua:506: in function '__index'
	[string "model=torch.load(tracking_model.dat)"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/sunny/torch/install/share/lua/5.2/trepl/init.lua:679: in function 'repl'
	...unny/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:204: in main chunk
	[C]: in ?</code></pre>

ptrblck: <p>It looks like you are using LuaTorch, which is not in active developement anymore.<br>
Are you using an older code base and would it be possible to port it to PyTorch?</p>
Is there no compatibility list from pytorch0.3.1 to 1.0.1

huihuiM: <p>my environment is CUDA10+python3.7+pytorch1.0.(because my INVIDIA only support CUDA10+)</p>
<p>Now I want run a demo code which is developed using pytorch0.3.1<br>
How can I upgrade the demo code so that it can run in my environment?</p>
<p>Are there some code modification gudience?<br>
I am a new user to pytorch.<br>
thank you</p>

ptrblck: <p>Have a look at <a href="https://github.com/pytorch/pytorch/wiki/Breaking-Changes-from-Variable-and-Tensor-merge" rel="nofollow noopener">these breaking changes</a> and the <a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">release notes</a>.<br>
In case you have trouble converting your code to the latest PyTorch version, please post in this board and we are happy to help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

huihuiM: <p>Thank you very much！<br>
for the package “torch.legacy.nn”,  it has been removed,<br>
and which class can replace it?<br>
the current code is like this:</p>
<pre><code class="lang-auto">from torch.legacy.nn import SpatialCrossMapLRN as SpatialCrossMapLRNOld
</code></pre>
<p>just like : <a href="https://github.com/tzzcl/RNN-HA/blob/master/LRN.py" rel="nofollow noopener">https://github.com/tzzcl/RNN-HA/blob/master/LRN.py</a></p>
<p>Thank you</p>
Abnormal memory cost

suke: <p>I face a strange problem, I have 2 models, model1 and model2, model1 is bigger than model 2.<br>
when I test forward time of different model with same input, i need copy to gpu memory.<br>
just like<br>
if self.args.cuda:<br>
device = torch.device(“cuda”)<br>
else:<br>
device = torch.device(“cpu”)<br>
input = input.to(device)</p>
<p>when i collect time cost for “input = input.to(device)”, i found for same input, 2 models cost different time in GPU case:<br>
for model1<br>
input = input.to(device)  prepareData Time: 0.12544s<br>
for model2<br>
input = input.to(device)  prepareData Time: 0.01899s</p>
<p>when i switch to cpu model, they are same:<br>
for model1<br>
input = input.to(device)  prepareData Time: 0.00001s<br>
for model2<br>
input = input.to(device)  prepareData Time: 0.00001s</p>
<p>as i know that is just copy to gpu memory, so i think that should be same. that time cost puzzled me.<br>
anyone know why?</p>

ptrblck: <p>CUDA operations are called asynchronously in PyTorch.<br>
If you would like to time certain operations, make sure to synchronize before starting and stopping the timer:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.time()
# CUDA op
torch.cuda.synchronize()
t1 = time.time()
</code></pre>
<p>Currently your code might accumulate other timings in a host to device copy and thus yield wrong timing results.</p>

suke: <p>yes, you are right, after torch.cuda.synchronize() time print is normal.<br>
the most time cost:<br>
time of torch.matmul: 0.02192 s<br>
one more question, does torch.matmul is very slower than conv2d ops.<br>
my test code:<br>
a = torch.randn(1, 153000, 1, 288)<br>
b = torch.randn(153000, 288, 3)<br>
<span class="hashtag">#a</span> = a.half()<br>
<span class="hashtag">#b</span> = b.half()<br>
if True:<br>
device = torch.device(“cuda”)<br>
else:<br>
device = torch.device(“cpu”)<br>
a = a.to(device)<br>
b = b.to(device)<br>
torch.cuda.synchronize()<br>
start = time.process_time()<br>
torch.matmul(a, b)<br>
torch.cuda.synchronize()<br>
end = time.process_time()<br>
print(“Elapsed time of {}: {} s”.format(‘torch.matmul’, np.round(end - start, decimals=5)))</p>
Parallel Implementation of Gaussian mixture models for working with multiple GPU's

Sedherthe: <p>Hello!<br>
I’m currently working on language recognition and I wanted to implement a Gaussian mixture model which can run parallelly on multiple GPUs(I have 2 GPU’s of 11 Gb each) . As of now, I have implemented a naive GMM comprising normal def functions which does everything on CPU without any parallelization and optimization(Like I don’t need to store the grads of tensors as well) .I wanted to get an idea or some suggestions about how I should go ahead with <strong>Parallelization implementation of GMM!</strong><br>
Thanks!</p>

ptrblck: <p>Would you like to apply some data parallelism (cloning the model on each GPU and splitting the data) or would you like to shard the model somehow (different GPUs compute different parts of your model)?</p>

Sedherthe: <p>I was thinking of the first way similar to nn.Dataparallel approach! This seems possible to me…<br>
I’m not sure how I can go about if I want to implement it in the second way though.</p>
Error while converting pytorch model to onnx

ayrus_anayaran: <p>m = torch.load(’./modell.pth’, map_location=lambda storage, loc: storage)</p>
<h1>Export the trained model to ONNX</h1>
<p>dummy_input = Variable(torch.randn(1, 3, 100, 100))<br>
torch.onnx.export(m(), dummy_input, “./models/ONNNNX.onnx”)</p>
<p>I am getting the error :<br>
File “convert.py”, line 28, in <br>
torch.onnx.export(m(), dummy_input, “./models/ONNNNX.onnx”)<br>
File “/home/surya/myenv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/surya/myenv/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 88, in forward<br>
raise NotImplementedError<br>
NotImplementedError</p>
QNNPACK examples

jinfagang: <p>Does there any qnnpack inference example?</p>
<p>We want do inference on embed devices such as raspberry pi or even small devices.</p>
<p>But libtorch is huge and does not support ARM very well.</p>
<p>QNNPACK is promising but so less documentations or example to get hands on… Does there any resources or links to get hands on wit hit?</p>
Help! How is F.upsample implemented?

zhoulukuan: <p>Since tensorrt does not support interploate/upsample operation, I want to use deconv to implement the bilinear upsamle.<br>
However, it was found that although the original model chose the bilinear mode during using F.upsample, the results are still very different.<br>
How is F.upsamle in mxnet implemented? Is it possible to get exactly the same result with transpose conv?<br>
My test code(refer: <a href="https://discuss.pytorch.org/t/nn-upsample-versus-upsampling-with-transposed-convolution/6813/4" class="inline-onebox">nn.Upsample versus upsampling with transposed convolution</a>):</p>
<pre><code class="lang-auto">def make_bilinear_weights(size, num_channels):
    factor = (size + 1) // 2
    if size % 2 == 1:
        center = factor - 1
    else:
        center = factor - 0.5
    og = np.ogrid[:size, :size]
    filt = (1 - abs(og[0] - center) / factor) * \
           (1 - abs(og[1] - center) / factor)
    print filt
    filt = torch.from_numpy(filt)
    w = torch.zeros(num_channels, 1, size, size)
    for i in range(num_channels):
        w[i, 0] = filt
    return w

x = np.array([[1, 2], [3, 4]], dtype=np.float32)
x = torch.from_numpy(x[np.newaxis, np.newaxis, :, :])

out1 = F.upsample(x, None, 2, 'bilinear', align_corners=True)
out2 = F.upsample(x, None, 2, 'bilinear', align_corners=False)
c = x.size(1)
out3 = F.conv_transpose2d(x, make_bilinear_weights(4, 1), stride=2, padding=1, groups=c)
print "Using usample and align_corners is True:"
print out1
print "Using usample and align_corners is False:"
print out2
print "Using conv transpose2d:"
print out3
</code></pre>
<p>Result:</p>
<pre><code class="lang-auto">Using usample and align_corners is True:
tensor([[[[1.0000, 1.3333, 1.6667, 2.0000],
          [1.6667, 2.0000, 2.3333, 2.6667],
          [2.3333, 2.6667, 3.0000, 3.3333],
          [3.0000, 3.3333, 3.6667, 4.0000]]]])
Using usample and align_corners is False:
tensor([[[[1.0000, 1.2500, 1.7500, 2.0000],
          [1.5000, 1.7500, 2.2500, 2.5000],
          [2.5000, 2.7500, 3.2500, 3.5000],
          [3.0000, 3.2500, 3.7500, 4.0000]]]])
Using conv transpose2d:
tensor([[[[0.5625, 0.9375, 1.3125, 1.1250],
          [1.1250, 1.7500, 2.2500, 1.8750],
          [1.8750, 2.7500, 3.2500, 2.6250],
          [1.6875, 2.4375, 2.8125, 2.2500]]]])
</code></pre>
GPU Strange Memory Usage

RebirthT: <p>Hi, I’m working on a model with two different module (A, B). Module A has 2 conv following by two FC layers and Module B is composed of 5 conv. The model is running on two GPUs.<br>
From my understanding, GPU is allocated when we new a model object with initialization. However, the memory increases about 600M (nvidia-smi) when forwarding thru the first conv(parameter: in_dim=256, out_dim=1024, kernel=3, stride=1) and not increasing any memory when passing thru module B in training phase.</p>

RebirthT: <p>Solved with:<br>
torch.backends.cudnn.benchmark = True</p>
<p>Do not know the reason but problem solved</p>
Network for downscaling followed by bicubic upscaling, can I use PIL or torch?

greenblock: <p>Hi all!</p>
<p>I am implementing a network for image downscaling, where a low resolution image (LR) is the output . However, for the loss function, I want to upscale the LR image back to its original resolution, using bicubic interpolation, and compare it with the original input.</p>
<pre><code class="lang-python">#e.g. where input is a 400x400 image, from a mini-batch of 16
LR = model(input) #where LR will be a 200x200 image
HR = upscale(LR, scale_factor=2, mode=bicubic) #HR will be a 400x400 image
loss = criterion(input, HR)
...
</code></pre>
<p>My question is, for the upscale method can I use Pillow’s bicubic interpolation? If not, I am aware of <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.interpolate" rel="nofollow noopener">torch’s interpolation</a>, however I am afraid that the method might mess with the gradient propagation (due to the nondeterministic behaviour warning).</p>
<p>Thank you for the attention.</p>
Backwards Pass Gradient Calculation for Part of a Layer

flarelink: <p>Hello all,</p>
<p>I’m currently met with a problem that I’m unsure of how to perform. In the backwards pass for PyTorch, the gradients are calculated to update the weights of each layer.</p>
<p>However, let’s say I take a layer and set part of the weights to a fixed-random value, by setting requires_grad = False, while leaving the rest of a layer to be trained, by leaving requires_grad = True, I can see that the values of the random weights aren’t changed throughout training and the trained weights are. Let’s call this network A. Similarly, if I had the same network with the same random seed but all weights are set to requires_grad = True, I would have all the weights tuned. Let’s call this network B.</p>
<p>I’ve confirmed that in both networks A and B the trained weights would be equivalent. This would mean that the network is looking at all of the weights to update its gradients and then the values of the trained weights.<br>
What I would like to do is have the trained weights only calculate the gradients relative to the weights that have requires_grad=True and have it ignore all the requires_grad=False weight so they don’t influence the gradient calculation. Is there a method to do this in PyTorch?</p>
<p>Please let me know if I can help to clarify what I would like to have done. It’s a little confusing, but in short I want to leave a part of a network set to random weights to still extract features while only having the trained weights perform calculations with respect to only the trained weights, not all weights which would include random and trained.</p>
Using BCEWithLogitsLoss for Multi-label problem

ahmed: <p>I have a dataset which contains images and xml files with associated labels and bounding boxes. Some of the images contain have more than one label.</p>
<p>I have designed a dataloader to extract the annotations from xml files. Based on the dataloader, the output looks like this for 1 batch:</p>
<pre><code class="lang-auto">{'boxes': tensor([[[444., 220.,  27.,  65.],
          [468., 220.,  26.,  66.],
          [415., 224.,  20.,  33.]]]), 
'id': tensor([[1., 1., 4.]])}
</code></pre>
<p>From my understanding, I would need to use <code>BCEWithLogitsLoss</code> for a multi-label approach.</p>
<p>The issue I am having is how to calculate the loss when there is more than one label. When it is a single label, I use the following approach with a <code>CrossEntropy</code> optimiser:</p>
<pre><code class="lang-auto">    for i, data in enumerate(trainloader, 0):
        inputs, labels = data

        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
</code></pre>
<p>Would this approach be correct for multi-labels?</p>
Why images.unsqueeze(1) adds 2 dimentions

Muhsinka: <p>In:<br>
img.shape</p>
<p>Out:<br>
torch.Size([1, 28, 28])</p>
<p>In:<br>
img = images.unsqueeze(1)</p>
<p>In:<br>
img.shape</p>
<p>Out:<br>
torch.Size([4, 1, 1, 28, 28])</p>
<p>Can anyone pls tell me why .unsqueeze fauntion adds 2 dimentions?<br>
batch size = 4</p>

ptrblck: <p>Are you sure you are not mixing up <code>img</code> and <code>images</code>?<br>
Additionally to the unsqueezed dimension <code>img</code> also has suddently a batch size of 4, which wasn’t there previously.<br>
Could you print <code>images.shape</code> before unsqueezing it?</p>
Clamp() function cannot handle 5 decimals

swordfish: <pre><code class="lang-auto">EPS = 1e-5

a = torch.tensor([1.]).type(torch.DoubleTensor)
a = torch.clamp(a, max=1.-EPS)

print(1. - EPS)  # This gives 0.99999
print(a)         # This gives tensor([1.0000], dtype=torch.float64)
</code></pre>
<p>Is there a way to avoid rounding off? However, if I do, I get the expected clamped value.</p>
<pre><code class="lang-auto">a = torch.tensor([0.]).type(torch.DoubleTensor)
a = torch.clamp(a, min=EPS)

print(a)         # This gives tensor([1.0000e-05])
</code></pre>

JuanFMontesinos: <p>Could you check it’s not a rounded due to print?</p>
What is the right approach for dealing with `retain_graph=True`

ljj7975: <p>I have simple GAN so I do following.</p>
<pre><code class="lang-auto">real_target = 1s
fake_target = 0s

gen_net.train()
dis_net.eval()
gen_optimizer.zero_grad()

generated = gen_net(noise)
gen_loss = loss_fn(dis_net(generated), real_target)

gen_loss.backward()

gen_net.eval()
dis_net.train()
dis_optimizer.zero_grad()

input_data = torch.cat(generated, real)
target = torch.cat(fake_target, greal_target)

output = dis_net(input_data)
dis_loss = loss_fn(output, target)

dis_loss.backward()
</code></pre>
<p>Yes, I am aware of the fact that once second <code>backward()</code> is called, the framework will attempt to backprop through <code>generated</code>, <code>gen_net</code> and <code>dis_net</code> twice and therefore raise warning.</p>
<p>Based on the code I have above, my understanding is that <code>gen_net</code> will be updated only from the first backward and <code>dis_net</code> will be updated only from the second backward as I call <code>.eval()</code> and <code>.train()</code> correctly prior to forward pass.<br>
So, if I am safe to set <code>retain_graph=True</code> to get the training I want. Is my understanding correct?</p>
<p>Furthermore, I would like to deal with this correctly.<br>
I believe if code is written correctly, I should not see <code>retain_graph=True</code> warning at all.<br>
How should I change the above code so that I do not see <code>retain_graph=True</code> warning without explicitly setting <code>retain_graph=True</code>?</p>

ptrblck: <p>If your generator was already trained in the first step, you could try to detach the <code>generated</code> tensor from it before feeding it to the discriminator:</p>
<pre><code class="lang-python">input_data = torch.cat(generated.detach(), real)
</code></pre>
<p>This will detach <code>generated</code> from the computation graph, so that the backward pass will stop at this point.</p>

ljj7975: <p>Thanks it worked.<br>
Out of curiosity, I have one more question</p>
<p>as you see, I am calling <code>.eval()</code>, and <code>.train()</code> correctly prior to each <code>loss.backward()</code>.<br>
Therefore, even though I just decided to add <code>retain_graph=True</code> to bypass the warning, the training should converge as if I detached the generated input.<br>
Am I correct here?</p>

ptrblck: <p><code>.eval()</code> and <code>.train()</code> are not related to the gradient calculation and will just switch the behavior of some layers. E.g. dropout will be disabled and batch norm layers will use their internal running estimates instead of the batch stats.<br>
The warnings/errors regarding a cleared graph should not change if you switch between eval and train.</p>
Implement custom metric in ignite

Luca_Pamparana: <p>Are there some examples on how to implement a custom metric in Ignite?</p>

vfdev-5: <p><a class="mention" href="/u/luca_pamparana">@Luca_Pamparana</a> to implement a custom metric is simple: you need to override <code>ignite.metrics.Metric</code> class and reimplement 3 methods: <code>reset</code>, <code>update</code> and <code>compute</code>:</p>
<pre><code class="lang-python">from ignite.metrics import Metric


class MyMetric(Metric):

    def __init__(self, output_transform=lambda x: x):
        self._var1 = None
        self._var2 = None
        super(MyMetric, self).__init__(output_transform=output_transform)

    def reset(self):
        self._var1 = 0
        self._var2 = 0

    def update(self, output):
        y_pred, y = output
        # ... your custom implementation to update internal state on after a single iteration
        # e.g. self._var2 += y.shape[0]
              
    def compute(self):
        # compute the metric using the internal variables
        # res = self._var1 / self._var2
        res = 0
        return res
</code></pre>
<p>For other examples, take a look at the source code :<br>
MSE: <a href="https://github.com/pytorch/ignite/blob/master/ignite/metrics/mean_squared_error.py" rel="nofollow noopener">https://github.com/pytorch/ignite/blob/master/ignite/metrics/mean_squared_error.py</a><br>
etc</p>
<p>HTH</p>

Luca_Pamparana: <p>Thank you for that reply!</p>
My network is outputing the same image for any input image

etekiller: <p>This seems like a crazy overfitting problem, but the loss isn’t even close to 0 yet.</p>
<p>I am new to Machine Learning and this is my second Neural Network.</p>
<p>What I am trying to do is to convert a grayscale image into an RGB image (doesn’t have to be perfect at all, I just want to see what the network comes up with for images it has never seen before, failure to replicate the original colors is the main interest of mine in this project).</p>
<p>The thing is, my current problem isn’t fun at all, because I get the same mash of colors as output for all the images I input into the network.</p>
<p>This is my network:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/peterworks/ml-practice" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/34659671?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/peterworks/ml-practice" target="_blank" rel="nofollow noopener">peterworks/ml-practice</a></h3>

<p>Contribute to peterworks/ml-practice development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The structure of the network itself is defined in ./classes. The implementation is in ./app.py. You can test it yourself if you want by putting images in the proper folders. ./guesses/set should contain testing set images, ./rooms/set should contain input images.</p>
<p>I would be happy if this network simply output different things for different input images, but it doesn’t do that.</p>
<p>Could you help me out with this problem? I would be grateful.</p>

etekiller: <p>I’m not sure if it’s okay but I wanted to bump this topic, since I can’t find help for this issue anywhere. No one is really replying on either SO or here.</p>
.data<float>[1] segfaults when CUDAType

Bart_Louwers: <p>When I print a tensor variable I get:</p>
<pre><code class="lang-auto"> 0.3403
 0.2788
-0.1594
[ Variable[CUDAType]{3,1} ]
</code></pre>
<p>but when I try to access the second element with <code>var.data&lt;float&gt;[1]</code> I get a segmentation fault. When I run it on the CPU it works fine.</p>
<p>Am I correctly accessing this tensor?</p>
<p>Note: I do not need fast access.</p>

Bart_Louwers: <p>I found the answer on StackOverflow:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/54200785/torch-c-getting-the-value-of-a-int-tensor-by-using-dataint" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5228890/afshin-oroojlooy" target="_blank" rel="nofollow noopener">
    <img alt="Afshin Oroojlooy" src="https://lh5.googleusercontent.com/-On5ouMp80l4/AAAAAAAAAAI/AAAAAAAAAAA/pCVMyVtcgDw/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/54200785/torch-c-getting-the-value-of-a-int-tensor-by-using-dataint" target="_blank" rel="nofollow noopener">Torch C++: Getting the value of a int tensor by using *.data&lt;int&gt;()</a>
</h4>

<div class="tags">
  <strong>pytorch, torch, libtorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/5228890/afshin-oroojlooy" target="_blank" rel="nofollow noopener">
    Afshin Oroojlooy
  </a>
  on <a href="https://stackoverflow.com/questions/54200785/torch-c-getting-the-value-of-a-int-tensor-by-using-dataint" target="_blank" rel="nofollow noopener">02:22PM - 15 Jan 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The Tensor class despreately needs documentation!</p>
Change torchvision model by ArgumentParser

duplex: <p>Hi,</p>
<p>I would like to add an argument to my script in which the user can quickly decide for a torchvision model.</p>
<pre><code class="lang-auto">parser = argparse.ArgumentParser()
parser.add_argument('--arch', dest='torchmodel', action='store')
torchmodel = args.torchmodel
</code></pre>
<p>The model is set by, e.g. for vgg19:<br>
model = models.vgg19(pretrained=True)</p>
<p>I would like make the models.attribute a placeholder, i.e. the string that is passed with “–arch ‘string’” is supposed to fill in the model-name in models.model_name(pretrained=True).</p>
<p>Is there a way to implement this? Thank you!</p>

shubhvachher: <p>Sure. Try <a href="https://github.com/psyec1/Lipreading-PyTorch/blob/master/main.py" rel="nofollow noopener">https://github.com/psyec1/Lipreading-PyTorch/blob/master/main.py</a></p>
<p>Instead of line 17, you load your model using an <code>if...elif...else</code> block.</p>

ptrblck: <p>Alternatively, use <code>models.__dict__</code> to select the specified model as shown <a href="https://github.com/pytorch/examples/blob/1de2ff9338bacaaffa123d03ce53d7522d5dcc2e/imagenet/main.py#L137" rel="nofollow noopener">here</a>.</p>

duplex: <p>Amazing, exactly what I was looking for! Thanks a lot!</p>
Support for Batch Renormalization

simon: <p>For the past few days, I’ve been training a model that uses batch normalization. While this normalization is crucial to speed up training, performance drops severely once I switch to <em>eval</em> instead of <em>train</em> mode. While browsing the PyTorch forums, I noticed that <a href="https://discuss.pytorch.org/t/batchnorm-eval-cause-worst-result/15948">a lot of other people</a> were having the same problem. The problem seems to be caused by the fact the running estimates are not reliable when using small batch sizes. For a lot of problems (e.g. segmentation), however, increasing the batch size is not feasible due to memory constraints. The authors of the batch normalization paper acknowledged this issue and wrote a follow-up paper about batch renormalization, a similar technique which should also work with smaller batches.</p>
<p>I was wondering if there were any plans to implement this batch renormalization in PyTorch? It seems like a feature that could be very helpful for all those users experiencing problems with regular batch normalization. I experimented with group, layer and instance normalization and neither of those performed as good as batch normalization, so batch renormalization may turn out to be very useful. I’m aware that it’s possible to implement this directly in Python, but other users already attempted this and reported <a href="https://discuss.pytorch.org/t/batch-renormalization-implementation-in-thcunn/5144">significantly higher memory usage and time consumption</a>. So, an implementation  directly for CPU/GPU seems necessary.<br>
Unfortunately, I’m afraid that I, like most users, lack the necessary skills to implement this feature myself for CPU/GPU in PyTorch.<br>
Hence my question if there were any plans to implement this or if anyone has any tips for me to make my model perform good in both eval and train mode with small batch sizes.<br>
Thanks in advance.</p>

tom: <p>You could always try instance norm (unless you have really few features per channel), which takes away the difference between training and evaluation.<br>
If you wanted to experiment with batch renorm, starting from current master makes that easier. In it’s native cuda kernels, we split the gathering the statistics from applying the transformation. So you could reuse the first and just adapt the pointwise transformation to use the amended formula. You can use the vanilla backward function (which is the “stop grad” wants you to ignore the amendments for the backward).<br>
I would recommend doing this as a custom op or extension module + torch.autograd.Function.</p>
<p>Best regards</p>
<p>Thomas</p>

Vincenzo_Lomonaco: <p>We also would love Batch Renorm support as it seems to be very important for Continual Learning over small batch sizes!</p>
Does model.eval() & with torch.set_grad_enabled(is_train) have the same effect for grad history?

rasbt: <p>For the sake of the example, let’s say I don’t use Dropout, BatchNorm etc, just a plain CNN.</p>
<p>According to the docs (in PyTorch 0.4),</p>
<pre><code>with torch.set_grad_enabled(is_train)
</code></pre>
<p>prevents tracking via autograd, which would make the inference mode more efficient (I assume). Now, if I would use <code>model.eval()</code>, would this have the same effect. E.g., does the following track gradients after <code>model.eval()</code></p>
<pre><code class="lang-python">model = CNN()
for e in num_epochs:
    # do training

# evaluate model:
model = model.eval()
logits, probas = model(testset_features)
</code></pre>
<p>or is it recommended, in addition, to do the following:</p>
<pre><code class="lang-python">model = CNN()
for e in num_epochs:
    # do training

# evaluate model:
model = model.eval()
with torch.set_grad_enabled(False):
    logits, probas = model(testset_features)
</code></pre>

Irfan_Bulu: <p>I think it is the latter. model.eval() has effect on dropout, batchnorm etc. You can use model.eval() in combination with with torch.no_grad() during inference phase.</p>
<p>from the docs:<br>
Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.</p>

SimonW: <p>eval doesn’t turn off history tracking.</p>

rasbt: <p>thanks <a class="mention" href="/u/irfan_bulu">@Irfan_Bulu</a> and <a class="mention" href="/u/simonw">@SimonW</a></p>

Shihab_Shahriar: <p>I also assumed that eval() mode automatically turns off gradient computation. Hopefully you can see why this might be confusing for us newcomers. I would request to emphasize this point in docs at nn.Module’s eval() function. Actually apart from FAQ, an article pointing out common mistakes and confusions would be great. Thanks.</p>

Shanenoluna: <p>Exactly! Emm, so is this article available now?</p>
<aside class="quote no-group" data-username="Shihab_Shahriar" data-post="5" data-topic="17183">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shihab_shahriar/48/10025_2.png" class="avatar"> Shihab_Shahriar:</div>
<blockquote>
<p>Actually apart from FAQ, an article pointing out common mistakes and confusions would be great.</p>
</blockquote>
</aside>
K-fold cross validation

Hodaya_Binyamini: <p>Hi,<br>
I’m using the code over my data:<br>
def prepare_sequence(seq, to_ix):<br>
idxs = [to_ix[w] for w in seq]<br>
return torch.tensor(idxs, dtype=torch.long)</p>
<p>training_data = [<br>
(“The dog ate the apple”.split(), [“DET”, “NN”, “V”, “DET”, “NN”]),<br>
(“Everybody read that book”.split(), [“NN”, “V”, “DET”, “NN”])<br>
]<br>
word_to_ix = {}<br>
for sent, tags in training_data:<br>
for word in sent:<br>
if word not in word_to_ix:<br>
word_to_ix[word] = len(word_to_ix)<br>
print(word_to_ix)<br>
tag_to_ix = {“DET”: 0, “NN”: 1, “V”: 2}</p>
<p>EMBEDDING_DIM = 6<br>
HIDDEN_DIM = 6<br>
class LSTMTagger(nn.Module):</p>
<pre><code>def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
    super(LSTMTagger, self).__init__()
    self.hidden_dim = hidden_dim

    self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

    # The LSTM takes word embeddings as inputs, and outputs hidden states
    # with dimensionality hidden_dim.
    self.lstm = nn.LSTM(embedding_dim, hidden_dim)

    # The linear layer that maps from hidden state space to tag space
    self.hidden2tag = nn.Linear(hidden_dim, tagset_size)

def forward(self, sentence):
    embeds = self.word_embeddings(sentence)
    lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
    tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))
    tag_scores = F.log_softmax(tag_space, dim=1)
    return tag_scores
</code></pre>
<p>I know that PyTorch can handle different length of sentences.<br>
The problem is that I want to add the option of cross-validation to improve my results.<br>
But when I used the line:</p>
<p>sentences = []<br>
tags = []<br>
for sentence, target in training_data:<br>
sentence_in = prepare_sequence(sentence, word_to_vec)<br>
targets = prepare_sequence(target, tag_to_ix)<br>
sentences.append(sentence_in)<br>
tags.append(targets)</p>
<p>y_pred = cross_val_predict(net, sentences, tags, cv=3)</p>
<p>The fail I get is:<br>
ValueError: Dataset does not have consistent lengths.</p>
<p>What can I do to use cross-validation?</p>
Print which tensors are passed into .backward()

paolo: <p>Is there a way to print which tensors are passed into .backward()? The problem is that, when I execute <code>loss.backward()</code>, the code prints out <code>RuntimeError: grad can be implicitly created only for scalar outputs</code>, meaning that the dimension of tensor passed into .backward() is not the same as the dimension of the tensor whose gradient is being calculated. Because of this, I would like to print which are the tensors used as input of the <code>backward()</code> function. Is this possible?</p>

sai_tharun: <p>can you post your code,it helps us what exactly is wrong</p>
RuntimeError: exp_vml_cpu not implemented for 'Long'

Halil_Yildirim: <p>I want to define an activation function for my model. The activation function I want to use is the sigmoid function.</p>
<p>I define this activation function.</p>
<pre><code class="lang-auto">import torch
def sigmoid(z):
   return 1/(1+torch.exp(-z))
X = np.array([0])
t = torch.from_numpy(X)
sigmoid(t)
</code></pre>
<p>RuntimeError: exp_vml_cpu not implemented for ‘Long’</p>
<blockquote>
<p>But when use X = np.array([0.]) I don’t get any error. Why that occurs?</p>
</blockquote>

ptrblck: <p><code>torch.from_numpy</code> keeps the <code>dtype</code>.<br>
If you initialize the numpy array with <code>0</code>, it will result in <code>t</code> being a <code>LongTensor</code>, while <code>0.</code> yields a <code>FloatTensor</code>.<br>
The method <code>torch.exp</code> is not implemented for <code>LongTensors</code>, which throws this error.</p>
RuntimeError: invalid argument 13

tournemonde: <p>Hi Pytorch fans ;D</p>
<p>I am running an encoder - decoder architecture and I am stuck on this error:</p>
<p>RuntimeError: invalid argument 13: ldc should be at least max(1, m=0), but have 0 at /Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THBlas.cpp:334</p>
<p>The traceback is the following:</p>
<p>File “/Users/…/code/classes/encoder_decoder.py”, line 130, in forward<br>
x = self.attn_linear(x.view(-1, self.hidden_size * 2 + self.T - 1))  # (batch_size * input_size) * 1</p>
<p>File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)</p>
<p>File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py”, line 67, in forward<br>
return F.linear(input, self.weight, self.bias)</p>
<p>File “/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py”, line 1352, in linear<br>
ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())</p>
<p>Any hint on how to tackle it is welcome!<br>
Thank you</p>

riley: <p>Also (sporadically) running into a similar error in the same .cpp file (I’m using a loss function based on stochastic indices into an embedding bank on Ubuntu 18.04). Would love to hear about these errors.</p>
<p>File “pytorch_fun.py”, line 541, in </p>
<pre><code>loss = LALoss(embeddings, emb_idx, toy_bank, cluster_I, args, val=False, centroid_index=centroid_index)
</code></pre>
<p>File “pytorch_fun.py”, line 504, in LALoss</p>
<pre><code>NNL_close_background -= torch.log(torch.exp((bank[list(close_background_I[emb_i])] @ embeddings[emb_i].unsqueeze(-1))/args['tau']).sum())
</code></pre>
<p>RuntimeError: invalid argument 13: ldc should be at least max(1, m=0), but have 0 at /pytorch/aten/src/TH/generic/THBlas.cpp:367</p>
<p>Update: I believe this error is occurring when matrix multiplications are attempted with tensors having a first dimension shape value of zero (i.e. when the list index into bank has a length of zero) with the other tensor having non-zero shape values in both of the dimensions of the tensor having a zero shape value (only when unsqueeze is used; see below). I was not aware that PyTorch supported tensors with shape values of zero.</p>
<p>Minimal code to reproduce:</p>
<p>a = torch.ones((4,4))<br>
b = torch.ones(4)</p>
<p>a@b<br>
tensor([4., 4., 4., 4.])<br>
a[[]]<span class="mention">@b</span><br>
&gt;&gt;&gt;tensor([])<br>
a[[]]<span class="mention">@b.unsqueeze</span>(-1)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: invalid argument 13: ldc should be at least max(1, m=0), but have 0 at /pytorch/aten/src/TH/generic/THBlas.cpp:367</p>

auroraustc: <p>I encountered the same problem using v1.1.0 stable version. When training on GPU, everything works fine. When I use CPU, then the error occurs.</p>
Cross validation using skorch

omarabdelaziz: <p>Hello,<br>
I want to make cross validation ofr my dataset so I chose the skorch lib but it require to pass the X and Y explicitly but My train_data is Dataloader that returns  image, caption, caption_lengths, video_length and I pass them to the model but how can I do this in skorch<br>
I get this error when I pass only the data_loader<br>
<code>ValueError: Stratified CV requires explicitely passing a suitable y.</code><br>
any Idea what should I do!</p>

karmus89: <p>I’ve only just recently started familiarizing myself with Skorch, but it seems that calls to <code>fit</code> or equivalent require passing <code>y</code> explicitly. Skorch allows passing PyTorch <code>Dataset</code>-classes directly but in turn requires the user to pass <code>y=None</code> in conjunction with the <code>Dataset</code>. Please see Skorch’s documentation for <a href="https://skorch.readthedocs.io/en/stable/user/neuralnet.html#fit-x-y" rel="nofollow noopener">how they handle parameters passed to <code>fit</code> method</a>.</p>
<p>IMHO you should maybe refrain from passing a <code>DataLoader</code> and instead let Skorch’s <code>NeuralNet</code> take care of wrapping the <code>Dataset</code> to a <code>DataLoader</code>. See their quick tutorial <a href="https://skorch.readthedocs.io/en/stable/user/FAQ.html#faq-how-do-i-use-a-pytorch-dataset-with-skorch" rel="nofollow noopener">here</a>.</p>
<p>Cheers,<br>
PN</p>

omarabdelaziz: <aside class="quote no-group" data-username="karmus89" data-post="2" data-topic="47384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karmus89/48/2210_2.png" class="avatar"> karmus89:</div>
<blockquote>
<p>IMHO you should maybe refrain from passing a <code>DataLoader</code> and instead let Skorch’s <code>NeuralNet</code> take care of wrapping the <code>Dataset</code> to a <code>DataLoader</code> . See their quick tutorial <a href="https://skorch.readthedocs.io/en/stable/user/FAQ.html#faq-how-do-i-use-a-pytorch-dataset-with-skorch" rel="noopener nofollow ugc">here</a>.</p>
</blockquote>
</aside>
<p>thank for your reply I also got this answer and it works <a href="https://stackoverflow.com/questions/56490278/passing-tensordataset-or-dataloader-to-skorch/56667050#56667050" class="inline-onebox" rel="noopener nofollow ugc">machine learning - Passing tensorDataset or Dataloader to skorch - Stack Overflow</a></p>
Stateless constant tensor w/ DataParallel, JIT

volcacius: <p>I’m currently writing a fairly complex library that exposes various Modules and I have the following requirement in multiple places that I have a hard time satisfying.<br>
I need to be able to define a constant tensor in a Module that is initialized with some user-defined value and shape. I want the tensor to move to the correct target hardware when doing model.to(some_device), but I don’t want the value to be part of the state_dict when the model is saved/loaded, so that if the user pass a different value at init time, that value is taken and not overwritten when a state_dict is loaded.<br>
There are possible ways around this, e.g. define a Buffer, manually exclude it from the state_dict when the model is saved, and the reload with strict=False. My issue with this is that I would basically pass on the problem to the library’s user, and having a requirement on strict=False would also expose me to all sort of other weaknesses. Instead, I want to be able to manage the problem from within the library, i.e. from the Module definition.<br>
My only idea at the moment is to use a Buffer and override the load_state APIs of the Module, but I have never seen it done anywhere. Any sort of pointer would be highly appreciated.<br>
The implementation should work with the JIT compiler in a DataParallel context.</p>
<p>Alessandro</p>

volcacius: <p>To answer myself, I partially solved the problem for by splitting the value into a buffer with value 0 and a python constant with the user-defined value, and then summing them in the forward pass. Hopefully the JIT is smart enough to see that they are both constant, and it propagates them when required, but I haven’t verified it yet.<br>
There is still has a requirement on strict=False though, since loading a pretrained model to which any of these layers have been added will complain about the missing buffer.<br>
It would be nice to see a proper API for this use case, especially give the fact that constant values could be easily optimized by the JIT.</p>
<p>Alessandro</p>
Autograd.grad works strange with increasing batch_sz

11158: <p>I am training GAN with gradient penalty for discriminator w.r.t. real images.</p>
<p>In my code I have snippet like this:</p>
<pre><code class="lang-auto">real_scores = discriminator(real_imgs)
...
r1_penalty = torch.autograd.grad(real_scores.sum(), real_imgs, retain_graph=True, create_graph=True)[0]
r1_penalty = torch.mean(r1_penalty ** 2)
loss = loss + self.r1_gamma * r1_penalty
</code></pre>
<p>The strange thing about this code is how the processing time grows with batch size: the same amount of real images are processed 4 times slower with batch size &gt;= 128 (e.g. 128, 256 or 512) then the batch size 64. If I remove this gradient penalty then all timings are pretty much the same.</p>
<p>That’s pretty strange for me. Any ideas how is it possible?</p>
Asking on dropout

NOP: <p>From  <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/functional.html</a> I see</p>
<pre><code class="lang-auto">[docs]@weak_script
def dropout(input, p=0.5, training=True, inplace=False):
    # type: (Tensor, float, bool, bool) -&gt; Tensor
    r"""
    During training, randomly zeroes some of the elements of the input
    tensor with probability :attr:`p` using samples from a Bernoulli
    distribution.

    See :class:`~torch.nn.Dropout` for details.

    Args:
        p: probability of an element to be zeroed. Default: 0.5
        training: apply dropout if is ``True``. Default: ``True``
        inplace: If set to ``True``, will do this operation in-place. Default: ``False``
    """
    if p &lt; 0. or p &gt; 1.:
        raise ValueError("dropout probability has to be between 0 and 1, "
                         "but got {}".format(p))
    return (_VF.dropout_(input, p, training)
            if inplace
            else _VF.dropout(input, p, training))
</code></pre>
<p>Where is the code for this function? In other words where is <code>_VF</code> object?</p>
<p>I ask because I wanted to confirm if this function is aware of training or not training stage?</p>

ptrblck: <p>You can find the implementation <a href="https://github.com/pytorch/pytorch/blob/1d705b4b075e32540293d1717b582442a66ffcce/aten/src/ATen/native/Dropout.cpp#L44" rel="nofollow noopener">here</a>.</p>
<p>A quick code sample also shows that the behavior is changed in training/eval:</p>
<pre><code class="lang-python">x = torch.randn(1, 10)
print(F.dropout(x, p=0.5, training=True))
&gt;tensor([[-0.0000,  1.2927, -0.0000,  0.0000,  1.2973, -0.0000, -1.5645, -0.0000,
          -0.0000, -0.0000]])
print(F.dropout(x, p=0.5, training=False))
&gt;tensor([[-0.4785,  0.6463, -0.4714,  1.4728,  0.6487, -2.0346, -0.7822, -1.0754,
          -0.7367, -0.8063]])
</code></pre>

NOP: <p>I am still in shock:</p>
<pre><code class="lang-auto">
import torch.nn.functional as F

x = torch.randn(1, 10)
print(F.dropout(x, p=0.5, training=True))

print(F.dropout(x, p=0.5, training=False))

# tensor([[ 0.0000, -2.0286, -2.6529,  0.0000,  0.6054,  0.0000, -0.0000, -0.5667,
#           0.3148,  1.6862]])
# tensor([[ 0.2737, -1.0143, -1.3264,  0.0609,  0.3027,  0.0464, -1.5175, -0.2833,
#           0.1574,  0.8431]])
</code></pre>
<p>Apparently on my end I don’t get 5/10 zeros, but sometime 1/10 (read: just one zero). Is this strange or OK. Here I showed 4 zero output.</p>

ptrblck: <p>It’s a random operation (thus you have to specify the probability), so you can’t expect to get strictly 5/10 zeros.</p>

NOP: <p>Nop, (no problem), thanks for the dropout. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
ShuffleNet classification

M_S: <p>Hi,<br>
I did a small experiment with the ShuffleNetV2 model, but I’m not sure about its results.<br>
From the torchvision models, I loaded aShuffleNetV2 instance pretrained on ImageNet. So I have a classification network across 1000 classes, right?</p>
<p>Then, I replaced the last FC layer:</p>
<p><code>model.fc = Linear(in_features=model.fc.in_features, out_features=1, bias=(model.fc.bias is not None))</code></p>
<p>Also I called <code>reset_parameters()</code> on all layers.</p>
<p>So now, supposedly, I have a random network for binary classification, right? So its output should vary between 0 and 1.<br>
But no. I figured that since the classification network probably has a softmax operator after the FC layer, then for a single output, I will got constantly 1.<br>
Then I tested my hypothesis with random inputs, and as results I get negative values!</p>
<pre><code class="lang-auto">model(torch.randn(1,3,300,300))
Out[310]: tensor([[-0.0318]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>How can it be?</p>

ptrblck: <p>If you set the number of output using to 1, you should use <code>nn.BCEWithLogitsLoss</code> as your criterion.<br>
Also, your target should have the same shape (<code>[batch_size, 1]</code>), have values in <code>[0, 1]</code>, and be a <code>FloatTensor</code>.</p>
<p>Alternatively, if you would like to stick to <code>nn.CrossEntropyLoss</code>, you should specify <code>out_features=2</code>, and your target should be a <code>LongTensor</code>, containing the class indices in <code>[0, 1]</code>, and have the shape <code>[batch_size]</code>.</p>

M_S: <p>Thanks. Regardless of the training criterion, I am discussion about an untrained network (as I reset the parameters).<br>
Does the call to the FC layer is followed by a softmax? (I didn’t find the model’s source code)<br>
If so, how come it results to a negative number?</p>

ptrblck: <p>No, the model returns the logits (<a href="https://github.com/pytorch/vision/blob/2b6da28c15b7b5ac6fc00b11a28908b5342a99a6/torchvision/models/shufflenetv2.py#L130" rel="nofollow noopener">line of code</a>).</p>

M_S: <p>I see. Thanks!<br>
So I’ll try to add a Sigmoid layer at the end (to get a probability of belonging to class 1, which then I can compare it to my ground truth class labels with values <code>[0, 1]</code>), and I’ll train using a BCELoss.</p>