akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Inconsistent results with cross_entropy

username: erickrf
cooked: <p>When running <code>cross_entropy</code> with indices to ignore and no reduction, I get inconsistent results in the ignored positions. For example:</p>
<pre><code class="lang-auto">In [547]: scores
Out[547]: 
tensor([[0.6453, 0.0335, 0.8880, 0.7096, 0.6212],
        [0.3379, 0.0559, 0.8633, 0.3935, 0.2379],
        [0.8813, 0.2753, 0.9798, 0.4739, 0.1096],
        [0.3567, 0.8076, 0.1867, 0.1396, 0.0971],
        [0.6270, 0.1037, 0.1204, 0.2856, 0.8055],
        [0.4446, 0.0313, 0.6597, 0.9088, 0.3563],
        [0.6929, 0.4242, 0.9795, 0.7334, 0.0760],
        [0.2506, 0.3346, 0.2624, 0.8448, 0.4569],
        [0.9056, 0.2093, 0.7258, 0.8028, 0.3810],
        [0.4425, 0.4331, 0.1988, 0.1506, 0.1138]])

In [548]: gold
Out[548]: tensor([ 2,  0,  0,  4, -1,  1,  0,  0, -1,  2])

In [549]: F.cross_entropy(scores, gold, ignore_index=-1, reduction='none')
Out[549]: 
tensor([1.3382e+00, 1.6877e+00, 1.3288e+00, 1.8669e+00, 9.9920e-16, 2.1011e+00,
        1.5427e+00, 1.8149e+00, 0.0000e+00, 1.6885e+00])

In [550]: F.cross_entropy(scores, gold, ignore_index=-1, reduction='none')
Out[550]: 
tensor([1.3382e+00, 1.6877e+00, 1.3288e+00, 1.8669e+00, 4.2039e-45, 2.1011e+00,
        1.5427e+00, 1.8149e+00, 0.0000e+00, 1.6885e+00])

In [551]: F.cross_entropy(scores, gold, ignore_index=-1, reduction='none')
Out[551]: 
tensor([1.3382e+00, 1.6877e+00, 1.3288e+00, 1.8669e+00, 2.4612e-28, 2.1011e+00,
        1.5427e+00, 1.8149e+00, 0.0000e+00, 1.6885e+00])

In [552]: F.cross_entropy(scores, gold, ignore_index=-1, reduction='none')
Out[552]: 
tensor([1.3382e+00, 1.6877e+00, 1.3288e+00, 1.8669e+00, 7.0065e-45, 2.1011e+00,
        1.5427e+00, 1.8149e+00, 2.2695e-21, 1.6885e+00])

In [553]: F.cross_entropy(scores, gold, ignore_index=-1, reduction='none')
Out[553]: 
tensor([1.3382e+00, 1.6877e+00, 1.3288e+00, 1.8669e+00, 4.2039e-45, 2.1011e+00,
        1.5427e+00, 1.8149e+00, 0.0000e+00, 1.6885e+00])

In [554]: F.cross_entropy(scores, gold, ignore_index=-1, reduction='none')
Out[554]: 
tensor([1.3382e+00, 1.6877e+00, 1.3288e+00, 1.8669e+00, 8.1065e+16, 2.1011e+00,
        1.5427e+00, 1.8149e+00, 2.3510e-38, 1.6885e+00])
</code></pre>
<p>In the last example, one of the ignored positions had a loss of 8e16!</p>
<p>So, is it working as expected, since these positions are to be ignored and the reduction apparently works fine? Even if it is, I find it very strange not to get always zeros in the ignored positions.</p>

username: ptrblck
cooked: <p>I tried to reproduce this issue, but get the same values for each call:</p>
<pre><code class="lang-python">tensor([1.338268518448e+00, 1.687772154808e+00, 1.328836917877e+00, 1.866955399513e+00,
        0.000000000000e+00, 2.101119518280e+00, 1.542701721191e+00, 1.814857602119e+00,
        0.000000000000e+00, 1.688515305519e+00])
</code></pre>
<p>Also, I set <code>torch.set_printoptions(precision=12, sci_mode=True)</code> to get these outputs.<br>
Which PyTorch version are you using?</p>

username: erickrf
cooked: <p>I was using torch 1.0.1. I tried 1.1.0 now, and it gives consistently the same result.</p>
Use of Dataloader and TensorDataset

username: Geeks_Sid
cooked: <p>Hi all,</p>
<p>So I was wondering the following two things.</p>
<ol>
<li>What is the use of DataLoader?</li>
<li>What is the use of TensorDataset?</li>
</ol>
<p>So the scenario begins where I have my current machine with 128GB of RAM.<br>
Is there a particular reason why I should use TensorDataset or DataLoader instead of converting my complete Dataset into a Tensor?<br>
Wouldn’t this be a considerably faster operation and also cause less usage of I/O inturn giving me speedups?</p>
<p>Am I missing something?<br>
Sounds like a good discussing point of pytorch to me.</p>

username: ptrblck
cooked: <p>Loading the complete dataset into memory might work for you, but won’t certainly work for a lot of use cases, e.g. dealing with 10,000,000 images.</p>
<p>Also, the initial loading of the whole dataset might slow down your iteration speed.<br>
I.e. if you are still experimenting with your code and would like to iterate quickly, waiting several minutes for the data loading just to see a code error might be annoying. Of course this can be avoided by loading only a subset of the data, but would need additional code.</p>
<p>Other benefits, e.g. shuffling and batching, will also be missing and you again would have to implement it manually.</p>
<p>The idea behind the <code>DataLoader</code> is to load your data using multiprocessing (and pinned memory) to asynchronously push your data batch onto the GPU during training so that you can basically hide the data loading time. This is of course the optimal use case and if you are working with a slow HDD, you will most likely notice the data loading time.</p>
<p>Anyway, the choice is of course yours, as PyTorch does not depend on a <code>Dataset</code> or <code>DataLoader</code> usage to work properly.</p>
<p>The <code>TensorDataset</code> is a convenient method to wrap already loaded tensors into a <code>Dataset</code> and e.g. to use a <code>Subset</code> or wrap it in a <code>DataLoader</code>.</p>

username: Geeks_Sid
cooked: <p>Yes, but let’s say my training set is of 7GB and I iterate over it a for 200 epochs, this means I have traversed through my harddisk or SSD for 1.4TB of data. Instead, if I just load this into the ram for 7~10GB(Considering overheads), would I not save myself from that hassle? Also, batchsize and shuffling can also be manually manipulated.<br>
My final question would then become,<br>
'Is this more energy efficient and will it give me a speedup? ’</p>
Why the loss is different even when I use torch.manual_seed(10)

username: Xin_Perry
cooked: <p>In order to get the same result during training, I use torch.manual_seed(10) at the beginning of the main code, but it seems doesn’t work as I expected, because when I run the same training code, the loss image is different .</p>

username: sbelharbi
cooked: <p>Check <a href="https://github.com/pytorch/pytorch/issues/7068" rel="nofollow noopener">this</a><br>
and Pytorch <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">note</a> on the subject.</p>
<p>There are more modules that you need to fix their seed.<br>
You have to fix the seed of any module that may introduce randomness.<br>
I recommend to start with <code>num_workers=0</code> or <code>1</code> to use the main process (for the dataloader). Once you obtain reproducible results, you can move to &gt; 1.</p>
Using resnet152 as a backbone of Faaster RCNN

username: solsol
cooked: <p>Hi there,</p>
<p>For training pretrain detector, I usesd following code using resnet 101 as a backbone:</p>
<p>from torchvision.models.resnet import resnet101<br>
self.features = load_resnet()<br>
self.compress = nn.Sequential(<br>
nn.Conv2d(1024, 256, kernel_size=1),<br>
nn.ReLU(inplace=True),<br>
nn.BatchNorm2d(256),<br>
)<br>
self.roi_fmap = nn.Sequential(<br>
nn.Linear(256 * 7 * 7, 2048),<br>
nn.RELU(inplace=True),<br>
nn.Linear(2048, 2048),<br>
nn.RELU(inplace=True),<br>
nn.Dropout(p=0.05),<br>
)<br>
How can I replace resnet 152 instead of 101?</p>
Use SVD with 2-D input (with batch_size) on GPU

username: Gyaya
cooked: <p>I want to use ‘torch.svd()’ on GPU, but the funcation only accept 2-D input. I have implemented the svd by a for loop, but it’s too slow. In <a href="https://github.com/gchanan/pytorch/wiki/Broadcasting-Notes" rel="nofollow noopener">https://github.com/gchanan/pytorch/wiki/Broadcasting-Notes</a>, it’s stated that “torch.svd” can be used in the same way as <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html" rel="nofollow noopener">numpy.linalg.svd</a>. But it causes an error when I pass a 3-D input to the function. Could you please help me?</p>
JIT tracing and parameter sharing

username: mtnk
cooked: <p>torch.jit.trace doesn’t allow us to trace a module that has shared parameters.<br>
Does someone know the reason?</p>
<p>I guess it aims to prevent the shared parameters from being concurrently updated (and destroyed).<br>
If so, I think parameter sharing is safe when we know the modules that access the parameters are NOT concurrently run.</p>
<p>For example, modules that run in a sequence never update their parameters concurrently.<br>
Could someone tell me if I can use tracing in that case (by removing the detection of shared parameters).</p>
A grid of grid using make_grid

username: vainaijr
cooked: <p>how to make a grid which contains grids, for example 25 images in the training set during meta learning,<br>
how to plot 25 grids</p>
<p>grid___ = make_grid(T.view(out.size(0) * out.size(1), 1, out.size(2), out.size(3)))<br>
print(“out size {}”.format(out.size()))<br>
print(“grid size {}”.format(grid___.size()))</p>
<p>writer.add_image(“Conv2d_fw_1”, grid___, 0, dataformats=‘CHW’)</p>
<p>out size torch.Size([25, 64, 84, 84])<br>
grid size torch.Size([3, 17202, 690])</p>
<p>currently I get one grid</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/32526acdf7618f8568885058c349359520bc98da.png" data-download-href="https://discuss.pytorch.org/uploads/default/32526acdf7618f8568885058c349359520bc98da" title="Screenshot%20(184)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/32526acdf7618f8568885058c349359520bc98da_2_690x388.png" alt="Screenshot%20(184)" data-base62-sha1="7barZPSqAyUkOdBTnHiwiIrKUYO" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/32526acdf7618f8568885058c349359520bc98da_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/32526acdf7618f8568885058c349359520bc98da_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/32526acdf7618f8568885058c349359520bc98da_2_1380x776.png 2x" data-dominant-color="D9D1C9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(184)</span><span class="informations">1920×1080 631 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
PyTorch bindings for Warp-ctc: c10/cuda/CUDAGuard.h: No such file or directory

username: Tzeviya
cooked: <p>I’m trying to install <a href="https://github.com/SeanNaren/warp-ctc" rel="noopener nofollow ugc">warp-CTC</a> for <a href="https://github.com/SeanNaren/deepspeech.pytorch" rel="noopener nofollow ugc">deepSpeech2</a>.</p>
<p>I’m using python 3.7.2,  pytorch 1.1.0, cuda 9.0.</p>
<p>I get the following error:</p>
<blockquote>
<p>src/binding.cpp:10:33: fatal error: c10/cuda/CUDAGuard.h: No such file or directory</p>
</blockquote>
<p>I’d appreciate any direction for solving this, thanks.</p>
Indexing a tensor based on another tensor

username: Ismail_Elezi
cooked: <p>Super basic question, but I am having problems with it.</p>
<p>Essentially I have a tensor called probs with shape (batch_size, classes) let’s say (4, 3), and another tensor called labs with shape (batch_size), let’s say (4). Let’s assume that the values of the tensors are:</p>
<pre><code>a = tensor([[0.1, 0.2, 0.3, 0.2, 0.2],
            [0.4, 0.3, 0.2, 0.1, 0   ],
            [0.1, 0.4, 0.3, 0.1, 0.1],
            [0.2, 0.2, 0.2, 0.1, 0.3]]
</code></pre>
<p>and</p>
<pre><code>labs = tensor([2, 1, 0, 3])
</code></pre>
<p>I want the result to be:</p>
<pre><code>result = tensor([0.3,
                [0.3,
                [0.1, 
                [0.1])
</code></pre>
<p>essentially getting the 2nd (zero indexing) value from the zeroth row, first value from the second row, zeroth value from the second row and the third value from the third row.</p>
<p>Does anyone know what is the best way of achieving this (not using for loops)? I thought that</p>
<pre><code>probs[labs.unsqueeze(1)]
</code></pre>
<p>should do the trick, but apparently not. Also, tried with gather, but no luck so far.</p>

username: xwasco
cooked: <p>you can create an auxiliary vector of indices:<br>
add_idx = torch.tensor([0,5,10,15])</p>
<p>then indices the tensor a in this way:<br>
result = a.flatten()[labs+add_idx]</p>

username: Ismail_Elezi
cooked: <p>After playing a bit with gather, I got something that almost gives me what I need:</p>
<pre><code>result = a.gather(1, labs.unsqueeze(1))
</code></pre>
<p>result has shape (4, 1) instead of (4) though. Obviously this can be solved by using torch.squeeze function, however I am interested to know if there is a more efficient version that achieves this.</p>

username: AreTor
cooked: <p>You can use <code>a[range(4), labs]</code>.</p>
Different results during inference

username: Catatafish
cooked: <p>Hello everyone,</p>
<p>I have a model that does a text generation on character level.</p>
<p>So far, I have been saving and loading the training models like this:</p>
<pre><code class="lang-auto">torch.save(model, 'model')

model = torch.load('model')
model.eval()
</code></pre>
<p>and it has been fine.</p>
<p>Now I am trying to do the same thing using the state_dict because it is more convenient but the text being generated during inference looks like a complete random text without any meaning.</p>
<p>The way I save and load the state dict is:</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), 'model_rnn.pt')

model = RNN(tokens=chars, hidden_size=hidden_size, n_layers=n_layers, dropout_prob=dropout_prob).to(device)
model.load_state_dict(torch.load('model_rnn.pt', map_location='cpu'))
model.eval()
</code></pre>
<p>I know that setting the model for evaluation mode is important because dropout shouldn’t be used during eval and I have that.</p>
<p>Here is some example of the text when I save and load the whole model:</p>
<pre><code class="lang-auto">Ти ми стои верност се појаваш
сам си ми се проклето сега
и пак да стави светот на порти
</code></pre>
<p>and here is an example with state_dict:</p>
<pre><code class="lang-auto">о2Е’лС’кокоЊХсьоTьiпоМьВьоџоЊьоЏХ’к’TP2СДоСьоЊџоЊzп́’z’PоTџz’оT’Э=ДЦДP2K’лоШДzьоTьоШХ’лџокьzџШльС2Сџд
</code></pre>
<p>I know most of you can’t read Cyrillic but still it is easy to see the difference</p>
<p>Does anyone know what might be the problem?</p>
Bckpropagation error through scatter_

username: MasterofPLM
cooked: <p>When I use .scatter_() as:</p>
<pre><code class="lang-auto">cosine = F.linear(F.normalize(input), F.normalize(self.weight))
sine = torch.sqrt(1.0 - torch.pow(cosine, 2))
phi = cosine * self.cos_m - sine * self.sin_m # phi = cos(theta + m)

one_hot = torch.zeros(cosine.size(), device='cuda')
one_hot.scatter_(1, label.view(-1, 1).long(), 1)
output = (one_hot * phi) + ((1.0 - one_hot) * cosine) 
</code></pre>
<p>gradient vanishing problem happens in</p>
<pre><code class="lang-auto">output = (one_hot * phi) + ((1.0 - one_hot) * cosine) 
</code></pre>
<p>so how can I do this correctly?</p>
TypeError: can't assign a str to a torch.LongTensor

username: omarabdelaziz
cooked: <p>Hello i’m trying assign list of strings into tensor but it gives me this error</p>
<pre><code>this is the code
</code></pre>
<pre><code class="lang-auto"># Merge captions (from tuple of 1D tensor to 2D tensor).
lengths = [len(cap) for cap in captions]
    targets = torch.zeros(len(captions), max(lengths)).long()
    for i, cap in enumerate(captions):
        end = lengths[i]
        targets[i, :end] = cap[:end]
</code></pre>

username: omarabdelaziz
cooked: <p>for people who might want the answer I converted the string to it’s unicode and stored it in the tensor<br>
here’s the answer</p>
<pre><code class="lang-auto">targets[i, :end] = torch.from_numpy(np.array(list(map(ord, cap[:end])))).to(torch.long)
</code></pre>
Freezing part of the weights in conv3d

username: isalirezag
cooked: <p>is it possible to freeze part of the weights in the 3d convolution?<br>
I like to freeze the weights for the 2d part and just train the temporal weights.<br>
not sure if this possible or not <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: JuanFMontesinos
cooked: <p>You can decompose a 3d convolution into a (2+1)D convolutions. There is a paper about it called <a href="https://arxiv.org/pdf/1711.11248.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1711.11248.pdf</a>. They claim to have better convergence and closer result to 3d convs.</p>
<p>If you definitely want to freeze weights partially you should realize 3d kernel is just a 3d kernel, how do you split “temporal” and “spatial” it’s learned together.</p>
How to set gradients manually and update weights using that?

username: rajcscw
cooked: <p>I have a situation that I compute the weights manually and want to update the weights using those. Here is what I did:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
param.grad = Variable(grad_tensor)
optimizer.step()
</code></pre>
<p>But the weights are not updated, is there something I’m missing?</p>

username: Separius
cooked: <p>1.Are you 100% sure that there is no difference in param after the step?, maybe you use an adaptive optimizer and the learning rate is almost zero?<br>
2.Are you 100% sure that the optimizer is responsible for this param update?</p>
<p>I wrote this code to test this: (I am using v0.4.0):</p>
<pre><code class="lang-auto">import torch

target = torch.randn(1, 10)
input = torch.randn(1, 20)
network = torch.nn.Linear(20, 10)
optimizer = torch.optim.SGD(network.parameters(), lr=0.01)
optimizer.zero_grad()
loss = ((network(input)-target)**2).sum()
loss.backward()
print(network.weight[0, 0])
optimizer.step()
print(network.weight[0, 0]) #works(it is changed)
network.weight.grad[0, 0] = 100
optimizer.step()
print(n.weight[0, 0]) #works(it is changed)
network.weight.grad = torch.randn(10, 20)
optimizer.step()
print(n.weight[0, 0]) #works(it is changed)
</code></pre>

username: rajcscw
cooked: <p>Thanks for your answer, I was indeed doing the same, though I was using setattr() to set the values dynamically, I found a bug in my logic. It is working fine now!</p>

username: yottabytt
cooked: <p>Do you have any idea of how to make it work for the following scenario?</p>
<pre><code class="lang-auto">import torch

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.l1 = nn.Linear(20, 15)
        self.l2 = nn.Linear(15, 10)
    
    def forward(self, input):
        out = self.l1(input).detach()
        return self.l2(out)

target = torch.randn(1, 10)
input = torch.randn(1, 20)
optimizer = torch.optim.SGD(network.parameters(), lr=0.01)
optimizer.zero_grad()
network = Network()
loss = ((network(input)-target)**2).sum()
loss.backward()
print(network.l1.weight[0, 0])
optimizer.step()
print(network.l1.weight[0, 0]) #doesn't work(it is not changed)
network.l1.weight.grad = torch.randn(15, 20)
optimizer.step()
print(network.l1.weight[0, 0]) #doesn't work(it is not changed)

</code></pre>
.cuda slow on GTX Titan vs. GTX 1080 Ti

username: mittimithai
cooked: <p>EDIT: This isn’t a docker issue, the slowdown is still there when I install outside the container.</p>
<p>Not much of a docker (or pytorch) expert but I have a text-to-speech codebase where I am shoe horning a general transformation (Griffin-Lim) into a torch.nn.Module. In one machine, runs on the order of a second (ignoring startup).  This code has been put into a docker image on a host with faster CPU and GPUs and I get an immense slowdown (1s to 18s).</p>
<p>I’ve profiled, and if one compares appropriately (a little bothered by why the number of calls isn’t the exact same), it looks like the culprit is the calls to the .cuda() method (3s on fast machine total vs. 21s on slow machine, pasted at end).</p>
<p>I’m currently using .cuda() as opposed to .to(“cuda”/device).  Is there something obviously idiotic I am doing that would result in such a slowdown in the docker image? The python versions between the machines differ slightly (fast machine = 3.6.3, slow machine = 3.6.7) but profile environment summary matches:</p>
<hr>
<h2>Environment Summary</h2>
<p>PyTorch 0.4.1 compiled w/ CUDA 9.0.176<br>
Running with Python 3.6 and CUDA 9.0.176</p>
<h2>Fast machine:</h2>
<h2>cProfile output</h2>
<pre><code>     1299411 function calls (1275111 primitive calls) in 17.041 seconds
</code></pre>
<p>Ordered by: internal time<br>
List reduced from 7173 to 15 due to restriction &lt;15&gt;</p>
<p>ncalls  tottime  percall  cumtime  percall filename:lineno(function)<br>
1    9.531    9.531    9.531    9.531 /lib/python3.6/site-pa<br>
ckages/numpy/linalg/linalg.py:1299(svd)<br>
324    3.486    0.011    3.486    0.011 {method ‘cuda’ of ‘torch._C._TensorBase’ objects}<br>
…</p>
<h2>Slow machine:</h2>
<h2>cProfile output</h2>
<pre><code>     1311902 function calls (1287710 primitive calls) in 28.400 seconds
</code></pre>
<p>Ordered by: internal time<br>
List reduced from 7152 to 15 due to restriction &lt;15&gt;</p>
<p>ncalls  tottime  percall  cumtime  percall filename:lineno(function)<br>
385   21.639    0.056   21.639    0.056 {method ‘cuda’ of ‘torch._C._TensorBase’ objects}<br>
1    3.791    3.791    3.791    3.791 /usr/local/lib/python3.6/dist-packages/numpy/linalg/linalg.py:1299(svd)</p>
<p>…</p>

username: mittimithai
cooked: <p>So the performance gap persists when I compare outside the docker container, I didn’t realize that the difference in the hardware was the issue. The Titan is much older than the 1080 Ti.</p>
<p>GeForce GTX 1080 Ti<br>
ncalls  tottime  percall  cumtime  percall filename:lineno(function)<br>
324    8.589    0.027    8.589    0.027 {method ‘cuda’ of ‘torch._C._TensorBase’ objects}</p>
<p>GeForce GTX TITAN<br>
ncalls  tottime  percall  cumtime  percall filename:lineno(function)<br>
324   21.860    0.067   21.860    0.067 {method ‘cuda’ of ‘torch._C._TensorBase’</p>
ATen create tensor of the same type/options

username: dashesy
cooked: <p>What is the difference between <code>.options()</code> and <code>.type()</code>:</p>
<p>Lets say</p>
<pre><code class="lang-cpp">    std::vector&lt;at::Tensor&gt; forward(at::Tensor xy, int height, int width) {
        auto tmp  = at::empty({10, 4, height, width}, xy.type());
        return {tmp}
    }
</code></pre>
<p>v.s.</p>
<pre><code class="lang-cpp">    std::vector&lt;at::Tensor&gt; forward(at::Tensor xy, int height, int width) {
        auto tmp  = at::empty({10, 4, height, width}, xy.options());
        return {tmp}
    }
</code></pre>
<p>Would the latter include not only CUDA/Cpu but also the device that xy is on? in that case which device will be used in the first case?</p>

username: colesbury
cooked: <pre><code class="lang-auto">Would the latter include not only CUDA/Cpu but also the device that xy is on?
</code></pre>
<p>Yes</p>
<pre><code class="lang-auto">in that case which device will be used in the first case?
</code></pre>
<p>The current CUDA device (if xy is a CUDA tensor). You can set the device with <code>DeviceGuard</code> in C++ or <code>torch.cuda.set_device</code> in Python.</p>

username: dashesy
cooked: <p>Thanks!</p>
<p>One more question:<br>
If I want to create a tensor with the same options but with type int, is there something like</p>
<pre><code class="lang-cpp">    std::vector&lt;at::Tensor&gt; forward(at::Tensor xy, int height, int width) {
        auto tmp  = at::empty({10, 4, height, width}, xy.options().as_int());
        return {tmp}
    }
</code></pre>

username: dashesy
cooked: <p>This is the solution</p>
<pre><code class="lang-auto">    std::vector&lt;at::Tensor&gt; forward(at::Tensor xy, int height, int width) {
        auto tmp  = at::empty({10, 4, height, width}, xy.options().dtype(at::kInt));
        return {tmp}
    }
</code></pre>
Running backward with different batch sizes

username: afshin67
cooked: <p>I use an instance of <code>torch::nn::sequential</code> to build a model. In the train step, I pass batch with different size into the model in different steps, as it is usual in most of the on-policy RL algorithm in family of actor-critic algorithms. This is fine as long as I use <code>pytorch</code>. Now, I am developing same algorithm with <code>libtorch</code>. In the first iteration I pass batch of size 24, and when in the second iteration I pass batch of size 13, I get:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
Signal: SIGABRT (Aborted)

Process finished with exit code 1
</code></pre>
<p>I checked the <code>backward</code> API and it does not have <code>retain_graph</code> (which is available in pytorch API); although it has <code>bool keep_graph</code>. However, I cannot use that too since it is the second input parameter and C++ does not allow to set only the second input without the first one, which is the <code>gradient</code> tensor.<br>
Beside, I am not sure if setting <code>keep_graph=true</code> helps to solve this problem.<br>
I appreciate any help or comment.</p>
Applying Function to Specific Tensor Elements with GPU Parallelization

username: andrerosete
cooked: <p>Hello, I have a set of values in a tensor and I want to apply a function that takes in several values and returns a value based on those. Basically, multiple elements as input to each output tensor element. I have attached an image of what I am trying to do. I want to take advantage of GPU parallelization as much as possible.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3a22c2a2c6a0b60487a3dfbdab693e42ef27ddb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a22c2a2c6a0b60487a3dfbdab693e42ef27ddb4" title="intention"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3a22c2a2c6a0b60487a3dfbdab693e42ef27ddb4.png" alt="intention" data-base62-sha1="8iialtxlJ1fZSg5c0PtcTzteTFG" width="463" height="500" data-dominant-color="ECEBE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">intention</span><span class="informations">705×761 5.48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Initialize specific index of a Parameter to zero with mask

username: mahsa
cooked: <p>I have a parameter named P with size n*n and I want to initialize the parameter with positive values. On the other hand, I have a mask which shows which indices of P are parameters and which indices are not.<br>
I have used the following code but I’ve got an error:</p>
<pre><code class="lang-auto">self.P.data.uniform_(0, stdv)
self.P = self.P * self.mask
</code></pre>
<p>Is there any way I can do it?</p>
Loss.backward() not updating model gradients

username: shlady
cooked: <p>I’m trying to use a pretrained keypoint detector model from torchvision to predict keypoints on a region extracted from an image. I’d then like to visualize a saliency map of the prediction. I have a loss function that is just the Euclidean distance of each predicted set of keypoints with the closest ground truth set. However, when I call loss.backward(), the printing model.layer.[weight/bias].grad yields nothing. I was trying this to make sure gradients were getting back to the input image. I’ve included code.</p>
<ul>
<li>dets_for_kp is a list of Tensors (CxHxW) that have requires_grad=True.</li>
<li>kp is the name of the model. It is a keypointrcnn_resnet50_fpn from torchvision models and is set to eval().</li>
</ul>
<pre><code class="lang-auto">for p in kp.parameters():
    p.requires_grad = True
    # run detections through keypoint detector
    kps = kp(dets_for_kp)

    # we could come up with a single keypoint estimate per image by either averaging
    # or a linear combination weighted by scores
    best = []
    valids = []
    for j, k in enumerate(kps):
        keypoints = k['keypoints']

        if len(keypoints) == 0:
            continue

        best.append(torch.round(keypoints[0]))

        valids.append(j)

    # convert keypoints back to image coordinates. keypoints are (x,y,visible)
    for i in range(len(valids)):
        j = valids[i]
        bbox, ph, pw = reverse_info[j]

        best[i][:,0] += (bbox[0] - pw[0])
        best[i][:,1] += (bbox[1] - ph[0])

    # loop over keypoints and compare them to the ground truth
    # since we don't know a priori which gt and which annotations are closest, and there aren't
    # too many, just loop over each to get centroids and use closest centroid of gt for each 
    # annotation. loss is L2
    anno_centroids = []
    for a in annos:
        x = np.array(a)[0::3].mean()
        y = np.array(a)[1::3].mean()
        z = np.array(a)[2::3].mean()
        anno_centroids.append(np.array([x,y,z]))

    k_centroids = []
    for b in best:
        centroid = b.mean(0)
        k_centroids.append(centroid)

    c_matches = []
    for jj, c in enumerate(k_centroids):
        closest_dist = 1e10
        closest_centroid = None

        for ii, ac in enumerate(anno_centroids):
            if torch.norm(c - torch.Tensor(ac)) &lt; closest_dist:
                closest_dist = torch.norm(c-torch.Tensor(ac))
                closest_centroid = ii

        c_matches.append((jj, ii))

    loss = torch.Tensor([0])
    for m in c_matches:
        loss += torch.norm(best[m[0]].view(-1) - torch.Tensor(annos[m[1]]))

    loss.backward()
</code></pre>
<p>You can check that the gradients are nothing by printing</p>
<pre><code class="lang-auto">kp.roi_heads.keypoint_predictor.kps_score_lowres.bias.grad
</code></pre>
<p>or</p>
<pre><code class="lang-auto">dets_for_kp[0].grad
</code></pre>
<p>I’ve tried making sure every parameter is set to require_grad=True, along with the input. I’ve wondered if I am doing the loss wrong. I tried using the MSELoss instead but had no luck. What am I doing wrong?</p>
Traning procedure stucked at the end of one epoch

username: ken_zhang
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0d85e024eaeb12ec58966a34a256bfa001f606e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d85e024eaeb12ec58966a34a256bfa001f606e2" title=""><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0d85e024eaeb12ec58966a34a256bfa001f606e2_2_690x347.png" alt="" data-base62-sha1="1VD27pOJRdiwJPnMQmgfJh9Sxk6" role="presentation" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0d85e024eaeb12ec58966a34a256bfa001f606e2_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0d85e024eaeb12ec58966a34a256bfa001f606e2_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0d85e024eaeb12ec58966a34a256bfa001f606e2.png 2x" data-dominant-color="0C0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">1270×640 96.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This error will happen on pytorch version greater than 0.4.1. I followed this <a href="https://github.com/pytorch/pytorch/issues/19457" rel="noopener nofollow ugc">instruction</a> to build 0.4.1 from source. Weirdly, there is no problem with the pip installed 0.4.1 version.</p>
<p>My pytorch version:</p>
<pre><code class="lang-auto">In [1]: import torch

In [2]: torch.__version__
Out[2]: '0.5.0a0+a24163a'
</code></pre>
<p>My code is heavily borrowed from <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="noopener nofollow ugc">pytorch-CycleGAN-and-pix2pix</a>. I use <code>torch.nn.DataParallel</code> to warp the module.</p>
<pre data-code-wrap="python"><code class="lang-python">    # create dataset
    # create dataloader
    # create model
    for epoch in range(1, opt.niter):
        for i, data in enumerate(dataset):
            ...
            model.set_input(data)
            model.optimize_parameters()
            ...
        model.save(...)
</code></pre>
<p>I have tried different dataloader implemention.</p>
<ol>
<li>
<p><a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/data/__init__.py#L62-L93" rel="noopener nofollow ugc">pytorch-CycleGAN-and-pix2pix implemention</a></p>
</li>
<li>
<p>vanilla version</p>
</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">def create_dataloader(dataset_opt):
    dataset_class = find_dataset_using_name(dataset_opt.dataset_mode)
    dataset = dataset_class(dataset_opt)
    print("dataset [%s] was created" % type(dataset).__name__)
    return torch.utils.data.DataLoader(
        dataset,
        batch_size=dataset_opt.batchSize,
        shuffle=dataset_opt.shuffle_data,
        num_workers=dataset_opt.nThreads,
        sampler=None,
        drop_last=True,
        pin_memory=False), len(dataset)
</code></pre>
LSTM Hidden State Changing Dimensions Error

username: Adrix
cooked: <p>I seem to be having an odd issue when using and LSTM. I pass in an initial hidden state of size (3,1,3) but pytorch throws an error saying it is only (1,3) and expected (3,1,3) even though I can verify before passing it in that the Tensor is (3,1,3).</p>
<pre><code class="lang-auto">class AdRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(AdRNN, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.rnn = nn.LSTM(input_size=input_size, num_layers=num_layers, hidden_size=hidden_size, batch_first=True)

    def forward(self, x):
        h_0 = torch.rand(self.num_layers, x.size(0), self.hidden_size)
        nn.init.xavier_normal_(h_0)

        print(h_0.shape)
        
        output, _ = self.rnn(x, h_0)
        return output
</code></pre>
<p>Note that I am printing my h_0 shape for debugging. It comes back as (3,1,3) but Pytorch raises the following error in the forward method when passing in h_0 to the LSTM:</p>
<pre><code class="lang-auto">RuntimeError: Expected hidden[0] size (3, 1, 3), got (1, 3)
</code></pre>
<p>If I change the rnn type to GRU or vanilla RNN in init everything works just fine but LSTM is being cranky. I am using using PyTorch 0.4.1. I can of course just not pass in h_0 and have the LSTM hidden init to zeros but I would rather not. Any thoughts?</p>

username: richard
cooked: <p>LSTM takes a tuple of hidden states:</p>
<p><code>self.rnn(x, (h_0, c_0))</code></p>
<p>it looks like you haven’t sent in the second hidden state?</p>

username: Adrix
cooked: <p>Aha! I didn’t realize the argument was a tuple for LSTM, I was thinking it was (x, h_0, c_0). So you have to provide either both the hidden and the cell state or none at all then (or I suppose pass in a zero tensor for one yourself if you want to just initialize the other).</p>
<p>That did it, thank you!</p>

username: richard
cooked: <p>Maybe we should have a better API for this, you’re right in that there is no way to just pass one state and not initialize the other one.</p>

username: Shandilya21
cooked: <p>just pass the (h_0, c_0) in your rnn cell :<br>
input_h, (input_h_t, input_c_t) = self.rnn(input_emb, (h_0, c_0))<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Difference between eval(), torch.no_grad() and with torch.set_grad_enabled()

username: n0obcoder
cooked: <p>i understood that</p>
<ul>
<li>eval() changes the bn and dropout layer’s behaviour</li>
<li>torch.no_grad() deals with the autograd engine and stops it from calculating the gradients, which is not compulsory but the recommended way of doing validation</li>
</ul>
<p>BUT, I didn’t understand the use of  <strong>with torch.set_grad_enabled()</strong></p>
<p>Can somebody pls explain what is its use and where exactly can it be used.</p>

username: n0obcoder
cooked: <aside class="quote" data-post="29" data-topic="19615">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n0obcoder/48/16606_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615/29">'model.eval()' vs 'with torch.no_grad()'</a> 
  </div>
  <blockquote>
    are you saying that torch.no_grad  and torch.set_grad_enabled(False) are the same ?
  </blockquote>
</aside>
Floating point handling in pytorch

username: Varkha_Agrawal
cooked: <p>Hi Team,</p>
<p>I am searching logic to handled floating point by using some mathematical manipulations in pytorch,<br>
below showing the example that i do manually for simple operation :<br>
<span class="hashtag">#define</span> FP_MUL(X,Y)     (((X)*(Y))&gt;&gt;8)<br>
<span class="hashtag">#define</span> FP_DIV(X,Y)     (((X)&lt;&lt;8)/(Y))</p>
<p>I need logic to handled deep learning functionality in embedded device where no floating point unit present. Such as how to handle activation function without floating point in hardware?</p>
Newbie question about autograd

username: kekday
cooked: <p>What kind of operations can I perform on tensors while still being able to compute gradients? For instance when I try</p>
<p>x = torch.rand(1, requires_grad=True)<br>
y = x+2<br>
print(y.grad_fn)</p>
<p>I get &lt;AddBackward0 object at 0x7fd459552320&gt;</p>
<p>but when I try something like</p>
<p>x = torch.rand(1, requires_grad=True)<br>
y = torch.tensor([x , 0])<br>
print(y.grad_fn)</p>
<p>I get None.</p>

username: ptrblck
cooked: <p>Recreating a <code>tensor</code> will break the computation graph.<br>
If you would like to concatenate two tensors, you should use <code>torch.cat</code> or <code>torch.stack</code> instead:</p>
<pre><code class="lang-python">x = torch.rand(1, requires_grad=True)
y = torch.cat([x , torch.tensor([0.])])
print(y.grad_fn)
&lt;CatBackward object at 0x7fdb08dca6d8&gt;
</code></pre>
PyTorch/Torch projects components map and history

username: Dawid_S
cooked: <p>It is hard for me to understand all the components, structure, and the history of PyTorch project.</p>
<p>I know there is Torch library that works with Lua. Somewhere there, there’s C part of the code, that works with CUDA (but CUDA is written in C++, so how it is?). Suddenly some people took some part of these things and created PyTorch. There is also something like ATen that seems to be important and is somehow rewritten part of Torch (torch7?) to C++</p>
<p>I can’t get all these things. Is there any place or description of all the dependencies, components, and history of the development? I am especially interested in Pytorch, Torch, torch7, ATen, &lt;TH/TH.h&gt; &lt;torch/torch.h&gt;, CUDA, CuDNN.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The initial torch7 (which is what you call Torch) library was written in lua and based on a pure C library called TH for cpu computation and cpp library called THC for gpu computations (this include all the cuda/cudnn dependancies). There was also THNN and THCNN for neural network specific ops. These C/cpp libraries are the one providing <code>TH/TH.h</code> or <code>THC/THC.h</code> (that should only be used internally by the way).</p>
<p>When pytorch was created, it was initially only a python wrapper + new autograd engine + TH, THC, THNN, THCUNN.</p>
<p>Given the complexity of all the interfacing with TH* libraries, Aten was created to provide a nice cpp interface to all of these. Note that now, new functions are implemented directly into Aten and not the TH* libraries.</p>
<p>More recently, to be able to do cpp-only tasks, libtorch has been introduced as a replacement for the python wrapper. It is a pure cpp library that replaces it. And it’s main include file is <code>torch/torch.h</code>.</p>
<p>Hope this helps, let me know if I missed anything!</p>

username: Dawid_S
cooked: <p>I have two more questions. What has ONNX to do with it and how is it related to torch JIT?</p>
Constraint optimization with torch.distributions.constraints.Constraint

username: Linchuan_XU
cooked: <p>Hi, I am new to pytorch.<br>
I have a need for constraint optimization, e.g., non-negativity of nn parameters.<br>
I have searched for solutions and found that most of you suggested manually clipping the values to be larger than zero.</p>
<p>By the way, I find that there is a constraint class torch.distributions.constraints.Constraint.<br>
I am wondering whether and how this class is related to the constraint optimization.<br>
Any help is appreciated!</p>
Broken pipe error ( Execution Error?)

username: lzozo95
cooked: <pre><code class="lang-auto">import txt_file_maker
import dataset_maker
import YOLO
import torch
from torch.utils.data import DataLoader
import time

image_paths = ['./datasets/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007', './datasets/VOCtrainval_11-May-2012/VOCdevkit/VOC2012', './datasets/VOCtest_06-Nov-2007/VOCdevkit/VOC2007', './datasets/VOC2012test/VOCdevkit/VOC2012']
txt_paths = ["./datasets/txt/voc2007trainval.txt", "./datasets/txt/voc2012trainval.txt", "./datasets/txt/voc2007test.txt", "./datasets/txt/voc2012test.txt"]

txt_make = 0 # If this is the first run, you should change it to 1 to make txt file to be used for making dataset

if txt_make == 1:
    print('txt file making start')
    for i in range(len(image_paths)):
        txt_file_maker.maker(image_path=image_paths[i], txt_path=txt_paths[i])
else: print("txt file already made")

print('txt file making complete!')

# YoloDataset(root=file_root, list_file='./codes/lab8/Generate_dataset/voc2007+2012.txt', train=True)

print('dataset making start!')

VOC2007_trainset = dataset_maker.YoloDataset(root=image_paths[0], list_file=txt_paths[0], train=True)
train_loader = DataLoader(VOC2007_trainset, batch_size=32, shuffle=True, num_workers=4)

print('dataset making complete!')

model = YOLO.yolo()

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

learning_rate = 1e-4
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=5e-4)
#optimizer.load_state_dict(state_dict['optimizer'])

print('pre_train_complete')
criterion = YOLO.yoloLoss(l_coord=5, l_noobj=0.5)

best = 1e+30
num_epochs = 10
print("training start!")

loss_per_epoch = 0

for i in range(num_epochs):
    model.train()
    start = time.time()
    for j, (img, target) in enumerate(train_loader, 0):
        img = img.to(device)
        target = target.to(device)

        optimizer.zero_grad()
        output = model.forward(img)
        loss = criterion.forward(output, target)
        loss_per_epoch += loss

        loss.backward()

        optimizer.step()
    print("Epoch {} loss : {} , Elapsed time : {}".format(i, loss_per_epoch / j, time.time() - start))
    loss_per_epoch = 0
</code></pre>
<p>If I run the code, there is broken pipe error.</p>
<p>I printed out the processing steps as like in above code.</p>
<p>txt file already made<br>
txt file making complete!<br>
dataset making start!<br>
dataset making complete!<br>
pre_train_complete<br>
training start!<br>
txt file already made<br>
txt file making complete!<br>
dataset making start!<br>
dataset making complete!<br>
pre_train_complete<br>
training start!</p>
<p>However, I only can figure out that it runs the code twice. I don’t know why.</p>
What are some good test cases for pytorch deep learning model and why?

username: highDopamine
cooked: <p>Hi all, I’m a beginner to Pytorch and Deep Learning. For me, testing the deep learning algorithms to make sure it works as intended is a very important step.<br>
So I want to create this topic for people to share their test cases (why use it and how to use it).</p>

username: Tony-Y
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/examples" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/examples" target="_blank" rel="nofollow noopener">pytorch/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>First, try MNIST.</p>
<p><a href="https://nextjournal.com/gkoehler/pytorch-mnist" rel="nofollow noopener">MNIST Handwritten Digit Recognition in PyTorch</a><br>
This article is helpful for beginners.</p>
Need help to explain difference between my BN and torch batchnorm2d during eval phase

username: KarenY323
cooked: <p>I want to optimize my model inference performance by combining the batch normalization layer into the conv2d layer during the eval phase. However, all my results are off by some vale range from 0.051 ~0.072 when compared to regular CONV+BN forward path.</p>
<p>My current approach is to update the CONV layer weights and bias using the pretrained model parameters from the BN layer, as show in the equations below:</p>
<ul>
<li>var_sqrt = torch.sqrt(BN_running_var + 1e-5)</li>
<li>w = w * (BN_weight / var_sqrt).reshape(dim)</li>
<li>b = (b - BN_running_mean) / var_sqrt * BN_weight  + BN_bias</li>
</ul>
<p>For example, I have some sample results from the first CONV/BN layer(s):<br>
Then:</p>
<ul>
<li>
<p>CONVlayer + torch batchnorm2d : [-0.1672, -0.1658, -0.1661,  …,  0.3378,  0.3378,  0.3378]</p>
</li>
<li>
<p>My combined CONV_BN layer: [-0.2186, -0.2173, -0.2176,  …,  0.2669,  0.2669,  0.2669]</p>
</li>
</ul>
<p><strong>As a mathematical sanity check,</strong>  I recorded the output of the first conv layer:</p>
<p>[1.4406e+02,  1.4436e+02,  1.4429e+02,  …,  2.5261e+02,  2.5261e+02,  2.5261e+02]</p>
<p>And using just the first value of the model 1.4406e+02, and find related values for batch normalization:</p>
<ol>
<li>model['bn1.weight’] : 0.3468</li>
<li>model['bn1.bias’]: -0.3069</li>
<li>model['bn1.running_mean’]: 124.3252</li>
<li>torch.sqrt(model['bn1.running_var’]+1e-05):77.5449</li>
</ol>
<p>I used the formula form torch batchnorm2d doc: <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a></p>
<pre><code>     Y = (144.06 - 124.3252) / 77.5449 * 0.3468 - 0.3069 =  **-0.21864 ,**  which matches my batch norm implementation, but is obviously different from pytorch’s  **batchnorm2D value of -0.1672.**
</code></pre>
<p>Note: I searched online and see some discussion. There is one related to BN difference between training and eval: <a href="https://www.google.com/url?q=https://github.com/pytorch/pytorch/issues/19902&amp;sa=D&amp;source=hangouts&amp;ust=1559607818373000&amp;usg=AFQjCNELQGNZSIIATVO28hU7CLiavnGTSA" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/19902</a>, but my case is on eval.</p>
<p>Really appreciate if you can help to explain the difference as the it is &gt; 1e-4.</p>
Guess I cannot run parallel synchronization with my Video card

username: pythonprojects2017
cooked: <p>Newbie Question:<br>
torch.cuda.synchronize()                              # check for cuda<br>
E:\ProgramData\Anaconda3\lib\site-packages\torch\cuda_<em>init</em>_.py:118: UserWarning:<br>
Found GPU0 GeForce GTX 650 which is of cuda capability 3.0.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability that we support is 3.5.<br>
warnings.warn(old_gpu_warn % (d, name, major, capability[1]))</p>
<p>When I install the latest CUDA supported card, will I have to reinstall all libraries and Anaconda/Spyder etc?</p>