akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: after reduction step 2: device-side assert triggered Pytorch on Jupyter Notebook

Flock1: <p>I am training a binary classifier using Pytorch on Jupyter Notebook. The following is the architecture:</p>
<pre><code class="lang-auto">class AlexNet(nn.Module):
    def __init__(self, num_classes=1):
        super(AlexNet, self).__init__()
        self.conv_base = nn.Sequential(
            nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=2, bias=False),
            nn.BatchNorm2d(96),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(96, 256, kernel_size=5, stride=1, padding=2, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(256, 384, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),

            nn.Conv2d(384, 384, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),

            nn.Conv2d(384, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.fc_base = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256*6*6, 4096),
            nn.ReLU(inplace=True),

            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),

            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.conv_base(x)
        x = x.view(x.size(0), 256*6*6)
        x = self.fc_base(x)
        return x
</code></pre>
<p>These are my parameters for training:</p>
<pre><code class="lang-auto">criterion = nn.BCELoss()

# specify optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=2e-5, momentum=0.9)
</code></pre>
<p>When I start the training, I get the following error:</p>
<blockquote>
<p>RuntimeError: after reduction step 2: device-side assert triggered</p>
</blockquote>
<p>Some weird things are also happening:</p>
<ol>
<li>
<p>It was training at one point in time, but the learning rate was small. So when I changed it and started the training process again, I got the error.</p>
</li>
<li>
<p>I’m also getting this error when I run this command:</p>
</li>
</ol>
<pre><code class="lang-auto">model = AlexNet()
model.cuda()
</code></pre>
<p>the error:</p>
<blockquote>
<p>RuntimeError: CUDA error: device-side assert triggered</p>
</blockquote>
<p>Some solutions that I found which worked temporarily are:</p>
<ol>
<li>Add  <code>os.environ['CUDA_LAUNCH_BLOCKING'] = '1'</code>  to the notebook. 2) GPU memory being used. So I removed the process that was running the notebook.</li>
</ol>
<p>However, nothing seems to be working permanently. Any suggestions?</p>
Understanding time taken from moving data from GPU to CPU

badesairam: <h2>
<img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=9" title=":question:" class="emoji" alt=":question:"> I am trying to check the time taken by two different models(ELMO vs BERT) while predicting in named entity recognition ,  It involves getting logits data from GPU to CPU which is taking different times with different models.</h2>
<p>this code snippet is same for both the models</p>
<pre><code class="lang-auto">       logits = torch.argmax(F.log_softmax(logits,dim=2),dim=2)
		batch_start_time  = time.time()
		logits = logits.detach().cpu().numpy() 
		print(logits.shape) 
		pred_batch = []
		print("Batch time before loop :: %s seconds"%(time.time()-batch_start_time)

</code></pre>
<p>output for model ELMO::</p>
<pre><code class="lang-auto">(32, 128)

Batch time before loop :: 0.0035903453826904297 seconds
</code></pre>
<p>output for model BERT::</p>
<pre><code class="lang-auto">(32, 128)

Batch time before loop :: 0.1682131290435791 seconds
</code></pre>
<p>why is this difference and are there any other factors that influence this time.</p>

ptrblck: <p>CUDA operations are called asynchronously, so you should synchronize the code before starting and stopping the timer using <code>torch.cuda.synchronize()</code>.</p>
<p>In your current code snippet <code>logits = logits.detach().cpu().numpy()</code> will create a synchronization point, so that your code will wait at this line of code for all preceding operations to finish (which also might include the forward pass of your models) while the timer was already started.</p>
Weird forward hook input shape

Stihanssen: <p>I have been using Module.register_forward_hook and Module.register_forward_pre_hook. I looked at the shape of the data I got from them and there is something I can’t figure out.</p>
<p>Say the shape of some data going into a module is: (2, 3, 256, 256), if I print the shape of the input data that I got from the hook I get: (1, 2, 3, 256, 256). The input data always comes in the format tuple(&lt;Tensor with data&gt;). I haven’t found a case where the tuple contain more than one entry.</p>
<p>Interestingly, this is not the case for output data. That I found to be in the format &lt;Tensor with data&gt;.</p>
<p>What is the purpose of the tuple?</p>
Data to device is quite slow

auroua: <p>I run the same code using two different device <code>rtx 2080ti</code> and <code>titan xp</code>.<br>
The time  cost of <code>data.to(device)</code> using <code>rtx 2080ti</code> is 0.1935, but using <code>titan xp</code>  is  1.0735. Why such huge difference?</p>

auroua: <p>Dose this caused by the <code>pci-e</code>? The RTX 2080 Ti work in PCI-E x16, but the Titan Xp work in PCI-E x8. When the data batch is small, there is no obvious difference.</p>
Loading pre trained model (vgg16) - RuntimeError: storage has wrong size: expected 0 got 512

danielafrimi: <p>Hi,<br>
When i load the model via my pc it works, but when i run it via the GPU cluster it fails.<br>
this is the code:</p>
<pre><code class="lang-auto">class _netVGGFeatures(nn.Module):
    def __init__(self):
        super(_netVGGFeatures, self).__init__()
        self.vggnet = models.vgg16(pretrained=True).cuda() #FAIL HERE

</code></pre>
<p>this is the error i get:</p>
<pre><code class="lang-auto">    self.vggnet = models.vgg16(pretrained=True).cuda() #Load the pre trained VGG16 model from pytorch on GPU
  File "/cs/usr/daniel023/.local/lib/python3.6/site-packages/torchvision/models/vgg.py", line 152, in vgg16
    model.load_state_dict(model_zoo.load_url(model_urls['vgg16']))
  File "/cs/usr/daniel023/.local/lib/python3.6/site-packages/torch/utils/model_zoo.py", line 67, in load_url
    return torch.load(cached_file, map_location=map_location)
  File "/cs/usr/daniel023/.local/lib/python3.6/site-packages/torch/serialization.py", line 368, in load
    return _load(f, map_location, pickle_module)
  File "/cs/usr/daniel023/.local/lib/python3.6/site-packages/torch/serialization.py", line 549, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 0 got 512
</code></pre>
<p>Thanks for any help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

InnovArul: <p>Can you verify if the torchvision version is same in your PC and GPU server?<br>
You can also verify if <code>torchvision</code> is loading the same version of <code>vgg</code> model. The pretrained models usually be downloaded to <code>~/.torch/models</code>.</p>
Question about MSELoss

ljj7975: <p>Here I have very simple test code</p>
<pre><code class="lang-auto">target = Variable(torch.tensor([
    [1, 0],
    [0, 1],
]).float(), requires_grad=False)

output = Variable(torch.tensor([
    [0, 1],
    [0, 1],
]).float(), requires_grad=True)

# loss = nn.L1Loss()(output, target)
loss = nn.MSELoss()(output, target)
print('LOSS\n', loss)

print(type(loss.grad_fn))

grad = loss.grad_fn(torch.tensor(1).float())
print('GRAD\n', grad)
</code></pre>
<p>where the output I got is</p>
<pre><code class="lang-auto">LOSS
 tensor(0.5000, grad_fn=&lt;MseLossBackward&gt;)
&lt;class 'MseLossBackward'&gt;
GRAD
 tensor([[-0.5000,  0.5000],
        [ 0.0000,  0.0000]], grad_fn=&lt;MseLossBackwardBackward&gt;)
</code></pre>
<p>Based on documentation (<a href="https://pytorch.org/docs/stable/nn.html#mseloss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#mseloss</a>) and its gradients definitions (<a href="https://stats.stackexchange.com/a/312997" rel="nofollow noopener">https://stats.stackexchange.com/a/312997</a>),<br>
I thought the right value for loss is 1 because sum of squared difference is 2 and N is 2. However it’s 0.5.<br>
Similarly, I was expecting to see [[-2, 2] [0,0]] for gradients but it is different.</p>
<p>I suspect that due to the averaging operation, <code>grad_fn</code> does some rescaling based on the gradient I provide (1 in this case). I want to understand that logic as well.</p>
<p>Does anyone know details about these Loss function implementation?<br>
Thank you</p>

acobobby: <p>For what concerns the loss value I believe the printed value (0.5) is correct.<br>
In fact if you calculate the loss <code>(output[i]-target[i])^2</code> element by element you obtain a tensor with values <code>1, 1, 0, 0</code> (read row-wise).<br>
So when you sum up you obtain <code>2</code> and you have to divide by the total number of elements (<code>4</code>) because you choose implicitly <code>reduction='mean'</code> when you create the <code>MSELoss</code> class. So in the end you obtain <code>0.5</code>.</p>
<p>I do not understand very well the meaning of the last line of your code:</p>
<blockquote>
<p>grad = loss.grad_fn(torch.tensor(1).float())</p>
</blockquote>

ljj7975: <p>Thank you for your response.<br>
Ya I suspected that its number of elements but the document say that it is batch size so I got confused.</p>
<pre><code class="lang-auto">grad = loss.grad_fn(torch.tensor(1).float())
</code></pre>
<p>This seems to be computing gradient by calling Backward and multiply the value I provide (which is 1 in this example).</p>
Weight replace manually

779d2bdf502fa0c323b7: <p>Hi, I wanted to add fully connected layer and weight initialization with weight manually.</p>
<p>Below are what I wanted to do.</p>
<ol>
<li>Save fully connected layer’s weights.</li>
<li>Trained model A.</li>
<li>In inference phase, add fully connected layer and replace fully connected layer’s weights to saved weights.</li>
</ol>
<p>Is there any similar topic or reference code?</p>
Help with torchvision

elibel: <p>Hi guys, I just started with PyTorch today and am going through the training tutorial at <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py</a>. In part 1, “Loading and normalizing CIFAR10”, I’m having issues with both of the torchvision function calls. I can’t download the file without putting in proxy info because of my work WiFi, and manually saving the file and changing the ‘./data’ portion of the code to my my home directory seems to not make a difference.</p>
<p>Is there some way to put in proxy information in this function in order to allow me to download through the function call? Or can someone assist with how this directory system works? Thank you very much!</p>
Torchvision dataset name as a variable

Kamil_Adamczewski: <p>Is it possible to have a variable which determines the name of the dataset loaded by torchvision.dataset?</p>
<p>e.g.<br>
dataset_var=‘ImageNet’</p>
<p>imagenet_data = torchvision.datasets.dataset_var(‘path/to/dataset/’)</p>
<p>That does not work but is there another way to make it work?</p>

ptrblck: <p>You could use the internal <code>__dict__</code> of the <code>torchvision.datasets</code> module, which is a bit hacky and most likely not the best approach:</p>
<pre><code class="lang-python">dataset_var = 'MNIST'
dataset = datasets.__dict__[dataset_var](
    root='./data'
)
</code></pre>
<p>Alternatively, just check <code>dataset_var</code> for a particular string and load the corresponding dataset after the if-condition.</p>

Kamil_Adamczewski: <p>Excellent reply! Thanks!!</p>
Cuda out of memory DQN training

Milky: <p>i am training my own DQN agent for a game i wrote in python, and the training loop works fine, but after about 150 episodes, cuda runs out of memory at optimize model function:</p>
<p>File “trainingAgent.py”, line 99, in <br>
optimize_model()<br>
File “trainingAgent.py”, line 67, in optimize_model<br>
next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()<br>
File “C:…\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:…\DQN.py”, line 28, in forward<br>
x = F.relu(self.bn3(self.conv3(x)))<br>
File “C:…\Python\Python37\lib\site-packages\torch\nn\functional.py”, line 862, in relu<br>
result = torch.relu(input)<br>
RuntimeError: CUDA out of memory. Tried to allocate 18.13 MiB (GPU 0; 8.00 GiB total capacity; 4.81 GiB already allocated; 17.54 MiB free; 1.43 GiB cached)</p>
<p>So yeah I have 8GB gpu memory, it is a gaming gpu. I’m not sure if this is normal but I guess not. Is it because my input screen matrix is too large? (300x300) cause i dont subtract the last screen from the current scren like in the example DQN algorithm (though i dont know if this lowers the input size?), but just use the current screen as the input for the network. Or is it some kind of memory leak? what could i do to solve the issue? I would like to be able to run at least 2000 episodes, and possibly 10,000 episodes if needed</p>

Kushaj: <p>I think gradient accumulation is the source of this problem. Seeing that the training worked for some epochs and you got the error after that.</p>
<p>Refer to this <a href="https://pytorch.org/docs/stable/notes/faq.html" rel="nofollow noopener">pytorch documentation</a> that can help if this is the case.</p>

Milky: <p>thanks for the answer, I am looking but I do not accumulate anything in my training loop, besides the frame count of the current game</p>

Milky: <p>My guess at the moment is the replayMemory memory, which at every frame pushes 2 states of my game and since each of my states are 300x300, the memory blows up</p>
<p>memory.push(state, action, next_state, reward)</p>
<p>edit: indeed, I commented this out, along with the optimizer function. Just commenting out the optimizer, i still ran out of memory, but once I commented out the memory push, my GPU memory was free for 1000 games. I guess I shall try to train using the difference of current and last screen like in the cartpool training</p>

Stone: <p>Hi, <a class="mention" href="/u/milky">@Milky</a>, I recently wrote a lib called <code>pytorch_memlab</code>, probably you can use the <code>MemReporter</code> to inspect which kind of tensors are eating up your GPU memory.</p>
Using DataParallel

david.bernstein: <p>Hi Everyone, I’m having a hard time getting DataParallel to work in my training script. The model is wrapped in DataParallel like this:</p>
<p>model = torch.nn.DataParallel(model, device_ids=device_ids).cuda()</p>
<p>where for experimentation purposes device_ids = [1].  In the training loop after the data is loaded it is sent to the gpu like this:</p>
<p>data = data.cuda()</p>
<p>When device_ids = [0] this is all fine but for any other setting I get an error saying</p>
<p>RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:0</p>
<p>I’ve tried various combinations of changing .cuda() to .to(device) where device is set in different ways but to no avail. I’ve looked at all the tutorials and documentation as well. I guess my confusion is when using a model wrapped in DataParallel where should the actual data be sent in the training loop?</p>

david.bernstein: <p>Well one thing that seems to work is the following: suppose that device_ids = [1,2]. If I send both the model and the data to device ‘cuda:1’ then the training runs on both gpus. I’m not seeing the speedup I expected but at least it’s running. Also the utilization of gpu 2 is significantly lower than that of gpu 1.  Is this correct? I.e., when using DataParallel do model.to(device) and data.to(device) where device is one of the gpus in device_ids? Or am I still missing something?</p>
<p>Well amend that. It runs but only on two gpus. The training machine has four gpus and when I try the above on 3 or 4 gpus it crashes the machine a few steps after training starts.</p>
Understanding stack_out

Thomas_Johnson: <p>Can someone explain how exactly the _out functions (stack_out, cat_out, etc) work?</p>
<p>Is it correct that torch::stack_out(my_output_tensor, my_vect) is exactly equivalent to torch::stack(my_vect) except that the output tensor will be stored in my_output_tensor, and my_output_tensor will be returned from the function?</p>
Torch.cuda.is_available() is returning false

ricky123: <p>I am facing a very strange issue. I am working with cuda 9.0 version of PyTorch and yesterday it was working properly in my system ( ubuntu 14.04). Today when I tried to run my model, I noticed that it is using CPU. When I looked into it, I found that <code>torch.cuda.is_available()</code> is returning <code>false</code> even though the cuda driver is available. How is it possible? It was working properly till yesterday. How can it change syddenly?</p>

ptrblck: <p>Did you update and NVIDIA drivers etc.?<br>
Could you try to restart your machine and check, if it’s working again?<br>
I had similar issues after Ubuntu updated some drivers.</p>

ricky123: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="46666">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>if it’s working again?</p>
</blockquote>
</aside>
<p>I was working on a remote machine. It will take some time to reboot and check. I will inform after checking. NVIDIA drivers are up to date</p>
Having trouble with VOC 2012 segmentation with the void = 255 label

dwkang: <p>Hello, I’m trying to implement a segmentation model using the VOC 2012 data, and I was having trouble with calculating the loss.</p>
<p>I am using cross entropy loss, but then I get the error</p>
<pre><code class="lang-auto">RuntimeError: Assertion `cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes' failed.  at ..\aten\src\THNN/generic/ClassNLLCriterion.c:92
</code></pre>
<p>when I try to set the number of classes to 22 for the output. This is because the data has a label of 255 for void, and thus pytorch thinks there are 256 classes. I was wondering how to rectify this? I loaded the data using torchvision.datasets.VOCSegmentation. Another thing I was not clear about was do people actually try to label the void pixels in practice? Thanks so much</p>

ptrblck: <p>You could pass <code>ignore_index=255</code> to the initialization of your criterion, if your model does not predict the void class.</p>

dwkang: <p>That worked! Thanks so much</p>

dwkang: <p>Oh another issue. How would you recommend computing the confusion matrix to calculate IoU? Would sklearn’s confusion_matrix function take care of this if we feed in a list of all the classes?</p>

ptrblck: <p><code>sklearn</code>'s confusion matrix should most likely work.<br>
If you would like to fill the confusion matrix in your evaluation loop, have a look at <a href="https://discuss.pytorch.org/t/how-to-check-and-read-confusion-matrix/41835/5">this post</a>.</p>

dwkang: <p>I’ve noticed both the scitkit method and the for loop method is a bit slow. Is there a vectorized way of implementing this or is this just the nature of constructing the confusion matrix? Also the confusion matrix is updated for<br>
every minibatch correct?</p>

ptrblck: <p>Yes, the confusion matrix is updated for each mini batch so avoid storing the values.<br>
Also there is a vectorized (and slightly more complicated way) to fill the confusion matrix using <code>.put_</code>:</p>
<pre><code class="lang-python"># Create dummy data
n_samples = 1000
n_classes = 5

conf_matrix = torch.zeros(n_classes, n_classes)

preds = torch.randn(n_samples, n_classes)
preds = torch.argmax(preds, 1)
labels = torch.randint(0, n_classes, (n_samples,))

# For loop
t0 = time.perf_counter()
for p, t in zip(preds, labels):
    conf_matrix[p, t] += 1
print('Took {}s'.format(time.perf_counter() - t0))

# put_ method
conf_matrix2 = torch.zeros(n_classes, n_classes)

t0 = time.perf_counter()
lin_index = preds * n_classes + labels  # You need to calculate the linear index here
conf_matrix.put_(lin_index, torch.tensor(1.).expand_as(lin_index), accumulate=True)
print('Took {}s'.format(time.perf_counter() - t0))
</code></pre>
Getting None for parameter gradient

Elas_Fi: <p>Hi Everyone,</p>
<p>I am new here and started using Pytorch on a project.</p>
<p>I am building my own loss function for a neural network . This is the forward function</p>
<pre><code>def forward(self, images, labels):
    self.noise = torch.randn(self.d_size).to(self.device) * torch.exp(self.sigma_posterior_)

    vector_to_parameters(self.flat_params + self.noise, self.model.parameters())
        
    outputs = self.model(images)
    
    loss = F.cross_entropy(outputs.float(), labels.long())

    return loss
</code></pre>
<p>The parameters of the loss function are<br>
‘flat_parameters’ :which i am getting by flattening neural network (self.model) weights<br>
“sigma_posterior” : which is a parameter i use for adding noise to network weights.</p>
<p>When trying loss.backward() I am getting the gradient according to the new weights. but I actually want also the gradient according to the previous weights and sigma_posterior.</p>
<p>This is what i am getting for gradient :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6437e502c78c03b6bbc5f84c9e8dab6907320179.png" alt="image" data-base62-sha1="eizv2mYevKQ2DnTkUMuLrEMDe0V" width="524" height="253"></p>
<p>Thanks for any answer, i will appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Collaborating with TVM and NNVM from DMLC

Travis_Oliphant: <p>Hey there,</p>
<p>As someone who has been in the Python data-science community for a long time, I love the energy around Machine Learning frameworks, but I’m also alarmed by the amount of fracturing, siloing, and lack of coordination that is taking place.   As I’ve said in several talks at this point, the division in Python machine-learning that exists right now makes my fears over a Numeric and Numarray split that led me to sacrifice my tenure track to develop NumPy seem rather “quaint”.</p>
<p>How can we foster cooperation and collaboration.  I’ve posted another message about Chainer specifically: <a href="https://chainer.org/" rel="nofollow noopener">https://chainer.org/</a></p>
<p>This message is to raise awareness to TVM and NNVM:  <a href="http://tvmlang.org/2017/10/06/nnvm-compiler-announcement.html" rel="nofollow noopener">http://tvmlang.org/2017/10/06/nnvm-compiler-announcement.html</a>.  Is ONNX the only way for PyTorch and these tools to be interoperable?</p>
<p>In my studies and research lately, I’ve found that TVM and NNVM to be fascinating projects.  I’d like to know what this community thinks of that work and if anyone sees potential for integrating those projects more clearly with PyTorch.</p>
<p>I have some potential funding (and intern talent) I can use to help with these efforts.</p>
<p>Thanks for your feedback.</p>
<p>-Travis</p>

smth: <p>On this topic, we are closely integrating with TVM now, with some engineering behind it. Please see <a href="https://github.com/pytorch/tvm" rel="nofollow noopener">https://github.com/pytorch/tvm</a></p>
<p>We are less inclined about NNVM – TVM provides Relay, which is a high-level IR.</p>
Non-deterministic result of LSTM after setting seed and option deterministic

Seth: <p>I was writing a LSTM-based model and experimented with k-fold cv, and I found that after setting all the seeds and options at the beginning of the code listed below, I still got  non-deterministic results. However, I can obtain reproductible result after these settings repeated each fold. I checked the issue posted on github and I noticed that torch.backends.cudnn.deterministic is a global option, but it did not work in my code. If anyone who meets same situation, please leave a msg.</p>
<p>random.seed(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
if torch.cuda.is_available():<br>
torch.cuda.manual_seed_all(seed)<br>
torch.backends.cudnn.deterministic = True</p>

erkanp: <p>I’m in the same situation, did you solve it?</p>

Seth: <p>I did not solve this issue. Here is the doc about the randomness in pytorch.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/notes/randomness.rst" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/docs/source/notes/randomness.rst" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/docs/source/notes/randomness.rst</a></h4>
<pre><code class="lang-rst">Reproducibility
===============

Completely reproducible results are not guaranteed across PyTorch releases,
individual commits or different platforms. Furthermore, results need not be
reproducible between CPU and GPU executions, even when using identical seeds.

However, in order to make computations deterministic on your specific problem on
one specific platform and PyTorch release, there are a couple of steps to take.

There are two pseudorandom number generators involved in PyTorch, which you will
need to seed manually to make runs reproducible. Furthermore, you should ensure
that all other libraries your code relies on and which use random numbers also
use a fixed seed.

PyTorch
.......
You can use :meth:`torch.manual_seed()` to seed the RNG for all devices (both
CPU and CUDA)::

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/notes/randomness.rst" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Different results when loading model

erkanp: <p>Hello</p>
<p>Yet another question regarding loading models with seed.</p>
<p>I have followed every post about the subject and trying to achieve deterministic results by using the proposed solution:</p>
<pre><code class="lang-auto">def seed_torch(seed=1029):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed) # if you are using multi-GPU.
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

seed_torch()
</code></pre>
<p>But without success, each time I get a different result when evaluating a single data point.</p>
<p>My model has been trained on another computer, but should that matter?</p>
<p>I have checked the values for each layer and they are the same on every load, it is just the output that differs.</p>
<p>Any ideas?</p>

erkanp: <p>Can it be because I’m using <code>nn.MaxPool1d</code> that might be mentioned under <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 2.1 documentation</a></p>
<blockquote>
<p>…  and many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions. ?</p>
</blockquote>
Single bounded output layer

Krishna_Chaitanya: <p>Hi,</p>
<p>I am looking for an output layer which has a single output that is bounded between (0,1).</p>
<p>I tried nn.linear, but its not bounded. nn.softmax doesnt work as its single output, its value is always 1.</p>

ptrblck: <p>You could try <code>torch.sigmoid</code>.</p>
Model no longer trainable with multi-GPUs

af3b04f0e6ef68be2e0e: <p>I have a model runs on single GPU and I am trying to speed-up with multi-GPUs implemented by DataParallel. The model’s training loss can’t reach the same level as what it did in the single-GPU case anymore.<br>
To analyze this, I’ve duplicated the model with the first one running on single-GPU and the second one running on multi-GPUs, both loaded well-trained weights. The result produced by the models were very different, compared by L2 norm. This is not expected because they have already loaded with the same weights, the only obvious difference would be batchnorm layers. I’ve done another experiment by setting the model to .eval() mode, the L2 difference goes to 0 this time!<br>
One interesting to know is only changing the track_running_stats setting of batchnorm does not help, it has to change the model’s mode to .eval() to eliminate the difference. This suggests the rootcause might not be like I thought, but I don’t know how to confirm this.<br>
.train() + track_running_stats=True --&gt; has difference<br>
.train() + track_running_stats=False --&gt; has difference<br>
<strong>.eval() + track_running_stats=True–&gt; no difference</strong><br>
.eval() + track_running_stats=False–&gt; has difference</p>
<p>I’d like to ask:</p>
<ol>
<li>except for batchnorm, what else can be affected by .eval() ? Any methods to check this?</li>
<li>is synchronized batchnorm necessary for multi-GPUs?</li>
<li>reduced batch size shouldn’t have a huge impact like this because the model was trainable on single-GPU if I set the batch size to half on purpose. -&gt; is this observation make sense?</li>
</ol>
<p>System:<br>
pytorch 0.4.0<br>
ubuntu 16<br>
python 3.5.2</p>
<p>Sorry for the lengthy and weird questions, appreciated for your patience.</p>

ptrblck: <p>Based on your experiments it looks like the batch statistics differ a lot between the single GPU run and the data parallel one.<br>
Using <code>.train()</code> or <code>track_running_stats=False</code> will use the current batch estimates, which might be skewed for smaller batch sizes.</p>
<p>You could try <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.SyncBatchNorm" rel="nofollow noopener">SyncBatchNorm</a> with <code>DistributedDataParallel</code>.</p>
Why my loss doesn't decrease when training

EEEEEEE: <p>When i was training my model, I found the loss doesn’t decrease, My model input is 3-D image, and the loss function is binary cross entropy. The optimizer is SGD with momentum 0.9, initial learning rate 0.01, batch size 256.</p>
<p>the loss in the 30 epochs is<br>
0.749619674<br>
0.693235771, 0.69320591, 0.693198329, 0.693196385, 0.693205666, 0.693200624<br>
0.693216185, 0.69323178, 0.693249031, 0.69317602, 0.693203645, 0.69319184<br>
0.693239689, 0.693188139, 0.693181048, 0.693167897, 0.693213663, 0.693183094<br>
0.693211535, 0.693212263, 0.69321184, 0.693209582, 0.693231421, 0.693211854<br>
0.693185517, 0.693207201, 0.693201362, 0.693189077, 0.693243751</p>
<p>Anyone can give me a suggestion? Thanks!</p>

ptrblck: <p>Try to play around with some hyperparameters, e.g. lower the learning rate.<br>
If that doesn’t help, try to scale down your use case (e.g. just use a tiny subset of your data) and check if your model is able to overfit this dataset.</p>
Problem of inplace operation with InstanceNorm

JiaBob: <p>Here is a problem I met when I implement GP-WGAN. It is something about inplace operation and second-order derivative. I simplify the problem code like below.  (Pytorch version 1.1.0)</p>
<pre><code class="lang-python">i = torch.randn((1,3,10,10), requires_grad=True)
main = nn.Sequential(
                    nn.Conv2d(3, 3, 3),
                    nn.InstanceNorm2d(3),
                    nn.ReLU(True),
                    nn.Conv2d(3, 3, 3),
                )
o = main(i)
gradients = torch.autograd.grad(outputs=o, inputs=i,
                                grad_outputs=torch.ones(o.size()),
                                retain_graph=True, create_graph=True, only_inputs=True)[0]
gradients.mean().backward()
</code></pre>
<p>The above code will cause</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>But if we use only first-order derivative of variable <code>o</code> or change the network layer part into</p>
<pre><code class="lang-python">main = nn.Sequential(
                    nn.Conv2d(3, 3, 3),
                    nn.BatchNorm2d(3),
                    nn.ReLU(True),
                    nn.Conv2d(3, 3, 3),
                )
</code></pre>
<p>Or</p>
<pre><code class="lang-python">main = nn.Sequential(
                    nn.Conv2d(3, 3, 3),
                    nn.ReLU(True),
                    nn.Conv2d(3, 3, 3),
                )
</code></pre>
<p>There will be no error. Of course, if change the inplace operation of ReLU to false or remove the last convolution layer, there will be no error either. Same case with <code>LeakyReLU</code>.<br>
May I ask why using the <code>InstanceNorm</code> causes such error while no problem for <code>BatchNorm</code>?</p>
Correct IOU method

David_Jitca: <p>Hello, I want to calculate intersection over union accuracy for segmentation. I have to method for IOU but I obtain something different. Can you help me?</p>
<pre><code class="lang-auto">Untitled 
#Method1
def jaccard_loss(logits, true, eps=1e-7):
  Args:
    true: a tensor of shape [B, H, W] or [B, 1, H, W].
    logits: a tensor of shape [B, C, H, W]. Corresponds to
      the raw output or logits of the model.
    eps: added to the denominator for numerical stability.
  Returns:
    jacc_loss: the Jaccard loss.
  """
  num_classes = logits.shape[1]
  if num_classes == 1:
    true_1_hot = torch.eye(num_classes + 1)[true.squeeze(1)]
    true_1_hot = true_1_hot.permute(0, 3, 1, 2).float()
    true_1_hot_f = true_1_hot[:, 0:1, :, :]
    true_1_hot_s = true_1_hot[:, 1:2, :, :]
    true_1_hot = torch.cat([true_1_hot_s, true_1_hot_f], dim=1)
    pos_prob = torch.sigmoid(logits)
    neg_prob = 1 - pos_prob
    probas = torch.cat([pos_prob, neg_prob], dim=1)
  else:
    true_1_hot = torch.eye(num_classes)[true.squeeze(1)]
    true_1_hot = true_1_hot.permute(0, 3, 1, 2).float()
    probas = F.softmax(probas, dim=1)
  true_1_hot = true_1_hot.type(logits.type())
  dims = (0,) + tuple(range(2, true.ndimension()))
  intersection = torch.sum(probas * true_1_hot, dims)
  cardinality = torch.sum(probas + true_1_hot, dims)
  union = cardinality - intersection
  jacc_loss = (intersection / (union + eps)).mean()
  return jacc_loss
  
  
  #Method2
def. ...
   loop dataset
       intersection = np.logical_and(mask, lbl_pred)
       union = np.logical_or(mask, lbl_pred)
       iou_score = np.sum(intersection.cpu().detach().numpy()) / np.sum(union.cpu().detach().numpy())
       to += iou_score
    return to/size
</code></pre>
<p>First method give me iou = 0,3 second iou= 0,56</p>
How to use nn.bottle?

Donggeun: <p>I want to:<br>
(32, 8, 16) -&gt; (32, 8, 64)</p>
<p>I want to change only the last dimension using nn.Linear.<br>
In the past, nn.Bottle(nn.Linear(16, 64))</p>

ptrblck: <p>What is <code>nn.Bottle</code> supposed to do and where did you find it (caffe2 maybe?)?</p>
Inconsistencies between opencv WarpAffine and grid_sample

Erez_Posner: <p>Hello, Im trying to reproduce an output from opencv WarpAffine on a reference image but get different results between opencv WarpAffine and Pytorch grid_sample function.</p>
<p>My transformation includes scaling, for some reason it does not seem to work using grid_sample function.<br>
see the results below<br>
On the right - the reference image size(64,64)<br>
Middle - transformed image to 320,320 with a minor rotation<br>
Left - grid_sample  function output.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9ab842f9c8aebd709f246c855b10345923017b6e.png" alt="image" data-base62-sha1="m4IjaoVsN4RZEqa5RSbBDBaoRpc" width="592" height="257"></p>
<p>My code is attached here - reading an affine transform and apply using both methods.</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt


def _transform_matrix(Hs, w, h):
    _Hs = np.zeros(Hs.shape, dtype=np.float32)
    for i, H in enumerate(Hs):
        H0 = np.concatenate((H, np.array([[0, 0, 1]])), axis=0)
        A = np.array([[2.0 / w, 0, -1], [0, 2.0 / h, -1], [0, 0, 1]])
        A_inv = np.array([[w / 2.0, 0, w / 2.0], [0, h / 2.0, h / 2.0], [0, 0, 1]])
        H0 = A.dot(H0).dot(A_inv)
        H0 = np.linalg.inv(H0)
        _Hs[i] = H0[:-1]
    return _Hs

#get affine transform from an array
H_e2e = np.copy(self.TODOInversemaskAlignMatrixs[i][j])

#convert to theta
theta = _transform_matrix(H_e2e[np.newaxis, 0:2, :], 320,320)
theta = (torch.from_numpy(theta)).cuda()
grid = F.affine_grid(theta, (1, 1, 320, 320))

#predmap in an array of reference images. get one
img_ = F.grid_sample(torch.from_numpy(predmap[:, :, 0]).unsqueeze(0).unsqueeze(0).cuda().float(),
                     grid.float(), mode='bilinear')

fig = plt.figure()
ax1 = plt.subplot(1, 3, 1)
ax1.imshow(img_.squeeze().detach().cpu().numpy())

#do the same using opencv
pred_e2e = cv2.warpAffine(predmap[:, :, 0], H_e2e[0:2], (width, height))
ax2 = plt.subplot(1, 3, 2)
ax2.imshow(pred_e2e)

ax2 = plt.subplot(1, 3, 3)
ax2.imshow(predmap[:, :, 0])
</code></pre>
Reshape time series data for RNNs

sachinruk: <p>This is a crosspost from <a href="https://stackoverflow.com/questions/56373603/train-time-series-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/56373603/train-time-series-in-pytorch</a>.</p>
<p>I wish to train a RNN model such that I can predict for T steps ahead in a time series model. Most of the examples that I have seen so far are centred around text.</p>
<p>The toy example that I have is to predict 3 sine waves as shown below:</p>
<pre><code class="lang-python">x = torch.arange(0,30,0.05)
y = [torch.sin(x), torch.sin(x-np.pi), torch.sin(x-np.pi/2)]
y = torch.stack(y)
y = y.t()
</code></pre>
<p>y is of shape <code>600,3</code>. However in order for the LSTM to accept it the input needs to be of shape <code>(seq_len, batch, input_size)</code>. I was wondering if there is a function in pytorch that converts them to required format. Suppose that in my case I want <code>seq_len=50</code> and <code>batch_size=32</code>.</p>
<p>This snippet of code from <a href="https://machinelearningmastery.com/time-series-prediction-with-deep-learning-in-python-with-keras/" rel="nofollow noopener">machinelearningmastery</a> was the only snippet of code I found.</p>
<pre><code class="lang-python"># convert an array of values into a dataset matrix
def create_dataset(dataset, look_back=1):
	dataX, dataY = [], []
	for i in range(len(dataset)-look_back-1):
		a = dataset[i:(i+look_back), 0]
		dataX.append(a)
		dataY.append(dataset[i + look_back, 0])
	return numpy.array(dataX), numpy.array(dataY)
</code></pre>
<p>Does <code>pad_packed_sequence</code> or anything similar in pytorch natively do this.</p>
<p>If anyone is interested, this is my LSTM model:</p>
<pre><code class="lang-python">class LSTM(nn.Module):
    def __init__(self, n_features, h, num_layers=2):
        super().__init__()
        self.lstm = nn.LSTM(n_features, h, num_layers)
        self.linear = nn.Linear(h, n_features)
    
    def forward(self, input, h=None):
        lstm_out, self.hidden = self.lstm(input, h)
        return self.linear(lstm_out)
</code></pre>
<p>[optional Q] For whatever solution that I end up with, is there a way to ensure that I can do stateful training?</p>
Backpropagation on separate GPUs

half-potato: <p>My backpropagation step is incredibly intensive, both for memory and GPU power. The data loading is not the bottleneck for my network architecture. How do I tell each GPU to calculate the loss step, then accumulate the gradients on the default GPU, rather than running the loss step on the default GPU?</p>

half-potato: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="32" height="32">
      <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" target="_blank" title="09:23AM - 15 October 2018" rel="nofollow noopener">Medium – 15 Oct 18</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/458;"><img src="https://cdn-images-1.medium.com/max/1200/0*f1Nz2zUOqpf5uUKk" class="thumbnail"></div>

<h3><a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" target="_blank" rel="nofollow noopener">💥 Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU...</a></h3>

<p>Training neural networks with larger batches in PyTorch: gradient accumulation, gradient checkpointing, multi-GPUs and distributed setups…</p>

  <p><span class="label1">Reading time: 9 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Why I can't transform a torch.Tensor to torch.cuda.Tensor

EEEEEEE: <p>I met a problem when I run pytorch code:<br>
RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument <span class="hashtag">#4</span> ‘mat1’.</p>
<p>That error means the net need torch.cuda.Tensor but I gave a torch.Tensor. However, before I send the data into the net, I have already copy the data to GPU memory, the error still occurs, Anyone can give a suggestion?<br>
the error code is below:</p>
<p>net = net.cuda()<br>
net = DataParallel(net)<br>
for i, (data,target) in enumerate(dataloader):<br>
if torch.cuda.is_available():<br>
data = data.cuda()<br>
target = target.cuda()<br>
output = net(data)<br>
…</p>
<p>I have already set the environment variable CUDA_VISIBLE_DEVICES<br>
my pytorch version is 0.4.0</p>

cyanM: <p>Instead, according to  error  the expected object of type is torch.FloatTensor.</p>

alex.veuthey: <p>Do you use manually tracked tensors in your network? For example in the <code>.forward()</code> method. If you use a tensor alone, without wrapping it in a <code>torch.nn.Parameter</code>, it won’t be “part” of the network, and won’t be mapped to GPU when you call <code>.cuda()</code>.</p>

EEEEEEE: <p>I think the error info means the input should be torch.cuda.FloatTensor instead of the torch.FloatTensor</p>

EEEEEEE: <aside class="quote no-group" data-username="alex.veuthey" data-post="3" data-topic="44885">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex.veuthey/48/12176_2.png" class="avatar"> alex.veuthey:</div>
<blockquote>
<p>and won’t be mapped to</p>
</blockquote>
</aside>
<p>You mean data.cuda() cannot always map the data into GPU? I don’t understand, I am new to pytorch</p>

dimaxano: <p>Can you please provide the code of your network?</p>

alex.veuthey: <p>(Your previous message) actually the error means that the matrix <code>mat1</code> in your model is of type <code>torch.FloatTensor</code> (CPU), while the input you provide to the model is of type <code>torch.cuda.FloatTensor</code> (GPU).<br>
The most likely scenario is that you have <code>nn.Parameter</code> or other modules such as <code>nn.Conv2d</code> defined in the <code>__init__()</code> method of your model, and additional weights or layers defined in the <code>forward()</code> method of your model.<br>
In this case, the layers defined in the <code>forward()</code> method are not modules of the model, and they won’t be mapped to GPU when you call <code>cuda()</code>, see <a href="https://discuss.pytorch.org/t/dynamic-parameter-declaration-in-forward-function/427/3">this answer</a> for the explanation.<br>
As mentionned in the linked topic, you also need to explicitely add the parameters to your optimizer if you want them to be updated with gradient descent.</p>

EEEEEEE: <p>Thank you. You are right, I found the error just like what you said.</p>
How to use multi-GPU with v1.1.0

Leonardo_Ma: <p>How to use multi-GPU to train our model with the newest stable version. Just cast the model to nn.DataParallel? Or plus manually converting BN to nn.SyncBatchNorm?</p>
<pre><code>        model = nn.DataParallel(model)
        model = nn.SyncBatchNorm.convert_sync_batchnorm(model) (Is this necessary?)
</code></pre>
<p>Moreover, do we need to set the batch-size to nGPUs times the single GPU batch-size?</p>

Leonardo_Ma: <p>Another thing weird is, when I used model = nn.DataParallel(model) without change the batch-size, it automatically distributed to 4 GPUs, with total memory-in-use about 2 times of memory in the single-GPU situation. And the training result is also a little different.<br>
One thing I found more efficient is<br>
model = nn.SyncBatchNorm.convert_sync_batchnorm(model)<br>
is not necessary when I have<br>
model = nn.DataParallel(model)<br>
at least in pytorch1.1.0.</p>
<p>So this issue is changed to how to handle batch-size with nn.DataParallel, and why the result using nn.DataParallel is different.</p>
How to index 2 dimensional Tensor using another 2 dimensional Tensor

Xuyang_Bai: <p>Hi all, I want to index 2 dimension tensor using another 2 dimension tensor. My first tensor points is <code>torch.Size([2048, 3])</code>, and second tensor neighbour is <code>torch.Size([2048, 27])</code>, I want to do a operation like this <code>points[neighbor, :]</code> , I think it can work in numpy but I do not know how to make it work in Pytorch.</p>