akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Functionality of tensor.index_copy_

Paul_Ha: <p>Hi,</p>
<p>I am trying to use tensor.index_copy_, but having some issues regarding tensor’s dimensions.</p>
<p>On the Pytorch documents, there is an example:</p>
<p>x = torch.zeros(5,3)<br>
t = torch.tensor([[1,2,3], [4,5,6], [7,8,9]], dtype=torch.float)<br>
index = torch.tensor([0, 4, 2])<br>
x.index_copy_(0, index, t)</p>
<p>These codes result in x as:<br>
x = [[1, 2, 3],<br>
[0, 0, 0],<br>
[7, 8, 9],<br>
[0, 0, 0],<br>
[4, 5, 6]]</p>
<p>I understood this example with the dimension of 0, but if I try to use it with dimension of 1 in similar example, I receive an runtime error that indicates some dimension issues. For the clarification, I will leave an example of this below:</p>
<p>x = torch.zeros(5,3)<br>
t = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10,11,12], [13,14,15]], dtype=torch.float)<br>
index = torch.tensor([0, 2, 1])<br>
x.index_copy_(1, index, t)</p>
<p>I expect to receive x as<br>
[[1,    3,    2],<br>
[4,    6,    3],<br>
[7,    9,    8],<br>
[10, 12, 11],<br>
[13, 15, 14],]<br>
but running these codes results in the following error:<br>
RuntimeError: index_copy_(): Source.destination tensor must have same slice shapes. Destination slice shape: [5] at dimension 1 and source slice shape: [3] at dimension 0.</p>
<p>Could someone explain what is wrong with this? Thank you!</p>

ptrblck: <p>Your code snippet seems to work and returns the right result:</p>
<pre><code class="lang-python">x = torch.zeros(5,3)
t = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10,11,12], [13,14,15]], dtype=torch.float)
index = torch.tensor([0, 2, 1])
x.index_copy_(1, index, t)
&gt; tensor([[ 1.,  3.,  2.],
        [ 4.,  6.,  5.],
        [ 7.,  9.,  8.],
        [10., 12., 11.],
        [13., 15., 14.]])
</code></pre>

Paul_Ha: <p>Thank you for your reply on my inquiry.<br>
When I run the same code on my environment, it raises with the error below.</p>
<hr>
<p>RuntimeErrorTraceback (most recent call last)<br>
 in ()<br>
----&gt; 1 x.index_copy_(1, index, t)</p>
<p>RuntimeError: index_copy_(): Source/destination tensor must have same slice shapes. Destination slice shape: [5] at dimension 1 and source slice shape: [3] at dimension 0.</p>
<p>I am running this code in torch version 0.4.0.<br>
What I notice is that if I run the same code on the torch version 0.3.1, then it runs as I expected, but it does not run properly once I use torch version 0.4.0. Do you know how to run this code on version 0.4.0?<br>
Thank you!</p>

ptrblck: <p>I’m not sure, but I would recommend to update to the latest stable release as described <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">here</a>.<br>
A lot of bug fixes were shipped in the versions after <code>0.4.0</code>, and this might be a particular bug.</p>

Paul_Ha: <p>Thank you for your kind reply!<br>
I hope it will be fixed as soon as possible.</p>
How to upgrade fastrnn's LayerNormLSTM to support PackedSequence?

Xingdong_Zuo: <p>In <a href="https://github.com/pytorch/benchmark/blob/master/rnns/fastrnns/custom_lstms.py" rel="nofollow noopener">fastrnn’s custom LSTM</a>, we could conveniently define JIT LayerNormLSTM. I am wondering that how could we also make it support PackedSequence as input data for varied lengths of tensors ?</p>

eellison: <p>This should be supported:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/21282" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/julioasotodv" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/20630819?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/21282" target="_blank" rel="nofollow noopener">Issue: TorchScript support for torch.nn.utils.rnn.pack_padded_sequence() and torch.nn.utils.rnn.pad_packed_sequence()</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/julioasotodv" target="_blank" rel="nofollow noopener">julioasotodv</a>
	on <a href="https://github.com/pytorch/pytorch/issues/21282" target="_blank" rel="nofollow noopener">2019-06-03</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/driazati" target="_blank" rel="nofollow noopener">driazati</a>
	on <a href="https://github.com/pytorch/pytorch/issues/21282" target="_blank" rel="nofollow noopener">2019-08-07</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🚀 Feature
Add TorchScript support for torch.nn.utils.rnn.pack_padded_sequence() and torch.nn.utils.rnn.pad_packed_sequence().
Motivation
Motivated by https://discuss.pytorch.org/t/torch-jit-script-for-torch-nn-utils-rnn-pack-padded-sequence/30668, this appears to be a desired feature given that tracing is...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">jit</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to visualize attention map

Ma_John: <p>Hi all.<br>
I have an image and its corresponding attention map, which is a [1, H, W] tensor and the attention map is supposed to tell me where in the image does the model think have the best exposure. I wonder if there is a way to visualize this attention, looking like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/84244f1581ce633bd65c8c3c94b95a90776939db.jpeg" alt="image" data-base62-sha1="iQYOGSkR4IEOM0tOhM6ZmW6afAn" width="568" height="278"><br>
Below are my image and its attention map.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/ba8a6df1658cc9cfac2926af81440786062132c6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ba8a6df1658cc9cfac2926af81440786062132c6" title="5Enhanced_Input"><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/ba8a6df1658cc9cfac2926af81440786062132c6.jpeg" alt="5Enhanced_Input" data-base62-sha1="qCdonHLkHYKwSmTPOmwunk39fqm" width="500" height="500" data-dominant-color="7D776A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">5Enhanced_Input</span><span class="informations">512×512 51.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dd5dbef784471bbcad7fc34e02bd178350080937.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dd5dbef784471bbcad7fc34e02bd178350080937" title="7A1"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/dd5dbef784471bbcad7fc34e02bd178350080937.jpeg" alt="7A1" data-base62-sha1="vAimZfdBqUMeQbD7YJOTdKGPfgj" width="500" height="500" data-dominant-color="565656"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">7A1</span><span class="informations">512×512 3.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for your advice.</p>

Ma_John: <p>Does anyone have a clue pls?</p>
Memory problems in torch.from_numpy()

Enrique: <p>Hi, this is my first post as often I encouter that all my PyTorch questions have been answered here at some point. Please accept my apologies if it happens this question has been already addressed.</p>
<p>In particular, I found a problem regarding converting variables from numpy to torch; I found that using torch.from_numpy() on big variables results in what appears to be a memory leakage. Here’s a very simple code in which I create a dictionary where each key will hold a 66x2 array. If I create the array beforehand and convert it through torch.from_numpy (Option 1), I find a MemoryError message when trying to dump the variable onto the file. If instead I convert each 66x2 array to torch “on-the-fly”, I find no errors when dumping the files. Does anyone have a clue about why this might be happening?</p>
<p>Many thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>import pickle, torch, numpy as np<br>
######### ----------- Option 1 ------------ #################<br>
keys = [‘{:06d}’.format(k) for k in range(10000)]<br>
db = {}<br>
points = np.random.randn(len(keys),66,2).astype(‘float’)<br>
points = torch.from_numpy(points).float()<br>
for i,k in enumerate(keys):<br>
db[k] = points[i,:,:]</p>
<p>pickle.dump(db,open(‘fromnumpy.pkl’,‘wb’))</p>
<blockquote>
<blockquote>
<blockquote>
<p>MemoryError</p>
</blockquote>
</blockquote>
</blockquote>
<p>######## ----------- Option 2 --------------##################<br>
keys = [‘{:06d}’.format(k) for k in range(10000)]<br>
db = {}<br>
points = np.random.randn(len(keys),66,2).astype(‘float’)</p>
<p>for i,k in enumerate(keys):<br>
db[k] = torch.from_numpy(points[i,:,:]).float()</p>
<p>pickle.dump(db,open(‘fromnumpy.pkl’,‘wb’))</p>
What is the validation loss mention in the lr_scheduler doc

JohnCwok: <p>Hello !</p>
<p>I hope I am not asking something dumb, but I could not find a clear answer despite a bit of googling so I’ll try my luck here :</p>
<p>When reading the doc about learning rate scheduling in pytorch, I see that the scheduler has to be provided with a validation error value. Unfortunately it is not describe anywhere what this corresponds to nor how to compute it so I don’t really know how to call <code>scheduler.step()</code> properly …</p>
<p>Can someone help me ?</p>
<p>Thanks in advance !</p>

ptrblck: <p>Some schedulers, e.g. <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau" rel="nofollow noopener">ReduceLROnPlateau</a> expect to track a metric in order to lower the learning rate.<br>
Usually you would split your dataset into training, validation (, and test) and use the validation loss to trigger the learning rate reduction.<br>
The idea behind it is that once your validation loss doesn’t decrease anymore (or not significantly), lowering the learning rate might help decreasing it further.</p>
<p>However, you could use whatever metric you like to use. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>PS: other learning rate schedulers do not expect any input and use some other schedule, e.g. epochs as milestones.</p>
Model.zero_grad() is not working if optimizer.step() is invoked

falmasri: <p>Hello I have a model call M. when I call it and call backward() to calculate the gradient and then I zero the gradient the model has no gradient. but once I call optimizer.step() on zero gradient the model update its parameters on something I don’t know so the next iteration the model has new params.</p>
<p>output = M(input,GT)<br>
error = L1(output,GT)<br>
error.backward()<br>
print(M.grad) —&gt; non-zero.<br>
M.zero_grad()<br>
print(M.grad) —&gt; zero.<br>
optimizer.step() -&gt; the model is update !!!</p>
<p>I don’t want to use with torch.no_grad() because I want to check each iteration.</p>

JamesTrick: <p>Hi <a class="mention" href="/u/falmasri">@falmasri</a></p>
<p>I think you’re on the right track with this, if you’re doing a typical training cycle - you don’t need to <code>zero_grad()</code> on your model, instead you zero out your optimizer.</p>
<p>Try this and let me know how you go:</p>
<pre><code class="lang-python"># I personally like to .zero_grad() as the first thing.
optimizer.zero_grad()
output = M(input) # Run input through. No need for the target here?
error = L1(output, GT)
# Remove the .zero_grad, backpropagate and step the optimizer.
error.backward()
optimizer.step()
</code></pre>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#update-the-weights" rel="nofollow noopener">This tutorial</a> has a lot more detail on this as well.</p>

falmasri: <p>It didn’t work, the model is still updating its params.  To make the model clear. M includes two sequential instantiated twice. if the model is called first M(input,1) the first is computed , while the second is computed with M(input,2).<br>
I’m doing this procedure to make  sure the first call or the second are not interfering but apparently yes.</p>

zhl515: <p>It depends your update rules (optimizer)  actually.  Many optimizers not only depend  on grad, such as  Nesterov-SGD, Adam, RMSProp eg:</p>
<pre><code class="lang-auto"># weight = weight - learning_rate * gradient, if gradients are zero, the weight will not update
optimizer = optim.SGD(model.parameters(), lr=0.01) 
# even gradients are zero, the weight will decay 
optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=0.1)
</code></pre>
<p>You can run below code, see the differences</p>
<pre><code class="lang-auto">model = nn.Sequential(
          nn.Linear(6, 2, bias=False),
          nn.Sigmoid(),
        )
input = torch.randn(6)
target = torch.randn(2)
#optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=0.1)
optimizer = optim.SGD(model.parameters(), lr=0.01)
criterion = nn.MSELoss()



output = model(input)
loss = criterion(output, target)
loss.backward()

print(list(model.parameters())[0].grad)
optimizer.zero_grad() 
print(list(model.parameters())[0].grad)


print("before step: ", list(model.parameters()))
optimizer.step()
print("after step: ", list(model.parameters()))
</code></pre>

falmasri: <p>Then maybe it is better to have two optimizers working on part of the model.  if once the backward is called there is no way to reset the optimizer parameters.</p>
Transformer train_model

slavavs: <p>Is cross_entropy built into the transformer or can I choose MSELos?</p>

spanev: <p>If you are talking about this training, I think you can just change it here:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/4581968193699de14b56527296262dd76ab43557/word_language_model/main.py#L104" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/4581968193699de14b56527296262dd76ab43557/word_language_model/main.py#L104" target="_blank" rel="nofollow noopener">pytorch/examples/blob/4581968193699de14b56527296262dd76ab43557/word_language_model/main.py#L104</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="94" style="counter-reset: li-counter 93 ;">
<li>###############################################################################</li>
<li># Build the model</li>
<li>###############################################################################</li>
<li>
</li>
<li>ntokens = len(corpus.dictionary)</li>
<li>if args.model == 'Transformer':</li>
<li>    model = model.TransformerModel(ntokens, args.emsize, args.nhead, args.nhid, args.nlayers, args.dropout).to(device)</li>
<li>else:</li>
<li>    model = model.RNNModel(args.model, ntokens, args.emsize, args.nhid, args.nlayers, args.dropout, args.tied).to(device)</li>
<li>
</li>
<li class="selected">criterion = nn.CrossEntropyLoss()</li>
<li>
</li>
<li>###############################################################################</li>
<li># Training code</li>
<li>###############################################################################</li>
<li>
</li>
<li>def repackage_hidden(h):</li>
<li>    """Wraps hidden states in new Tensors, to detach them from their history."""</li>
<li>
</li>
<li>    if isinstance(h, torch.Tensor):</li>
<li>        return h.detach()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


slavavs: <p>I have a sequence of inputs equal to 50. Each input has a vector of dimension 6. This is src_vocab. tgt_vocab is a number from 0 to 1 that corresponds to each sequence.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a6ef70e8f3a3e7d67bc4974d2add663b400779de.png" alt="3" data-base62-sha1="nOMg8a5rxEbY5ms2zh5M8Ucn9Mi" width="486" height="236"></p>
<p>I have a lot of such sequences and each of them corresponds to a number from 1 to 0</p>
<p>To the network, I submit a two-dimensional vector 50X6, at the output I get a number from 1 to 0</p>
<p>Loss I need to calculate as MSELoss.<br>
If you compare with the translation, then I have a sentence, and at the output as a translation I get a number.</p>
Temporal median ignore zero

rapidproto: <p>Hi,</p>
<p>I am using pytorch to generate a temporal median across “n” images in an array and this works very well. But now I would like to mask out image regions I dont need to compute the median for and set these regions to zero in the image array, ie. mask.</p>
<p>Now I have a stacked array of images that are mostly zero values, but the pytorch median function takes same time to compute.</p>
<p>How do I construct a torch.median(image_array) to ignore zeros, but retain the image shape in the result. If I do torch.nonzero I get tuples and if I set as_tupples=False, pytorch says there is no such thing as “as_tupples”.</p>
<p>Any help would be appreciated. Image array is [1920x1080x1] by 21 frames stacked into a buffer, converted to torch tensor, then apply torch.median to the buffer. This works fine and retains shape provided I use np.array(tensor) to convert back to an image.</p>
<p>thanks in advance</p>

ptrblck: <p>What would be the result, if a certain pixel location contains all zeros?<br>
You are most likely looking for <code>as_tuple=False</code> (note the missing s at the end).</p>

rapidproto: <p>thanks for you quick response ptrblck,</p>
<p>The result I am getting is,</p>
<p>TypeError: nonzero() got an unexpected keyword argument ‘as_tuple’</p>
<p>I am using PyTorch on ARM64, version  1.1.0</p>

ptrblck: <p>The <code>as_tuple</code> argument was introduced in <code>1.2.0</code> (<a href="https://pytorch.org/docs/stable/torch.html#torch.nonzero" rel="nofollow noopener">docs</a>), so you should update to the latest stable version.</p>

rapidproto: <p>Hi ptrblk,</p>
<p>Yes, upgrading to 1.2 fixes the missing option as_tuple, thank you for pointing this out, much appreciated.</p>
<p>I still have a problem with maintaining or reconstructing the result back to the original image shape. It appears that the result becomes a 1D array, whereas the original image was a 2D array.</p>
<p>Can you tell me what is required to reconstruct the image array from the nonzero median result [1D array].</p>
<p>If I do not use the torch.nonzero step, a 2D image format is preserved.</p>
<p>This is my test code  (buffer = "stacked array of 2D images, each image has the same masked area having zero values)</p>
<p>buffer2 = torch.nonzero(buffer, as_tuple=False)<br>
median = torch.median(buffer2, 0)[0]  # this is a temporal median along axis 0<br>
torch.cuda.synchronize()<br>
median2 = median.cpu()<br>
median3 = np.array(median2)<br>
cv2.imshow(“median”, median3)</p>
<p>Error:</p>
<p>ValueError: Image must be 2D (grayscale, RGB, or RGBA).</p>
<p>Regards rapidproto</p>
Requires_grad only on subset of weight matrix

Knievel: <p>Is there a way to have one weight matrix that is partly trainable and partly not?</p>
<p>Im trying to implement a RNN module that incorporates multiple time scales (as proposed in <a href="https://arxiv.org/pdf/1402.3511.pdf" rel="noopener nofollow ugc">A Clockwerk RNN</a>) but struggle on weight matrices that are partitioned in blocks of trainable weights and untrainable zeros (W_H in the image below).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d2045fcc512346219e3a6d73332c7115babe676f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2045fcc512346219e3a6d73332c7115babe676f" title="30"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d2045fcc512346219e3a6d73332c7115babe676f_2_442x179.png" alt="30" data-base62-sha1="tXTDI4PPiEAw1875j4jsr8KAlvp" width="442" height="179" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d2045fcc512346219e3a6d73332c7115babe676f_2_442x179.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d2045fcc512346219e3a6d73332c7115babe676f_2_663x268.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d2045fcc512346219e3a6d73332c7115babe676f_2_884x358.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">30</span><span class="informations">980×380 24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried to just initialize the Parameters as individual blocks and concatenate them, but using torch.cat on Parameters doesn’t work.<br>
Is there a nice way of handling this or do I have to do all calculations separately?</p>
Pytorch fft on multiple gpus

Tim_Zhang: <p>I am using pytorch function torch.rfft() and torch.irfft() inside the forward path of a model. It runs fine on single GPU. However, when I train the model on multiple GPUs, it fails and gave the error:</p>
<p>RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR</p>
<p>Does anybody has the intuition why this is the case? Thanks!</p>

pietern: <p>Hi <a class="mention" href="/u/tim_zhang">@Tim_Zhang</a> – are you using <code>torch.nn.DataParallel</code> for training on multiple GPUs? If so, this could be some sort of initialization bug where cuFFT is initialized on the first device only and not others.</p>

chenzhuo1011: <p>I have the same problem here, when using DataParallel with torch.fft() or torch.rfft(), it generates error without error message. I.e. the visual studio pops up saying" An unhandled win32 exception ocurred in python.exe"</p>

pietern: <p>I managed to reproduce this issue and reported it at <a href="https://github.com/pytorch/pytorch/issues/24176" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/24176</a>.</p>
TorchScript: indexing question / filling NaNs

achim: <p>Hi!</p>
<p>I’m trying to <code>jit.script</code>-compile a model which uses the <code>t[t != t] = ...</code> trick to fill the nans of a tensor with a default value. However, torchscript does not seem to appreciate this kind of indexing. Does anyone know a solution/workaround?</p>
<p>Thank you!</p>
<p>Here’s a small example plus error message:</p>
<pre><code class="lang-python"> @torch.jit.script 
 def f(): 
     t = torch.tensor(0.) 
     t[t!=t] = 7 
</code></pre>
<pre><code class="lang-nohighlight">RuntimeError: 
Arguments for call are not valid.
The following operator variants are available:
  
  aten::index_put_(Tensor(a) self, Tensor?[] indices, Tensor values, bool accumulate=False) -&gt; (Tensor(a)):
  Expected a value of type 'Tensor' for argument 'values' but instead found type 'int'.
  
  aten::index_put_(Tensor(a) self, Tensor[] indices, Tensor values, bool accumulate=False) -&gt; (Tensor(a)):
  Expected a value of type 'List[Tensor]' for argument 'indices' but instead found type 'List[Optional[Tensor]]'.

The original call is:
at &lt;ipython-input-21-45d08c8a21f7&gt;:4:5
@torch.jit.script
def f():
    t = torch.tensor(0.)
    t[t!=t] = 7
    ~~~~~~~~~~~ &lt;--- HERE
</code></pre>

ptrblck: <p>I’m not sure if it’s the indexing or rather the rhs of the assignment, since the error says:</p>
<blockquote>
<p>Expected a value of type ‘Tensor’ for argument ‘values’ but instead found type ‘int’.</p>
</blockquote>
<p>This seems to work:</p>
<pre data-code-wrap="python"><code class="lang-python">@torch.jit.script 
def f(): 
    t = torch.tensor([0.])
    t[t!=t] = torch.tensor(7.)
    return t

f()
</code></pre>

achim: <p>Your interpretation of the error msg makes a lot more sense. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"><br>
Thank you!</p>
Layer convolution

slavavs: <p>Сan i make a convolution of the layer like in the picture?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/10c72a110c7ab229997d902bc7b7a9511a5c71b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/10c72a110c7ab229997d902bc7b7a9511a5c71b7" title="rnn5"><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/10c72a110c7ab229997d902bc7b7a9511a5c71b7.png" alt="rnn5" data-base62-sha1="2oqlwkx2V687U3ubyvD6INCQ7RB" width="690" height="479" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rnn5</span><span class="informations">1347×936 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could use a linear layer or a conv layer with a specific kernel size:</p>
<pre><code class="lang-python">x = torch.randn(1, 16, 7)
lin = nn.Linear(7, 1, bias=False)
output = lin(x)

conv = nn.Conv2d(1, 1, kernel_size=(1, 7), stride=(1, 1), bias=False)
with torch.no_grad():
    conv.weight.copy_(lin.weight.unsqueeze(1).unsqueeze(2))
x_conv = x.unsqueeze(1)
output_conv = conv(x_conv)

print(torch.allclose(output, output_conv.squeeze(1)))
</code></pre>

slavavs: <p>I do not understand why we do it:<br>
<em>with torch.no_grad():</em><br>
<em>conv.weight.copy_(lin.weight.unsqueeze(1).unsqueeze(2))</em><br>
<em>x_conv = x.unsqueeze(1)</em><br>
<em>output_conv = conv(x_conv)</em></p>
<p>this way will not work?:<br>
<em>conv = nn.Conv2d(1, 1, kernel_size=(1, 7), stride=(1, 1), bias=False)</em><br>
<em>output = conv(x)</em></p>

slavavs: <p>I understood, conv2d takes 4 parameters</p>

ptrblck: <p>It will work. I just used this code snippet to set the weights according to the linear layer’s, so that we can make sure both results are equal.<br>
Of course you don’t have to set the weights manually.</p>

slavavs: <p>if you do this (print(output_conv(x)):<br>
<em>RuntimeError: Expected 4-dimensional input for 4-dimensional weight 1 1 1, but got 3-dimensional input of size [1, 16, 7] instead</em></p>

ptrblck: <p><code>output_conv</code> is the result of the convolution so I assume you are using <code>conv</code> instead.<br>
As shown in my code, you would have to <code>unsqueeze</code> <code>x</code> in dim1.</p>

slavavs: <p>How can I use a transformer instead of conv?<br>
I have a sequence of 16 inputs. Kadzhi entrance has a length of 7.<br>
To use trabsformer I have to specify src and tgt. I can’t understand how src and tgt should look. I can assume that src = 7, but what is tgt?</p>
How to launch two distributed programs on single machine?

lxtGH: <p>I try to train ImageNet on 8-gpu server. However, <strong>I have enough memory</strong> after I start to train resnet50 using distributed training. So I want to launch another training, but the error shows that the address has been used. <strong>Can we achieve two distributed training in a single machine</strong> ?</p>

shubhvachher: <p>How are you launching the distributed training processes?</p>
<p>If manually using <code>torch.distributed</code>, try setting the <code>master_port</code> setting. The way to do this differs. Refer <a href="https://pytorch.org/docs/stable/distributed.html#tcp-initialization" rel="nofollow noopener">here</a>.</p>
<p>If you’re doing it using <code>torch.distributed.launch</code> utility, then try setting the <code>--master_port</code> flag.</p>
<p>Finally, it isn’t always all about the GPU memory left unused: Check if your data loader is fast enough? You could do this by checking your CPU usage. If all your CPUs are maxing out, or waiting on data io or whatever the case: if your GPU is idle waiting on data, then launching another training will just load the CPUs more, thus using even lesser GPU since the GPU would be idle most of the time waiting for data loading.</p>
Which Python module can I use after calling model.cuda()?

Wayne_Mai: <p>Hi, I am a newcomer to PyTorch.</p>
<p>As <code>cuda()</code> give a better performance since it’s on GPU, I wonder if I can call other module during CUDA calculation.</p>
<p>I have known that currently we can not use numpy directly in GPU mode, and if I want to do that I have to use <code>.cpu().numpy()</code> to migrate the data to CPU first.</p>
<p>So I am curious about other module. For example, just like using  <code>queue</code> module as following:</p>
<pre><code class="lang-auto">import queue
x=torch.cuda.FloatTensor([1.0])
q=queue.Queue()
q.put(x)
</code></pre>
<p>Above code didn’t get any errors. So does it run on GPU exactly? Which means I have created a Queue object on the GPU? And is it advisable to do that? Does it mean that some modules are supported by torch cuda but others are not ?</p>
<p>What’s more, if I hard code a list, dict or something else (eg. <code>x=[1,2,3,4,5]</code>) in my model code, will it (<code>list[1,2,3,4,5]</code>) be migrated to GPU memory automatically when use <code>model.cuda()</code> ?</p>
<p>Thank you very much.</p>

ptrblck: <p>What are you trying to achieve using the queue?<br>
Have a look at the <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing">Multiprocessing Best Practices</a>, as it might get tricky.</p>
<aside class="quote no-group" data-username="Wayne_Mai" data-post="1" data-topic="53105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayne_mai/48/14905_2.png" class="avatar"> Wayne_Mai:</div>
<blockquote>
<p>What’s more, if I hard code a list, dict or something else (eg. <code>x=[1,2,3,4,5]</code> ) in my model code, will it ( <code>list[1,2,3,4,5]</code> ) be migrated to GPU memory automatically when use <code>model.cuda()</code> ?</p>
</blockquote>
</aside>
<p>No, plain Python types won’t be pushed to the GPU. You would have to create a tensor and register it as an <code>nn.Parameter</code> (if trainable) or buffer (using <code>self.register_buffer</code>, if not trainable).</p>

Wayne_Mai: <p>Thank you very much! Actually I’m trying to implement a simple breadth first search algorithm, I’ll try if I can speed it using torch.multiprocessing.</p>
Need teammate for Pytorch Hackathon, creating Sparse Library

Santosh-Gupta: <p>I have an idea for a sparse training library for pytorch.</p>
<p>A lot of the work will involve speed/performance optimization, particularly parallelizing IO memmap operations so that it doesn’t slow down training.</p>
<p>The library will then be used to train embeddings for ~40 million research papers.</p>
<p>If you haven’t heard of the Facebook Pytorch Hackathon, Google ‘Facebook Pytorch Hackathon’</p>
RuntimeError: invalid argument 4: expecting vector of indices at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:16

Teresa_Ho: <p>I have two mask tensors which both have size of [b,c,w,h] (c=63*63)<br>
weight_1 and weight_2 are their corresponding label [b,1,w,h] ] indicating which position in mask should I pick<br>
Because mask_1 and mask_2 are paired (mask_1[i] is paired with mask_2[i] ), I want to select common min numbers of mask for each pair<br>
below is my code</p>
<pre><code class="lang-auto">def select_mask(mask_1, weight_1, mask_2, weight_2, o_sz=63, g_sz=127):
   
    mask_1 = mask_1.permute(0, 2, 3, 1).contiguous()
    mask_2 = mask_2.permute(0, 2, 3, 1).contiguous()

    sel_pm_1 = torch.tensor([])
    sel_pm_2 = torch.tensor([])
    sel_pm_1 = torch.autograd.Variable(sel_pm_1).cuda()
    sel_pm_2 = torch.autograd.Variable(sel_pm_2).cuda()
    for i in range(weight_1.size(0)):
        tmp = weight_1[i][0].view(-1) #[w*h] (w=25,h=25)
        tmp2 = weight_2[i][0].view(-1) #[w*h]
        tmp = Variable(tmp.data.eq(1).nonzero())  
        tmp2 = Variable(tmp2.data.eq(1).nonzero())
        n_tmp = tmp.nelement()
        n_tmp2 = tmp2.nelement()
        mi = min(n_tmp, n_tmp2)
        
        tmp = tmp[:mi].squeeze()
        tmp2 = tmp2[:mi].squeeze()

        T = mask_1[i].view(25*25,63*63)
        T2 = mask_2[i].view(25*25,63*63)
        tmp_pm_1 = torch.index_select(T,0,tmp)
        tmp_pm_2 = torch.index_select(T2,0,tmp2)
        sel_pm_1 = torch.cat([tmp_pm_1,sel_pm_1])
        sel_pm_2 = torch.cat([tmp_pm_2,sel_pm_2])

    if sel_pm_1.nelement()==0 or sel_pm_2.nelement()==0: return None,None,False

    #sel_pm_1 = [n,63*63]
    sel_pm_1 = sel_pm_1.view(-1, 1, o_sz, o_sz)
    sel_pm_1 = nn.UpsamplingBilinear2d(size=[g_sz, g_sz])(sel_pm_1)
    sel_pm_1 = sel_pm_1.view(-1, g_sz , g_sz)

    sel_pm_2 = sel_pm_2.view(-1, 1, o_sz, o_sz)
    sel_pm_2 = nn.UpsamplingBilinear2d(size=[g_sz, g_sz])(sel_pm_2)
    sel_pm_2 = sel_pm_2.view(-1, g_sz , g_sz)
    
    return sel_pm_1, sel_pm_2, True
</code></pre>
<p>Forwarding seems no problem , however , I got the following error message<br>
Where could possibly go wrong?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/teresa/SiamMask/tools/train_siammask.py", line 301, in &lt;module&gt;
    main()
  File "/home/teresa/SiamMask/tools/train_siammask.py", line 169, in main
    train(train_loader, dist_model, optimizer, lr_scheduler, args.start_epoch, cfg)
  File "/home/teresa/SiamMask/tools/train_siammask.py", line 255, in train
    loss.backward()
  File "/home/teresa/anaconda3/envs/siammask/lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/teresa/anaconda3/envs/siammask/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: invalid argument 4: expecting vector of indices at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:16
</code></pre>
How to properly install and utilize torch 1.1.0?

Deb_Prakash_Chatterj: <p>I am installing torchvision, pandas, PIL and other things, here is a glimpse -</p>
<pre><code class="lang-auto">!pip install -q pandas
# http://pytorch.org/
from os.path import exists
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag

platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\10/'    
accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision
import torch
!pip install -q Pillow==4.3.0
!pip install -q PIL
!pip install -q image
import PIL

#%reload_ext autoreload        &lt;------------— comment out 
#%autoreload 0                 &lt;------------— comment out
%matplotlib inline

!pip install --no-cache-dir -I pillow
</code></pre>
<p>But when I run this, the result is this -</p>
<blockquote>
<p>ERROR: torchvision 0.3.0 has requirement torch&gt;=1.1.0, but you’ll have torch 1.0.0 which is incompatible.<br>
ERROR: albumentations 0.1.12 has requirement imgaug&lt;0.2.7,&gt;=0.2.5, but you’ll have imgaug 0.2.9 which is incompatible.<br>
ERROR: Could not find a version that satisfies the requirement PIL (from versions: none)<br>
ERROR: No matching distribution found for PIL</p>
</blockquote>
<p>But I have torch 1.1.0 successfully installed -</p>
<pre><code class="lang-auto">!pip install -q torch

import torch
print(torch.__version__)
</code></pre>
<p>The result of the print is 1.1.0. Then why it is showing this result? I am doing all this in google colab. Thanks in advance.</p>

LeviViana: <p>You could use the cmd in the official <a href="https://pytorch.org/" rel="nofollow noopener">PyTorch’s website</a>.</p>

Deb_Prakash_Chatterj: <p>Solved, the main problem was in torch vision versions. Just had to update torchvision version to the latest edition.</p>

John_Deterious: <p>How did you update torchvision?</p>

ecdrid: <p><code>pip install torchvision==version or pip install torchvision --upgrade</code> ? For Linux, you can now get torchvision 0.4.0 directly as well with PyTorch-v1.2.0</p>
How can i continue my script even with manual interruption?

ecdrid: <p>Hello All <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>What i am trying to achieve is basically interrupting my model training in-between without getting the script killed (obviously weights will be cached by default in the loop itself) (i.e. the lines/functions after the training loop etc should execute without failure).</p>
<p>What are few ways that we can do this in Pytorch?</p>
<p>My Current guess is that we can write a custom callback and do this but what would the key stroke to trigger the same and how to handle that etc is what i am wondering…</p>
<p>Thanks Folks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Can I use torch.save() and torch.load to keep training my model?

cat-loves-donuts: <p>Well, I saved my model with torch.save(). And I can also test this model by using torch.load(). However, I just  wanna ask, can I keep training this model?Because when I did this, the code was “stopped”… I mean it still running, but it looks like the code was stocking there.</p>
<p>By the way, I am using Adam method…Do I need to save the optimizer too?</p>

ptrblck: <p>Have a look at the <a href="https://github.com/pytorch/examples/blob/4581968193699de14b56527296262dd76ab43557/imagenet/main.py#L249" rel="nofollow noopener">ImageNet example</a> to see, how to store all necessary information in order to be able to continue the training.<br>
Also, I would recommend to save and load the <code>state_dict</code> instead of the complete model.</p>

cat-loves-donuts: <p>Thank you so much for telling me those. Well, I used state_dict to save the model in the previous version.</p>
<p>Anyway, so it is possible to keep train the model which saved by torch.save() function?</p>

ecdrid: <p>you should check this out <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">saving_loading_models</a></p>
<p>Pytorch docs are just too good <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Populating a 4D tensor from 3D tensors using stack

Alex_Ge: <p>I am mostly certain this is something trivial, yet I can’t find much on Google about it.<br>
I’m populating 3D tensors from BGR data, which I need to place in a 4D tensor to transform into a batch for evaluation/testing purposes.<br>
I know how to get my 3D tensor:</p>
<pre><code class="lang-auto">img   = Image.open(file)
in_t  = self.img_tf(img).cuda(non_blocking=True).float()
</code></pre>
<p>And I know the size of the batch:</p>
<pre><code class="lang-auto"> def make_batch(self, faces):
        data  = []
        for (uuid, hwid, file) in faces:
            img   = Image.open(file)
            in_t  = self.img_tf(img).cuda(non_blocking=True).float()
            print(in_t.shape)
            data.append(in_t)
        self.input = torch.tensor(data)
        return self.input
</code></pre>
<p>My problem of course is how to stack the <code>in_t</code> in <code>self.input</code> so that it makes sense to use for propagation right afterwards:</p>
<pre><code class="lang-auto"> def run_batch(self):
        with torch.no_grad():
            output = self.net(self.input)
            pred   = output.argmax(dim=1, keepdim=True)
</code></pre>
<p>Placing them in a list dosn’t work, passing them one at a time by unsqueezing makes no sense since I know I have a batch for evaluation, so AFAIK I need to stack the BGR 3 channel tensor as a 4D tensor, with 1st dimension being the batch?</p>

InnovArul: <p>Have you looked into the function <code>torch.stack()</code>?</p>

Alex_Ge: <p>Yes, that’s what I’ve been trying to use, but I keep getting the dimensions mixed up.<br>
I’ve found another post on here asking the same question, and his solution was to just setup a 4D tensor and populate it directly:</p>
<pre><code class="lang-auto">        data  = torch.zeros([len(faces), 3, 224, 224])
        self.face_map = []
        i = 0
        for (uuid, hwid, file) in faces:
            img   = Image.open(file)
            in_t  = self.img_tf(img)
            data[i] = in_t
            i += 1
</code></pre>
<p>Like so. I imagine the “proper” approach would be to use <code>stack</code> instead?</p>

ptrblck: <p>This approach is fine. Alternatively, you could append the tensors in a list and use <code>torch.stack</code> on the list to get the same result.</p>

Alex_Ge: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> I’ll stick to using that one then. I’ve tried to use torch.stack but I seem to be getting the dimensions wrong, because every time I get an error/exception.<br>
What would be a minimal example of how to stack 3D (BGR 3, 224, 224) into a 4D batch tensor?</p>

ptrblck: <p>This would be a simple example:</p>
<pre><code class="lang-python">x = []
for _ in range(10):
    x.append(torch.randn(3, 224, 224))
x = torch.stack(x)
print(x.shape)
&gt; torch.Size([10, 3, 224, 224])
</code></pre>

Alex_Ge: <p>Aha! Now I see what I did wrong. Thanks a lot, this is actually cleaner code.</p>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1544199946412/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:111

Momofilm: <p>When train fcn, it happended cuda runtime error(59).Then, I tried to run it on CPU, getting errors as follows:<br>
0it [00:00, ?it/s]/home/zqc/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:2351: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.<br>
warnings.warn(“nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.”)<br>
/home/zqc/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.<br>
“See the documentation of nn.Upsample for details.”.format(mode))<br>
/home/zqc/anaconda3/lib/python3.6/site-packages/torch/nn/_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction=‘sum’ instead.<br>
warnings.warn(warning.format(ret))<br>
/opt/conda/conda-bld/pytorch_1544199946412/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [29,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.<br>
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544199946412/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=111 error=59 : device-side assert triggered</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 278, in <br>
main()<br>
File “train.py”, line 223, in main<br>
loss = loss_fn(output, target)<br>
File “/home/zqc/yxh/PaperExperiment/semseg/loss.py”, line 27, in cross_entropy2d<br>
weight=weight, size_average=False)<br>
File “/home/zqc/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py”, line 1790, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1544199946412/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:111</p>

ptrblck: <p>Your target tensor seems to contain out of bounds class indices.<br>
Make sure that your class indices are in the range <code>[0, nb_classes-1]</code>.</p>

Momofilm: <p>Thank you！I checked out the value of image, but forgot to check the “nb_classes”.</p>
PyTorchWrapper library

jkoutsikakis: <p>Hey all,</p>
<p>I developed PyTorchWrapper, a helper library for pytorch. It provides a systematic and extensible way to build, train, evaluate, and tune deep learning models. It also provides several ready to use modules and functions for fast model development.</p>
<p>If you’d like to check it out follow the following links.</p>
<p>install with: pip install pytorch-wrapper<br>
GitHub: <a href="https://github.com/jkoutsikakis/pytorch-wrapper" rel="nofollow noopener">https://github.com/jkoutsikakis/pytorch-wrapper</a><br>
docs: <a href="https://pytorch-wrapper.readthedocs.io/en/latest/" rel="nofollow noopener">https://pytorch-wrapper.readthedocs.io/en/latest/</a></p>
How does pytorch count batch size in SGD with multiGPU when it does Batch Norm?

Andrew_Paint: <p>When pytorch does the batch normalization in SGD, what batch size it use? Per GPU batch size, or total batch size over all GPUs?<br>
For example, suppose I set the per-GPU batch size as 32, and I use 8 GPUs. When Pytorch does batch normalization, what batch size it use, 32 or 32 x 8?<br>
My concern comes from a facebook’s paper “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”. If Pytorch’s implementation is the same with what this paper describes, I can modify the parameters of my model accordingly; if not, I need to know. Thanks!</p>

rasbt: <p>I am pretty sure that if you e.g., set the minibatch size to 32 in the dataloader and have say 8 GPUs, you get 4 data points per GPU during DataParallel (you can kind of see that based on the number of training instances per minibatch and also memory use) . I am wondering if you are referring to DataParallel in your question?</p>
<p>In practice, I found that yes, using e.g., 4 GPUs can maybe speed up computation by ~2.5 times, but then I also need to train 1.5 more epochs to get comparable results (compared to 1 GPU).</p>
<p>Also, I usually use minibatchsize * number of GPUs as my minibatch param to make best use of your GPUs. I.e., if you have a model with a current batch size that fills up 80% of the GPU memory, and you want to maintain that level of utilization, you’d need to increase the batch size then of course as the batch gets scattered across the GPUs.</p>

MariosOreo: <p>Hi,</p>
<p>Thanks for you detailed explanation.</p>
<aside class="quote no-group" data-username="rasbt" data-post="2" data-topic="37995">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/48/60_2.png" class="avatar"> rasbt:</div>
<blockquote>
<p>I usually use minibatchsize * number of GPUs as my minibatch param</p>
</blockquote>
</aside>
<p>What does the <code>minibatch param</code> means ? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> the epochs the model to train?</p>
<p>Thank you.</p>

rasbt: <p>sorry that was maybe a bit confusing. With minibatch param, I meant the minibatch size. E.g., if my minibatch size is 256 for 1GPU, I use 4*256 for four GPUs.</p>

Andrew_Paint: <p>What I’m asking is which batch size is used in Batch Normalization, when there are more than one GPU? Does Pytorch do batch normalization per GPU, or it do batch normalization accross all GPUs?</p>
<p>For example, let’s assume batch size per GPU is 32, and there are 4 GPUs, so, total batch size is 32 * 4 =128. So, when Pytorch does batch normalization, which ‘batch’ does it use? 32 or 128?</p>

Elegant_Lin: <p>Yeah, this is also what I found. So you mean we cannot implement linear acceleration using the current PyTorch?</p>