akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error while loading the image

Carolmeir: <p>Hello,</p>
<p>I am trying to fine-tune <a href="https://github.com/ialhashim/DenseDepth" rel="noopener nofollow ugc">this</a>. I am getting this error while loading the data. Please let me know how to go on with this.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Error:</p>
<blockquote>
<p>Loaded (1) images of size (480, 640, 4).<br>
Traceback (most recent call last):<br>
File “test.py”, line 34, in <br>
outputs = predict(model, inputs)<br>
File “/home/kbdp5524/Downloads/DenseDepth-master/utils.py”, line 12, in predict<br>
predictions = model.predict(images, batch_size=batch_size)<br>
File “/home/kbdp5524/anaconda3/envs/densedepth/lib/python3.7/site-packages/keras/engine/training.py”, line 1441, in predict<br>
x, _, _ = self._standardize_user_data(x)<br>
File “/home/kbdp5524/anaconda3/envs/densedepth/lib/python3.7/site-packages/keras/engine/training.py”, line 579, in _standardize_user_data<br>
exception_prefix=‘input’)<br>
File “/home/kbdp5524/anaconda3/envs/densedepth/lib/python3.7/site-packages/keras/engine/training_utils.py”, line 145, in standardize_input_data<br>
str(data_shape))<br>
ValueError: Error when checking input: expected input_1 to have shape (None, None, 3) but got array with shape (480, 640, 4)</p>
</blockquote>

ptrblck: <p>It seems you are using Keras, so you might get a better answer in their discussion board or on StackOverflow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to set padding to "same" on nn.Conv2d?

marksasp: <p>I’m a bit of a newbie on Pytorch with some TF background. What’s the equivalent of TF’s (or conceptually speaking) padding=“same” on Pytorch? I don’t really understand the explanation of padding in the docs.</p>

ptrblck: <p>Since the input shapes are not statically defined, you would have to set the padding size manually to get the same output size.<br>
You can find some convenience methods to calculate the padding for a specific input shapes here in the forum.</p>
Step size for Cyclic scheduler

Upgrade_Yourself: <p>What I would like to know is how to choose a reasonable number for the step size. This is a multilabel classification with 21 labels.<br>
My batch size is 8. And the number of images is 7000 images. Number for gradient accumulation<br>
is 4.<br>
Thank you</p>

ptrblck: <p>I think <code>CyclicLR</code> is usually called after each batch (and the docs also say so):</p>
<blockquote>
<p>Cyclical learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.</p>
</blockquote>

Upgrade_Yourself: <p>I didn’t formulate my question well.<br>
I was talking about the upper step size for the increasing half of the cycle.<br>
I choose it when initiating the scheduler.<br>
I’m not talking about scheduler.step().<br>
So in order to go from minimum lr to the maximum lr, how many steps should I choose?</p>

ptrblck: <p>Thanks for clarification.<br>
From the <a href="https://arxiv.org/abs/1506.01186">paper</a>:</p>
<blockquote>
<p>The length of a cycle and the input parameter stepsize<br>
can be easily computed from the number of iterations in<br>
an epoch. An epoch is calculated by dividing the number<br>
of training images by the batchsize used. For example,<br>
CIFAR-10 has 50, 000 training images and the batchsize is<br>
100 so an epoch = 50, 000/100 = 500 iterations. The final<br>
accuracy results are actually quite robust to cycle length but<br>
experiments show that it often is good to set stepsize equal<br>
to 2 − 10 times the number of iterations in an epoch. For<br>
example, setting stepsize = 8 * epoch with the CIFAR-10<br>
training run (as shown in Figure 1) only gives slightly better<br>
results than setting stepsize = 2 * epoch.</p>
</blockquote>
Specify size of per-class samples used for training

wenyugu: <p>I’m trying to explore using different size of per-class samples for transfer learning. For example, I want to only take 10 images per class for training versus 50 images per class. How do I specify the number of per-class samples without reorganizing the dataset?</p>

kl_divergence: <p>One thing you can do is what I did <a href="https://github.com/prajjwal1/autonomous-object-detection" rel="nofollow noopener">here</a>. You can randomly select image paths of 10 images of a particular class. After you have a list which has paths of these image files, you can define the rule in <code>__getitem__</code>. Dataloader will open the image on the fly using the path (by iterating over the list) and converting it into a tensor. You can have a look in the repo for more details. The concept is same.</p>
Tensorboard video tensor format?

pablovela5620: <p>I’m a little confused about the format of a video tensor for the function <code>add_video</code></p>
<p>The documentation states that</p>
<pre><code class="lang-auto">vid_tensor: (N, T, C, H, W). The values should lie in [0, 255] for type uint8 or [0, 1] for type float
</code></pre>
<p>I would image that N=number of frames, C=channels, H=Height, W=Width</p>
<p>What is T? frame rate?</p>

RylanSchaeffer: <p>I would guess T is number of frames.</p>

gebrahimi: <p>Never worked with video, but N should be batch size and T should be number of frames.</p>
Using model parallel for hessian vector product

gebrahimi: <p>I have alexnet:</p>
<pre><code>(0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
(1): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
(2): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(3): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(5): Linear(in_features=9216, out_features=4096, bias=True)
(6): Linear(in_features=4096, out_features=4096, bias=True)
(7): Linear(in_features=4096, out_features=102, bias=True)
</code></pre>
<p>which I am trying to calculate hessian vector product for. The Conv layers are put on <code>cuda:0</code> and the linear layers on <code>cuda:1</code>. I have the following code, which make sure the vector and the gradient are on the same gpu, but I get an error. <code>stored_grad</code> is a vector of all the gradients.</p>
<pre><code>class HVPOperator(object):

def __init__(self, model, loss, device):
    
    self.device = device
    self.model = model
    self.loss = loss  # loss function
    self.stored_grad = None  # stored gradient (on CPU)
    self.count_params = sum(p.numel() for p in self.model.parameters() if p.requires_grad)

    
def structured_vector(self, flat_vector):
    structured =[]
    start = 0
    for param in self.model.parameters():
        if param.requires_grad:
            size = param.shape.numel()
            vec = torch.reshape(flat_vector[start: (start+size)], param.shape)
            structured.append(vec.double().to(param.device))
            start += size
    return structured


def Hvp(self, vec, storedGrad=False):
    # Returns H*vec where H is the hessian of the loss w.r.t. the vectorized model parameters

    # convert numpy array to torch tensor
    if type(vec) is np.ndarray:
        vec = torch.from_numpy(vec)
    vec = vec.to(self.device)
    vec = vec.double()  # convert to double

    # compute original gradient, tracking computation graph
    grad_dict = self.structured_vector(self.stored_grad)

    self.zero_grad()
    gvp = [torch.sum(g*v) for g,v in zip(grad_dict, vec)]
    grad_dict = torch.autograd.grad(gvp, self.model.parameters(), create_graph=True)
    hessian_vec_prod = [v.double() for v in grad_dict]


    return hessian_vec_prod.data.double()
</code></pre>
<p>I get the following error which happens at index 10, meaning the kernel weights for first linear layer. Any idea how to solve this?</p>
<pre><code>grad_dict = torch.autograd.grad(gvp, self.model.parameters(), create_graph=True)
File "/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py", line 157, in grad
inputs, allow_unused)
RuntimeError: Function CatBackward returned an invalid gradient at index 10 - expected device cuda:1 but got cuda:0</code></pre>
No grad accumulator for a saved leaf! error

divinho: <p>My network applies modules <code>a</code>, <code>b</code> and <code>c</code> upon the input in that order.<br>
<code>c</code> looks like:</p>
<pre><code class="lang-auto">    def forward(self, anch, pos, neg):
        pos_out = self.fc(anch) + self.fc(pos)
        pos_out = to.sigmoid(self.fc_final(fu.relu(pos_out)))
        neg_out = self.fc(anch) + self.fc(neg)
        neg_out = to.sigmoid(self.fc_final(fu.relu(neg_out)))
        binary_loss = (-to.log(pos_out + 1e-7) - to.log(1 - neg_out)).mean()
        return binary_loss
</code></pre>
<p>When I wrap <code>b</code> (just does some indexing operations) in <code>torch.no_grad()</code>, I get an error <code>RuntimeError: No grad accumulator for a saved leaf!</code>. It works when I don’t wrap it, or when I change forward to concat the inputs before calling <code>self.fc</code> (a fully connected layer, of course the input dimension is changed when I do that).</p>
<p>What’s going on? And is wrapping with <code>torch.no_grad()</code> the right thing when I don’t want some operations to have an effect on the gradient?</p>

albanD: <p>Hi,</p>
<p>There is no <code>b</code> in your code sample.<br>
Could you share both code: the one that works and the one that fails so that we can reproduce this? Thanks !</p>

divinho: <p>This is <code>b</code>:</p>
<pre><code class="lang-auto">def create_pos_neg(x):
    """ x has shape (N, T, C) """
    with to.no_grad():
        sz = x.size(1) - 1
        idx = to.randint(0, sz, size=(2,)).cuda()
        idx_offset = to.randint(max(-idx[0].item(), -3), min(sz-idx[0].item(), 3), size=(1,))
        if idx_offset.item() == 0:
            idx_offset = 1
        else:
            idx_offset = idx_offset.item()
        indcs = to.randperm(x.size(0)).cuda()
        anchor = x[:, idx[0]]
        pos = x[:, idx[0].item()+idx_offset]
        neg = x[indcs, idx[1]]
    return anchor, pos, neg
</code></pre>
<p>It works if I don’t use <code>no_grad()</code> or change <code>c</code> to (concatening inputs)</p>
<pre><code class="lang-auto">def forward(self, anch, pos, neg):
        pos_out = self.fc(to.cat((anch, pos,), dim=-1))
        pos_out = to.sigmoid(self.fc_final(fu.relu(pos_out)))
        neg_out = self.fc(to.cat((anch, pos,), dim=-1))
        neg_out = to.sigmoid(self.fc_final(fu.relu(neg_out)))
        binary_loss = (-to.log(pos_out + 1e-7) - to.log(1 - neg_out)).mean()
        return binary_loss
</code></pre>

albanD: <p>Sorry, I can’t find any variables called a, b or c. Not sure what you mean here.<br>
Can you give a code sample that I can run to make this clearer?</p>

divinho: <p>Sorry I should have been clearer, a, b and c are not variables but each a group of operations (either a function or a nn.Module::forward). The point I was trying to make was that first I do operations where I want the gradient to be tracked (in <code>a</code>), then I do some where I don’t want them to be tracked (<code>b</code>), then I want them to be tracked again (<code>c</code>).</p>
<p>I’ll try and create a simple example tomorrow.</p>
Unable to clean CUDA cache with torch.cuda.empty_cache()

DzReal: <p>Dear all,</p>
<p>I ran into a situation where I need to duplicate a large tensor many times. To get around this, I only create a small number of duplicates each time in a loop. To ensure sufficient memory, <code>torch.cuda.empty_cache()</code> is called right before duplicating the tensor. The code is something like this</p>
<pre><code class="lang-auto">a = torch.rand(1, 256, 256, 256).cuda()
for ...
    torch.cuda.empty_cache()
    b = torch.cat([a]*100, 0)  # CUDA out of memory at this line
    # Do some operation with b
    # The resultant tensor from the operation is reasonably small
    ...
    # Clean up
    del b
</code></pre>
<p>However, I still ran out of memory because some cache are not cleaned for some reason. The error is shown below.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 7.63 GiB 
(GPU 0; 11.92 GiB, total capacity; 679.74 MiB already allocated; 7.51 GiB free; 3.29 GiB cached)
</code></pre>
<p>Since the version of my CUDA driver is 9.0, which doesn’t support the latest pytorch build, I can’t really use more sophisticated tools like <code>torch.cuda.memory_stats()</code>. So, I’m wondering what kind of cache can actually be cleaned with <code>torch.cuda.empty_cache()</code>?</p>

DzReal: <p>I managed to find the culprit of the problem. It’s not actually related to any of the code I posted above, but something upstream. But it does appear that <code>torch.cuda.empty_cache()</code> cannot clean all cached memory. I’ve searched through most of the documentations available, and the best I got is</p>
<blockquote>
<p>Calling <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.empty_cache" rel="noopener nofollow ugc"> <code>empty_cache()</code> </a> releases all  <strong>unused</strong>  cached memory from PyTorch so that those can be used by other GPU applications</p>
</blockquote>
<p>I’m not quite sure what kind of cached memory is <strong>used</strong>. It’ll be great if someone can clarify</p>
LSTM with 3D CNN for activity recognition

joekid: <p>Hi friends. I like to recognize activity  in video data using Conv3D + LSTM.</p>
<p>Only for testing, I coded:</p>
<blockquote>
<p>conv1 = nn.Conv3d(in_channels=3, out_channels=64, kernel_size=3, padding=1)<br>
pool1 = nn.MaxPool3d(kernel_size=2)</p>
<p>conv2 = nn.Conv3d(in_channels=64, out_channels=32, kernel_size=3, padding=1)<br>
pool2 = nn.MaxPool3d(kernel_size=2)</p>
<p><span class="hashtag-raw">#assume</span> we have two video data separated<br>
vid1 = torch.rand(1, 3, 25, 220, 240) # (N_samples, Ch, Frame, H, W)<br>
vid2 = torch.rand(1, 3, 30, 220, 240)</p>
<p>x1 = conv1(vid1) # 25 Frames to be processed<br>
x1 = pool1(x1)<br>
x1 = conv2(x1)<br>
x1 = pool2(x1) # output shape torch.Size([1, 32, 6, 55, 60]) - 32 channels, 6 Frames</p>
<p>x2 = conv1(vid2) # 30 Frames to be processed<br>
x2 = pool1(x2)<br>
x2 = conv2(x2)<br>
x2 = pool2(x2) # output shape torch.Size([1, 32, 7, 55, 60]) - 32 channels, 7 Frames</p>
</blockquote>
<p>After reading the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="noopener nofollow ugc">LSTM</a> docs, it has two basic parameters <code>input_size</code> and <code>hidden_layers</code> .</p>
<p>But I can figure out:</p>
<ol>
<li>how to setup correctly the LSTM parameters?</li>
</ol>
<pre><code class="lang-auto">    input_size = 32ch * 6fram * H * W ?
    hidden_size = what value, only intuition, ex. 100 ?
</code></pre>
<p>Or it exists some relation among <code>input_size</code> and <code>hidden_size</code> ?</p>
<ol start="2">
<li>And I have another issue,  the number of frames should be equals for input to LSTM ? And for working over time).</li>
</ol>
<p>Someone can help me to setup a basic example? Thanks.</p>

G.M: <ol>
<li>There isn’t a necessary relation between input and hidden size. But we usually set it to numbers like 32, 64, 128(u can try what number is better).</li>
<li>They must be presented the same in the batch, but they don’t have to be the same. You can, for example, pad inputs with different lengths into a certain max_length.</li>
</ol>

joekid: <p>Thank you sir, I understood the item 1.</p>
<p>Related 2, it is still unclear for me. I see that the batch should be (B, C, n_frames, H, W) with n_frames equals for all samples.</p>
<p>By padding, do you say that a short sequence can be completed with frames of zeros until it achieves the desired n_frames equal for all sequences. Please, give a little feedback on this point.</p>

G.M: <p>Yes, pad the short sequences to a longer sequence. But u don’t have to pad it to global maximum, u can pad them only to the maximum in the batch. There are many ways to pad in addition to zero-padding, u can do some experiment with that.</p>

joekid: <p>Ok, thank you. I will work on it.</p>
How to weight my own multiple losses?

qiangqiang_wu: <p>I have multiple losses, and i want to weight them seperately. More specifically, i want to set the weight of loss1 to 1, and the weight of loss2 to 0.5, is that correct?<br>
loss1 = nn.L1Loss(B, A)<br>
loss2 = nn.L1Loss(C, A)<br>
loss2 = loss2*0.5<br>
loss1.backward(retain_variables=True)<br>
loss2.backward()<br>
optimizer.step()</p>

albanD: <p>Hi,</p>
<p>Yes that would work.<br>
By the way, a more efficient way to do it is:</p>
<pre><code class="lang-auto">loss1 = nn.L1Loss(B, A)
loss2 = nn.L1Loss(C, A)
final_loss = loss1*1 + loss2*0.5
# Don't forget the optimizer.zero_grad() somewhere before the backward
final_loss.backward()
optimizer.step()
</code></pre>

qiangqiang_wu: <p>Thanks for your reply:grinning:</p>

Quintia_Zhao: <p>do you maybe have any idea, that, the weight of loss can also be learnt by the net?<br>
like:<br>
loss1 = nn.L1loss(B,A)<br>
loss2 = nn.L1loss(C,A)<br>
final_loss = alpha * loss1 + beta * loss2<br>
??<br>
how to deal with the alpha and beta?</p>

albanD: <p>You can either have them as Parameter or add a network that outputs the two values.</p>
Optimisation on aliased parameters

heavyfranz: <p>Hi everyone,<br>
I would like to constrain my optimisation on a subset of the training params, also imposing a relation among them, in order to reduce the number of DoF on my transform.<br>
I have searched for “alias”, “parameter sharing” but nothing seems to fit.</p>
<p>For example, using STN components I would like to learn just 3 variables (e.g. 2 offsets and 1 scale), instead of the full 6 parameters. I thought that it could be done building the parameter matrix by hand.<br>
In order to do so in my model  <strong>init</strong>:</p>
<pre><code class="lang-auto">def __init__:
  # [...]
  self.mytensor = torch.rand(N, C, H, W) #learn this
  self.offset = torch.randn(N, 2).double().cuda() # learn this
  self.scale = torch.ones(1).double().cuda() # learn this
  self.mytensor.requires_grad = True
  self.scale.requires_grad = True
  self.offset.requires_grad = True
  # [...]
</code></pre>
<p>while in my model <strong>forward</strong> I compose the temporary tensor reading the actual learned values</p>
<pre><code class="lang-auto">def forward(self, index):
  with torch.no_grad():
    # temporary affine transform matrix
    cm = torch.zeros(1,2,3, dtype=torch.double, device=self.mytensor.device, requires_grad=True)
    cm[0,0,0], cm[0,1,1] = self.magfactor[0], self.magfactor[0]
    cm[0,:,2] = self.offval[index]
  cm.requires_grad = True
  # affine transform matrix is ready for actual forward pass  
  grid = torch.nn.functional.affine_grid(cM, self.mytensor.size())
  y = torch.nn.functional.grid_sample(self.mytensor, grid)
  # some operations on y
  retval = y**2
  return retval
</code></pre>
<p>In my main script</p>
<pre><code class="lang-auto">model = myModel()
modelparams = [model.mytensor, model.scale, model.offset]
optim = torch.optim.Adam(modelparams_par, amsgrad=False)
loss = #...
[...] 
loss.backward()
print(model.offval.grad, model.magfactor.grad) # print None
</code></pre>
<p>But the gradients on those two parameters are None. If instead of building a temporary tensor, I use a complete theta tensor, (as in a plain STN), everything works fine.<br>
I thought that was a legit alias but it seems not.<br>
So how to optimise just  model.offval and model.magfactor? Thank you in advance</p>

heavyfranz: <p>I will reply myself:</p>
<p>in the model forward, the correct code is:</p>
<pre><code class="lang-auto">def forward(self, index):
  # temporary affine transform matrix, no more within no_grad
  cm = torch.zeros(1,2,3, dtype=torch.double, device=self.mytensor.device, requires_grad=True)
  cm = cm + 0.0 ### here is the change
  cm[0,0,0], cm[0,1,1] = self.magfactor[0], self.magfactor[0]
  cm[0,:,2] = self.offval[index]
  # affine transform matrix is ready for actual forward pass  
  grid = torch.nn.functional.affine_grid(cM, self.mytensor.size())
  y = torch.nn.functional.grid_sample(self.mytensor, grid)
  # some operations on y
  retval = y**2
  return retval
</code></pre>
Solve the system of linear equations with strictly upper triangular matrix

linyu: <p>Now, I have a system of linear equations Ax = y, where A is a strictly upper triangular matrix, I want to solve x efficiently by using the special structure of A. Pay attention to that, in general, this can be solved by x = torch.gesv(y, A), but this function can not utilize the strictly upper triangular structure of A.</p>

tom: <p>Does <a href="https://pytorch.org/docs/stable/torch.html#torch.trtrs" rel="nofollow noopener"><code>torch.trtrs</code></a> fit your purpose?</p>
<p>Best regards</p>
<p>Thomas</p>

linyu: <p>Thank you for your reply. This is my need.</p>

eusouoandre: <p>now I believe it’s being renamed to <code>torch.triangular_solve</code></p>
Can we use activation layer before Global average pooling?

Sanjayvarma11: <p>Hello,I am trying to get 99.4% accuracy in mnist dataset.I tried using different architectures but on one architecture i got 99.4% accuracy.In that one i used a nn.AdaptiveAvgPool2d(1).Here before this layere i used relu layer and send that output as input to this nn.AdaptiveAvgPool2d(1) layer.Can we use relu before global average pooling.<br>
Piece of code is</p>
<pre><code class="lang-auto">        x=self.conv7(x)
        x=F.relu(x)
        x=self.avgpool(x)
        x = x.view(-1, 10)
        return F.log_softmax(x)
</code></pre>
<p>here  self.avgpool = nn.AdaptiveAvgPool2d(1)</p>

vgsprasad: <p>You can use ReLU before adaptive average pool. Check the well known architectures like ResNet and SqueezeNet. They are using adaptive average pool at the end.</p>

Sanjayvarma11: <p>Thank you prasad for answering</p>
Variational dropout RNN (Gal NeurIPS 2016)

marga: <p>I am looking for a pytorch implementation of an RNN module with variational dropout (= SAME dropout mask at each timestep AND recurrent layers) as proposed by Gal and Ghahramani in the paper <em>A Theoretically Grounded Application of Dropout in Recurrent Neural Networks</em> <a href="https://papers.nips.cc/paper/6241-a-theoretically-grounded-application-of-dropout-in-recurrent-neural-networks.pdf" rel="nofollow noopener">https://papers.nips.cc/paper/6241-a-theoretically-grounded-application-of-dropout-in-recurrent-neural-networks.pdf</a> .</p>

ptrblck: <p>Would one of the approaches from <a href="https://discuss.pytorch.org/t/variational-dropout/23030/9">this topic</a> work or are you looking for another implementation? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Iridium_Blue: <p>As <a class="mention" href="/u/ptrblck">@ptrblck</a> indicated, I’ve been using the LSTM implementation with much success for the last few months, <a href="https://discuss.pytorch.org/t/variational-dropout/23030/9" class="inline-onebox">Variational dropout?</a> .</p>
<p>If it gives you any trouble I’d be happy to try and help.</p>

marga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/iridium_blue">@Iridium_Blue</a> thank you both for your reply. I saw the <a href="https://github.com/keitakurita/Better_LSTM_PyTorch/blob/master/better_lstm/model.py" rel="nofollow noopener">Better_LSTM_Pytorch</a> code, and I was wondering if this is indeed a correct implementation of recurrent dropout.<br>
VariationalDropout class: <code>        Applies the same dropout mask across the temporal dimension     See https://arxiv.org/abs/1512.05287 for more details.     Note that this is not applied to the recurrent activations in the LSTM like the above paper.     Instead, it is applied to the inputs and outputs of the recurrent layer.    </code></p>

Iridium_Blue: <p>It’s not a literal application of the paper, no.   Better_LSTM changes the dropout strategy from in a natural way - which I have found works much better for NLP anyway.</p>
<p>For a faithful implementation of the paper see AWD-LSTM.  Here’s a great writeup on it I found - <a href="https://yashuseth.blog/2018/09/12/awd-lstm-explanation-understanding-language-model/" rel="nofollow noopener">https://yashuseth.blog/2018/09/12/awd-lstm-explanation-understanding-language-model/</a></p>

marga: <p>Thank you <a class="mention" href="/u/iridium_blue">@Iridium_Blue</a>. I started implementing it myself based on the AWD-LSTM code. You may find my implementation for the paper <a href="https://github.com/mourga/variational-lstm" rel="nofollow noopener">here</a>.</p>
Tensorboard JSON dump of all scalars

ludwigwinkler: <p>In essence I simply want to dump all scalars in a json file such that I can import them quickly to matplotlib to create more flexible plots.</p>
<p>Doing that in hindsight after completing the experiment is cumbersome so I was hoping that I could do it at the end of the running experiment circumventing all the laborous tracking of paths-to-file etc.</p>
<p>I am aware that tensorboardX offers the <a href="https://tensorboardx.readthedocs.io/en/latest/tensorboard.html#tensorboardX.SummaryWriter.export_scalars_to_json" rel="nofollow noopener">export_scalars_to_json</a> function but that limits it down to tensorboardX and writing_to_scalars is required.</p>
<p>Does anybody have any experience with the EventAccumulator code in tensorboard who could help me out?</p>

ludwigwinkler: <p>I wrote a function which directly exports the data from the SummaryWriter.</p>
<p>In case somebody stumbles over it.</p>
<pre><code class="lang-auto">def export_jsondump(writer):

	assert isinstance(writer, torch.utils.tensorboard.SummaryWriter)

	tf_files = [] # -&gt; list of paths from writer.log_dir to all files in that directory
	for root, dirs, files in os.walk(writer.log_dir):
		for file in files:
			tf_files.append(os.path.join(root,file)) # go over every file recursively in the directory

	for file_id, file in enumerate(tf_files):

		path = os.path.join('/'.join(file.split('/')[:-1])) # determine path to folder in which file lies
		name = os.path.join(file.split('/')[-2]) if file_id &gt; 0 else os.path.join('data') # seperate file created by add_scalar from add_scalars

		# print(file, '-&gt;', path, '|', name)

		event_acc = event_accumulator.EventAccumulator(file)
		event_acc.Reload()
		data = {}

		hparam_file = False # I save hparam files as 'hparam/xyz_metric'
		for tag in sorted(event_acc.Tags()["scalars"]):
			if tag.split('/')[0] == 'hparam': hparam_file=True # check if its a hparam file
			step, value = [], []

			for scalar_event in event_acc.Scalars(tag):
				step.append(scalar_event.step)
				value.append(scalar_event.value)

			data[tag] = (step, value)

		if not hparam_file and bool(data): # if its not a hparam file and there is something in the data -&gt; dump it
			with open(path+f'/{name}.json', "w") as f:
				json.dump(data, f)
</code></pre>
The difference of "=" and "clone" in the computation graph?

Liangzhi_Tang: <p>Here are two cases use “=” and “clone”:</p>
<p>(1)<br>
x1 = torch.randn(4)<br>
x1.requires_grad()<br>
y1 = x1<br>
z1 = y1.mean().backward()</p>
<p>(2)<br>
x2 = torch.randn(4)<br>
x2.requires_grad()<br>
y2 = x2.clone()<br>
z2 =</p>
<p>Since x1 and x2 can get their grads (which are equal), and my questions are:</p>
<ol>
<li>what’s the difference in the two cases?</li>
<li>which one is preferred when creating a computation graph?</li>
</ol>

G.M: <p>(1) Using <code>=</code> is simply setting y1 to point to the object that x1 points to, so y1 and x1 both points to the same Python object.<br>
(2) <code>Clone</code> made a copy to x2, so even though they are still equal, they aren’t the same object. y1 points to the cloned object while x1 points to the original one.</p>

Liangzhi_Tang: <p>Thanks for the explanation! And which way is better when create a computation graph in your option?</p>

G.M: <p>It depends on your purpose because they have different behavior. If you are only doing simple calculations, I suggest (1). One thing to note is that because (2) uses <code>clone</code>, (2) uses more memory.</p>
OOM error even though there is free space

Ali2500: <p>I’m getting the following OOM error:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 450.00 MiB (GPU 0; 15.78 GiB total capacity; 12.43 GiB already allocated; 171.12 MiB free; 980.30 MiB cached)</p>
</blockquote>
<p>can someone explain the mechanics behind this? Even if the cached memory is discounted, there should still be 15.78 - 12.43 - 0.98 = 2.37GB of free memory. Does it have to do with all the available memory being fragmented into blocks smaller than 450MB? But even so, why does it say there is only 171MB of free space?</p>
MaxUnpool3d throws index error

aktgpt: <p>I’m running the follwing code:</p>
<pre><code class="lang-auto">pool = torch.nn.MaxPool3d((4, 4, 4), return_indices=True)
unpool = torch.nn.MaxUnpool3d((4, 4, 4))
o, i = pool(x)      #x is tensor of size [1, 4, 256, 256]
y = unpool(o, indices=i, output_size=x.size())
</code></pre>
<p>But the unpool operation throws follwing error:<br>
<code>IndexError: tuple index out of range</code></p>
<p>I’m not sure what I’m doing wrong. I think I provided the indices and the output size correctly. Has anyone faced a similar problem? What can be the possible solution?</p>

ptrblck: <p>You are using 3d layers, while your input would work with 2d layers.<br>
If you add another dimension, the code should work:</p>
<pre><code class="lang-python">x = torch.randn(1, 4, 256, 256, 256)
pool = torch.nn.MaxPool3d((4, 4, 4), return_indices=True)
unpool = torch.nn.MaxUnpool3d((4, 4, 4))
o, i = pool(x)      
y = unpool(o, indices=i, output_size=x.size())
</code></pre>

aktgpt: <p>Thank you for your reply. However, I forgot to specify that given the input of size <code>[1, 4, 256, 256]</code>, I wanted the output size of <code>[1, 1, 64, 64]</code>,i.e., get the max-pooling along channels as well. But after seeing your approach, I added one more dimension at index zero and then it seems to work fine. I forgot about adding the batch dimension. Thank you for your help.</p>
Pytorch mertic learning package problem

Yonatan: <p>I’ve tried to use pytorch-mertic-learning module, but after install it with the command:<br>
conda install pytorch-metric-learning -c metric-learning<br>
I couldn’t import it. I also notice that if I’m searching for conda list in terminal I got the following result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e68300ebe02ea80c77ab64ba1e6a2c6bd7571c03.png" data-download-href="https://discuss.pytorch.org/uploads/default/e68300ebe02ea80c77ab64ba1e6a2c6bd7571c03" title="Screen Shot 2020-02-09 at 11.14.39"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e68300ebe02ea80c77ab64ba1e6a2c6bd7571c03.png" alt="Screen Shot 2020-02-09 at 11.14.39" data-base62-sha1="wTcuvHZrXYhu6BxuUYvLJccJm8z" width="690" height="17" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-09 at 11.14.39</span><span class="informations">732×19 1.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Importance thing to say is that I got this issue on Jupiter NoteBook.</p>
<p>thanks!</p>

ptrblck: <p>These import errors are often created, if your Jupyter notebook uses a different environment (and thus kernel) as the one you’ve installed the package in.<br>
Make sure to select the right environment in your notebook or restart Jupyter in your virtual environment.</p>

Yonatan: <p>Hi,</p>
<p>Thanks for answering!<br>
If I got you right, I use conda environment so I made an installation using conda command.</p>
<p>Am I missing something?</p>

ptrblck: <p>This should usually work. However inside your notebook you should have a menu where you can select different kernels.<br>
I’ve seen Jupyter sometimes select the system Python installation instead of the conda one, so I stick to put everything in a virtual environment. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Yonatan: <p>Thanks for helping <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=9" title=":relaxed:" class="emoji" alt=":relaxed:"></p>
Network speed at different batch

slavavs: <p>I studied a lot of forum posts on this topic, but I can not find a good answer. I can’t understand why at bacht 50, 100, 400 I get the same speed on one epoch. I value num_workers me, but was getting even more work time. I have one video card. I want to buy 2 or 4 video cards, but I don’t understand if I will have acceleration or not? What determines the speed of work? I have a lot of data and one era lasts 25 minutes, it is very long, I need to make 5000 eras at least.</p>

Mazhar_Shaikh: <p>Hi, if I have understood this correctly, the problem you are facing is due to your data i/o being slower than the gpu ops. This could be happening due to a variety of reasons, including expensive read operation due to on the fly decompression, slow storage device, other programs using up cpu resources. You could try to check the speed of your data i/o without passing any data to your model, to see if it stays constant across batch sizes.<br>
Hope this gives you the right direction to solve this problem.</p>

slavavs: <p>I read data from RAM. (dataload = np.load(‘data.npz’), dtl = torch.from_numpy(data[‘arr_0’])<br>
I made timings on every line of my code. Time is spent only on forward pass and backward.<br>
Since I have reinforcement learning, I do forward pass first, get action and reward, then load the data into the second dataload, then forward pass and backward again. But it bothers me that changing batch does not speed things up. I think this is related to the size of my network.</p>
Custom autograd function with extra flags

deyan: <p>It’s possible to define a custom autograd function as shown here:</p>
<pre><code class="lang-auto">class Custom(Function):

    @staticmethod
    def forward(ctx, input):
        // Compute output
        return output

    @staticmethod
    def backward(ctx, grad_output):
       // Compute grad
        return grad
</code></pre>
<p>What should I do if I need my function to have a bool flag? That is,</p>
<pre><code class="lang-auto">class Custom(Function, flag):

    @staticmethod
    def forward(ctx, input, flag):
         if flag:
             // Compute output
         else:
            // Compute output
        ctx.flag = flag
        return output

    @staticmethod
    def backward(ctx, grad_output):
         flag = ctx.flag
         if flag:
             // Compute grad
         else:
            // Compute grad
        return grad
</code></pre>

G.M: <p>Just save the flag in <code>ctx</code>.</p>
DataParallel - Get GPU Number

soulslicer: <p>I am trying to understand how to use DataParallel API</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a></p>
<p>Currently, I have two GPU’s, and I am passing an object to the model. I can see that two GPU’s are now being used. I know that Pytorch launches to “threads” for each GPU. I want to have an API to get that number in my forward pass so I can allocate the GPU memory correctly. How can I know it?</p>

ptrblck: <p>What kind of number would you like to get in the forward pass and why do you want to allocate device memory manually?</p>
Debugging nan in computing loss

kl_divergence: <p>I’m using a transformer, using alternatives of softmax like sparsemax etc. With softmax, things work out smoothly, but with the other function, I’m getting nan somewhere during my training. I tried with anomaly detection, it says:<br>
<code>Function 'BinaryCrossEntropyWithLogitsBackward' returned nan values in its 0th output. </code><br>
This doesn’t happen when I use softmax. I’m now using anomaly detection on the attention directly. Now training takes 4x more time. What can be done to pin point the source of nan ?</p>
How to initialise the value of a tensor where a certain value doesn't exist for it

bing: <p>I want to initialise the value of a tensor which doesn’t get value from a certain function as below -</p>
<pre><code class="lang-auto">img_path = "Path_to_single_image"
mask = function(img_path)
class0,class1 = mask.unique(return_counts = True)
class0,class1
</code></pre>
<p>The general default output looks like this -</p>
<pre><code class="lang-auto">tensor([0, 1, 2, 3]) tensor([304960, 13746, 10765, 489])  
</code></pre>
<p>I want to loop through multiple files and the issue is many files don’t have all the the same tensors -<br>
so some file may just have only have data as below -<br>
<code>tensor([0, 1, 2]) tensor([304960, 13746, 10765]) </code></p>
<p>I need the above tensor values for some additional operation and the data like above is creating issue.<br>
<code>class0</code> will always same 4 values if it is present in the file - <code>tensor([0, 1, 2, 3])</code><br>
while values of  <code>class1</code> may differ.<br>
If any of the value of class0 is present , then I want class1 to be initialised to 0<br>
like</p>
<pre><code class="lang-auto">`tensor([0, 1, 2,3]) tensor([304960, 13746, 10765,0]) ` 
</code></pre>
<p>But I am not able to do it.<br>
I also tried initialising it using the below method but was not able to do it.</p>
<pre><code class="lang-auto">img_path = "Path_to_single_image"
mask = function(img_path)
class0 = torch.tensor([0,1,2,3],dtype = torch.float32)
class1 = torch.zeros([4],dtype = torch.float32)
class0,class1 = mask.unique(return_counts = True)
class0,class1
</code></pre>
<p>Output for above code -<br>
<code>tensor([0, 1, 2]) tensor([304960, 13746, 10765]) </code></p>

ptrblck: <p>You could use the returned unique values to index another tensor and add the counts to it:</p>
<pre><code class="lang-python">counts = torch.zeros(4)

data1 = torch.randint(0, 4, (1000, ))
u, c = data1.unique(return_counts=True)
counts[u] += c

data2 = torch.randint(0, 3, (1000, ))
u, c = data2.unique(return_counts=True)
counts[u] += c
</code></pre>
Across multiple nodes

slavavs: <p><code>nn.DataParallel</code>  requires that all the GPUs be on the same node.<br>
<code>nn.DistributedDataParallel</code> - GPUs can be distributed across multiple nodes.<br>
Explain what is meant by nodes?</p>

midhunharikumar: <p>Nodes in this context refers to the physical machine or VM. <code>nn.DataParallel</code> performs model replication and data distribution within the GPU’s connected to a single machine. All operations happen within a single system process. There is no multi-process synchronization. <code>nn.DistributedDataParallel</code> can syncronize models across multiple machines. Each GPU on a machine can be configured to run on a single process and weights are synchronized after backgprop. This synchronization can also happen over the network across multiple machines. The former is a single line conversion. The latter requires you to startup processes yourself although <code>torch.distributed.launch</code> is a quick way to do this.</p>
Gradient accumulation and scheduler

Upgrade_Yourself: <p>I’m using gradient accumulation and torch.optim.lr_scheduler.CyclicLR.<br>
Is there a special thing to consider when using gradient accumulation in this case?<br>
scheduler.step() should come after optimizer.step(). However because of gradient accumulation, this may not be the case.<br>
Thank you</p>

danitamm: <p>Hello,<br>
I was wondering the same thing with respect to huggingface transformers’ scheduler. I found that one of this library’s <a href="https://github.com/huggingface/transformers/blob/master/examples/run_glue.py" rel="nofollow noopener">examples</a> addresses this in the scheduler constructor by dividing the “pre-accumulation” number of steps by <em>gradient_accumulation_steps</em>:</p>
<pre><code class="lang-auto">t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs
scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
)
</code></pre>
<p>They then call <em>scheduler.step()</em> immediately after each <em>optimizer.step()</em>.</p>
<p>In your case with CyclicLR, I think you’ll want to divide your original <em>step_size_up</em> and <em>step_size_down</em> by <em>gradient_accumulation_steps</em>.</p>

Upgrade_Yourself: <p>Thank you! What you said is reasonable and matches what’s implemented in lr_finder fast_ai package.They basically train a number of batches equal to the number of gradient accumulation steps. Then they do scheduler.step().</p>
Normalize vectors to [-1,1] or [0,1]

HT_Liu: <p>I am sorry that the question may be easy.<br>
But I can not find the api in pytorch that normalize a vector into a range, such as into [0,1] or [-1,] which is useful for training</p>
<p>for example:<br>
a_i / sqrt(sum(a_i^2))</p>

HT_Liu: <p>It seems that there is no a api, so I use :</p>
<pre><code class="lang-auto">vec.div_(torch.norm(vec,2))
</code></pre>

Martin_Mundt: <p>I don’t think there currently is an in-build PyTorch function for what you want.<br>
If you really want a specific function for it, instead of simply doing this yourself, “torchsample”: <a href="https://github.com/ncullen93/torchsample" rel="nofollow noopener">https://github.com/ncullen93/torchsample</a> offers a “RangeNormalize” function where you can just specify min and max range values (tuples per channel: i.e. ((0,0,0),(1,1,1)) or just floats (0, 1)</p>

HT_Liu: <p>Wow, That looks very convience for the fresh pytorcher~</p>

Ambareesh_S_J: <p>For Normalization btw [0,1]</p>
<pre><code class="lang-auto">x = x/x.sum(0).expand_as(x) 
x[torch.isnan(x)]=0   #if an entire column is zero, division by 0 will cause NaNs
</code></pre>
<p>For Normalization btw [-1,1]</p>
<pre><code class="lang-auto">x = x/x.sum(0).expand_as(x) 
x[torch.isnan(x)]=0   #if an entire column is zero, division by 0 will cause NaNs
x = 2*x - 1
</code></pre>

stas: <blockquote>
<p>x = x/x.sum(0)</p>
</blockquote>
<p>I saw you post this in a few places, but it doesn’t look right - why are you dividing by a sum? And you’re not taking into account negative values. i.e. the above code works only under certain conditions, and while it does make the vector’s max value lesser than 1, it could end up being lesser than 0.0001.</p>
<p>The following code would work correctly for any given 2d vector:</p>
<pre><code class="lang-auto"># rescale vectors to a desired range
x = torch.tensor([2, -1.0, 5, 6, 7])
if not all(x == 0): # non all-zero vector
    # linear rescale to range [0, 1]
    x -= x.min() # bring the lower range to 0
    x /= x.max() # bring the upper range to 1
    x # tensor([0.3750, 0.0000, 0.7500, 0.8750, 1.0000])
    # linear rescale to range [-1, 1]
    x = 2*x - 1
    x # tensor([-0.2500, -1.0000,  0.5000,  0.7500,  1.0000])
</code></pre>

Ambareesh_S_J: <p>You are right, mine doesn’t look right. I was using this for a specific use-case. Your snippet is the one to follow. Thank you for pointing this out! Cheers!</p>
Jacobian gradient matrix

Sam3: <p>Hi ,<br>
i have this code to calculate the image gradient output  w.r.t input  (for every pixel from output image w.r.t each pixel in input image)  output image and input image have the same size (28 *28)<br>
but i have  missmatch error.  Mismatch in shape: grad_output[0] has a shape of torch.Size([784]) and output[0] has a shape of torch.Size([]).</p>
<p>my code</p>
<p>def jacobian(y, x, create_graph=False):</p>
<pre><code>image = Image.open('/Users/lionardo/Desktop/4_.png')
</code></pre>
<p>x = TF.to_tensor(image)<br>
x = x.unsqueeze(0)<br>
x.requires_grad_(True)</p>
<p>y = cnn(x)<br>
jac = []<br>
flat_y = y.reshape(-1)<br>
flat_x = x.reshape(-1)<br>
grad_y = torch.zeros_like(flat_y)</p>
<p>for i in range(len(flat_y)):</p>
<pre><code>for j in range (len (flat_x)):
    
    grad_y[i] = 1. 
   
     grad_x,= torch.autograd.grad(flat_y[i] ,flat_x[j], grad_y,retain_graph=True, create_graph = True )

jac.append(grad_x.reshape(x.shape))  

grad_y[i] = 0. 
   
#return torch.stack(jac).reshape(y.shape + x.shape) 
</code></pre>
<p>return torch.stack(jac).reshape(y.shape + x.shape)</p>
<pre><code># return jacobian(jacobian(y, x, create_graph=True))
</code></pre>
<p>print (jacobian)</p>
Finding gradient of a temporary variable without adding it to computational graph

curious: <p>I am trying to find the value of loss for a temporary set of variables in a neural network so that I can have their gradients but I do not want these variables in the computational graph. I have a matrix of weights, I want to know their gradients but this matrix is not in the neural network’s model parameters.</p>

albanD: <p>If these are used during the computation of the loss in a differentiable manner, you can get:</p>
<ul>
<li>call <code>.retain_grad()</code> on these intermediary elements so that their <code>.grad</code> field will be populated when you call <code>loss.backward()</code>.</li>
<li>Get the grad only for these with <code>grads = autograd.grad(loss, interm_tensors)</code>.</li>
</ul>

curious: <p>What should be the format of interm_tensors to do that?</p>

albanD: <p>You can do that on any Tensor that has <code>requires_grad=True</code>. Note that this field is set automatically when a Tensor is computed in a differentiable manner.</p>
Install pytorch 0.4.1 with CUDA 10.0 from source

Yilin_Ji: <p>Hi, I am trying to install pytorch 0.4.1 with CUDA 10 from source, I follow <a href="https://github.com/pytorch/pytorch/issues/19457" rel="nofollow noopener">instructions</a>, but I met some error:</p>
<pre><code class="lang-auto">make install -j12
[100%] Generating lib/libnccl.so
make[3]: warning: -jN forced in submake: disabling jobserver mode.
Compiling src/libwrap.cu &gt; /home/yji/pytorch-0.4.1/third_party/build/nccl/obj/libwrap.o
nvcc fatal : Unsupported gpu architecture 'compute_75'
Makefile:137: recipe for target '/home/yji/pytorch-0.4.1/third_party/build/nccl/obj/libwrap.o' failed
make[3]: *** [/home/yji/pytorch-0.4.1/third_party/build/nccl/obj/libwrap.o] Error 1
CMakeFiles/nccl.dir/build.make:60: recipe for target 'lib/libnccl.so' failed
make[2]: *** [lib/libnccl.so] Error 2
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/nccl.dir/all' failed
make[1]: *** [CMakeFiles/nccl.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
Failed to run 'bash tools/build_pytorch_libs.sh --use-cuda --use-nnpack nccl caffe2 nanopb libshm gloo THD c10d'
</code></pre>
<p>Do you have any idea about this error, I have tried long time to solve it ?<br>
My setup: RTX5000, Ubuntu 18.04, Python 3.6, CUDA 10.0 (I know pytorch 0.4.1 only support CUDA&lt;10.0, but I have to use both CUDA 10.0 and torch 0.4.1)<br>
Thanks in advance.</p>

albanD: <aside class="quote no-group quote-modified" data-username="Yilin_Ji" data-post="1" data-topic="69139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yilin_ji/48/14261_2.png" class="avatar"> Yilin_Ji:</div>
<blockquote>
<p>nvcc fatal : Unsupported gpu architecture ‘compute_75’</p>
</blockquote>
</aside>
<p>It looks like your cuda version does not support your GPU. You might want to upgrade to latest CUDA. Also make sure that the <code>nvcc</code> that is used is the one with the latest CUDA version.</p>

Yilin_Ji: <p>GPU: Quadro RTX 5000, CUDA compiler tool 10.0</p>
<p>CUDA 10.0 should support RTX <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

albanD: <p>Hmm 7.5 is the latest compute capability, you might want 10.1 or even 10.2.<br>
I couldn’t find the info online. But given the error message, I would bet on this.</p>