akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loss in self-supervised training

shrutee: <p>I am training a self-supervised learning model. The <code>training loss</code> is moving within a range of values, even after training for 50+ epochs. And not decreasing after a certain point.</p>
<p>I want the model to overfit. I have set dropout to 0. Will try playing with the learning rate. What else can be done?</p>
<p>Training loss currently</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f9732f33754a5f1b0ff7870bd8c226e3ac15dea.png" alt="xyz" data-base62-sha1="dDDiJix0bipKwWbJ4zPT2hkK0xY" width="378" height="252"></p>
RNN backward can only be called in training mode, even if I just wanted to compute grdient w.r.t. inputs

Huimin_ZENG: <p>Hello everybody,</p>
<p>I understand that we need to set RNN in training mode to optimize the weights of the neural network. However, in my case, I was trying to use <code>torch.autograd.grad</code> to compute the gradient w.r.t. the inputs. It still returned the same error.</p>
<pre><code class="lang-auto">embedded_grad = torch.autograd.grad(loss, embedding_temp)[0]
</code></pre>
<p>Is this normal or I have a bug in my code?</p>
<p>thanks!</p>
Apply transformations on a set of points (boundary box)

Neofytos: <p>Hi everyone,</p>
<p>I’ve come across an issue that I can’t wrap my head around.</p>
<p>A part of my network outputs the (x, y, width, height) of a boundary box which I then use to localize the corresponding part of the initial image.</p>
<p>The issue is that I am also applying random transformations (RandomHorizontalFlip, RandomVerticalFlip,         transforms.RandomRotation, RandomColorJitter) and so before I can go on and use the coordinates of the boundary box, I need to reverse the transformations that were applied. The reason for this is that I am dealing with huge images (whole slide images, digital pathology) and so I am always reading the images from the source.</p>
<p>Is there any direct or indirect way I can go about this? Also, if it’s of any help, I have stored the random seed that was used to generate the transformations. If I knew which transformations were applied and with what parameters, I would be able to reverse the transformations but I don’t think this is currently possible.</p>
<p>The only solution I can see is to implement the transformations myself. Any ideas on less time-consuming approaches?</p>
<p>Let me know if this is not clear enough. Thanks in advance!</p>

ptrblck: <p><a class="mention" href="/u/sgugger">@sgugger</a> created recently a <a href="http://docs.fast.ai/vision.image.html#Applying-transforms">transformation package</a> for fast.ai which seems to be working on images, bounding boxes, segmentation maps etc.<br>
It seems to be the tool you are looking for. As I’m not that familiar with fast.ai I don’t know how easy it would be to use these transformations in PyTorch directly.</p>
<p>Let me know, if that would work for you!</p>

Neofytos: <p>I am looking into it. Thanks for the fast reply! I will provide an update once I have something <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ham952: <p>its late though …</p>
<p>but for those who are still facing same problem … try this</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/Paperspace/DataAugmentationForObjectDetection" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/9925401?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Paperspace/DataAugmentationForObjectDetection" target="_blank" rel="nofollow noopener">Paperspace/DataAugmentationForObjectDetection</a></h3>

<p>Data Augmentation For Object Detection. Contribute to Paperspace/DataAugmentationForObjectDetection development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Does Transformations = Augmentations?

alx: <p>I was wondering if pytorch transformations are equivalent to augmentations.</p>
<p>Say I use:<br>
<code>torchvision.transforms.RandomHorizontalFlip(p=0.5)</code></p>
<p>Does that mean that an image will be flipped horizontally and passed through the model or that both the original and flipped image are passed?</p>
<p>Case A - One image is passed.<br>
Case B - Two images are passed.</p>

AladdinPerzon: <p>I believe it works in the way that each time you pass through an image there’s a 50% probability that it will be flipped horizontally. In other words a single image is sent in and mapped to a single image with varying outcomes (50% flipped, 50% normal).</p>

alx: <p>That is per epoch. So if I have a epoch=1 and p=0.5, my model will see 50% of the images flipped and 50% non flipped. That would be considered transformations.</p>
<p>However, if I have more epochs I guess almost every image will be seen in their original and transformed form. So it would be an augmentation.</p>
<p>Can anyone confirm?</p>

ham952: <p>yeah … its like that …</p>
<p>Augmentation would not increase the number of images i-e Case A is applicable …</p>
<p>the main benefit of augmentation here lies in the fact that for next epoch , a transformed image (maybe, based on probability ) is passed … hence model is practically trained on a new image</p>
Quantization for conv models with custom modules

pytos: <p>In per-channel quantization for conv models (done following <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a>), the scales and zero points of weights and intermediate outputs of the model are always 1 and 0, i.e. the range is same as in the float model. I have a custom module in between convbnrelu modules, which I want to quantize separately. But the quant and dequant stubs are only rounding off or float typecasting the outputs. Since there is no scale and zero point attached, I am not able to quantize my custom model properly. Is there any other way to quantize so I can get scales and zero points (the output/weight range is spread out to -127 to 128 or 0 to 255 rather than being same) for the output of the quantized convbnrelu layer?</p>

dskhudia: <p>scales and zero_points are 1 and 0 by default. Did you run calibration step, quantize step?</p>
How to see the quantized weights?

HumberMe: <p>like tensor.int_repr() can return the int values of quantized tensor, how can we get the int values of the quantized weights(_packed_params).<br>
while printing the _packed_params, we can get a uint8 tensor, what does it stand for?</p>

dskhudia: <p>Yes. Please use &lt;q_module&gt;.weight(). It returns a quantized tensor of weights and you can use int_repr() to get the int8_t values of weights. Note: Weights are usually quantized with dtype = int8_t.</p>
Model converging to similar output regardless the loss magnitude

manurare: <p>Hi,</p>
<p>I have a multiple term loss where the term3 is weighted by a hyperparameter w.</p>
<pre><code class="lang-auto">loss = term1+0.006*term2+w*term3
</code></pre>
<p>I’m trying to do a grid search of my w parameter using values from 0.001 to 1x10^17 in steps multiples of 10. My model was pretrained previously in another dataset but with the term3 set to zero.</p>
<p>Now I would like to load the pretrained model and retrain my model for 25 epochs on each value in the grid search with the influence of term3. Surprisingly, the absurd high values converge into almost the same results as the low values. For a better understanding, my model is an encoder/decoder network. I thought that the higher the term3 is weighted in the loss the most tweaked my model parameters will be and eventually, the more weirdest-looking outputs I would get in the generated samples. Am I missing anything here?</p>
<p>Learning rate and optimizers are kept the same during all training.</p>
Android and PyTorch

ntinos: <p>I have trained a custom model and wanna add it to my android app. I saved the model with jet but when I run it on my android device I get this error:</p>
<p>com.facebook.jni.CppException: empty not implemented for TensorTypeSet(VariableTensorId, CUDATensorId) (empty at aten/src/ATen/Functions.h:3999)</p>
<p>Any fixes ?</p>

albanD: <p>Hi,</p>
<p>It seems like you’re trying to use a CUDA Tensor in your mobile app. I’m pretty sure that is not supported. You might want to make sure that your model is on the CPU before doing the export.</p>

ntinos: <p>Any code advice how to do that ?<br>
When I trained my model I moved it to the cpu and then extracted with this code:</p>
<pre><code class="lang-auto">model.cpu()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("app/src/main/assets/model.pt")
</code></pre>

albanD: <p>That looks right. And when you use this model you see the error above?<br>
How do you create the input to the model on the mobile side?</p>

ntinos: <p>Im using the sample PyTorchDemo app from pytorch. Just changed the model’s name in the code nothing else.</p>

albanD: <p>This is weird, maybe someone from the mobile team will have a better idea?</p>
<p>You can also try to double check your model to make sure it does not contain any weight that is on the gpu?</p>

ntinos: <p>hmm okay thanks for your help.<br>
is there any easy way how to check the location of the weights ?</p>

albanD: <p>You can print the <code>.device</code> field of any Tensor to check where it lives.<br>
Also you can try to install a cpu-only build of pytorch and try to load your model in there. If there is cuda weights in it, it will fail as well, but with a friendlier error message.</p>

Untitled: <p>Hi! I have the same problem, even with model.cpu().<br>
Did you find a solution?</p>
Low test accuracy in multi-class classification

joepareti54: <p>This is a multi-class classification of 224x224 images, I am using VGG16 and only training the classifier.<br>
While the losses decrease, the test accuracy stays almost constant at low vlaues-<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6af8a2d064461fbef71f4ab4bdd4ed80939acbf2.png" data-download-href="https://discuss.pytorch.org/uploads/default/6af8a2d064461fbef71f4ab4bdd4ed80939acbf2" title="test0"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6af8a2d064461fbef71f4ab4bdd4ed80939acbf2.png" alt="test0" data-base62-sha1="fgjjlq8hy6gTn8LS1AmPRSm7Sgi" width="602" height="500" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test0</span><span class="informations">725×602 26.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

alx: <p>I think there’s a few mistakes in your code. Where’s the validation? Is that what you call test?</p>
<p>Try to emulate this:<br>
<a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/pytorch_with_examples.html</a></p>

AladdinPerzon: <p>The loss seems to be going down nicely but there seems to be something wrong with the accuracy. Could you check that training accuracy actually increases first?</p>
<p>Also perhaps just to structure the code a little nicer you could use something like this (I have a little difficult time reading your code now)</p>
<pre><code class="lang-auto">def check_accuracy(loader, model):
    num_correct = 0
    num_samples = 0
    model.eval()
    
    with torch.no_grad():
        for x, y in loader:
            x = x.to(device=device)
            y = y.to(device=device)
            
            scores = model(x)
            _, predictions = scores.max(1)
            num_correct += (predictions == y).sum()
            num_samples += predictions.size(0)
        
        print(f'Got {num_correct} / {num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}') 
    
    model.train()
</code></pre>
<p>Edit: Included <code>model.train()</code> at end of code.</p>
Doubt about multi-layer RNN

jordiae: <p>Hi everyone! I have found something in PyTorch’s documentation that challenges my view on the computation that multi-layer RNNs are supposed to do. I would greatly appreciate it if you could correct me if I don’t understand it properly.</p>
<p>In RNN’s documentation, it is said that each layer receives the input and h_(t-1), which is “the hidden state of the previous layer at time t-1”.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f42b4de2b801a74d143fe179e7bbaa884188e5ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/f42b4de2b801a74d143fe179e7bbaa884188e5ef" title="Screenshot_20200410_140741"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f42b4de2b801a74d143fe179e7bbaa884188e5ef_2_690x398.png" alt="Screenshot_20200410_140741" data-base62-sha1="yQ1hsQndt8u1OhAPX0h1gqJdq3B" width="690" height="398" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f42b4de2b801a74d143fe179e7bbaa884188e5ef_2_690x398.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f42b4de2b801a74d143fe179e7bbaa884188e5ef_2_1035x597.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f42b4de2b801a74d143fe179e7bbaa884188e5ef.png 2x" data-dominant-color="F4F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20200410_140741</span><span class="informations">1093×631 75.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I understand that, for the first layer, x_t is the original input, and for the rest, x_t is the <strong>current</strong> hidden state of the <strong>previous</strong> layer.</p>
<p>However, regarding h_(t-1), isn’t it a bit weird to receive the previous hidden state of the <strong>previous</strong> layer? Shouldn’t it receive its own previous hidden state? Also, this definition does not seem to be compatible with single-layer RNNs.</p>
<p>From this post: <a href="https://discuss.pytorch.org/t/what-is-num-layers-in-rnn-module/9843/2" class="inline-onebox">What is num_layers in RNN module? - #2 by SimonW</a> I understand that it uses its <strong>own previous</strong> hidden state, not the <strong>previous</strong> hidden state of the <strong>previous layer</strong>.</p>
<p>Many thanks in advance.</p>

googlebot: <p>h[t] and h[t-1] from different layers would mean a non-recurrent formula for a deeper layer, so it must be a documentation issue or poor wording.</p>

jordiae: <p>I see, thanks for your answer!</p>
Correct way to compute derivative of outputs w.r.t inputs

tomergolany: <p>The input to my network is of shape [batch_size, 1], and the output is of shape [batch_size, 3].<br>
That is the network gets as input a single scalar and outputs a 3d vector.<br>
My goal is to get the derivative of each of the elements in the output w.r.t to the input. I expect that the dimensions of the output will be [batch_size, 3] as well.</p>
<p>Currently, this is how I do it:</p>
<pre><code class="lang-auto">def d_network_d_input(network_obj, input_to_network):
    """
    :param network_obj: netowrk object that returns outpout of shape: [batch_size, x, y, z]
    :param input_to_network: shape [batch_size, T]
    :return: Gradient of output w.r.t to input - [batch_size, 3]
    """

    network_obj.zero_grad()
    output_from_network = network_obj(input_to_network)

    if input_to_network.grad is not None:
        input_to_network.grad.data.zero_()
    for single_batch in output_from_network:

        output_x, _, _ = single_batch
        output_x.backward(retain_graph=True)

    dxdt = input_to_network.grad.clone()

    input_to_network.grad.data.zero_()
    for single_batch in output_from_network:

        _, output_y, _ = single_batch
        output_y.backward(retain_graph=True)

    dydt = input_to_network.grad.clone()

    input_to_network.grad.data.zero_()
    for single_batch in output_from_network:
        _, _, output_z = single_batch
        output_z.backward(retain_graph=True)
    dzdt = input_to_network.grad.clone()

    return dxdt, dydt, dzdt, output_from_network
</code></pre>
<p>Is this the correct way, and is there any efficient way?</p>
<p>Thanks!</p>
Math background of SVD autograd

godlike: <p>Hi, I am working on problems about SVD autograd. I want to ask how  torch computes the gradient of SVD , I mean, the math background.</p>

albanD: <p>Hi,</p>
<p>You can find good ressources online about this. For example: <a href="https://j-towns.github.io/papers/svd-derivative.pdf">https://j-towns.github.io/papers/svd-derivative.pdf</a></p>

godlike: <p>Thanks for your share. I get it.</p>
For FasterRCNN, do you need to do bounding box regression yourself?

damonbla: <p>My training code is simply adding up all of the losses returned from the pretrained fasterrcnn model and sending that backwards, like this:</p>
<pre><code class="lang-auto">loss_dict = model(images, targets)

loss_classifier = loss_dict['loss_classifier']
loss_box_reg = loss_dict['loss_box_reg']
loss_objectness = loss_dict['loss_objectness']
loss_rpn_box_reg = loss_dict['loss_rpn_box_reg']

total_loss = loss_classifier.mean() + loss_box_reg.mean() + loss_objectness.mean() + loss_rpn_box_reg.mean()
total_loss.backward()
</code></pre>
<p>I’m only extracting and updating the final layer of the model for training. Is that the correct way to do it, or do I need to some bounding box regression on my own, or some other method?</p>
<p>The training on my own objects, with 1220 annotated images, is just not that great. It’s barely 50% correct, and that’s in easy images.</p>
How could I return a tensor from c++ to python?

coincheung: <p>Hi,</p>
<p>I am trying to  load data with c++ and return the c++ data to pytorch directly as a tensor. Is there any tutorial on how could I do this?  By the way, is pytorch tensor always contiguous in the memory ? Namely, if I need a (N, M) tensor, is it likely that I prepare N c++ pointers each of which holds the data for a (1, M) tensor and I pass these pointers to some api to compose a pytorch tensor without copying memory?</p>

albanD: <p>Hi,</p>
<p>The pytorch Tensor will require a contiguous chunk of memory. But you can specify size and stride so it does not have to be stored in a conitguous way in that contiguous chunk of memory.<br>
You have some information in the cpp extension doc <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a> or in the cpp API doc <a href="https://pytorch.org/cppdocs/">here</a>.<br>
But the simplest solution is most likely going to be to create a Tensor of the right size and then copy your content into it.</p>
GPU to cpu tensor transfer

Balamurali_M: <p>I have two GPUs, and I am using cuda:1 for validation. The model and dataset ran in cuda:1. But when I use the .to(‘cpu’) it uses some memory from cuda:0. Is this normal or I am doing some mistake ?</p>

albanD: <p>Hi,</p>
<p>It may. You want to make sure that you set the cuda device to be <code>1</code> to avoid using the 0th one.<br>
Note that if you never want to use the other device, the best way is to set the env variable <code>CUDA_VISIBLE_DEVICES=1</code> and you won’t even see the GPU 0 anymore.</p>
Num_workers=0 use multiple processes..?

sh0416: <p>Hi,</p>
<p>I use dataloader with <code>num_workers=0</code>, but CPU usage is more than 100%.</p>
<p>Is it possible??</p>
<p>Thanks,</p>

sh0416: <p><code>torch.set_num_threads(1)</code> works.</p>
Edit huge tensor piece-wise by using references

bokeh: <p><strong>Question</strong>: How can I assign values to tensor <code>p</code>, while not loosing the shared storage with <code>t</code>, such that every change on <code>p</code> will be visible in <code> t</code>? Note that I need to avoid element-wise iteration.</p>
<p><code>Context</code>:<br>
I have a huge tensor <code>t</code> with dim n x m. And another huge tensor <code>v</code> with dim l x k, containing information that has to be parsed somehow into <code>t</code>.<br>
During a loop, I always get a piece of <code>t</code> out. I call it <code>p</code>. Similar to the following code (assuming that the array pos is given):</p>
<pre><code class="lang-auto">for i in range(0, len(pos)-1):
   p = t.narrow(0, pos[i], pos[i+1])
   # now assign values to p that come from v by some parsing logic.
</code></pre>
<p>Since <code>t</code> and <code>p</code> share the same underlying storage:<br>
If I assign values to <code>p</code> these should be also assined to <code>t</code>. (Which in fact works, if I manually do <code>p[0][0]=1,</code> for example.)<br>
Due to efficiency problems I cannot loop through every element of t by using indices.</p>
<p>Since my parsing logic is encoded in an index array, I thought that<code> torch.gather</code> would perfectly fit my needs.<br>
Unfortunately the returned tensor is a new copy of the data. Such that <code>p</code> is assigned correctly to the values but <code>t</code> is not (since the reference got lost).</p>
<p>Something like:<br>
<code>p = v[index].clone().detach()</code><br>
Did not work as well.</p>
<p><strong>Note</strong>: I cannot operate on <code>t</code> directly due to some program logic.</p>

googlebot: <p>Hi, perhaps you’re looking for p.copy_ method</p>

bokeh: <p>Thank you. This works as I requested.<br>
Unfortunately the Backwards pass does not work anymore:</p>
<p><code>RuntimeError: leaf variable has been moved into the graph interior</code></p>
<p>Do you have a suggestion on how to overcome this issue?</p>

googlebot: <p>Ouch, supporting backprop through such a “buffer” is tough, I don’t think pytorch has a solution for that. Maybe writing your “gatherings” inside autograd.Function could work, then in backward function you’d have to scatter gradients to original positions in v (torch.gather does backward with scatter_add_, but your case is more complex, so you’d have to figure how to map indexes).</p>
Convolution for a batch of images

denexo: <p>I was wondering how the convolution operation is implemented for a batch of N images, i.e. for a tensor of shape (N, C, H, W). Is it fully vectorized or is there a for-loop over N images? Also, if we have more filters than one, does it use one additional for-loop over the filters?</p>

ptrblck: <p>There are a variety of convolution algorithms.<br>
Some are calling an (implicit) im2col and use a matrix multiplication afterwards. Other might use an FFT  or Winograd approach etc.</p>
<p>Also, if you are using <code>torch.backends.cudnn.benchmark = True</code>, the first iteration for a new input shape will run some benchmarking and select the fastest kernel.</p>

denexo: <p>Thanks ptrblck! In case of <code>im2col</code> (I couldn’t find the source code, but even if I find it I am afraid it is implemented in c++ and I won’t be able to understand it), if we pass a batch of N images, is it ran N times over each image using a for-loop? Also if we have more than one filters, will it require another for-loop? Or is <code>im2col</code> somehow cleverly vectorized so that it can handle tensors of shape (N, C, H, W) without for-loops (just a single matrix multiplication)?</p>

ptrblck: <p><code>im2col</code> can be applied via <code>unfold</code>. There should be no loops in the matrix multiplication and the unfolding should be implicit for performance reasons.</p>
<p>The usage of loops might slow down the execution in a lot of use cases, but there might be use cases, where it would make sense to use a loop instead of a dense operation.</p>
nn.LSTM seems to produce different results with/without using PackedSequence

vincentzlt: <p>Here is the code example where I tried to encode a sequence using <code>nn.LSTM</code>:</p>
<pre><code class="lang-python">import torch
from torch import nn

src = torch.tensor([[1, 2, 3, 0, 0], [2, 3, 4, 5, 1], [1, 3, 6, 4, 0]])
src_len = torch.tensor([3, 5, 4])

embed = nn.Embedding(7, 4, 0)
lstm = nn.LSTM(4, 4)

src_embedded = embed(src)

# without PackedSequence
output, hidden = lstm(src_embedded)

# with PackedSequence
src_embedded_packed = nn.utils.rnn.pack_padded_sequence(
    src_embedded, src_len, True, False)
output_packed, hidden = lstm(src_embedded_packed)
output_unpacked, unpacked_len = nn.utils.rnn.pad_packed_sequence(
    output_packed, True)

assert output.equal(output_unpacked)
</code></pre>
<p>I got <code>AssertionError</code> when trying to compare the output computed with <code>PackedSequence</code> or without. Is this normal behavior? Thanks!</p>
Indexing a tensor with another tensor vs an array

denexo: <p>What is the exact reason for treating tensors and (numpy) arrays differently for <em>indexing</em> of tensors? For example:</p>
<pre><code class="lang-auto">indices = torch.tensor([[0, 1], [0, 2]])

t = torch.arange(1,28).reshape(3,3,3)
# tensor([[[ 1,  2,  3],
#          [ 4,  5,  6],
#          [ 7,  8,  9]],

#         [[10, 11, 12],
#          [13, 14, 15],
#          [16, 17, 18]],

#         [[19, 20, 21],
#          [22, 23, 24],
#          [25, 26, 27]]])



&gt;&gt;&gt; a[indices.numpy()]
tensor([[ 1,  2,  3],
        [16, 17, 18]])

&gt;&gt;&gt; a[indices]
tensor([[[[ 1,  2,  3],
          [ 4,  5,  6],
          [ 7,  8,  9]],

         [[10, 11, 12],
          [13, 14, 15],
          [16, 17, 18]]],


        [[[ 1,  2,  3],
          [ 4,  5,  6],
          [ 7,  8,  9]],

         [[19, 20, 21],
          [22, 23, 24],
          [25, 26, 27]]]])

</code></pre>
<p>I understand how it works, but is there a particular reason for this? Isn’t having this differentiation a little bit confusing and potentially can lead to errors?</p>

denexo: <p>Moreover, using bool arrays doesn’t work:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; b = t &lt; 15

tensor([[[ True,  True,  True],
         [ True,  True,  True],
         [ True,  True,  True]],

        [[ True,  True,  True],
         [ True,  True, False],
         [False, False, False]],

        [[False, False, False],
         [False, False, False],
         [False, False, False]]])

&gt;&gt;&gt; t[b.numpy()]
IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)
</code></pre>
<p>But it works just fine if we use bool tensor without converting it to numpy array:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t[b]
tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14])
</code></pre>
<p>Why is that happening? Both the bool tensor <code>b</code> and its numpy version <code>b.numpy()</code> are exactly the same in shape and values.</p>
How can I buy ptrlblck a drink?

darek.kleczek: <p><span class="mention">@ptrblck_de</span> I’m getting started with Pytorch and your posts on this forum helped me SO MUCH!!! I really wanted to appreciate it! If there’s a way to do something virtually as a token of gratitude (e.g. donation etc.) please let me know! Hope you’re safe, have a great day! Cheers, Darek</p>

ptrblck: <p>Hi Darek,</p>
<p>haha thanks, but no need to buy me a drink! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Welcome to the community and it would make me happy to see some posts of you here! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Proper BatchNorm2d Mode during Fine-Tuning

wwaayyaaww: <p>During fine-tuning, if I set the BatchNorm2d in evaluation mode during fine-tuning on different datasets, the accuracy is increased from 63% to 80%.</p>
<p>However, if I leave the BatchNorm2d in the training mode, the performance drops drastically from 63% (before fine-tuning started) to only 39% in the first iteration.</p>
<ol>
<li>
<p>How if I wanted to maintain the performance while leaving the BatchNorm2d in the training mode during fine-tuning?</p>
</li>
<li>
<p>What’s the BatchNorm2d setting equivalence if we set it to evaluation mode?</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>How large is your batch size during fine-tuning? Note that the updates for the running stats might be off, if the batch is too small. You could change the <code>momentum</code> to smooth the updates a bit in this case.<br>
Also, does the accuracy recover at some point?</p>
</li>
<li>
<p>During evaulation the running estimates of mean and std are used instead of the current batch statistics.</p>
</li>
</ol>

wwaayyaaww: <p>Hi thank you for quick response.</p>
<ol>
<li>
<p>The batch size is set to 64 during fine-tuning in my experiments. And yes, I tried different values of momentum from 0.01, 0.1, 0.2, up to 1, and it is observed that the best accuracy is only 72% (momentum = 0.1) compared to 80% for BatchNorm2d in evaluation mode.</p>
</li>
<li>
<p>Yes, I tried to manually code the evaluation mode to see how batch-norm works by setting child_layer.momentum = 0, child_layer.weight,requires_grad = False, child_layer.bias.requires_grad = False, but it seems like the results are totally different with that of simply setting child_layer.eval(). May I have your advice on this?</p>
</li>
</ol>

ptrblck: <p>Batchnorm layers contain trainable parameters (<code>weight</code> and <code>bias</code>), if <code>affine=True</code> (default), as well as running estimates (<code>running_mean</code> and <code>running_var</code>).<br>
<code>bn.train()/eval()</code> changes the behavior of the running estimates. The affine parameters will not be changed by it.</p>
What is the reason of default bias initialization in Conv2d/ Linear?

KamWoh_Ng: <pre><code class="lang-auto">def reset_parameters(self):
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)
</code></pre>
<p>I am not sure if someone else has asked the question, may I know why is the bias in Conv2d is not initialized as 0 but initialized using <code>uniform_(-bound, bound)</code>. Can someone share the knowledge on it? Thanks!</p>
Any guideline on load a pretrained Pytorch NLP model to evaluate new dataset

zorozy: <p>Hi,</p>
<p>I am very new to NLP and would like to ask a very stupid question.</p>
<p>After I trained one NLP model, like GRU, what should I save and how could I reload the model again later on a new test dataset?</p>
<p>Besides, what should I do to record the prediction result for each input in the original sequence?</p>
<p>I am so sorry for this baby question.</p>

ptrblck: <p>To save and reload a model you should serialize its <code>state_dict</code>, which contains all parameters and buffers, as explained in the <a href="https://pytorch.org/docs/stable/notes/serialization.html">serialization docs</a>.</p>
<aside class="quote no-group" data-username="zorozy" data-post="1" data-topic="76105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zorozy/48/13373_2.png" class="avatar"> zorozy:</div>
<blockquote>
<p>Besides, what should I do to record the prediction result for each input in the original sequence?</p>
</blockquote>
</aside>
<p>You could append all outputs or predictions in a list after detaching them and just store the list.<br>
Detaching is necessary, as otherwise the whole computation graph will be added to the list and thus your memory usage might grow:</p>
<pre data-code-wrap="python"><code class="lang-python">outputs = []
for data in loader:
    output = model(data)
    outputs.append(output.detach().cpu())
</code></pre>
<p>If you are wrapping the code in a <code>with torch.no_grad()</code> block, as is common during evaluation, you don’t need to detach the outputs.</p>
Output of my network is 0 for some reason

shamoons: <pre><code class="lang-auto">        self.linear_right = torch.nn.Sequential(
            torch.nn.Linear(20, (feature_dim + 20) // 2),
            torch.nn.ReLU(),
            torch.nn.Linear((feature_dim + 20) // 2, feature_dim),
            torch.nn.ReLU6()
        )
...
        mean_out = torch.mean(down_out, 2)

        linear_out = self.linear_right(mean_out)
        print('mean')
        print(mean_out)
        print(self.linear_right[2].weight)
</code></pre>
<p>I have something like that outputs:</p>
<pre><code class="lang-auto">mean
tensor([[0.00e+00, 0.00e+00, 3.66e+01, 0.00e+00, 0.00e+00, 0.00e+00, 7.40e+01, 5.98e+01, 5.52e+01, 0.00e+00,
         4.82e+00, 0.00e+00, 7.89e+00, 0.00e+00, 0.00e+00, 1.05e+00, 1.77e+01, 0.00e+00, 3.64e+01, 0.00e+00]],
       grad_fn=&lt;MeanBackward1&gt;)
Parameter containing:
tensor([[1.79e-02, 7.82e-02, -1.99e-02,  ..., 4.72e-02, 3.25e-03, 3.68e-02],
        [1.23e-01, 7.37e-02, -4.71e-02,  ..., 1.88e-02, -9.74e-02, -3.74e-02],
        [8.93e-02, 7.38e-02, -4.53e-02,  ..., 2.13e-02, -1.54e-02, -3.55e-02],
        ...,
        [-9.18e-02, 2.46e-02, 6.96e-02,  ..., -4.98e-02, -9.11e-02, -2.09e-02],
        [-4.82e-02, -9.72e-02, 8.65e-02,  ..., -6.63e-02, -5.12e-02, -6.78e-02],
        [3.11e-03, 1.28e-02, -6.40e-02,  ..., 4.17e-03, 5.25e-02, -1.37e-02]],
       requires_grad=True)
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
       grad_fn=&lt;HardtanhBackward0&gt;)
</code></pre>
<p>So the output of the <code>linear_out = self.linear_right(mean_out)</code> is all zeros, but the input isn’t and the weights aren’t. So why is it?</p>

ptrblck: <p>The last <code>ReLU6</code> activation might return all zeros, if input activation is negative.<br>
Could you remove it for the sake of debugging and check the output again?</p>
Differentiable image compression operations in PyTorch

spaul13: <p>During a CNN classification model training while calculating the loss I am applying the encoding jpeg compression on the image in PyTorch. While I call loss.backward() it must also backpropagate through encoding and compression operation performed on the images.</p>
<p>Are those compression algorithms (e.g. encoding and JPEG compression) are differentiable otherwise how to backpropagate the loss gradient through those operations?</p>
<p>If those operations are not differentiable is there any differentiable compression algorithm that exists in PyTorch which performs H.264 encoding and JPEG compression? Any suggestions will be highly helpful.</p>

ptrblck: <p>To the best of my knowledge I don’t think encoding and compression is differentiable out of the box and it seems it’s an <a href="https://openreview.net/forum?id=HyeG9lHYwH">ongoing research</a>.</p>
Index selection usage

braindotai: <p>I’m trying to select values out of a tensor based on in indices stored in another tensor.</p>
<pre><code class="lang-python">outputs = torch.FloatTensor([[0.3, 0.7], [0, 1], [0.4, 0.6]])
labels = torch.Tensor([0, 1, 1]).long()
</code></pre>
<p>I want to select based on labels as indices at the 1st dimension. So I want outputs to be <strong><code>torch.tensor[[0.3], [1], [0.6]]</code></strong>.</p>
<p>I tried the code below:</p>
<pre><code class="lang-python">outputs = torch.FloatTensor([[0.3, 0.7], [0, 1], [0.4, 0.6]])
labels = torch.Tensor([0, 1, 1]).long()
print(torch.index_select(outputs, 1, labels))
</code></pre>
<p>but the output is -</p>
<pre><code class="lang-python">tensor([[0.3000, 0.7000, 0.7000], 
        [0.0000, 1.0000, 1.0000], 
        [0.4000, 0.6000, 0.6000]])
</code></pre>
<p>So how can I get desired results?</p>
<p>Thanks</p>

ptrblck: <p><code>torch.gather</code> should do the job:</p>
<pre><code class="lang-python">outputs = torch.FloatTensor([[0.3, 0.7], [0, 1], [0.4, 0.6]])
labels = torch.Tensor([0, 1, 1]).long()
print(outputs.gather(1, labels.unsqueeze(1)))
&gt;tensor([[0.3000],
        [1.0000],
        [0.6000]])
</code></pre>

braindotai: <p>Thanks as always <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Sampling STudent T giving an error

Jordan_Howell: <p>Hello,</p>
<p>I’m trying to sample from the Student T distribution.</p>
<p>I use the following code:</p>
<pre><code class="lang-auto">m = StudentT(torch.tensor([4846]))
m.sample()
</code></pre>
<p>That is derived from the docs:</p>
<p><a href="https://pytorch.org/docs/stable/distributions.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/distributions.html</a></p>
<p>Here is the error in and the traceback:</p>
<pre><code class="lang-auto">--------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-73-3011aef638a1&gt; in &lt;module&gt;()
      6 annual_investment = torch.tensor([0])
      7 m = StudentT(torch.tensor([4846]))
----&gt; 8 m.sample()
      9 # for x in range(iterations):
     10 

2 frames
/usr/local/lib/python3.6/dist-packages/torch/distributions/distribution.py in sample(self, sample_shape)
    117         """
    118         with torch.no_grad():
--&gt; 119             return self.rsample(sample_shape)
    120 
    121     def rsample(self, sample_shape=torch.Size()):

/usr/local/lib/python3.6/dist-packages/torch/distributions/studentT.py in rsample(self, sample_shape)
     68         #   Y = X / sqrt(Z / df) ~ StudentT(df)
     69         shape = self._extended_shape(sample_shape)
---&gt; 70         X = _standard_normal(shape, dtype=self.df.dtype, device=self.df.device)
     71         Z = self._chi2.rsample(sample_shape)
     72         Y = X * torch.rsqrt(Z / self.df)

/usr/local/lib/python3.6/dist-packages/torch/distributions/utils.py in _standard_normal(shape, dtype, device)
     39         return torch.normal(torch.zeros(shape, dtype=dtype, device=device),
     40                             torch.ones(shape, dtype=dtype, device=device))
---&gt; 41     return torch.empty(shape, dtype=dtype, device=device).normal_()
     42 
     43 

RuntimeError: _th_normal_ not supported on CPUType for Long
</code></pre>

ptrblck: <p>Pass the degrees of freedom as a <code>float</code> or <code>FloatTensor</code> and it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Error: 'indexing' is not a member of 'torch'

Edwardmark: <p>I use libtorch in <a href="https://download.pytorch.org/libtorch/cu92/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcu92.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cu92/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcu92.zip</a></p>
<p>when thy to use torch::index, the compiler complains that:<br>
error: ‘indexing’ is not a member of ‘torch’</p>
<p>why is that?</p>
Gradient norm as a loss function

Kangil_Lee: <p>def gradient_calc(self, outputs, inputs, create_graph=False):<br>
gradient = grad(outputs=outputs, inputs=inputs, grad_outputs=torch.ones(outputs.size()).cuda(self.args.device), retain_graph=True, create_graph=create_graph, only_inputs=True)[0]<br>
gradient = gradient.view(gradient.size(0), -1)<br>
grad_norm = (gradient.norm(2, dim=1)**2)<br>
grad_norm_mean = grad_norm.mean()</p>
<pre><code>    return gradient, grad_norm_mean

def grad_annealing(self, outputs, inputs, anneal_factor, std_factor):
    gradient, grad_norm_2 = self.gradient_calc(outputs, inputs, False)
    return gradient, 0.5*anneal_factor*grad_norm_2*std_factor

def approx_constraint(self, std, gradient, std_factor):
    # with torch.no_grad():
    rv = 3.5*(std*std_factor)*np.random.randn(self.args.batch_size, 3072) 
    # mean_vect = rv.mean(axis=0)
    # rv = rv-mean_vect
    rv = torch.from_numpy(rv).float()
    rv = rv.to(self.args.device)
    const_term = torch.mean((torch.bmm(gradient.unsqueeze(1), rv.unsqueeze(-1)).squeeze()).abs())
    return const_term
</code></pre>
<p>Hi guys. I m using gradient norm as a second loss function of my research topic.<br>
But still, I m somewhat confused about ‘create_graph’ in autograd.grad.<br>
Is it right using gradient norm as a loss function with ‘create_graph=False’ ?<br>
Should I set ‘create_graph=True’ to use the norm of the gradient for loss function?</p>