akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loss drops for the first epoch after loading a saved model

username: bar13290
cooked: <p>Hey,<br>
I train a RESNET34 with angular softmax loss for speaker identification.<br>
a very strange effect happens while I stop the training and save the already trained model using “torch.save” and load the saved model to continue training (with the same parameters and continue with the last LR with the same optimizer). The test loss and train loss are decreasing significantly (so it’s not an overfitting) just for the first epoch of the second training session and then increase on the second and third epochs until it stabilises and then continues decreasing in the way I expect.<br>
After the decreasing of the first epoch the model, the test accuracy is much better and so the model (If I stop it and check on the test test) better then even 5 or 6 epoch after the decreasing.</p>
<p>attached is a print screen of the train and val loss. I stopped and save the training at the end of epoch 12 and then I load it and continue to epoch 13 (you can see the drop). also I stop it on epoch 42 and the continue on epoch 43 with the same effect.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e4986ecdd0f516930934b6ad61366b5361b83e4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2e4986ecdd0f516930934b6ad61366b5361b83e4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e4986ecdd0f516930934b6ad61366b5361b83e4_2_681x500.jpeg" alt="image" data-base62-sha1="6BtuFkolJSpVnIIdtJIXGNxLede" width="681" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e4986ecdd0f516930934b6ad61366b5361b83e4_2_681x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e4986ecdd0f516930934b6ad61366b5361b83e4.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e4986ecdd0f516930934b6ad61366b5361b83e4.jpeg 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">990×726 55.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you have an idea why is this drop happens?</p>

username: Ben_Marinberg
cooked: <p>Are you sure LR is loaded correctly?</p>

username: bar13290
cooked: <p>Yes, for sure. The LR loaded and continued directly from the end of the last epoch, according to the optimizer.</p>
Manual Change to Gradient Doesn't Affect Performance?

username: wwaayyaaww
cooked: <p>Hi. I am trying the gradient mannual to explore how the modified gradient changes the ultimate performance.</p>
<p>I am wondering why it doesn’t show any difference in terms of performance it is multipled by 0.000001?</p>
<pre><code class="lang-auto">        optimizer.zero_grad()
        loss_batch.backward()
        
        for param in model_head.parameters():
                param.grad.data *= 0.000001

        optimizer.step()
</code></pre>
<p>Would it be possible if it is due to the Adam optimizer?</p>
<p>Thank you.</p>
<p>Best Regards</p>
Question about the num_workers in DataLoader

username: imaginist
cooked: <p>I got a question about how num_workers works during training.</p>
<p>Assume that batch size = 1, num_workers = 2.<br>
In my opinion, the process should be:<br>
worker1: 1 mini-batch (1 sample)<br>
worker2: 1 min-batch (1 sample)<br>
So, in the beginning, two workers should fetch <strong>2</strong> samples for initialization.</p>
<p>But, when I try to prove it by coding, the result is different.<br>
I saw that two workers fetched <strong>4</strong> samples at first for initialization, as shown below.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self):
        self.data = np.arange(20)

    def __getitem__(self, index):
        print("get index{}: ".format(index), self.data[index])
        return self.data[index]

    def __len__(self):
        return 20

if __name__ == '__main__':
    train_dataset = MyDataset()
    train_loader = DataLoader(train_dataset, batch_size=1, shuffle=False, num_workers=2)

    for _, data in enumerate(train_loader):
        print("training: ", data)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">get index0:  0
get index1:  1
get index2:  2
get index3:  3
training:  tensor([0])
training:  get index4:  4tensor([1])

get index5:  5
training:  get index6:  6
tensor([2])
training:  tensor([3])
get index7:  7
training:  get index8:  8
tensor([4])
training:  get index9:  9
tensor([5])
training:  get index10: tensor([6]) 
10
training:  tensor([7])get index11:  11

training:  get index12:  12
tensor([8])
training:  get index13:  13
tensor([9])
training:  tensor([10])get index14: 
 14
training:  get index15:  15
tensor([11])
training:  get index16: tensor([12]) 
16
training:  tensor([13])
get index17:  17
training:  get index18:  18
tensor([14])
training:  tensor([15])
get index19:  19
training:  tensor([16])
training:  tensor([17])
training:  tensor([18])
training:  tensor([19])
</code></pre>
<p>Can you explain this phenomenon to me?<br>
Thanks!</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> is prefetching samples, which can be set by <code>prefetch_factor</code> as described in the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader">docs</a>.<br>
The default value is set to <code>2 * num_workers</code> so 4 in your case, which explains the loading of 4 samples at the beginning.</p>
How to implement floatArray to tensor in android through java

username: codingHappily
cooked: <p>Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap, TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);<br>
float[] inputs = inputTensor.getDataAsFloatArray();<br>
Map&lt;String, IValue&gt; outTensors = module.forward(IValue.from(inputTensor)).toDictStringKey();<br>
Tensor outputTensor = outTensors.get(“out”).toTensor();<br>
float[] scores = outputTensor.getDataAsFloatArray();</p>
<p>this is my code, what shall I do next?</p>
Where do the VGG16 weights come from?

username: wsouza
cooked: <p>Hello,</p>
<p>Does anyone know if the VGG16 weights from PyTorch are loaded from the Caffe model proposed by the authors (<a href="https://www.robots.ox.ac.uk/~vgg/research/very_deep/" class="inline-onebox" rel="noopener nofollow ugc">Visual Geometry Group - University of Oxford</a>)?</p>
<p>I was comparing the results from PyTorch and TensorFlow and despite the prediction classes being the same, the probability is different. Since the VGG16 preprocessing from PyTorch and TensorFlow are different, I am wondering if they trained the VGG16 instead of loading the weights from somewhere else?</p>
<p>Thank you.</p>

username: tom
cooked: <p>TorchVision models are trained the models using the scripts in the <a href="https://github.com/pytorch/vision/tree/master/references/classification" rel="noopener nofollow ugc">references directory</a> of the source tree.</p>
<p>Best regards</p>
<p>Thomas</p>

username: wsouza
cooked: <p>Thank you, Thomas.</p>
<p>Best<br>
Wallace</p>
How can I measure how much time one of the GPU is straggling?

username: lmatz
cooked: <p>When I am training a job with multiple GPUs on a single server, I suspect that one of the GPUs is running somehow slower than the other GPU.</p>
<p>For example, I suspect that 3 of 4 GPUs finish the forward and backward in 1s while the rest completes by (1 + X) s. During X, 3 of 4 GPUs are waiting without any GPU utilization. After X, PyTorch starts to synchronize the model weights/parameters among 4 GPUs.</p>
<p>How can I measure the X? Thanks!</p>

username: ptrblck
cooked: <p>You could use e.g. <a href="https://developer.nvidia.com/nsight-systems">Nsight Systems</a> to create a timeline for the workload and see the workloads of each GPU.</p>
Can the input tensor of torch.svd() run in GPU?

username: kun_yang1
cooked: <p>I put a tensor data into the GPU through .cuda(), and then use the torch.svd (), but the speed didn’t improve. I found that the CPU was still working at 100%。can’t torch.svd () work in GPU？</p>

username: ptrblck
cooked: <p>We are actively working on adding <code>cuSolver</code> methods to the <code>torch.linalg</code> namespace and the PR for <code>svd</code> is <a href="https://github.com/pytorch/pytorch/pull/48436">here</a>.<br>
Currently MAGMA would be used, which can fallback to the CPU for smaller sizes of the inputs.</p>
The output of roi_align function when the input size is smaller than the output size

username: hanhaowen
cooked: <p>The class torchvision.ops.RoIAlign outputs the result of roi_align based on the input feature map. In tradition, the output size is smaller than the input size when using roi_align. So my question is what will happen if the output size is larger than the input size. Will it just divided the input feature map into output_size[0] * output_size[1] equal parts and then do the same thing as it will do when the output size is   smaller than the input feature map size?</p>
How to forward nn.Transformer in inferenecing time?

username: chois_good
cooked: <p>I have two non-equal-length sequence data, one for the input and the other for the target. This data is not nlp data, but multivariate time series data. The data is for predicting values of the next M days(regression) using the features of the past N days.</p>
<p>In this case, I wonder how to handle the data in inferencing <code>nn.Transformer</code> or <code>nn.TransformerDecoderLayer</code>.</p>
<p>It doesn’t have any token like , so I have no idea how can I prepare the <code>tgt</code> when calling decoder’s <code>forward()</code></p>
<p>Need your advice or refenrece code or source</p>

username: lucastononrodrigues
cooked: <p>If I am not mistaken the dimension of tokens in your decoder input doesn’t have to be the same as the dim in your encoder input.</p>
<p>In your encoder: X Input_shape = (Num_of_tokens,Batch_size,Embed_size)<br>
In your decoder: Y Input_Shape = (Num_of_tokens_2,Batch_size,Embed_size)</p>
<p>Take a look at <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" class="inline-onebox" rel="noopener nofollow ugc">MultiheadAttention — PyTorch 1.7.0 documentation</a></p>
<p>So when data meet you will have Keys and Values coming from the encoder and Queries from the decoder</p>
<p>Q<em>K^T will give you a ( Num_of_tokens_2 x  Num_of_tokens) matrix and then<br>
Sm(Q</em>K^T) * V will give you a (Num_of_tokens_2 x Embed_size) matrix which will be the shape of the output of the decoder after all the layers.</p>
<p>Obs: all the layers conserve the shape of the input.</p>
<p>I would also highly recommend you to code the Transformer from scratch, is not that complicated and it will give you a better understading of the input/output shapes.</p>
<p>Here’s a good reference (but pay attention and check if it corresponds to the paper, there might be a few incoherences with the original model or oversimplifications)</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://miro.medium.com/1*m-R_BkNf1Qjr1YbyOIJY2w.png" class="site-icon" width="32" height="32">
      <a href="https://medium.com/the-dl/transformers-from-scratch-in-pytorch-8777e346ca51" target="_blank" rel="noopener nofollow ugc" title="10:29PM - 13 December 2020">Medium – 13 Dec 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:652/500;"><img src="https://miro.medium.com/max/1200/0*K2pL1e1ED12sPEW0" class="thumbnail" width="652" height="500"></div>

<h3><a href="https://medium.com/the-dl/transformers-from-scratch-in-pytorch-8777e346ca51" target="_blank" rel="noopener nofollow ugc">Transformers from Scratch in PyTorch</a></h3>

<p>Join the attention revolution! Learn how to build attention-based models, and gain intuition about how they work.</p>

  <p><span class="label1">Reading time: 7 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Hope it was somehow helpful.</p>
Create a masked view of a tensor

username: gekko404
cooked: <p>I have the following tensor:<br>
tensor([[   nan],<br>
[0.2000],<br>
[   nan],<br>
[0.3000],<br>
[0.4000]])</p>
<p>Now I want to get only non-nan values, run some processing on them (keeping same dimension) and get the resulting tensor as my input but with updated non-nan values.<br>
I was thinking just to create a view of that tensor with only non-nan values, so then the changes to the original data will automatically propagate, but I didn’t find how to create a “masked view” of a tensor.</p>
Yuv2tensor function seems to tortue the RGB space (mobile)

username: Silence-Monk
cooked: <p>I am currently trying to deploy some realtime style-transfer model to process yuv pictures frame by frame from camera on mobile devices, and use</p>
<pre><code class="lang-auto">TensorImageUtils.imageYUV420CenterCropToFloat32Tensor() 
</code></pre>
<p>to convert yuv to tensor, but the RGB colorspace of the pictures generated afterwards are always tortuous. I fixed this problem by finding some other ways to convert yuv to bitmap and use</p>
<pre><code class="lang-auto">TensorImageUtils.bitmapToFloat32Tensor()
</code></pre>
<p>instead. It might not affect much for classification or object detection tasks, but it worths to check out if there’s really a bug in it.</p>

username: IvanKobzarev
cooked: <p>Hello <a class="mention" href="/u/silence-monk">@Silence-Monk</a>,<br>
Thanks for reporting it.</p>
<p>There was a bug in yuv to rgb conversion, the fix was landed in master recently <a href="https://github.com/pytorch/pytorch/pull/50871" class="inline-onebox" rel="noopener nofollow ugc">[android] Fix YUV camera image to tensor by IvanKobzarev · Pull Request #50871 · pytorch/pytorch · GitHub</a></p>
<p>Next nightly build of</p>
<pre><code class="lang-auto">    implementation 'org.pytorch:pytorch_android_torchvision:1.8.0-SNAPSHOT'
</code></pre>
<p>will contain that fix.</p>
PyTorch for Tesla k40c with cuda 11.2

username: Saurav_Gupta1
cooked: <p>I am trying to train a model on multiple GPUs. I have 2 Tesla k40c and 1 GeForce GTX 1080. My PyTorch version is 1.7.1 with CUDA 11.2.<br>
PyTorch 1.3.1 onwards has stopped support for GPUs with compute capability of 3.5, which means I am unable to use Tesla k40c. Is there any way I can use Tesla k40c with CUDA 11.2 and PyTorch?</p>

username: ptrblck
cooked: <p>You could build PyTorch from source for the compute capability 3.5 as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>

username: Saurav_Gupta1
cooked: <p>As recommended by you I have built PyTorch from source, I am able to train on Tesla k40c but when trying to run on GTX 1080 I get the following error:</p>
<pre><code class="lang-auto">GeForce GTX 1080 with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_35.
If you want to use the GeForce GTX 1080 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: no kernel image is available for execution on the device
</code></pre>

username: ptrblck
cooked: <p>You can specify multiple GPU architectures by building with:</p>
<pre><code class="lang-python">TORCH_CUDA_ARCH_LIST="3.5 6.1" python setup.py install
</code></pre>
<p>where you can pass multiple architectures to the env var (as well as <code>+PTX</code>).</p>
Where deprecated torch.uint8 warning comes from, possibly from numpy conversion?

username: martinr
cooked: <p>I have a rather large code base, and upon upgrading to pytorch 1.2 I started receiving thousands of warnings:</p>
<blockquote>
<p>/pytorch/aten/src/ATen/native/IndexingUtils.h:20: UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead.</p>
</blockquote>
<p>Now, I was using torch.uint8 in one place, I replaced that with torch.bool. I was also using np.uint8 in one array which I later used to create PyTorch tensor. I changed that to np.bool_ now.</p>
<p>The problem is that I still have thousands of the same messages and as I am not explicitly using uint8 anywhere now, I wonder where it might come from? Does PyTorch still convert some numpy data to torch.uint8? I am converting a lot of numpy data to pytorch tensors. The problem is that there is no extra info or trace of this warning. Also, my code base is huge, so not possible to include it here.</p>
<p>Does anyone have any clue what might cause the error as I do not explicitly use torch.uint8 or numpy.uint8 anywhere?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can try to make python stop on warning to get the stack trace of where that happens or add something like <a href="https://stackoverflow.com/questions/22373927/get-traceback-of-warnings" rel="nofollow noopener">that</a> that prints a traceback for each warning.</p>

username: martinr
cooked: <p>Thanks! It was an np.array that used <em>np.uint8</em> for booleans and that got automatically converted into <em>torch.uint8</em> later when doing <code>var.to(device)</code>, <em>var</em> being this <em>np.array</em> of type <em>np.uint8</em>. It was hard to trace as the numpy array was saved off to H5PY file and I only loaded this array from file using dataloader. Converting from <em>np.uint8</em> to <em>np.bool</em> before doing <code>var.to(device)</code> solved the problem.</p>

username: Ohmar
cooked: <p>Hello<br>
I have the same problem, you could report the exact place where you changed from np.uint8 to np.bool, go to the h5py folder and change all the np.uint8 that I found but the problem still persists.</p>
<p>tanks!</p>

username: martinr
cooked: <p>h5py is the library that allows to save data in HDF5 format. I did not change anything in h5py library source code, but in the data file that I was loading with h5py. It had some data in it that was saved using np.uint8. If you can, change the file generation so that it will use np.bool instead. If not, you need to convert the data to np.bool or torch.bool when loading the data.</p>
<p>PS! If you are just running someone else’s code and do not care about maintaining it, then just suppress this warning when running:</p>
<p><code>python -W ignore::UserWarning pythonscript.py</code></p>
<p>It will also suppress all other UserWarnings though…</p>

username: mikey_t
cooked: <p>I’m not sure if this is still helpful to anyone, but I was getting this error in my code, so I changed from<br>
<code>torch.zeros(..., dtype=torch.uint8)</code> to <code>torch.zeros(..., dtype=torch.bool)</code></p>
Numpy.interp vs torch.nn.functional.interpolate

username: sidnanda
cooked: <p>How can I translate the numpy implementation to torch?</p>
<p><a href="https://numpy.org/doc/stable/reference/generated/numpy.interp.html" class="onebox" target="_blank" rel="nofollow noopener">https://numpy.org/doc/stable/reference/generated/numpy.interp.html</a></p>
<p><a href="https://pytorch.org/docs/stable/nn.functional.html#interpolate" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.functional.html#interpolate</a></p>
<p>The torch version doesn’t seem to take in discrete data points.</p>
Designing custom dataset using multiple different data tables

username: swlee
cooked: <p>Hello. Currently I’m working on a recommendation system, and I’m trying to make custom dataset which uses multiple data tables. There are three data tables.</p>
<ul>
<li><code>rating_data</code> : contains rating data between users and items. First column means userID and second one means item ID. type of <code>np.array </code>, shape of [Nx2]. (N ratings)</li>
<li><code>negative_data</code> : Contains negative items for each users. i’th row is list of itemIDs, and these items are negative items of user whose ID is i. type of <code>np.array</code>. The number of rows is same as number of users and the number of columns is varies from users.</li>
<li><code>image_path_list</code> : 1D array that Contains path to image data corresponding to items. i’th element is path to the image that describes item whose ID is i. type of <code>np.array</code> and size is same as number of items.</li>
</ul>
<p>In each iteration, I need 1) pair of user ID and positive item ID from <code>rating_data</code> first of all. 2) Then I have to  sample 4 negative items from <code>negative_data[userID]</code>. 3) Finally I need image paths corresponding to positive item and 4 negative items from <code>image_path_list</code></p>
<p>I wrote my code like this.</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
from PIL import Image
import numpy as np

class CustomDataset(Dataset):
    def __init__(self, rating_data, negative_data, image_path_list, transform=None):
        super(CustomDataset, self).__init__()
        self.rating_data = rating_data
        self.negative_data = negative_data
        self.image_path_list = image_path_list
        self.transform = transform

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, index):
        # Get (userID, positive itemID) pair. (e.g, user=3, item_p=0)
        user, item_p = self.rating_data[index]
        
        # negative items of specific user.
        ng_pool = self.negative_data[user]
        # sample 4 negative items (e.g, item_n = [1,2,7,10])
        idx = np.random.choice(len(ng_pool),4,replace=False) 
        item_n = ng_pool[idx].tolist() 

        p_path = self.image_path_list[item_p] # path to image describes item_p
        n_path = self.image_path_list[item_n] # paths to images describe item_n

        img_p = Image.open(p_path)
        img_n = Image.open(n_path) # Error occurs here

        if self.transform is not None:
            img_p = self.transform(img_p)
            img_n = self.transform(img_n)

        return user, item_p, item_n, img_p, img_n
</code></pre>
<p>Since there are 4 negative items, <code>n_path</code> is array type contains four paths. And <code>Image.open()</code> seems to take only str type of input. When I run</p>
<pre><code class="lang-auto">dataset = CustomDataset()
dataset[0]
</code></pre>
<p>Here is the error message.</p>
<blockquote>
<p>*** AttributeError: ‘numpy.ndarray’ object has no attribute ‘read’</p>
</blockquote>
<p>How can I make <code>Image.open()</code> work on multiple paths? Can anybody help me to fix this code?</p>
[ux] Mix of nn.Sequential and nn.ModuleDict?

username: vadimkantorov
cooked: <p>Often modules have structure like this (so that <code>state_dict()</code> key names are readable):</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
  def __init__(self):
    super().__init__()
    self.module1 = nn.Linear(1, 4)
    self.act1 = nn.ReLU()
    self.module1 = nn.Linear(4, 8)
    self.act2 = nn.ReLU()

  def forward(self, x):
    x = self.module1(x)
    x = self.act1(x)
    x = self.module2(x)
    x = self.act2(x)
    return x
</code></pre>
<p>Such forward method could go away if we could have some mix of nn.Sequential and nn.ModuleDict, i.e. having a forward method for ModuleDict that would rely on the OrderedDict and implement forward as a Sequential, and still enjoy good module names.</p>
<p>Currently it errors out with a unclear error message (it should rather cry “ModuleDict forward is not supported by default”):</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.nn.ModuleDict(dict(a = torch.nn.Linear(2, 2), b = torch.nn.Linear(2, 4)))(torch.rand(3, 2))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/miniconda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() takes 1 positional argument but 2 were given
</code></pre>
<p>I think starting Python 3.7 (?) insertion order of dictionaries is preserved, and kwarg order is also preserved since <a href="https://www.python.org/dev/peps/pep-0468/" rel="noopener nofollow ugc">https://www.python.org/dev/peps/pep-0468/</a></p>
<p>This would also simplify static analysis and graph transformations of such models</p>

username: ptrblck
cooked: <p>Would it work to pass the <code>OrderedDict</code> directly to <code>nn.Sequential</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html?highlight=sequential#torch.nn.Sequential">docs</a>?</p>
<pre><code class="lang-python"># Example of using Sequential with OrderedDict
model = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ]))
</code></pre>

username: vadimkantorov
cooked: <p>Would nn.Sequential accept a regular dict as well? (given that insertion order is preserved in modern Pythons)</p>

username: ptrblck
cooked: <p>You can find the feature request <a href="https://github.com/pytorch/pytorch/issues/49530">here</a>.</p>
Weight_decay implementation in optimizers

username: iamgroot42
cooked: <p>I was going through how weight_decay is implemented in optimizers, and it seems that it is applied per batch with a constant that ideally should be for the whole loss. For instance, if training a model with CE loss, one would expect the formulation to be:</p>
<pre><code class="lang-auto">mean(CE, X, Y) + weight_decay * norm(W, 2)
</code></pre>
<p>Since models are trained in batches, the first term is loss per batch. However, the second term should ideally also be scaled by the number of batches. For instance, if training a model with batch size 64 will penalize weight_decay X times, training with a batch size of 256 will do it X/4 times, which seems a bit odd?</p>
<p>Is this an implementation bug, or is it assumed that the developer will scale weight_decay while passing it to the optimizer (which kind of makes sense, since the optimizer does not know how many batches are present, but it would be nice to mention this somewhere in the documentation)?</p>

username: SumanthRH
cooked: <p>Hey! The CE loss considered is the mean value and not the sum of the loss values for each example. Thus the scale of the first term remains the same (with changes in batch size). So the second term need not be scaled by the number of batches.</p>

username: iamgroot42
cooked: <p>Correct, but in essence the model is being penalized for larger weights much more frequently if the batch size is small. Over one epoch, a model with BS=16 might get penalized 32 times, whereas BS=128 would make that only 4 times. I get that the loss considers mean and we are doing batch-gradient descent after all, but should there not be some sort of scaling for the regularization as well?</p>

username: SumanthRH
cooked: <p>Oops! Sorry for misunderstanding your point earlier. I think what you are raising ties to the difference between weight decay and L2 regularization? And that PyTorch currently implements L2 regularization but it’s still labelled as “weight decay”?</p>

username: iamgroot42
cooked: <p>No. I guess what you mentioned earlier makes sense now, since per epoch the model only really sees one batch of data, and it makes sense for the weight decay constant to not vary.</p>
How not to consider padding during training?

username: Bez
cooked: <p>Hi, I have a dataset with 2D samples of shape <code>(X, 100)</code> each, where X is an arbitrary number.<br>
I want to feed a neural network using batches, but the samples can have variable sequence length and for this reason I used pad_sequence method to create the batch tensor with padded samples (of shape <code>(batch_size, sequence_length, 100)</code>).<br>
Example (here each sample has shape (x, 3)):</p>
<p><code>L = [tensor([[ 0.7900, -0.7206,  1.2947]]),          tensor([[ 0.5230, -0.3547, -1.8458],                     [ 0.2626,  0.0604,  0.1883]])]</code></p>
<p>L is the list of my samples, as you can see, the first has shape (1,3) and the second has shape (2,3)<br>
Applying pad_sequence method the result is:</p>
<p><code>tensor([[[ 0.7900, -0.7206,  1.2947],                [ 0.0000,  0.0000,  0.0000]],               [[ 0.5230, -0.3547, -1.8458],                [ 0.2626,  0.0604,  0.1883]]])</code></p>
<p>The neural network has two nested networks which results are summed together and concatenated to output a score. Does the network’s result depend on padding? How can not consider it?<br>
I know that Keras has a masking layer, but I don’t know how to implement it on PyTorch.<br>
Thank you in advance.</p>
Check weights after laoding state_dict

username: snowe
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a pretrained model that I would like to use on a testset. I do the following to check if the model did actually load the weights:</p>
<pre><code class="lang-auto">checkpoint = torch.load(checkpoint_path, map_location=device) 
model = Model()
print(model.conv1.weight[0]) # (1) see the weights of the random initialisation

model.load_state_dict(checkpoint['model_state'], strict=False)
print(checkpoint['model_state']['module.conv1.weight'][0]) # (2) see the weights of the pretrained model
print(model.conv1.weight[0]) # I would expect this to be the same as (2) but it is the same as (1)

self.model.eval()
</code></pre>
<p>What am I doing wrong?</p>
<p>Any help is very much appreciated!</p>
<p>All the best,<br>
snowe</p>

username: user_123454321
cooked: <p>I maybe wrong but I think you are trying to load a DataParallel object into one that is a normal model. Can you please give more information on how you are saving the model and some idea about the model ? Also, can you try with the strict not as False?</p>

username: snowe
cooked: <p>You are right, the checkpoint file is from a DataParallel object. Does that mean I have to make one again and then load the weights?</p>
<p>I save like this:</p>
<pre><code class="lang-auto">checkpoint = {
                'run': self.run_count,
                'epoch': self.epoch_count,
                'model_state': self.model.state_dict(),
                'optimizer_state': self.optimizer.state_dict(),
                'accuracy_val': self.accuracy_val
            }
torch.save(checkpoint, 'filename.pth')
</code></pre>
<p>The <code>strict=False</code> is needed since I load weights from a ResNet with one fully-connected layer into a ResNet with a different fully-connected layer in the end.</p>

username: snowe
cooked: <p>So I made the following changes and now it seems to work:</p>
<pre><code class="lang-auto">checkpoint = torch.load(checkpoint_path, map_location=device) 
model = Model()
model = nn.DataParallel(model, device_ids=device_ids) 
model.to(self.device) 
print(model.module.conv1.weight[0]) # shows random weights

model.load_state_dict(checkpoint['model_state'], strict=False)
print(checkpoint['model_state']['module.conv1.weight'][0]) # shows pretrained weights of checkpoint

print(model.module.conv1.weight[0]) # shows pretrained weights in model
</code></pre>
<p>The problem was that I had to initialise my model as a <code>DataParallel</code> object and additionally call <code>.module</code> when accessing the weights of a specific layer.</p>
Keypoints RCNN multiple classes with different number of keypoints

username: Traceur27
cooked: <p>Hello,</p>
<p>I’m interested in creating a Keypoints RCNN detecting objects and keypoints for two classes: pants and t-shirts. There are different number of keypoints associated with each class (7 and 10). What I tried is to first train for separate classes (one model for pants and one for t-shirts) and was able to succeed. However, I wonder if it’s possible to do this together. My approach here was to simply create an array big enough to store both numbers of keypoints (17) and simply set first 7 to 0 and not visible, then proper keypoints locations. Similar thing for pants, where first 7 points have correct coordinates and the rest is 0s with not visible. Unfortunately I wasn’t able to train any working model, even for a very small dataset (20 examples). Is it possible to approach this problem in another way?</p>
How to change all Relu to TLU in pretrained models?

username: mobassir94
cooked: <p>i was trying to replace all the Relu activation functions with TLU of skresnet34 model(from timm library) as shown here : <a href="https://github.com/yukkyo/PyTorch-FilterResponseNormalizationLayer/blob/master/frn.py" class="inline-onebox" rel="noopener nofollow ugc">PyTorch-FilterResponseNormalizationLayer/frn.py at master · yukkyo/PyTorch-FilterResponseNormalizationLayer · GitHub</a> and in filter response normalization layer paper.</p>
<p>i was trying this :</p>
<pre><code class="lang-auto">
class TLU(nn.Module):
    def __init__(self, num_features = 32):
        """max(y, tau) = max(y - tau, 0) + tau = ReLU(y - tau) + tau"""
        super(TLU, self).__init__()
        self.num_features = num_features
        self.tau = nn.parameter.Parameter(
            torch.Tensor(1, num_features, 1, 1), requires_grad=True)
        self.reset_parameters()

    def reset_parameters(self):
        nn.init.zeros_(self.tau)

    def extra_repr(self):
        return 'num_features={num_features}'.format(**self.__dict__)

    def forward(self, x):
        return torch.max(x, self.tau)
    

def convert_relu_to_tlu(model):
    for child_name, child in model.named_children():
        if isinstance(child, nn.ReLU):
            setattr(model, child_name, TLU())
        else:
            convert_relu_to_tlu(child)

def getter(model, name):
    layer = model
    for attrib in name.split("."):
        layer = getattr(layer, attrib)
    return layer

def setter(model, name, layer):
    try:
        attrib, name = name.rsplit(".", 1)
        model = getter(model, attrib)
    except ValueError:
        pass
    setattr(model, name, layer)

for name, module in model.named_modules():
    if isinstance(module, nn.ReLU):
        relu = getter(model, name)
        print(relu)
        #tlu = TLU(num_features = relu.num_features)
        tlu = TLU()
        print("changing {} with {}".format(relu, tlu))

        setter(model, name, tlu)
</code></pre>
<p>it replaces all relu with TLU but i can’t pass exact value of num_features in TLU while replacing relu with tlu,how to modify this line of code :</p>
<pre><code class="lang-auto">
tlu = TLU(num_features = relu.num_features)
</code></pre>
<p>for changing all relu with tlu?</p>
Right way of comparing tensor sizes without tracer warnings?

username: adanielnoel
cooked: <p>In some of my forward functions I have assert statements to check that the tensors passed as input have the expected shape. The asserts work as expected, but when exporting the model to ONNX the tracer throws the warning:</p>
<pre><code class="lang-auto">TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert x.shape == y.shape
</code></pre>
<p>I have tried wrapping the assert into a <code>with torch.nograd()</code> block and also doing <code>assert list(x.shape) == list(y.shape)</code>, but the warning perseveres.</p>
<p>Is there a more standardised approach within pytorch to do these checks?</p>
How to split MNIST images based on their labels?

username: nil_tdr
cooked: <p>Dear Altruists,</p>
<p>I am currently working with MNIST dataset. I am able to download and load training data. For my project, I need to train my model with images belonging to a subset of labels, not all of them. Is there a way to do it?</p>
<p>I am new to Pytorch. Thank you in advance!</p>

username: ptrblck
cooked: <p>You could filter out the unwanted labels by directly accessing the <code>dataset.targets</code> and/or wrap it in a custom <code>Dataset</code>.</p>

username: nil_tdr
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks a lot for your help. This is how I did it:</p>
<pre><code class="lang-auto">train_data = datasets.MNIST(root='~/.pytorch/MNIST_data/', train=True,
                                   download=True, transform=transform)
train_set = [] 

for i in range(0,len(train_data)):
    if train_data[i][1] &lt; 7:
        train_set.append(train_data[i])
</code></pre>
Storing proprietary data inside a bundle

username: Arkady
cooked: <p>Hi,</p>
<p>We currently develop a Glow backend for CEVA target DSP that supports AOT (offline) compilation mode.</p>
<p>During the bundle creation stage, our backend generates metadata which is required by the target DSP during runtime.</p>
<p>We need to access this data from the main application on the ARM host before it calls the bundle entry point to start the actual Glow inference process.</p>
<p>Is there an existing mechanism in Glow that allows the backend to store proprietary raw data inside the bundle or the weights file, and then access it during inference initialization?</p>
<p>If not, is it possible then for the backend to write the data to a “private” file and then open it for read during runtime?</p>
<p>Thanks</p>

username: jfix
cooked: <p>Hi, AFAIK currently there is no such ability to save the metadata, at least not in a hacky way (e.g. create some Constant blob that has a single Save user that will then be written out as part of the Constant weights). However it should be relatively simple for the backend to do this – just save it to some location on disk, and then in the driver file that loads the weights you can also add your own custom handling to load the file and do with it whatever you need.</p>

username: Arkady
cooked: <p>Thanks for your reply.</p>
From onnx to pytorch

username: braindotai
cooked: <p>Is it possible to import a onnx model to pytorch as pytorch module?<br>
I want to convert pretrained models for video recognition from mxnet to pytorch… So…</p>

username: ebarsoum
cooked: <p>No, PyTorch only support exporting to ONNX. For your scenario, you will need to dump the mxnet model weights to a file, create a similar architecture in PyTorch and replace the model parameters with the mxnet one.</p>

username: Talmaj
cooked: <p>There’s a new library <a href="https://github.com/ToriML/onnx2pytorch" rel="noopener nofollow ugc">onnx2pytorch</a> that helps you transform onnx models to PyTorch.</p>
How to optimize parameters in a specific range in PyTorch?

username: Z_Mario
cooked: <p>Here is a toy example:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.p = nn.Parameter(torch.tensor(0.5))

    def forward(self, x):
        y = x * self.p
        return y

model = Model()
optimizer = torch.optim.Adam(model.parameters())
loss_func = torch.nn.MSELoss()
y = torch.tensor([0.1,0.2,0.3])
x = torch.tensor([1.,2.,3.])
model.train()
for i in range(3):
    pred = model(x)
    optimizer.zero_grad()
    loss = loss_func(pred, y)
    loss.backward()
    optimizer.step()
</code></pre>
<p>I want to optimize <code>p</code> in a certain range. For example, the value of <code>p</code> must between 0.2 to 0.6 in during training process.</p>
The same CUDA version, different Pytorch versions, but different outputs

username: cfh3c
cooked: <p>I run the code below by using Pytorch 0.4.1 and Pytorch 1.0.0 respectively with cuda 10.1.</p>
<pre><code class="lang-auto">model = nn.Linear(3,2).cuda()
x = torch.FloatTensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]).cuda()
with torch.no_grad():
    model.weight = torch.nn.Parameter(torch.cuda.FloatTensor([[1.0, 2.0, 1.0], [2.0, 3.0, 3.0]]))
    model.bias = torch.nn.Parameter(torch.cuda.FloatTensor([1.0, 2.0]))
    y = model(x)
    print(y)
</code></pre>
<p>The problem is that the outputs are different:</p>
<pre><code class="lang-auto">tensor([[17., 36.],
        [41., 84.]], device='cuda:0') # pytorch 0.4

tensor([[ 9., 19.],
        [21., 43.]], device='cuda:0') # pytorch 1.0
</code></pre>
<p>It’s quite amazing. Is there anyone can help me to solved it? Thanks.</p>

username: ptrblck
cooked: <p>I would recommend to stick to the latest release (<code>1.7.1</code>) and not use old releases such as <code>0.4</code> or <code>1.0</code>, as these might have been known issues and should be already fixed.</p>

username: cfh3c
cooked: <p>Yeah, thank ptrblck.</p>
Is this code snippet compatible with backward?

username: saul_santos
cooked: <p>Hi, it seems that my model fails to learn i.e. the loss keeps constant after a few epochs. I’m wondering whether I have any bug on my code but can’t find any. Just to make sure is this code snippet compatible with autograd?</p>
<pre><code>def forward(self, x):
    with torch.set_grad_enabled(True):
        self.n = x.shape[0]
        if self.forced:
            self.d = x.shape[1] // (3*2)
            q_past, qk, q_next, u_past, u, u_next =  torch.split(x,self.d,1)
        else:
            self.d = x.shape[1] // (3)
            q_past, qk, q_next =  torch.split(x,self.d,1)
            
        qk  = qk.requires_grad_(True)
        if len(x)==1:
            q_next = q_next.requires_grad_(True)
        
        q1, qd1 = self.input_scaling(qk, q_next)
        q2, qd2 = self.input_scaling(q_past, qk)
        
        L1 = self.lagrangian(q1, qd1)
        L2 = self.lagrangian(q2, qd2)
        
        #Discrete Euler-Lagrange equations
        if self.forced:
            f = L1 + L2
            DEL = grad(f.sum(), qk, create_graph=True)[0] + (u_past + u) + 0.25*(u + u_next)
        else:
            DEL = grad(f.sum(), qk, create_graph=True)[0]
        # The loss layer will be applied outside Network class
        
        if len(x) ==1:
        
            J = []
            for i in range(self.d):
                D1_i = DEL[:,i][:,None]
                J_i = grad(D1_i[:].sum(), q_next, create_graph=True)[0]
                J.append(J_i)
            J_i=torch.stack((J[0], J[1]), 0)
            Jac=J_i.permute(1,0,2)
            return DEL, Jac
        else:
            return DEL</code></pre>
How to transform two serial models to ONNX

username: yangyi_hpu
cooked: <p>Dear.<br>
I am in trouble.  It around me two weeks.  I have a project that uses Yolov4 in pytorch for the Deepstream. So the yolo model should transform to ONNX, then Engine file by TensorRT.</p>
<p>But I want add some  preprocessing for yolov4 model, then transform the preprocessing and yolo model to ONNX together.  I don’t know how to do that.</p>
<p>Actually,  I try to modify the YoloV4 code and hope to add a preprocess before the yolov4 model. But I get the error</p>
<p>“torch.nn.modules.module.ModuleAttributeError: ‘Darknet’ object has no attribute ‘myPreProcessing’”</p>
<p>So I want to build the preprocess model and make the two model serially.</p>
<p>Please help me. Thank you very much.</p>
GAN based caption generation (how to propagate Adversarial loss gradients correctly)?

username: malla_srikanth
cooked: <p>Hi,<br>
I am looking at these papers (GAN based image to caption generation):</p><aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1703.06029.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1703.06029.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1703.06029.pdf" target="_blank" rel="noopener nofollow ugc">1703.06029.pdf</a></h3>

<p class="filesize">2.10 MB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1804.00861.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1804.00861.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1804.00861.pdf" target="_blank" rel="noopener nofollow ugc">1804.00861.pdf</a></h3>

<p class="filesize">4.97 MB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>that produce multiple captions for a single image using GAN based architecture, they encode the predicted caption from generator (using RNN) for feeding it into the discriminator.</p>
<p>My Question:</p>
<ol>
<li>When encoding the predicted caption, torch.max indices/ argmax of generator outputs will convert the logits into a sentence. Which can be encoded and used in discriminator. The discriminator output is used in the adversarial loss for the generator. But the torch.max is not differentiable, So it doesn’t make sense (not sure how these papers implemented it).</li>
<li>Instead of taking argmax, if we encode the logits directly from the generator, wouldn’t it be very easy for the discriminator to know these are (not ones and zeros) so it must be from generator/fake and (these are ones and zeros) must be from ground truth.</li>
</ol>
<p>Anyone with ideas or knowledge in this topic (generator with a classification outputs in GAN). please share your ideas.</p>
<p>Thank you,<br>
Srikanth</p>
How to get the index of the unique values in the origin tensor

username: cslxiao
cooked: <p>for example,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([4, 1, 4, 3, 2, 3], dtype=torch.long)
&gt;&gt;&gt; b,c =torch.unique(a,return_inverse=True)
&gt;&gt;&gt; a
tensor([4, 1, 4, 3, 2, 3])
&gt;&gt;&gt; b
tensor([1, 2, 3, 4])
&gt;&gt;&gt; c
tensor([3, 0, 3, 2, 1, 2])
</code></pre>
<p>I want to get the index of each value of <code>b</code> in <code>a</code><br>
i.e., <code>d</code> such that <code>a[d]=b</code></p>
<pre><code class="lang-auto">&gt;&gt;&gt; d=torch.tensor([1,4,3,0])
&gt;&gt;&gt; a[d]
tensor([1, 2, 3, 4])
</code></pre>
Why does the collate_fn need to be pickled?

username: ataxias
cooked: <p>If I understand correctly, each worker process runs the <code>__get_item__</code> of the <code>Dataset</code> on the indices it has received (a subset of the batch indices), and creates a list of outputs. What is unclear to me is <strong>when</strong> the <code>collate_fn</code> is called to assemble the list of outputs into a batch. It would be reasonable for this function to be called by the main process, once it has received all the outputs corresponding to a single batch from the worker processes. However, the documentation states that any custom <code>collate_fn</code> needs to be picklable, and thus I assume that each worker process calls <code>collate_fn</code> on the portion of the batch it has processed, and these portions are then sent to the main process, which then stacks the tensors? It doesn’t make much sense, but I can’t think of another reason why <code>collate_fn</code> has to be declared at the top level and be picklable. Can someone clarify when/where <code>collate_fn</code> is called and why it needs to be pickled? Can it be a member function of the <code>Dataset</code>, and use member variables?</p>