akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does a `DataLoader` created from `ConcatDataset` create a batch from a different files, or a single file?

username: Santosh-Gupta
cooked: <p>I am working with multiple files, and multiple training samples in each file. I will use <code>ConcatDataset</code> as described here:</p>
<aside class="quote" data-post="7" data-topic="11769">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/7">DataLoaders - Multiple files, and multiple rows per column with lazy evaluation</a> 
  </div>
  <blockquote>
    I created one dataset for each file, and if there’s only 3000 files then it isn’t that much to hold it inside an array (object that has a reference to). 
If you wrote your DataSet with linecache, then it won’t read each file into memory. 
At least this is my observation after reading more files than my computer’s memory can support.
  </blockquote>
</aside>

<p>I need to have negative samples in addition to my true samples, and I need my negative samples to be randomly selected from all the training data files. So, I am wondering, would the returned batch samples just be a random consecutive chuck from a random single file, or would be batch span across multiple random indexes across all the datafiles?</p>
<p>If there are more details needed about what I am trying to do exactly, it’s because I am trying to train over a TPU with Pytorch XLA.</p>
<p>Normally for negative samples, I would just use a 2nd <code>DataSet</code> and <code>DataLoader</code>, however, I am trying to train over TPUs with Pytorch XLA (alpha was just released a few days ago <a href="https://github.com/pytorch/xla" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/xla: Enabling PyTorch on XLA Devices (e.g. Google TPU)</a> ), and to do that I need to send my <code>DataLoader</code> to a <code>torch_xla.distributed.data_parallel.DataParallel</code> object, like <code>model_parallel(train_loop_fn, train_loader)</code> which can be seen in these example notebooks</p>
<p><a href="https://github.com/pytorch/xla/blob/master/contrib/colab/resnet18-training-xrt-1-15.ipynb" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/pytorch/xla/blob/master/contrib/colab/resnet18-training-xrt-1-15.ipynb</a></p>
<p><a href="https://github.com/pytorch/xla/blob/master/contrib/colab/mnist-training-xrt-1-15.ipynb" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/pytorch/xla/blob/master/contrib/colab/mnist-training-xrt-1-15.ipynb</a></p>
<p>So, I am now limited to a single <code>DataLoader</code>, which will need to handle both the true samples, and negative samples that need to be randomly selected from all my files.</p>
How to mask for each batch?

username: izuna385
cooked: <p>Currently I has  index of <code>batch_size * candidate_size</code> and desired_candidate : <code>batch_size * top_5</code></p>
<p>like, if <code>batch_size</code> == 16 and candidate_size == 15</p>
<pre><code class="lang-auto">&gt;&gt; index.size()
torch.Size([64, 15])

&gt;&gt; desired_candidate.size()
torch.Size([64, 5])

&gt;&gt; index[0]
tensor([104, 171, 182,   3,  56, 178,   6,   6,   4,  21,  30, 182,  27,  39,  56], device='cuda:0', dtype=torch.int32)

&gt;&gt; desired_candidate[0]
tensor([171,   4, 182, 102,  61], device='cuda:0')
</code></pre>
<p>I’d like to filter index by its desired_candidate, so the desired one is</p>
<pre><code class="lang-auto">&gt;&gt; index_[0]
tensor([104, -1, -1,   3,  56, 178,   6,   6,   -1,  21,  30, -1,  27,  39,  56], device='cuda:0', dtype=torch.int32)
</code></pre>
<p>for each batch. But I can’t find out how to do this.<br>
If someone would know how to implement this, I’d appreciate it.<br>
Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>It might depend a bit on the memory limitations you have. But the following should work (the sizes are very small for printing purposes):</p>
<pre><code class="lang-auto">import torch

b = 2
c = 3

index = torch.rand(b, c).mul(5).long()
desired_candidate = torch.rand(b, 5).mul(5).long()

print(index)
print(desired_candidate)


expanded_size = (b, c, 5)
expanded_index = index.unsqueeze(2).expand(expanded_size)
expanded_desired_candidate = desired_candidate.unsqueeze(1).expand(expanded_size)

mask = expanded_index.eq(expanded_desired_candidate).any(-1)

index[mask] = -1
print(index)
</code></pre>
<p>Let me know if it fits your needs !</p>

username: izuna385
cooked: <p>Thanks for great advice! I’ll try it.</p>
Cross Entropy Loss for multiclass classification

username: Er_Hall
cooked: <p>Hi all,</p>
<p>I have a question I have a multiclass classification problem where each sample belongs only to one class, my targets are in [0, C-1] format.</p>
<p>My question is: as input the cross entropy loss must take logits? so as a last layer I need to have a sigmoid layer?</p>

username: albanD
cooked: <p>You can see the doc for what exactly each one is doing: <a href="https://pytorch.org/docs/stable/nn.html#crossentropyloss" rel="nofollow noopener">cross entropy</a>, <a href="https://pytorch.org/docs/stable/nn.html#nllloss" rel="nofollow noopener">nll</a>, <a href="https://pytorch.org/docs/stable/nn.html#bceloss" rel="nofollow noopener">bce</a> and <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="nofollow noopener">bce with logits</a> should be the ones you want to check.</p>

username: ricardborras
cooked: <p>With cross entropy loss, it is enought to input class labels directly. Also, softmax is not needed for cross entropy loss.</p>
Sum true after mask

username: snip3r77
cooked: <p>Hi guys,</p>
<pre><code class="lang-auto">x = torch.tensor([[1,2], [3, 4], [5, 6]]) 
(x&lt;2).numpy().sum()
</code></pre>
<p>I’m able to do the above but is there a way to do it just using torch?</p>
<p>Thanks.</p>

username: albanD
cooked: <p>The sum() method exists in torch. Simply removing the <code>.numpy()</code> will make it work.<br>
Do you see an error when you do that?</p>

username: snip3r77
cooked: <p>Indeed it is. Thanks</p>
Pytorch and SQL

username: grrr
cooked: <p>We sometimes train models using annotations from multiple datasets. Merging multiple datasets into 1 and building dataloaders take a lot of effort and many, many for loops. I (only) recently found that organizing datasets into SQL tables and do merges/queries greatly reduces the amount of code I have to write and that probably saved a lot of my hairs.</p>
<p>In fact something like COCO annotations</p>
<blockquote>
<p>{“license”: 5,“file_name”: “COCO_train2011<br>
4_000000057870.jpg”,“coco_url”: “https://images_cocodataset_org/train2014/CC<br>
OCO_train2014_000000057870.jpg”,“height”: 480,“width”: 640,“date_captured”"<br>
: “2013-11-14 16:28:13”,“flickr_url”: “https://farm4_staticflickr_com/3153//<br>
2970773875_164f0c0b83_z.jpg”,“id”: 57870}, …</p>
</blockquote>
<p>can be represented as a SQL table with fields “license”, “file_name”, “coco_url”, “height”, “width”, etc. Merging annotations from two different datasets can be done by <a href="https://www.w3schools.com/sql/sql_join.asp" rel="noopener nofollow ugc">SQL joins</a> based on image names/indexes. Dataloaders could be written to process and return rows of a data table.</p>
<p>I think PyTorch could implement something like “torch.db” to do this. Traditional databases can’t efficiently handle GPU tensors, so an opportunity for PyTorch is to enable fast joins of tables with tensors and potentially optimally manage GRAM/RAM/disk access.</p>

username: RicCu
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/20822" rel="nofollow noopener">This issue</a> talks about an internal FB framework that could benefit you. They were wondering if publishing it made sense and they were asking for input from the community, so you might want to check it out and comment on your use case.<br>
I don’t really know the extent of that project nor whether it will ever be open source but it might be what you’re looking for.</p>
Looking for parallel implementation for a piece of code

username: gggggxz
cooked: <pre data-code-wrap="python"><code class="lang-python">import torch
x = torch.arange(25).view((5, 5))
y = torch.tensor([[3, 4, 2, 2], [0, 4, 1, 1], [2, 4, 3, 1]])  # shape: (3, 4)
result = torch.zeros(3, 4)
for i in range(1, 4):
    current_y = y[:, i]  # shape: (3,)
    prev_y = y[:, i - 1]  # shape: (3,)
    result[:, i] = x[prev_y, current_y]
print(result)
</code></pre>
<p>Here is the running result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/496a61ee11a6e3618163c4b9275641eefb81c496.png" data-download-href="https://discuss.pytorch.org/uploads/default/496a61ee11a6e3618163c4b9275641eefb81c496" title="%E5%9B%BE%E7%89%87"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/496a61ee11a6e3618163c4b9275641eefb81c496.png" alt="%E5%9B%BE%E7%89%87" data-base62-sha1="atsMi1c9uUiToXWqMAwfW6lIkp8" width="690" height="497" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E5%9B%BE%E7%89%87</span><span class="informations">692×499 9.74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I think this procedure can be executed in parallel, but I don’t know how. Can anyone help me?</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">result2 = torch.zeros(3, 4)
result2[:, 1:] = x[y[:, :-1], y[:, 1:]]
print((result==result2).all())
&gt; tensor(True)
</code></pre>

username: gggggxz
cooked: <p>Thank you for your answer, you solved my problem. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Old version of pytorch (v0.1.12) with CUDA 10.1

username: Samarth_Mishra
cooked: <p>Hi,</p>
<p>I need to use an old version of pytorch (0.1.12) for running an author’s implementation. Can I install it from source with CUDA 10.1?</p>
<p>I cloned the pytorch repository, and checked out v0.1.12. I then ran the following</p>
<pre><code class="lang-auto">conda install numpy pyyaml mkl setuptools cmake gcc cffi
</code></pre>
<p>where I got an error saying that package ‘gcc’ wasn’t found.</p>
<p>I tried ignoring this and running the following</p>
<pre><code class="lang-auto">conda install numpy pyyaml mkl setuptools cmake cffi
conda install -c pytorch magma-cuda100

export CMAKE_PREFIX_PATH=~/anaconda3/
python setup.py install
</code></pre>
<p>But this installs pytorch without CUDA/CUDnn. What can I do to install pytorch v0.1.12 with my current version of CUDA (which is on a server where I do not have root access)?</p>
<p>Any help would be appreciated. Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Have you tried installing the cuda packages in conda? That should make sure that it is detected during installation from source.</p>

username: Samarth_Mishra
cooked: <p>Hi, thanks for the response.<br>
I later just tried to compile it with an older version of CUDA (7.5). Realized I needed to install just the CUDA toolkit at a non-root location and it works with the Nvidia driver I have (418.67). Haven’t tried it with cuda 10.1 toolkit yet, but for now, I can just run with the cuda7.5</p>
Pytorch Crush using backward with exit code -1073741819 (0xC0000005)

username: _ike
cooked: <h2>Bug</h2>
<p>I am trying to use DCGAN with DCGAN TUTORIAL: <a href="https://github.com/pytorch/tutorials/blob/master/beginner_source/dcgan_faces_tutorial.py" rel="nofollow noopener">https://github.com/pytorch/tutorials/blob/master/beginner_source/dcgan_faces_tutorial.py</a><br>
But there occurs an error: Process finished with exit code -1073741819 (0xC0000005) During loss.backward() running.</p>
<h2>To Reproduce</h2>
<p>My problem is I can only train it on CPU. It raises an error when I try to train it on GPU.And my cuda is available.<br>
Here is the error.</p>
<p><code>Preformatted textProcess finished with exit code -1073741819 (0xC0000005)</code></p>
<p>the code is here <a href="https://github.com/IkeYang/AndrewNg_ML_practise/blob/master/GAN" rel="nofollow noopener">https://github.com/IkeYang/AndrewNg_ML_practise/blob/master/GAN</a></p>
<h2>Environment</h2>
<p>My environment is python3.6.9 windows10 torch1.2.0. cuda9.2</p>
<p>Please help solve this, thanks a lot.</p>

username: albanD
cooked: <p>Duplicate of <a href="https://discuss.pytorch.org/t/please-help-me-solve-this-error-process-finished-with-exit-code-1073741819-0xc0000005/57932" class="inline-onebox">Please Help me solve this error Process finished with exit code -1073741819 (0xC0000005)</a></p>
Is there a piece of code which splits backward process layer by layer?

username: DreamCaCao
cooked: <p>Hi!<br>
I want to insert some codes between layers in backward process of any model (in this case, I use ResNet-50). I considered backward hook, but it is not what I want. So I have searched /torch/csrc/autograd, /torch/c10, ./torch/aten, but I cannot find the piece of code which splits backward process layer by layer (like for statement). I also followed codes by GDB after building source by DEBUG mode, but it’s difficult to me because of python codes.<br>
I just found out that autograd is executed by node by node (I’m not sure). Then, does this node of graph represents layer?</p>
<p>Is there anybody who knows where is the piece of code which splits backward process layere by layer, or the fact that there is no such piece of code?</p>
<p>p.s.  I am mainly examining /torch/csrc/autograd/engine.cpp file  (especially Engine::execute and Engine::thread_main in this file). Is it the right path?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What do you mean by “layer by layer”? Do you mean the layers from <code>nn</code> ? Or each operation that is performed?</p>
<p>An easy way to do this could be to add Tensor hooks to all the points where you want to stop and you can fire pdb debug in the hook. Or do what you want to do.</p>
Pytorch squeeze doesn't work

username: zimmer550
cooked: <pre><code class="lang-auto">    def forward(self, input_seq, encoder_outputs, hidden=None):
        outputs, hidden = self.gru(input_seq, hidden)
        outputs = outputs[:, :, :self.hidden_size] + outputs[:, :, self.hidden_size:]
        attn_weights = self.attn(outputs, encoder_outputs)
        context = attn_weights.bmm(encoder_outputs.transpose(0, 1))
        print(context.size())
        context = context.squeeze(1)
        new_outputs = outputs
        new_outputs = new_outputs.squeeze(0)
        print(outputs.size())
        print(new_outputs.size())
        print(context.size())
        concat_input = torch.cat((new_outputs, context), 1)
        concat_output = torch.tanh(self.concat(concat_input))
        outputs = self.out(concat_output)
        return outputs, hidden
</code></pre>
<p>The output:</p>
<pre><code class="lang-auto">torch.Size([5, 1, 50])
torch.Size([5, 5, 50])
torch.Size([5, 5, 50])
torch.Size([5, 50])
</code></pre>
<p>the context tensor is squeezed but the output tensor is not squeezed (size of new_output should be 5,50 and not 5,5,50). Why is this happening?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Squeeze only works if the size of a given dimension is 1.<br>
If you want to remove a dimension of size &gt; 1 then you need to use a function to do that reduction like sum or max or mean.</p>
AssertionError: nn criterions don't compute the gradient w.r.t. targets - please mark these tensors as not requiring gradients

username: yunpei
cooked: <p>When I use the nn.MSELoss(), just as follows:<br>
<strong>nn.MSELoss(feature1, feature2)</strong></p>
<p>where both the feature1 and the feature are need to be differentiated here. That means feature1 is the output of model1 and feature2 is the output of model2.</p>
<p>But the error is like this:<br>
<strong>AssertionError: nn criterions don’t compute the gradient w.r.t. targets - please mark these tensors as not requiring gradients</strong></p>
<p>How should I solve it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Which version of pytorch are you using?</p>

username: yunpei
cooked: <p>Thanks for replying.</p>
<p>pytorch0.4</p>

username: albanD
cooked: <p>Ok, I think this has been fixed in the latest versions.<br>
Can you try upgrading to see if it fixes the issue?</p>
autograd.Variable in training loop

username: tomt
cooked: <p>Hi,</p>
<p>I’m looking at some earlier github repos where people have used this in a training loop, e.g.:</p>
<pre><code class="lang-auto">Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.Tensor
for epoch in range(0, epochs):
    for i, data in enumerate(trainloader):
        # Configure model input
        #imgs_lr = Variable(imgs["lr"].type(Tensor))
</code></pre>
<p>where imgs is a dataloader. Have been reading a bit about Variable in the docs, where it is no longer used. If I want to apply this to my code, where I’m using the newest PyTorch version, is this somehow equivalent:</p>
<pre><code class="lang-auto">imgs_lr = imgs["lr"].type(Tensor)
</code></pre>
<p>where I don’t need to use Variable, and from the docs it will automatically go through gradient ?<br>
And is this equivalent?</p>
<pre><code class="lang-auto">fake = Variable(Tensor(np.zeros((LR.size()[0],*discriminator.output_shape))),requires_grad=False) 
  fake = torch.zeros((LR.size()[0],*discriminator.output_shape), requires_grad =False).type(Tensor)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Yes you can just drop the Variable now.<br>
The <code>requires_grad=False</code> is the default so you don’t need to specify it when creating your Tensor.<br>
Also factory functions like <code>torch.zeros()</code> accept a dtype argument that you can use to directly create a Tensor of the right type and so avoid the <code>.type()</code> call.</p>

username: tomt
cooked: <p>If <code>requires_grad=False</code> by default, shouldn’t my <code>imgs_lr = imgs["lr"].type(Tensor)</code> have set it to True, or am I misunderstanding?</p>
<p>Is it also acceptable to use .type(Tensor), or do I need to use dtype argument instead?</p>

username: albanD
cooked: <p>You don’t usually require gradients for the input to train neural networks. The parameterd inside your model are <code>requires_grad=True</code> though.</p>
<p>You can use any type conversion functions. They are a few of them for historical reasons, but there is no benefit of using one against the other.</p>
Feature map visualization error

username: Surekha_Gaikwad
cooked: <p>Hi I am trying to use the same code as mentioned in one of the post of this discussion forum for visualizing feature map but I am kind of getting this error "hook_result = hook(self, input, result)</p>
<p>TypeError: ‘NoneType’ object is not callable"</p>
<p>Below is the code</p>
<pre><code class="lang-auto">activation = {}
    def get_activation(name):
        def hook(model, input, output):
            activation[name] = output.detach()
            return hook
        
    model.dc.register_forward_hook(get_activation('dc'))
    
    dataiter = iter(trainloader)
    images,labels = dataiter.next()
    imshow(images[0],labels[0])
    print(images[0].unsqueeze_(0).shape)
    
    output = model(images[0].unsqueeze_(0))
    print(output.shape)

    
    act = activation['dc'].squeeze()
    fig, ax = plt.subplots(act.size(0))
    for idx in range(act.size(0)):
        ax[idx].imshow(act[idx])
</code></pre>
<p>Model</p>
<pre><code class="lang-auto">class NormalCNN(nn.Module):
    def __init__(self, args, classes):
        super(NormalCNN, self).__init__()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.args = args
        self.class_num = 10
        self.classes = classes
        
        resnet18 = models.resnet18(pretrained=True)
        
        self.backbone = nn.Sequential(*list(resnet18.children())[0:5])
        self.dc = nn.Conv2d(64, 1 , kernel_size = 1, stride=1, padding=1)
        
        self.mlp = nn.Sequential(
                nn.Linear(64, 64),
                nn.ReLU(),
                nn.Linear(64, 10))

        ### ----------------------------------------------

    def forward(self, imgs):

        v = self.backbone(imgs)
            
        v_out = self.dc(v)
        v_out = v_out/8
            
        out = nn.Upsample(size=(8, 8), mode='bilinear')(v_out)
        out = out.view(-1,1*8*8)
        cls_scores = self.mlp(out)
        
        return cls_scores # Dim: [batch_size, 10]
</code></pre>

username: ptrblck
cooked: <p>You have a small indentation error in your hook method:</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook
</code></pre>
<p>The return statement should be called from <code>get_activation</code>.</p>
How to plot different channels in grid?

username: Frida
cooked: <p>Hi All,<br>
If I have tensor [1,16,16,16] i.e., 1 batch, 16 channels and image size 16 on 16. How is it possible to show plot in which each channel can be see separately?<br>
Img_0 -  tensor[:,0,:,:]<br>
Img_1 -  tensor[:,1,:,:]</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>You could use <code>matplotlib</code> and plot each channel in a subplot:</p>
<pre><code class="lang-python">imgs = torch.empty(1, 16, 16, 16).uniform_(0, 1)

f, axarr = plt.subplots(imgs.size(1))

for idx in range(imgs.size(1)):
    img = imgs[:, idx].squeeze().numpy()
    axarr[idx].imshow(img)
</code></pre>
How to prevent a variable from automatic differentiation?

username: dl_noob
cooked: <p>Hi,<br>
In my implementation, I have following piece</p>
<pre><code class="lang-auto">dec = decoder_output.clone().squeeze()
for i in self.state_history:
    dec[i] = -100000

topv, topi = dec.topk(1)
action = topi.squeeze().detach() 
</code></pre>
<p><code>decoder_output</code> is output of <code>F.log_softmax</code><br>
My question is, is <code>dec</code> recorded for automatic differentiation and does it affect the differentiation for <code>decoder_output</code> and rest of the graph? What is the best way to prevent it from automatic differentiation?<br>
Also, is accessing <code>dec[i]</code> inside a for loop possible if dec is a torch.cuda tensor?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="dl_noob" data-post="1" data-topic="57895">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/eada6e/48.png" class="avatar"> dl_noob:</div>
<blockquote>
<p>is <code>dec</code> recorded for automatic differentiation and does it affect the differentiation for <code>decoder_output</code> and rest of the graph?</p>
</blockquote>
</aside>
<p>The assignment should be recorded and the inplace operation could yield an error.<br>
You could <code>detach</code> <code>dec</code> from <code>decoder_output</code>, if you don’t want to track this assignment.</p>
<aside class="quote no-group" data-username="dl_noob" data-post="1" data-topic="57895">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/eada6e/48.png" class="avatar"> dl_noob:</div>
<blockquote>
<p>Also, is accessing <code>dec[i]</code> inside a for loop possible if dec is a torch.cuda tensor?</p>
</blockquote>
</aside>
<p>Yes, that’s possible.</p>
nn.Transformer attention mask transposed?

username: DonCano
cooked: <p>in nn.Transformer the method “generate_square_subsequent_mask” outputs a square matrix with the first column with all 0, second column with -inf and all 0, and so on.</p>
<p>if we are working column wise (ie the input is SEQ_LEN, BATCH_SIZE, E_DIM) shouldn’t it be transposed?</p>

username: DonCano
cooked: <p>Sorry, just went through the code. It’s okey the way it is</p>
Full loss Hessian spectrum estimation

username: LeviViana
cooked: <p>For those interested in analyzing the loss landscape of modern neural nets, it is worth noting that <a href="https://arxiv.org/pdf/1901.10159.pdf" rel="nofollow noopener">this article</a> has introduced a method for estimating the full loss Hessian spectrum. I’ve implemented a simple version of this method here: <a href="https://github.com/LeviViana/torchessian" rel="nofollow noopener">https://github.com/LeviViana/torchessian</a>.</p>
<p>This method brings unprecedented clarity on understanding very high dimensional losses’ landscapes, and even the simple examples I’ve published in the repo weren’t feasible about a year ago.</p>
<p>As an example of application, you can study how batchnorm layers change the shape of the loss landscape, or how the spectrum of a single batch can approximate the global dataset landscape, or how decreasing the learning rate pushes the neural net to a point where the curvatures are not necessarily bigger, or how hardly separable data points can generate big eigenvalues on the Hessian, etc etc.</p>
<p>I hope you’ll have fun as much as I do when playing around with losses’ landscapes !</p>

username: ptrblck
cooked: <p>Thanks for sharing your code here! It looks really interesting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Tsanley: A tensor shape checker and annotator

username: ekshaks
cooked: <p>We’ve been working on a tool <code>tsanley</code>  to enable finding subtle shape errors in your deep learning code quickly and cheaply. The key idea is to label tensor variables with their <em>expected</em> shapes (e.g., <code>x : 'b,t,d' = ...</code>) using optional types in Python 3.x and let <code>tsanley</code> perform shape validity checks at runtime automatically. Works with small and big tensor programs.</p>
<p><strong>repository</strong>: <a href="https://github.com/ofnote/tsanley" rel="nofollow noopener">https://github.com/ofnote/tsanley</a></p>
<p><strong>examples</strong>: <a href="https://github.com/ofnote/tsanley/tree/master/models" rel="nofollow noopener">models</a> (Resnet, GraphNNs, Transformers)</p>
<p><strong>Quick example</strong>:</p>
<pre><code class="lang-python">def foo(x):
    x: 'b,t,d'               #expected shape of x is (B, T, D).                
    y: 'b,d' = x.mean(dim=0) * 2        # error!         
    z: 'b,d' = x.mean(dim=1)           # ok
    return y, z
</code></pre>
<p>Function <code>foo</code> contains tensor variables labeled with their <em>named shapes</em> using a <em>shorthand</em> notation. It has a subtle shape error in the assignment to <code>y</code>: we expect the shape of <code>y</code> to be <code>(B,D)</code>, however <code>mean</code> got rid of the first, and not the second, dimension. <code>pytorch</code> won’t flag this as an error: instead, we will get a weird shape inconsistency error somewhere downstream.</p>
<p><code>tsanley</code> finds such unexpected bugs quickly at runtime:</p>
<pre><code class="lang-auto">Update at line 37: actual shape of y = t,d 
  &gt;&gt; FAILED shape check at line 37 
  expected: (b:10, d:1024), actual: (100, 1024) 
  
Update at line 38: actual shape of z = b,d 
  &gt;&gt; shape check succeeded at line 38 
</code></pre>
<p>Writing these named shape annotations <em>manually</em> can also get tedious.  <code>tsanley</code> can <strong>auto-annotate</strong> the tensor variables in your (or someone else’s) code, if the code is executable. This is especially useful when trying to dig deep into or adapt an existing code / library for your project.</p>
<p>The tool builds upon the <a href="https://github.com/ofnote/tsalib" rel="nofollow noopener">tsalib</a> library, which introduced a shorthand notation for labeling tensor variables with their <em>named</em> shapes, irrespective of the backend tensor library used.</p>
<p>We would love feedback on <a href="https://github.com/ofnote/tsanley" rel="nofollow noopener">tsanley</a> and hope it is useful for your coding/debugging workflow.</p>

username: ptrblck
cooked: <p>Thanks for sharing! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Functional random_crop

username: Kong
cooked: <p>how can i do the random crop using functional ?<br>
<a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomCrop" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomCrop</a></p>

username: ptrblck
cooked: <p>Have a look at the <a href="https://github.com/pytorch/vision/blob/7ae1b8c9c00e2bec1b0c111cd1299415519ef002/torchvision/transforms/transforms.py#L473" rel="nofollow noopener">implementation</a> to get a good idea, how the random cropping is applied internally.</p>
Backward function of median

username: mxh000
cooked: <p>It would be nice if I could take the median of losses in a batch, instead of the mean, and get gradients based on that. But does torch.median() have a useful backward function, or is it only indexing like torch.max()? Thanks for your help.</p>

username: tom
cooked: <p>You already can, but the median gradient is generally very sparse by the nature of the median: only the element (pytorch does not interpolate for an even number of inputs) that has the value of the median has a gradient. The way ties for the are handled in pytorch likely is not based on rigorous mathematics, probably because the case lacks practical relevance.</p>
<p>Best regards</p>
<p>Thomas</p>

username: AntreasAntoniou
cooked: <p>How does the gradient computation work exactly here? I’ve spent hours researching for an answer but I was not able to match pytorch’s gradients from torch.median().</p>

username: tom
cooked: <p>Consider</p>
<pre><code class="lang-python">a = torch.randn(3,4, requires_grad=True)
b = a.median(1)
print("a =", a)
print("b = ",b)
</code></pre>
<p>giving</p>
<pre><code class="lang-auto">a = tensor([[-0.8903,  1.3827,  0.8102,  0.2972],
        [-0.5800, -0.3084, -0.6165, -0.5376],
        [-0.5144, -1.6414, -0.1507,  0.5274]], requires_grad=True)
b = torch.return_types.median(
values=tensor([ 0.2972, -0.5800, -0.5144], grad_fn=&lt;MedianBackward1&gt;),
indices=tensor([3, 0, 0]))
</code></pre>
<p>Then the backward will just be the grad_out scattered to the appropriate indices:</p>
<pre><code class="lang-python">b.values.sum().backward()
print(a.grad)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 1.],
        [1., 0., 0., 0.],
        [1., 0., 0., 0.]])
</code></pre>
<p>That also is why the indices are non-optional: They need to be computed for the backward, anyways.</p>
<p>Best regards</p>
<p>Thomas</p>

username: AntreasAntoniou
cooked: <p>Thank you Tom. I figured this out shortly after asking this question. :). For those interested:</p>
<pre><code class="lang-auto">def backward_median(x, v):
    median_x = np.median(x)

    v_original_order = np.zeros(x.shape)

    v_original_order_hits = v_original_order.copy()

    v_original_order_hits[x != median_x] = 0.

    v_original_order_hits[x == median_x] = v[x == median_x]

    return v_original_order_hits
</code></pre>
<p>This is a numpy version.</p>
Save dataset into .pt file

username: himat
cooked: <p>I am taking MNIST data and performing some processing on it.<br>
Instead of doing this processing every time the image is loaded, I want to just save it as a new dataset so that I can just directly read it the next time.</p>
<p>What is the proper way of saving a dataset? I can’t seem to find any examples.</p>

username: ptrblck
cooked: <p>You approach so store the data as tensors seems like a good idea!<br>
If you save the data as <code>tensors</code> and load them in your next run, you could pass them into a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.TensorDataset">TensorDataset</a>.</p>

username: RobertYu
cooked: <p>How about using pickle library?</p>
<p><a href="https://docs.python.org/3/library/pickle.html" rel="nofollow noopener">https://docs.python.org/3/library/pickle.html</a></p>

username: Frida
cooked: <p>Thanks a lot, can you share some example?<br>
What should be done after calling:<br>
torch.utils.data.TensorDataset(data)<br>
Out[14]: &lt;torch.utils.data.dataset.TensorDataset at 0x1d6c4522ef0&gt;</p>

username: ptrblck
cooked: <p>Once you’ve loaded the tensors and created a <code>TensorDataset</code>, you could pass it to a <code>DataLoader</code> and start the training. Have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data loading tutorial</a> for more information.</p>

username: Frida
cooked: <p>Thanks but how can I use it in a separate session?</p>

username: ptrblck
cooked: <p>You could just load the tensors again and create the Dataset in the same way in another script.</p>
Gradient of Standard Deviation is nan

username: marcel1991
cooked: <p>I encountered a strange behaviour that I don’t reall understand. When I want to normalize a Variable I run into a zero division error during the backward pass when the standard deviation is zero.</p>
<p>For example:</p>
<pre><code class="lang-auto">a = Variable(torch.ones(1), requires_grad=True)
b = (a - a.mean())/(a.std() + 1e-4)
b.backward()
print(a.grad) # gives nan
</code></pre>
<p>I tracked down this problem and it even occurs if I just divide by the std with adding of some high epsilon</p>
<pre><code class="lang-auto">a = Variable(torch.ones(1), requires_grad=True)
b = 1/(a.std() + 1)
b.backward()
print(a.grad) # gives nan
</code></pre>
<p>But the epsilon should actually prevent the division by zero of the derivative right?<br>
So what is happening here? And how can I normalize without running into this error?</p>

username: SimonW
cooked: <p>Well, sqrt(x)'s derivative at x=0 is undefined.</p>

username: marcel1991
cooked: <p>Thank you, sure of course. I totally forgot about the sqrt(x)</p>

username: unnir
cooked: <p>sorry, could you share a link where I can read about that. Since, I thought we can find a derivative of square root.</p>

username: SimonW
cooked: <p>the left limit doesn’t exist…</p>

username: marcel1991
cooked: <p>The sqrt(x) is not defined for values smaller or equal zero (at least for real numbered values). Doing this operation with such values results in nan. Therefore the derivative does only exist for x &gt; 0.</p>

username: fermat97
cooked: <p>Hi, how could you get rid of this issue? I need to use std, is there any solution to avoid nan in the backward pass?</p>
How to measure the executive time (training and inference)

username: Wong
cooked: <p>Hi everyone,</p>
<p>We aim to measure the executive time in the GPU mode. We take Mnist as an example. The code is as follows.</p>
<pre><code class="lang-auto">109     for epoch in range(1, args.epochs + 1):
110         # start time
111         torch.cuda.synchronize()
112         since = int(round(time.time()*1000))
113         train(args, model, device, train_loader, optimizer, epoch)
114         torch.cuda.synchronize()
115         time_elapsed = int(round(time.time()*1000)) - since
116         print ('training time elapsed {}ms'.format(time_elapsed))
117
118         # start time
119         torch.cuda.synchronize()
120         tsince = int(round(time.time()*1000))
121         test(args, model, device, test_loader)
122         torch.cuda.synchronize()
123         ttime_elapsed = int(round(time.time()*1000)) - tsince
124         print ('test time elapsed {}ms'.format(ttime_elapsed))
</code></pre>
<p>The result is as follows.</p>
<pre><code class="lang-auto">training time elapsed 13325ms
testing time elapsed 2115ms
</code></pre>
<p>However, we find that there is no difference between training time and inference time when we caculate the average time  (total time/num_samples). That is, 13325/60000 and 2115/10000.  We are confused with the fact.</p>

username: tom
cooked: <p>I think your basic timing setup makes sense (congrats!).</p>
<p>You’d have to figure in things such batch size, number of synchronizations during training etc.<br>
For MNIST, the main thing might not be the actual compute, but the latency of getting things to the GPU and getting results back to the CPU. If you then have the same batch size, you would expect proportional results regardless of the extra computation the backward takes.</p>
<p>Best regards</p>
<p>Thomas</p>
Model.cuda() is very slow

username: hij
cooked: <p>I just updated my pytorch to 1.3 and it takes a long time (5~10mins) to call cuda() on my quite large model. Before the update, its almost instantaneous. I am using a titan x pascal with cudatoolkit 10.1.168 if that helps.</p>
<p>Once the model is loaded, the training itself seems fine.</p>

username: dhpollack
cooked: <p>I am also experiencing this issue.  I posted an issue to the pytorch github.  The quick fix is to downgrade to CUDA 10.0</p>
<pre><code class="lang-auto">conda install -c pytorch pytorch cudatoolkit=10.0
</code></pre>

username: smth
cooked: <p>not entirely sure why yet, but we’re looking into it.</p>
<p>tracking in: <a href="https://github.com/pytorch/pytorch/issues/27807" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/27807</a></p>
Does pytorch support training with low-precision INT8?

username: rps
cooked: <p>I am trying to experiment low-precision training - INT8. Is there support for this in pytorch?. Also what quantization methods are supported?</p>

username: Marat
cooked: <p>Training with int8 - no chance due to numerical stability limitations I guess, but inference on int8 is very interesting.</p>

username: linsonghere
cooked: <p>however when I use INT8 to compute, it doesn’t faster even slower, i wonder why</p>

username: Marat
cooked: <p>Pytorch does not support efficient INT8 scoring, and if you do not have Volta you will not gain any speed gain on both train and score on fp16. If you want fast scoring in int8 consider using of TensorRT you will get up to 3x faster scoring on ResNet like nets on INT8 with “slightly” lower accuracy</p>

username: polarbaryon
cooked: <p>I am not aware of any native 8 bit or lower training, or for that matter, inference, as compared with something like tflite, which only supports it in specific instances.  Partially I assume this has to do with the fact that there is no canonical method for quantization.  There are a number of implementation examples though, see, e.g., <a href="https://github.com/eladhoffer/quantized.pytorch" rel="nofollow noopener">https://github.com/eladhoffer/quantized.pytorch</a>, or Glow.</p>

username: tom
cooked: <p>If you look at the github issues or PRs or even the git tree’s test directory, you’ll find there is good progress towards a comprehensive solution of the various quantisation strategies.</p>
<p>Best regards</p>
<p>Thomas</p>

username: linsonghere
cooked: <p>I have tried to transform my pytorch model into ONNX model, and transform it into TensorRT model, but I met an unexpected error(using yolov3.onnx downloaded from offical web), it said “ERROR: Network must have at least one outout”, have you ever met this problem?</p>
Torch model downloaded as string to bytes

username: Bastulli
cooked: <p>Hello, I am downloading a model via Dropbox url and saving the data as a system string.  In order to load the model into Pytorch I have to convert the data from string to byte data.  When I use .encode() with ‘UTF-8’ codec to convert the string data to byte data and try to load it into BytesIO I get UnpicklingError: invalid load key, ‘&lt;’</p>
<p>I wish I could just use Python request and use the method .content to get the raw byte info but I am stuck with string data on the platform I am using.</p>
<p>Any other ideas on how to convert string data to byte?</p>
<pre><code class="lang-auto">data_str = self.Download("https://www.dropbox.com/s/examplepath/checkpoint.pth?dl=1")

data_byte = data_str.encode()

self.agent.local.load_state_dict(torch.load(BytesIO(data_byte), map_location=lambda storage, loc: storage))
</code></pre>
<blockquote>
<p>UnpicklingError: invalid load key, ‘&lt;’</p>
</blockquote>

username: tom
cooked: <p>There are plenty of examples and utility functions for downloading and opening models in <a href="https://github.com/pytorch/pytorch/blob/master/torch/hub.py" rel="nofollow noopener">the utils for the torch model hub</a>. The key likely is to download them as an octet stream into bytes. You are probably best off using the api rather than reinventing the implementation.</p>
<p>Best regards</p>
<p>Thomas</p>
Setting one optimizer for a list of Models

username: Deeply
cooked: <p>I have a few models defined in list:</p>
<p><code>G = [ SomeResNet().cuda() for i in range(no_models)]</code></p>
<p>I need to define the <code>Adam optimizer</code> for all these models, ie <code>G[0], G[1],...</code><br>
Is the following correct?</p>
<pre><code class="lang-auto">x=[]
for i in range(no_models):
    x= itertools.chain(x, G[i].parameters())

optimizer = torch.optim.Adam(
    x, lr= 0.01, betas=(0.8, 0.9)
)
</code></pre>

username: ptrblck
cooked: <p>If you want to get the parameters of all modules, you could also use an <code>nn.ModuleList</code> and just pass the <code>.parameters()</code> of this <code>ModuleList</code> to the optimizer.<br>
Using this approach would also make sure to push all submodules to the CPU/GPU if needed.</p>

username: Deeply
cooked: <p>Great idea! Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>; this worked for me:</p>
<p><code>optimizer = torch.optim.Adam( nn.ModuleList(G).parameters(), lr= 0.01, betas=(0.8, 0.9) )</code></p>
NLP basic array shape

username: Tank
cooked: <p>I am looking into building an RNN where the inputs are words. As you know not every word is the same size and therefore when I am one hot encoding each letter I end up having to pad the end of most words with zero arrays. Is there a better way to do this? Does a tensor even work if the array is not consistent. Where consistent is  defined as having the same dimensions.</p>
<p>In other words I am currently doing the following:<br>
number_of_words * len(longest_word) * number_of_chars</p>
<p>since I do not believe the following is possible<br>
number_of_words * len(word_i) * number_of_chars</p>

username: vdw
cooked: <p>I’m not sure what you’re trying to do. I seems to me that you’re mixing words and characters. What is your “unit of interest” w.r.t. each time step of the RNN? I see just two alternatives</p>
<ul>
<li>
<strong>Characters</strong>, i.e., at each time step, you feed the next character to the RNN. Here, you can simply one-hot encode each character. Word lengths are in this case not important.</li>
<li>
<strong>Words</strong>, i.e., at each time step, you feed the next word to the RNN. Just use an <code>Embedding</code> layer to map each word to a vector representation of a fixed size. For example, each word is mapped to a 100-dim vector. Again, the actual lengths of words is not important.</li>
</ul>

username: dhpollack
cooked: <p>You need to create word vectors with something like word2vec or fasttext or create character vectors in a similar fashion.  Then each of your vectors will be <code>num_words</code> x <code>word_vector_size</code> or more generically <code>length</code> x <code>feature_dim</code>.  You can see examples in the pytorch tutorials for this:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html</a></p>
<p><a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>But the general idea is that you want to avoid using one hot-encodings and instead use dense vector representations.</p>
Computing covarince matrix from batches of cholesky upper matrices

username: Bhavya_Bhatt
cooked: <p>I have a batch of upper triangular matrix of shape (B, N, N) and I want to calculate a batch of positive definite matrices (covariance matrix) using cholesky decomposition such that each matrix in resultant batch comes from corresponding upper triangular matrix in the given batch of upper triangular matrices.</p>

username: KFrank
cooked: <p>Hi Bhavya!</p>
<aside class="quote no-group" data-username="Bhavya_Bhatt" data-post="1" data-topic="57964" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bhavya_bhatt/48/14371_2.png" class="avatar"> Bhavya_Bhatt:</div>
<blockquote>
<p>I have a batch of upper triangular matrix of shape (B, N, N) and I want to calculate a batch of positive definite matrices (covariance matrix) using cholesky decomposition such that each matrix in resultant batch comes from corresponding upper triangular matrix in the given batch of upper triangular matrices.</p>
</blockquote>
</aside>
<p>Let me say what I think your problem is:</p>
<p>You have a batch (batch-size B) of NxN upper triangular matrices<br>
<em>that are  stored as full NxN matrices,</em> for which the elements in<br>
the lower triangle are zero.</p>
<p>We think of these matrices as having come from the Cholesky<br>
decomposition of symmetric-positive-definite (SPD) matrices,<br>
and you wish to recover those SPD matrices as explicit, full<br>
NxN matrices.</p>
<p>Let <code>S</code> be the desired full SPD matrix, and <code>L</code> be its upper-triangular<br>
Cholesky decomposition.  Then, by definition, <code>S = L x L-transpose</code>.</p>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.matmul" rel="noopener nofollow ugc">torch.matmul()</a> does matrix multiplication on a batch basis, and you<br>
can use <a href="https://pytorch.org/docs/stable/torch.html#torch.transpose" rel="noopener nofollow ugc">torch.transpose()</a> to perform matrix transposition on a batch<br>
basis by specifying the dimensions to swap.</p>
<p>So, letting <code>l</code> be your <code>(B, N, N)</code> batch of upper-triangular matrices,<br>
<code>s = torch.matmul (l, torch.transpose (l, 1, 2))</code><br>
should give you <code>s</code>, the <code>(B, N, N)</code> batch of your SPD matrices.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Bhavya_Bhatt
cooked: <p>This is perfectly what I was looking for !!<br>
Thanks K. Frank !!</p>