akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The efficiency of torch.cat

Laine_J: <p>Hi, I implemented a network, which used torch.cat()  to concatenate the input. when I train this network, I found it’s too slow. Is torch.cat inefficient, and are there any other ways to do the same?<br>
The following is the code, thank you</p>
<pre><code class="lang-auto">class AdaptationNN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(AdaptationNN, self).__init__()
        self.fc1 = nn.Linear(input_dim, 1000, bias=False)
        self.fc1.weight = torch.nn.Parameter(torch.Tensor(1000, input_dim).uniform_(
            -np.sqrt(0.01 / (input_dim + 1000)), np.sqrt(0.01 / (input_dim + 1000))))
        self.fc1.bias = torch.nn.Parameter(torch.zeros(1000))
        self.bn1 = nn.BatchNorm1d(1000, momentum=0.05)
        self.act1 = nn.ReLU()
        #the size of speaker code is 50
        self.fc2 = nn.Linear(1050, 1000, bias=False)
        self.fc2.weight = torch.nn.Parameter(torch.Tensor(1000, 1050).uniform_(
            -np.sqrt(0.01 / (1000 + 1050)), np.sqrt(0.01 / (1000 + 1050))))
        self.fc2.bias = torch.nn.Parameter(torch.zeros(1000))
        self.bn2 = nn.BatchNorm1d(1000, momentum=0.05)
        self.act2 = nn.ReLU()
        self.fc3 = nn.Linear(1050, output_dim, bias=True)
        self.fc3.weight = torch.nn.Parameter(torch.Tensor(output_dim, 1050).uniform_(
            -np.sqrt(0.01 / (1050 + output_dim)), np.sqrt(0.01 / (1050 + output_dim))))
        self.fc3.bias = torch.nn.Parameter(torch.zeros(output_dim))

    def forward(self, speaker_codes, input):
        x = self.act1(self.bn1(self.fc1(torch.cat((speaker_codes, input),1))))
        x = self.act2(self.bn2(self.fc2(torch.cat((speaker_codes, x), 1))))
        x = self.fc3(torch.cat((speaker_codes, x), 1))
        return x
</code></pre>

Anand_Ramachandran: <p>The following issue indicates that cat is slow on CPU: <a href="https://github.com/pytorch/pytorch/issues/18634" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/18634</a></p>
<p>I found that a custom cat function is slightly faster on CPU:</p>
<pre><code class="lang-auto">def customCat(allTensors):
    totalSize = sum([t.shape[0] for t in allTensors]);
    newTensor = torch.zeros(totalSize, allTensors[0].shape[1], allTensors[0].shape[2]);
    counter = 0;
    for t in allTensors:
        newTensor[counter:counter+t.shape[0]] = t;
        counter += t.shape[0];
    return newTensor;
</code></pre>
<p>It definitely isn’t a generic replacement for cat, and I haven’t tested this code.</p>
<p>You also have the option of doing concatenations in the DataLoader via a user-defined collate_fn. DataLoader will be able to parallelize the cat operations across multiple batches. It may hide any latency you see.</p>
How to get the same GPU index order as nvidia-smi?

Poisson_Chasseur: <p>I work with a GPU server where many of us can use it and we should not use all GPUs for a single user. So I’m trying to do something that will be able to automatically use the GPU with the least memory used by other users.</p>
<p>I saw that it is possible to get the same GPU usage information as with “nvidia-smi” with functions like the package [nvgpu](<a href="https://pypi.org/project/nvgpu" rel="nofollow noopener">https://pypi.org/project/nvgpu</a> /) or an example using [subprocess.call to execute the command “nvidia-smi” and place the output in a csv file] (<a href="https://discuss.pytorch.org/t/why-the-two-gpus-on-my-machine-have-the-same-id-so-that-pytorch-can-only-choose-one/6494/2" class="inline-onebox">Why the two GPUs on my machine have the same ID, so that Pytorch can only choose one?</a>). So with that, I can know which GPU has the least memory used.</p>
<p>The only trouble is that the order of the devices of PyTorch is not the same as on “nvidia-smi” (can be seen by the order of names in my case). <strong>So I’m looking for a way to convert the GPU order from “nvidia-smi” to the one used by PyTorch to properly select the GPU with the least memory used.</strong></p>
<p>Note: In my case, the server has 2 GPU “GTX 1080 Ti” and 1 GPU “TITAN RTX”, so I can see the difference between the names, but not between the 2 GPU “GTX 1080 Ti”.</p>

Poisson_Chasseur: <p>I found the solution to the problem that has already been solved as presented in the following links:</p>
<ol>
<li>
<p><a href="https://stackoverflow.com/questions/52815708/order-of-cuda-devices" rel="nofollow noopener">Order of CUDA devices</a></p>
</li>
<li>
<p><a href="https://github.com/matterport/Mask_RCNN/issues/109" rel="nofollow noopener">How to setting the GPU No. for training?</a></p>
</li>
<li>
<p><a href="https://discuss.pytorch.org/t/gpu-devices-nvidia-smi-and-cuda-get-device-name-output-appear-inconsistent/13150">Gpu devices: nvidia-smi and cuda.get_device_name() output appear inconsistent</a></p>
</li>
</ol>
<p>The solution is simply to make the first line of the following code BEFORE calling functions related to “torch.cuda” since its initialization is performed only once.</p>
<pre><code class="lang-auto">#Change the order so that it is the one used by "nvidia-smi" and not the 
#one used by all other programs ("FASTEST_FIRST")
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"

#Check that it is the same order as for "nvidia-smi":
[torch.cuda.get_device_name(i) for i in xrange(0,torch.cuda.device_count())]

</code></pre>
Using a pytorch model for inference

Vishal_Ahuja: <p>I am using the fastai library (<a href="http://fast.ai" rel="nofollow noopener">fast.ai</a>) to train an image classifier. The model created by fastai is actually a pytorch model.</p>
<pre><code>type(model)
&lt;class 'torch.nn.modules.container.Sequential'&gt;
</code></pre>
<p>Now, I want to use this model from pytorch for inference. Here is my code so far:</p>
<pre><code>torch.save(model,"./torch_model_v1")
the_model = torch.load("./torch_model_v1")
the_model.eval() # shows the entire network architecture
</code></pre>
<p>Based on the example shown here: <a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#sphx-glr-beginner-data-loading-tutorial-py" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#sphx-glr-beginner-data-loading-tutorial-py</a>, I understand that I need to write my own data loading class which will override some of the functions in the Dataset class. But what is not clear to me is the transformations that I need to apply at test time? In particular, how do I normalize the images at test time?</p>
<p>Another question: is my approach of saving and loading the model in pytorch fine? I read in a pytorch tutorial that the approach that I have used is not recommended. The reason is not clear though.</p>

austin: <p>Hi Vishal,</p>
<p>It is not recommended to save the entire model (architecture and weights) the way that you did because that method will not work if you try to load the model in a different project. For example, if you try to send your model file <code>./torch_model_v1</code> to me and I try to load it with <code>torch.load("./torch_model_v1")</code> I will get an error because it’s likely my project won’t have the exact same directory structure as your project.</p>
<p>Instead you should save only the model weights (state dict), define the architecture in code, then load the weights into the new models state dict.</p>
<pre><code class="lang-python">## save 
torch.save(model.state_dict(), "./torch_model_v1.pt")

## load
model = Model() # the model should be defined with the same code you used to create the trained model
state_dict = torch.load( "./torch_model_v1.pt")
model.load_state_dict(state_dict)

</code></pre>

Mohammed_Awney: <p><a class="mention" href="/u/austin">@austin</a> Is it necessary to save optimizer state dict for inference as listed in pytorch tutorial " When saving a general checkpoint, to be used for either inference or resuming training, you must save more than just the model’s  <em>state_dict</em> . It is important to also save the optimizer’s  <em>state_dict</em> , as this contains buffers and parameters that are updated as the model trains." ? or this just it is important for completing the training?</p>

ptrblck: <p>It’s important to resume the training. If you just want to use the model for inference, loading the <code>state_dict</code> and setting the model to <code>eval()</code> should be enough.</p>
Question about gan

Frida: <p>Is anyone know what enforces the generated images to be at the correct range of [0,1]?</p>

ptrblck: <p>You could try to use a <code>sigmoid</code> at the end of your generator.<br>
In the <a href="https://github.com/pytorch/examples/blob/ee964a2eeb41e1712fe719b83645c79bcbd0ba1a/dcgan/main.py#L142" rel="nofollow noopener">DCGAN example</a> a <code>tanh</code> is used to get predictions in <code>[-1, 1]</code>, which might work better.</p>
Importing dataset that contain training set with sub-folders

yusufbrima: <p>Hi guys,<br>
I am currently working on a MRI classification project.<br>
I need some suggestions  on how to import dataset in a folder structure given below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edf46479176940dde5b7436e81a5ad76786e8e33.png" data-download-href="https://discuss.pytorch.org/uploads/default/edf46479176940dde5b7436e81a5ad76786e8e33" title="readingDataset"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edf46479176940dde5b7436e81a5ad76786e8e33.png" alt="readingDataset" data-base62-sha1="xX2LFRlOxx8tK0VfL0QcnRJWWK7" width="196" height="500" data-dominant-color="000F05"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">readingDataset</span><span class="informations">271×689 4.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
A Very strange phenomenon I met in training machine translation

DoubtWang: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/fab996a0d0d6656e80d40cecd25a7147ccd99e40.png" alt="image" data-base62-sha1="zM0Zjmd3IcBednO6RKAFfPmvRh6" width="312" height="210"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6d2385ea0fb5cc2e29eae5388f07cd906babf11f.png" alt="image" data-base62-sha1="fzu9qXHjX5OBBV0NRsIPHuZLynJ" width="318" height="215"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f917cdd32ceed90f51e43d86280694aa7f54da1d.png" alt="image" data-base62-sha1="zxzTb69bYQ32jqj4gzqrm1zpESx" width="306" height="215"></p>
<blockquote>
<p>Namespace(batch_size=32, cuda=True, d_inner_hid=2048, d_k=64, d_model=512, d_v=64, d_word_vec=512, data=‘data/multi30k.atok.low.pt’, dropout=0.1, embs_share_weight=False, epoch=10, label_smoothing=True, log=None, max_token_seq_len=102, n_head=8, n_layers=6, n_warmup_steps=4000, no_cuda=False, proj_share_weight=True, save_mode=‘best’, save_model=‘data/trained’, src_vocab_size=28699, tgt_vocab_size=52799)<br>
cuda device count: 2<br>
[ Epoch 0 ]</p>
</blockquote>
<ul>
<li>(Training)   :  79%|▊| 7034/8883 [56:34&lt;15:09,  2.03it/s]</li>
</ul>
<p>learning rate update code:</p>
<pre><code class="lang-auto">    def _get_lr_scale(self):
        return np.min([np.power(self.n_current_steps, -0.5), np.power(self.n_warmup_steps, -1.5) * self.n_current_steps])

    def _update_learning_rate(self):
        ''' Learning rate scheduling per step '''
        self.n_current_steps += 1
        lr = self.init_lr * self._get_lr_scale()
        #
        for param_group in self._optimizer.param_groups:
            param_group['lr'] = lr
</code></pre>
<p>I  use transformer model to train machine traslation, but in first epoch i meet the above strange phenomenon/problem.</p>
<ol>
<li>Why does loss increase?<br>
2.Why does accuracy  decrease?</li>
</ol>

Abhilash_Srivastava: <p>It is not necessary that your loss should decrease for every batch within each epoch (it can go up for different batches), but it should decrease across different epochs.<br>
If your loss is not decreasing across different epochs, learning rate could be a problem</p>

DoubtWang: <p>yeah, the learning rate is a problem.<br>
I have set lr to min(self.init_lr * self._get_lr_scale(),  0.00001 ). this phenomenon disappear</p>
Unimodal output from a model trained on multimodal input without forward pass

S_Krishna: <p>Hi,</p>
<p>I have trained a model to learn correlation between (audio, image) input pairs. While the original use case is for audio-visual correlation, I also want to use the  embeddings from an intermediate layer of this trained model for unimodal similarity (e.g  image-image or audio-audio similarity). Is it possible to do this without running a forward pass and if so I would like some guidance on this.</p>
<p>My model code appears below (MMDNet). After I  train this model, I load it with the saved weights. Then I want to use it for image similarity as follows. However, this generates an error “DataParallel object has no attribute getImgEmb” even though the model has that method. I could run forward pass and get the embeddings, but the forward pass would require me to input audio. I want to know if I can extract the image embeddings from the visual subnetwork (which is what getImgEmb() is doing), without having to pass audio input.</p>
<p>thank you</p>
<p>#================code for image similarity from pretrained MMDNet=======<br>
<span class="hashtag">#load</span> pretrained model<br>
model = MMDNet(2)<br>
device = torch.device(“cuda” if use_cuda else “cpu”)<br>
num_gpus = torch.cuda.device_count()<br>
gpu_list = list(range(num_gpus))<br>
model = torch.nn.DataParallel(model, device_ids=gpu_list)<br>
model.to(device)<br>
model.load_state_dict(saved_wts)<br>
model.eval()</p>
<p><span class="hashtag">#get</span> Image embeddings, however this generates an error “DataParallel object has no attribute getImgEmb”<br>
emb1  = model.getImgEmb(src_img)<br>
emb2 = model.getImgEmb(dest_img)</p>
<p><span class="hashtag">#get</span> euclidean distance between (emb1, emb2) to compute similarity</p>
<p>#====mulimodal correlation model ===#<br>
class MMDNet(nn.Module):<br>
def <strong>init</strong>(self, n_cls):</p>
<pre><code>  super(MMDNet, self).__init__()

  self.num_class = n_cls
  
  self.vfeatures = VisNet()  #visual subnetwork
  self.afeatures = AudNet()  #audio subnetwork 

  self.av_classifier = nn.Linear(1, n_cls)          
</code></pre>
<p>def forward(self, x_v, x_a):</p>
<pre><code>  v_out = self.vfeatures(x_v)
  a_out = self.afeatures(x_a)

  #euclidean distance between embeddings
  av_dist = F.mse_loss(v_out, a_out, reduction='none').mean(1)
  out = av_dist.view(av_dist.shape[0], 1)

  out = self.av_classifier(out)
  
  return out
</code></pre>
<p>def getImgEmb(self, x_v):<br>
“”“Get visual embeddings for a given image from the visual subnetwork”""</p>
<pre><code>  v_emb = self.vfeatures(x_v)
  
  return v_emb</code></pre>

ptrblck: <p>Since you’ve wrapped the model in <code>nn.DataParallel</code>, you would have to access the methods via the <code>.module</code> attribute:</p>
<pre><code class="lang-python">emb1 = model.module.getImgEmb(src_img)
emb2 = model.module.getImgEmb(dest_img)
</code></pre>
Is grad_fn=<UnsafeViewBackward> problematic?

Yue_Zhao: <p>When implementing the attention block as used in transformers, I saw the output of the following layer has “grad_fn=&lt;UnsafeViewBackward&gt;”:</p>
<pre><code class="lang-auto">import math
import torch
import torch.nn as nn
import torch.nn.functional as F

class ScaledDotProduct(nn.Module):
    def __init__(self,dropout=0.1):
        super(ScaledDotProduct, self).__init__() 
        self.drop_out = nn.Dropout(p=dropout)
 
    def forward(self,Q,K,V,d_k,d_v):
        d_model = Q.size(2)
        W_Q = nn.Linear(d_model,d_k)
        W_K = nn.Linear(d_model,d_k)
        W_V = nn.Linear(d_model,d_v)        
        dot1 = W_Q(Q)
        dot2 = W_K(K)
        dot3 = W_V(V)
        p_attention = F.softmax(dot1.matmul(dot2.transpose(1,2))/math.sqrt(d_k),dim=2)
        attention1 =  self.drop_out(p_attention).matmul(dot3)
        #attention2 = self.drop_out(p_attention.matmul(dot3))
        return attention1

Q = torch.randn(1, 3, 5) 
K = torch.randn(1, 3, 5)
V = torch.randn(1, 3, 5)
d_k = 2
d_v = 2 
attn = ScaledDotProduct()
print (attn.forward(Q,K,V,d_k,d_v))
</code></pre>
<p>Is UnsafeViewBackward bad? It seems to come from the line</p>
<pre><code class="lang-auto">attention1 =  self.drop_out(p_attention).matmul(dot3)
</code></pre>
<p>in the forward function where the dropout layer is multiplied with the Value matrix. I also have a second closely related question regarding where the dropout comes in in the scaled dot product attention. In the paper “Attention is All You Need”, the authors say in the Residue Dropout section  that “We apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized.” My interpretation is that the dropout is applied after the product of softmax(QK)V. Yet all the implementation I have seen so far, including pytorch’s implementation on line 3367 of the <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py" rel="nofollow noopener">code</a> for multi_head_attention_forward, apply the dropout on softmax(QK), and then multiply with V. I wonder why is the dropout is applied here instead of on softmax(QK)V? When I followed the Pytorch implementation, I got grad_fn=&lt;UnsafeViewBackward&gt;, whereas when I followed my interpretation, had “grad_fn=&lt;MulBackward0&gt;”, which seems more normal.</p>

ptrblck: <p>It shouldn’t be problematic as described <a href="https://discuss.pytorch.org/t/unsafeviewbackward-in-grad-fn/25992/3">here</a>.</p>
Trying to implement a local-gram loss

Haoyu_Ma: <p>I am trying to reproduce a paper and I have a question when trying to implement the local gram loss. Global Gram loss is implemented in the online tutorial of style (transferhttps://pytorch.org/tutorials/advanced/neural_style_tutorial.html), and a local gram is implemented to every small sliding window of the image. To provide a detailed description: an nn.functional.conv2d can be considered as a weighted summation applied to every sliding window with kernel weights, and a local gram can be considered as a gram matrix loss applied to every sliding window with kernel size, and losses are then averaged among all channels and all pixels.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4dbe0737863bfc3a659679c379647e9ab0ffd55.png" alt="Untitled" data-base62-sha1="un2j47PdtiRTh0D5iqz4dAPaMwR" width="328" height="47"></p>
<p>It is definitely possible to implement by creating a nn.functional called ‘local-gram’ and derive forward and backward for it. But is it possible to assemble one with existing nn.functionals? I think it should be something similar to pandas.DataFrame.apply but didn’t find it in the documentations? Thank you.</p>
Transfer learning with NLP on different context using Pytorch

qnanswer: <p>Hi, I managed to create a model using BiDAF and Pytorch.</p>
<p>BiDAF: Bi-Directional Attention Flow<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/galsang/BiDAF-pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/6680061?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/galsang/BiDAF-pytorch" target="_blank" rel="nofollow noopener">galsang/BiDAF-pytorch</a></h3>

<p>Re-implementation of BiDAF(Bidirectional Attention Flow for Machine Comprehension, Minjoon Seo et al., ICLR 2017) on PyTorch. - galsang/BiDAF-pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The model has been trained on SQuAD: The Stanford Question Answering Dataset (<a href="https://rajpurkar.github.io/SQuAD-explorer/" rel="nofollow noopener">https://rajpurkar.github.io/SQuAD-explorer/</a>)</p>
<p>Now, I would like to do transfer learning on a different context - financial regulation documents</p>
<p>My question is can anyone provide a ‘big picture’ view on what I should to to enable transfer learning so that the model can answer questions in relation to financial documents?</p>
<p>For example, do I have to create question answers pair with regards to the financial documents manually? If yes, do I have to clean up the documents (there are hundreds of them) and what software/technique should I use to clean up the data?</p>
<p>Any links and resources to read up?</p>

Abhilash_Srivastava: <p>It might be a good idea to use a pre-trained BERT model and apply fine-tuning on top of it. The BERT model is pretrained on a large corpus (some of which are finance related), so knowledge transfer should be possible. Also, BERT has performed really well on SuperGLUE for QnA.</p>
<p>You definitely need a dataset (larger the better) for transfer learning to work. Weak supervision may be helpful to create a decent quality dataset. Using crowdsourcing is another way (will cost some money). But, even with a small manually labeled dataset you may be able to achieve decent results.</p>
Anyone working with a data pipeline of CPU -> GPU? I am developing a library of methods for faster transfer to GPU. In some cases, 370x faster than used Pytorch's Pinned CPU Tensors

Santosh-Gupta: <p>Let me know what your pipeline is and I’ll try to add methods for it. Just show me your code.</p>
<p>I am developing methods for fast transfer from CPU and GPU, and currently coding the methods for it. Show me your code (A Colab notebook would be really helpful) and I’ll see how to incorporate the library into it, for faster data transfer.</p>

juanmed: <p><a class="mention" href="/u/santosh-gupta">@Santosh-Gupta</a> Hi, thanks for your work and proposal. Actually I am very interested on tools to speed up CPU-GPU-CPU data transfers. I have written the details of my application in <a href="https://discuss.pytorch.org/t/cpu-x10-faster-than-gpu-recommendations-for-gpu-implementation-speed-up/54980">this post</a>. There you will find details on what I tried and access to the code.</p>
<p>Basically, im trying to do real-time control <span class="mention">@100hz</span> on constrained devices (Jetson Nano) using pytorch for computation of the control laws on GPU. I am sure there are several bottlenecks but one of them is the CPU-GPU and GPU-CPU data transfers.</p>
<p>I am starting to rewrite the code to use torch JIT, but I am not quite sure about the speed up improvements. I will appreciate your support on trying out SpeedTorch (of which I read about on <a href="https://discuss.pytorch.org/t/introducing-speedtorch-4x-speed-cpu-gpu-transfer-110x-gpu-cpu-transfer/56147">your post</a>) and hopefully get some nice performance increase.</p>

Santosh-Gupta: <p><a class="mention" href="/u/juanmed">@juanmed</a> it’s out</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Santosh-Gupta/SpeedTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/5524261?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Santosh-Gupta/SpeedTorch" target="_blank" rel="nofollow noopener">Santosh-Gupta/SpeedTorch</a></h3>

<p>Library for fastest pinned CPU &lt;-&gt; GPU transfer . Contribute to Santosh-Gupta/SpeedTorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How do you interpret this loss graph?

Jiho_Noh: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a2f831f5ee1c47669ef48832188d2b55f1ba2c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a2f831f5ee1c47669ef48832188d2b55f1ba2c2" title="run0910_loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a2f831f5ee1c47669ef48832188d2b55f1ba2c2_2_662x500.png" alt="run0910_loss" data-base62-sha1="f9mpbKajdTWac7aGDXfH1mPBpT4" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a2f831f5ee1c47669ef48832188d2b55f1ba2c2_2_662x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a2f831f5ee1c47669ef48832188d2b55f1ba2c2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a2f831f5ee1c47669ef48832188d2b55f1ba2c2.png 2x" data-dominant-color="F8F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">run0910_loss</span><span class="informations">817×617 38.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is a language model based on transformer architecture. It is extremely imbalanced data, but still I have no idea how to interprete this and what to do next.</p>
<p>First guess is the learning rate and scheduler. For this, I used Adam (learning rate: 0.00025 and cosine scheduler). I thought the first phase of train loss increase was due to the imbalanced data, but that should be applied same to the validation data. The suddens peaks occur at the very beginning of epochs. I can’t explain why that happens too.</p>

LeviViana: <p>Can you explain what the axes and the colors mean in the graph ?</p>

Abhilash_Srivastava: <p>The spikes in the graphs are bothering me.<br>
Are you sure, you’re using <code>optimizer.zero_grad()</code> correctly? (or <code>model.train()</code> and <code>model.eval()</code> are used at correct places).</p>
3D bounding box IOU Loss function

DerekGloudemans: <p>This topic isn’t strictly related to pytorch, but to computer vision. The 2D bounding box intersection over union (IOU) is relatively straightforward to calculate for a ground truth bounding box versus a model output, and thus translates neatly into a loss function. For 3D object detections, it would be nice to extend the IOU concept into 3D. Unfortunately, as far as I can tell there isn’t a (simple) way to calculate the intersection between two 6-sided 3-dimensional space; even 2D IOU with arbitrary quadrilaterals instead of axis-constrained rectangles is somewhat difficult.</p>
<p>I am wondering if anyone knows either: a. how the 3D bounding box IOU is calculated (I have found papers that reference it but never with a description of how it is calculated) or b. what loss functions are used instead of one based on 3D volume IOU.</p>
Advice on cuda and memory management?

Faolian: <p>I’m running an off policy rl algorithm with deepminds pysc2, and i am finding im quickly running out of gpu memory. My pc does only have 4 gig of vram, so if this is a bad plan from the start just let me know.</p>
<p>Essentially the run loop of the program goes:<br>
Actor and critic initialised on gpu</p>
<ol>
<li>observe environment</li>
<li>process observations (into cuda tensors, such as minimap_features, a 1 x 4 x 64 x 64 tensor)</li>
<li>actor.forward(processed observations)</li>
<li>store a bunch of information in the replay buffer (am storing, for example  minimap_features.cpu())</li>
</ol>
<p>the problem im having appears to be that the memory is never being deallocated, with the cuda.memory_allocated and max_memory_allocated remaining (almost) the same as eachother and constantly, linearly increasing. My guessing is that something I am doing is maintaining references to previously calculated variables which should have been cleared. Any advice? (if you want to know more about a specific part of the code i can show)</p>

ptrblck: <p>If you see an increase in memory usage, you are most likely right in maintaining some references to variables.<br>
Could you check your code for storing tensors which are not detached from the computation graph, e.g. <code>losses.append(loss)</code>? This is often the cause of increasing memory usage, as <code>loss</code> still holds the whole computation graph, if you don’t call <code>detach()</code> on it.</p>

Faolian: <p>I solved this, it turns out that my storing in the replay buffer by using .cpu() on the tensors was keeping a reference, using .cpu().data fixed this</p>

ptrblck: <p>Good to hear it’s fixed. However, the usage of the <code>.data</code> attribute is not recommended so I would use <code>.detach()</code> instead. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to do a backward pre hook

MatthieuBizien: <p>I have some custom operations that needs to be configured using environment variables. It’s easy to do that in the forward pass, but much harder to do it in the backward, because the orchestration happens in the C code.</p>
<p>Using myfunction.register_backward_hook(hook), the hook is called after the output gradient are computed, so it don’t work. I didn’t something similar to register_forward_pre_hook for the backward pass.</p>
<p>I can’t use custom torch.autograd.Function because Cudnn is not called inside it (<a href="https://github.com/pytorch/pytorch/issues/26537" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/26537</a>).</p>
<p>How can I set my environment variables in the backward pass?</p>
Pack padded sequence last hidden state shape

Lewis: <p>Hi,<br>
It is mentioned in the documentation of an LSTM, that if <code>batch_first = True</code> for <code>pack_padded_sequence</code> input to LSTM (bi-directional), the last hidden state output is also of shape <code>(batch, num_directions, hidden_size)</code>. However, the output of the last hidden state appears to be of shape <code>(num_directions, batch, hidden_size)</code>, even though <code>batch_first</code> is set to true. This is the code:</p>
<pre><code class="lang-auto">import torch
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence, PackedSequence
from torch import nn

sequences = torch.FloatTensor([[1, 2, 0, 0, 0, 0],  # length 2
                               [3, 4, 5, 0, 0, 0],  # length 3
                               [5, 6, 0, 0, 0, 0],  # length 2
                               [8, 9, 10, 11, 12, 0]
                               ])  # length 5
seq_lengths = torch.LongTensor([2, 3, 2, 5])

rnn = nn.LSTM(1, 5, 
              batch_first=True, 
              bidirectional = True)

packed_sequences = pack_padded_sequence(sequences.unsqueeze(2),
                                        lengths=seq_lengths,
                                        batch_first=True,
                                        enforce_sorted=False)  

rnn_output, (hn,cn) = rnn(packed_sequences)
print(hn.shape)   # torch.Size([2, 4, 5]). It should be torch.Size([4, 2, 5])? 
</code></pre>

Lewis: <p><a class="mention" href="/u/alband">@albanD</a> may you please help?</p>

albanD: <p>Hi,</p>
<p>Unfortunately I have no experience with the recurrent modules and the pack padded sequence mechanics.<br>
From what I see, could it be the case that this distinction <code>batch_first</code> is to compare with the sequence dimension that you can have sometimes? and that the bidirectional will always add an extra dimension of size 2 at position 0 on top of whatever you already had?</p>

Lewis: <p>Thank you for your answer. Most probably this is what happens… However, even if the <code>bidirectional</code> is set to <code>False</code>, the output would still be of shape <code>(1,4,5)</code>. Transposing the matrix to be <code>(4,1,5)</code> would solve the problem i believe. I just wanted to make sure i’m doing the right thing. Thanks a lot.</p>

albanD: <p>So after asking around, basically, hidden_state is just not influenced by the batch_first argument. So it will have this shape whatever you pass.<br>
This inconsistency is tracked in the following <a href="https://github.com/pytorch/pytorch/issues/4145" rel="nofollow noopener">issue</a>.</p>
RuntimeError: invalid gradient at index 0 - expected type torch.cuda.FloatTensor but got torch.FloatTensor

vainaijr: <p>got error on doing this</p>
<pre><code class="lang-auto">from flashtorch.utils import apply_transforms, load_image
from flashtorch.saliency import Backprop
train_dataset = torchvision.datasets.CIFAR10(root='./cifar10/',
                                           train=True,
                                           transform=transforms.Compose([
                                                                         transforms.ToTensor()
                                                                         ]),
                                           download=True)
...
backprop = Backprop(model)
backprop.visualize((train_dataset[0][0].unsqueeze(0)).to('cuda'), 6, guided=True, use_gpu=True)
</code></pre>
<p>where do I need to set to(‘cuda’)?</p>
I think tensor.item() method have some errors

Yupjun: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f590e6f7fcc3f77fec7af878d9658871ad79683a.png" data-download-href="https://discuss.pytorch.org/uploads/default/f590e6f7fcc3f77fec7af878d9658871ad79683a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f590e6f7fcc3f77fec7af878d9658871ad79683a_2_690x298.png" alt="image" data-base62-sha1="z2nqLZ1VOJ8QzJVBHw7X9Uhy7yy" width="690" height="298" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f590e6f7fcc3f77fec7af878d9658871ad79683a_2_690x298.png, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f590e6f7fcc3f77fec7af878d9658871ad79683a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f590e6f7fcc3f77fec7af878d9658871ad79683a.png 2x" data-dominant-color="F4F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">912×394 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The current status is that we want to sum convolution layer feature maps based on index<br>
to get the sum of the certain part of the feature maps.</p>
<p>I can’t understand the error situation that I posted on the image.<br>
If the fake.item() got zero, Then we should get division by zero error not “float division by zero” error.<br>
Is inferring tensor.item() on specific variable on various time is dangerous?</p>

phan_phan: <p>If <code>(fake + real)</code> is equal to zero, then why do you think the problem comes from the <code>.item()</code> method ?</p>

Yupjun: <p>Because when python divide value by zero, the name of the error is “division by zero” where as when divide float by zero, the output is “float divided by zero”<br>
for example<br>
“3/0” &gt; “division by zero” error<br>
“1.5/0” &gt; “float division by zero” error<br>
“0/0” &gt; “division by zero” error</p>

spanev: <p>Hi <a class="mention" href="/u/yupjun">@Yupjun</a><br>
Try:</p>
<pre><code class="lang-auto">fake.dtype
</code></pre>
<p>if it returns <code>torch.float32</code>, <code>fake.item()</code> will return you a Python <code>float</code> scalar (.<br>
And in Python:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; 0.0/0
ZeroDivisionError: float division by zero
</code></pre>
<p>so everything is expect here.</p>
Plotting losses and accuracy in Cycling learning rate

Mohammed_Awney: <p>How can I plot training and validation losses in cycling learning rate? if I plot  it every batch it would be noisy</p>

KFrank: <p>Hi Mohammed!</p>
<aside class="quote no-group" data-username="Mohammed_Awney" data-post="1" data-topic="56433" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohammed_awney/48/8656_2.png" class="avatar"> Mohammed_Awney:</div>
<blockquote>
<p>How can I plot training and validation losses in cycling learning rate? if I plot  it every batch it would be noisy</p>
</blockquote>
</aside>
<p>I’m not sure exactly what you’re asking, but let me make a<br>
few comments that may help address your question.</p>
<p>First, I don’t think “cycling learning rate” really has anything<br>
to do with this.  At least my comments would apply equally<br>
well to other training schedules.</p>
<p>If I understand your issue:  Averaging the loss over a small<br>
training batch produces a noisy “measurement” of the loss.<br>
This is true.</p>
<p>What I do (when I care about this) is perform a separate<br>
calculation that I don’t use for gradients (or otherwise have<br>
affect my actual training) where I average my loss over the<br>
entire training set (or at least over a subset that is significantly<br>
larger that a batch).</p>
<p>This can be expensive, so, if that’s a problem, you can perform<br>
this big loss calculation after every n batches.  For example,<br>
you could do this once every epoch.</p>
<p>As a legitimate expedient, you can store the per-batch losses<br>
and after every n steps, you can average them together, and<br>
plot your n-batch average loss.  This makes good sense, but<br>
you have to bear in mind that this is not the loss for a model<br>
with any one specific set of weights.  It is, of course, the loss<br>
for one set of weights applied to one batch, a somewhat<br>
different set of weights applied to the next batch, and so on.</p>
<p>(For the validation loss I just calculate the loss for my entire<br>
validation set.  That “measurement” still has noise, but, other<br>
than increasing the size of the validation set, you can’t do<br>
anything about it.  Also, these comments about the loss<br>
apply equally well to calculating the accuracy.)</p>
<p>Good luck.</p>
<p>K. Frank</p>
Deepcopy not showing up in model

Mgorb: <p>I’m trying to load 8 pretrained models into another model in PyTorch.  The definition of the models I want to load (loaded on top of a vgg net):</p>
<blockquote>
<pre><code>self.output_layer=nn.Sequential(nn.Conv2d(512,512, kernel_size=3,padding=2,dilation = 2),
                            nn.Conv2d(512,256, kernel_size=3,padding=2,dilation = 2),
                            nn.Conv2d(256,128, kernel_size=3,padding=2,dilation = 2),
                            nn.Conv2d(128,64, kernel_size=3,padding=2,dilation = 2),
                            nn.Conv2d(64, 1,kernel_size=1))
</code></pre>
</blockquote>
<p>I’m deep copying these into an array, then loading the weights from the pretrained models, and each gets used on the forward pass as an expert in a gating block.</p>
<pre><code>  self.branches=[]
  for i in range(8):
        self.branches.append(copy.deepcopy(self.output_layer))
</code></pre>
<p>Forward pass:</p>
<pre><code>    x = self.frontend(x)
    regressors=[]
    for i in range(8):
        regressors.append(self.branches[i](x)) 
    stack=torch.cat(regressors, dim=1)
</code></pre>
<p>…<br>
final=(gate[:,:,None,None]*stack)<br>
output=torch.sum(final, dim=1, keepdim=True)</p>
<p>These pretrained layers are not showing up in my model.  When I run torchsummary on the model, I am only seeing the vgg and gating layers.  The outlayer_layer is missing.</p>
<p>Without deepcopying the output_layer, I see all the layers and parameters in the torchsummary (10,798,408 with deepcopy vs. 42,067,280 without deepcopy.).  However, I need deepcopy to load the weights of each model and not just create a list of references to the same output_layer with the same weights.</p>
<p>In the meantime as a workaround I defined 8 different self.output_layers and added them to the branches array.  This seems to work out okay.</p>

albanD: <p>Hi,</p>
<p>You will need to use <code>nn.ModuleList()</code> for the submodules (each of your regressor) to be found properly.<br>
You can simply replace <code>self.branches=[]</code> by <code>self.branches=nn.ModuleList()</code> and it will work as expected.</p>
Numpy array to Long Tensor

zahra: <p>Hi,</p>
<p>I am reading a file includes class labels that are 0 and 1 and I want to convert it to long tensor to use <strong>CrossEntropy</strong> by the code below:</p>
<pre><code class="lang-auto">def read_labels(filename):
    lists = deque()
    with open(filename, 'r') as input_file:
        lines_cache = input_file.readlines()
        for current_line in lines_cache:
            sp = current_line.split()
            sp = map(float, sp)
            sp = list(sp)
            lists.append(sp)

        current_label_array = np.array(lists)

    return current_label_array

current_label = read_labels('./train/labels1.txt')
current_label = torch.as_tensor(current_label, dtype=torch.long, device=torch.device("cuda:0"))
</code></pre>
<p>but I receive this error. How could I do it?</p>
<pre><code class="lang-auto">TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, and uint8.

</code></pre>
<p>Thanks</p>

ptrblck: <p>Could you print the <code>dtype</code> of <code>current_label</code> before passing it to <code>torch.as_tensor</code>?<br>
It seems you are reading mixed object types, which would create this error (i.e. not all your values are represented as a numerical dtype).</p>

zahra: <p>Yes, you are right. It returns object . I checked the file. There are 2 empty lines at the end.</p>
<p>Thanks Ptrblck.</p>