akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataParallel Tutorial usage of model.to(device)

dsethz: <p>Hey,</p>
<p>I am new to torch and multi-GPU usage. I went through the <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">Tutorial</a> and I am confused by the usage of <code>model.to(device)</code> in the multi-GPU case. Removing some intermediate lines of code, we are left with something like the this:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

all_available_devices = [0,1,2,3]
model = some_NN()
model = nn.DataParallel(model, device_ids=all_available_devices)

model.to(device)
</code></pre>
<p>. Now, this last line of code is confusing me. Maybe my understanding of <code>model.to()</code> is not correct. In the single-GPU case <code>model.to(device)</code> allocates the model, gradients, and feature maps (in case of a CNN) to our <code>device</code>, correct? Now what does this line do in the multi-GPU case, where <code>model</code> is saved on <code>all_available_devices</code>?</p>
<p>In addition, in the single-GPU case I allocate the e.g. training data on a specific device (i.e. <code>data.to(device)</code>). Can this impede the data flow that is handled by <code>DataParallel()</code> behind the curtains?</p>
<p>Cheers</p>

PistonY: <p><code>nn.DataParallel</code> will “copy” model to multi-gpu automatically, <code>.to(device)</code> will load model to <code>main</code> device.</p>

dsethz: <p>Thank you, and just to be clear: <code>device</code> from the above example would be <code>GPU-1</code> in the graphic of <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="noopener nofollow ugc">this post</a>?</p>
<p>And about the 2nd question: Following the graphic of the same post, is it also the fastest approach to allocate the data on the same device as the model?</p>

PistonY: <p>It’s just a way to use mulit-gpu, nothing about <code>fast</code> or not.</p>
Batching for stateful lstm

mizunt: <p>I have one large file that has one measurement each second for 100,000 seconds. Each measurement captures 3 features.<br>
I am going to create an LSTM that will look at 30 seconds in one go. i.e.<br>
seq_len = 30, input_size = 3.<br>
As I have a single file with continuous data measurements being taken for 100,000 seconds, after passing in the first set of data points in to this lstm of len 30, I am going to pass on the hidden state when examining time points 31 to 60.<br>
If I wanted to batch this process, to look at two different 30 second windows at the same time, is there a DataLoader that I can use to ensure that between each seq_len, the data is correlated, but between different batches, the data is independent?<br>
so my input size will be<br>
(seq_len=30, batch=2, input_size=3)<br>
two hidden states will be passed on to the next two batches, and the next two set of seq_lens are from time points that follow the first two sets of seq_lens.</p>
How to different image size prediction fix?

themcanv: <p>Hello,</p>
<p>I trained <strong>mask rcnn resnet50</strong> to my model. During the training, the input images are 512.1024 in size.<br>
When I give an input picture of this size, the results are very good. However, the input image 4096x8192 cannot find any guesses.</p>
<p>What can I do.</p>

ptrblck: <p>You could reshape the new images to the trained shape. These larger images would be new to the model and it might thus fail, as e.g. the feature extraction might not be able to find any useful features if the resolution changes “a lot”. Note that the decrease in performance might not be that drastic if you slightly change the resolution.</p>

themcanv: <p>Thank you for answer.</p>
<p>Before start traning i have set anchor sizes and aspect ratio for example:</p>
<p>(<em>I didn’t give it according to a rule, I just wanted it to learn in different scales.</em>)</p>
<pre><code class="lang-auto"> anchor_sizes : ((32,), (64,), (128,), (256,), (512,))
 aspect_ratio : ((0.15, 0.2, 0.25, 0.5, 1.0, 2.0, 2.5, 3.0, 4.0, 5.0 ,6.0, 8.0),)
</code></pre>
<p>I thought it might work but it didn’t yield any results</p>
<p><strong>Besides, These anchor settings how i can make according to my datasets.</strong></p>
<p>My Dataset information,</p>
<p>images 512x1024 and generally labeled objects large 30x120, 45x150 etc</p>
<p><strong>Critic</strong>,<br>
I don’t want to give resize images or tile images while prediction because i am missing object or can’t merging detected objects in tiles image</p>
Square_subsequent_mask BoolTensor alternative

gautierdag: <p>Hi,<br>
I’ve been implementing a transformer model but came across the function <code>generate_square_subsequent_mask bool</code> in both the PyTorch library and the Sequence-to-Sequence tutorial.</p>
<p>The current implementation generates a square mask matrix as follows:</p>
<pre><code class="lang-auto">def generate_square_subsequent_mask(self, sz: int) -&gt; Tensor:
        """Generate a square mask for the sequence. The masked positions are filled with float('-inf').
            Unmasked positions are filled with float(0.0).
        """
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask
</code></pre>
<p>For sz=2 this would give a matrix = [[0.0, -inf], [0.0, 0.0]]</p>
<p>I understand what it does but would like to understand if there was any reason not to do the following instead:</p>
<pre><code class="lang-auto">def generate_square_subsequent_mask(self, sz: int) -&gt; Tensor:
        """Generate a square mask for the sequence. The masked positions are filled with True.
            Unmasked positions are filled with False.
        """
        mask = (torch.triu(torch.ones(sz, sz)) == 0).transpose(0, 1)
        return mask
</code></pre>
<p>For sz=2 this would give a matrix = [[False, True], [False, False]]</p>
<p>According to the nn.Transformer documentation, this should be equivalent since <code>If a BoolTensor is provided, positions with True are not allowed to attend while False values will be unchanged.</code></p>
<p>I’ve confirmed it on the tutorial and obtain the same results with both.</p>
<p>Using the BoolTensor is simpler and would be slightly faster at the mask creation, is there an underlying reason to not recommend it or to prefer the float mask over the other?</p>
<p>Thanks!</p>
Can't import torch

shikharp: <p>I’m Running 64 Bit Windows With CUDA 9.2 support , with Conda as my preferred package manager.<br>
Used ’ conda install pytorch -c pytorch’ and managed to install Pytorch 0.4.1 .<br>
When i try to import torch , I get the “module not found error” , I can’t install Torchvision either<br>
<a>IMG_20180809_224122|666x500</a></p>

ptrblck: <p>Could you try to create a new and clean conda environment and reinstall pytorch and torchvision there?<br>
You will find more information on managing conda environments <a href="https://conda.io/docs/user-guide/tasks/manage-environments.html">here</a>.</p>

shikharp: <p>I created a new Conda environment and installed Pytorch , I still can’t import torch or install Torchvision</p>
<p><a>15339895805591337528185|374x500</a></p>

ptrblck: <p>Do you get any error messages?<br>
Could you paste the commands and output as code with three backticks ```?<br>
Your images don’t seem to work.</p>

shikharp: <blockquote>
<p>conda create --name pytorch</p>
</blockquote>
<blockquote>
<p>activate pytorch</p>
</blockquote>
<blockquote>
<p>conda install pytorch cuda92 -c pytorch</p>
</blockquote>
<p>After Pytorch and all the necessary packages were downloaded and the transaction was succesfully executed,<br>
i did the following:</p>
<blockquote>
<p>ipython</p>
</blockquote>
<blockquote>
<p>import torch</p>
</blockquote>
<pre data-code-wrap="ModuleNotFoundError"><code class="lang-ModuleNotFoundError">&lt;ipython-input-1-eb42ca6e4af3&gt; in &lt;module&gt;()
----&gt; 1 import torch
ModuleNotFoundError: No module named 'torch'</code></pre>

ptrblck: <p>Did you get any output after the conda install?<br>
Could you try using <code>python</code> instead of <code>ipython</code>?</p>
<p>If you still get import errors, could you try to install the CPU version:</p>
<pre><code class="lang-auto">conda install pytorch-cpu -c pytorch 
</code></pre>

shikharp: <p>using python instead of ipython worked<br>
installed torchvision as well!!<br>
Thank You!</p>

lingvisa: <p>Hey, I am having the same issue. Yes, python works, but I prefer iPython terminal. Why can’t iPython find the torch module?</p>

lingvisa: <p>Reinstalled fixed the problem. Thanks.</p>

NA_GONG: <p>Works for me as well. Thanks!</p>

Alexey_Demyanchuk: <aside class="quote no-group" data-username="shikharp" data-post="7" data-topic="22814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/7bcc69/48.png" class="avatar"> shikharp:</div>
<blockquote>
<p>using python instead of ipython worked</p>
</blockquote>
</aside>
<p>A little side note for those who are interested why. If conda <code>custom</code> virtualenv doesn’t have ipython package installed, when you fire up ipython from within your <code>custom</code> env it will silently use ipython from conda <code>base</code> environment and <code>base</code> python runtime for this instance of ipython. So, if your <code>base</code> doesn’t have pytorch installed the import error occured.</p>
Why does KL vanishing occur?

Sine: <p>Hi, I am trying to implement the CVAE-based dialog model described in Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders.</p>
<p>I am confused that the different computation methods of the reconstruction error affect the KL divergence.</p>
<p>When I use the nll_loss function to obtain the mean of reconstruction error, the KL divergence gradually becomes 0, which causes the KL vanishing problem. The code is as below.</p>
<pre><code class="lang-auto">avg_recon_loss = F.nll_loss(outputs.contiguous( ).view(-1, outputs.size(-1)), targets.reshape(-1), ignore_index=self.pad_id)
</code></pre>
<p>The value of  avg_recon_loss is between 4 and 2.</p>
<p>However, I borrow the method from <a href="https://github.com/ruotianluo/NeuralDialog-CVAE-pytorch" rel="noopener nofollow ugc">NeuralDialog-CVAE-pytorch</a>, which calculates the sum of the reconstruction error. As a result, the KL reaches a non-zero lower bound, such as 28.<br>
The code is as bellow:</p>
<pre><code class="lang-auto">recontruction_loss = F.nll_loss(outputs.contiguous().view(-1, outputs.size(-1)), targets.reshape(-1), reduction='none')
recontruction_loss = recontruction_loss.view(outputs.size()[:-1])
label_mask = (targets != self.pad_id).float()
recontruction_loss = torch.sum(recontruction_loss*label_mask, 1)
avg_recon_loss = torch.mean(recontruction_loss)
</code></pre>
<p>The value of  avg_recon_loss is between  77 and 20.</p>
<p>Thanks.</p>
Different output between pytorch and caffe2

XiaXuehai: <p>In this page,<a href="https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html#running-the-model-on-mobile-devices" rel="nofollow noopener">TRANSFERING A MODEL FROM PYTORCH TO CAFFE2 AND MOBILE USING ONNX</a>, the output is different.</p>
<pre><code class="lang-auto"># Verify the numerical correctness upto 3 decimal places
np.testing.assert_almost_equal(torch_out.data.cpu().numpy(), c2_out, decimal=3)

print("Exported model has been executed on Caffe2 backend, and the result looks good!")
</code></pre>
<p>I just copy the code, and the output is :</p>
<pre><code class="lang-auto">(mismatch 99.88728564342404%)
 x: array([ 0.24 ,  0.332,  0.502, ...,  0.025, -0.106, -0.055], dtype=float32)
 y: array([ 0.24 ,  0.665,  1.075, ...,  0.087,  0.369, -0.055], dtype=float32)
</code></pre>

ChamSu: <p>Hi <a class="mention" href="/u/xiaxuehai">@XiaXuehai</a>, I run into the same issue, any suggestions?</p>
PyTorch's DataParallel is only using one GPU

seankala: <p>Hi. This question seems to have been asked a lot but I’m still facing some trouble. I’m trying to use two GPU’s using <code>torch.nn.DataParallel</code> but when I wrap my model <code>nvidia-smi</code> says I’m only using one.</p>
<p>The code I have looks something like:</p>
<pre><code class="lang-auto">import torch.nn as nn


model = SomeModel()

if args.multiple_gpu: # Boolean
    os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
    model = nn.DataParallel(model)

model = model.to('cuda')
</code></pre>
<p><code>nvidia-smi</code> says I’m only using GPU 0 and not 1. I feel like I’m overlooking something in plain sight but I can’t grasp what. Any opinions are appreciated, thanks.</p>

seungjun: <p>You should specify device ids.</p>
<pre><code class="lang-python">model = nn.DataParallel(model, device_ids=[0, 1])
</code></pre>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html?highlight=dataparallel#torch.nn.DataParallel" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html?highlight=dataparallel#torch.nn.DataParallel" target="_blank" rel="noopener nofollow ugc">DataParallel — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


seankala: <p>I’ve tried that as well, and by default <code>device_ids</code> is set as all available devices.</p>

seungjun: <p>I tried the below snippet which is similar to yours but it works with multi-gpu on my device.</p>
<pre><code class="lang-python">import torch.nn as nn

model = nn.Conv2d(3,3,1,1)
model = nn.DataParallel(model).cuda()

input = torch.randn(64,3,8,8).cuda()
output = model(input)
</code></pre>
<p>I’m not sure what the difference between mine and yours could be.</p>