akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to determine the proper number of hidden layers, and

morphism: <p>Hello, everyone</p>
<p>I am doing project whose data has several hundred variables (many of them are categorical)<br>
and the model is binary classification</p>
<p>I am using deep learning with Pytorch</p>
<p>In this case, I want to know</p>
<ol>
<li>
<p>how many hidden layers should I use?</p>
</li>
<li>
<p>how many nodes should I use for each hidden layer?</p>
</li>
</ol>
<p>Is there any general theory or generally accepted methods to do these? I really want to know.</p>
<p>Thank you in advance and have a nice day!!!</p>
State_dict for weight norm Conv2d doesn't give weight_g and weight_v keys

tr3sleches: <p>Hi, I’m trying to recreate a decoder with pretrained weights and I’m using a Conv2d layer with weight_norm. In the state_dict for the layer I created, there is only a single “weight” key when I’m expecting there to be a “weight_v” and “weight_g” as in the original model. Any ideas what could be causing this and how I can get the state_dict to take the “weight_v” and “weight_g” values. I need it to load pretrained weights onto the model I’m creating</p>

crcrpar: <p>As you know, basically weight-normalized modules have <code>weight_g</code> and <code>weight_v</code> unless <code>nn.utils.remove_weight_norm</code> is applied to those modules, as follows.</p>
<pre><code class="lang-auto">In [5]: wn_linear = weight_norm(nn.Linear(2, 2, bias=False))

In [6]: wn_linear.state_dict()
Out[6]:
OrderedDict([('weight_g',
              tensor([[0.3002],
                      [0.3428]])),
             ('weight_v',
              tensor([[-0.2730, -0.1248],
                      [-0.3416, -0.0280]]))])

In [7]: nn.utils.remove_weight_norm(wn_linear).state_dict()
Out[7]:
OrderedDict([('weight',
              tensor([[-0.2730, -0.1248],
                      [-0.3416, -0.0280]]))])
</code></pre>
<p>So, I’m not quite confident but one trick in my mind is, register weight normalization after <code>load_state_dict</code> as follows:</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/crcrpar/d3493b56364df1ddf413e25e0013f68e" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/crcrpar/d3493b56364df1ddf413e25e0013f68e" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/crcrpar/d3493b56364df1ddf413e25e0013f68e</a></h4>
<h5>state-dict-for-weight-norm-conv2d-doesnt-give-weight-g-and-weight-v-keys.ipynb</h5>
<pre><code class="Jupyter Notebook">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "correct-hebrew",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://discuss.pytorch.org/t/state-dict-for-weight-norm-conv2d-doesnt-give-weight-g-and-weight-v-keys/113876\n",</code></pre>
This file has been truncated. <a href="https://gist.github.com/crcrpar/d3493b56364df1ddf413e25e0013f68e" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Network output with 100% probability on one class

FilipBd: <p>Quite new to PyTorch and DL, but trying to do an MLP with 256 classes (possible values for one byte). The desired output is a tensor with length 256 with probabilities for each class, so the sum of the output tensor should be 1 (right?), using Softmax. In our model, we have softmax on the output layer activation function and for loss function we are using nn.NLLLoss()(torch.log(y), y_target).</p>
<p>During the training, the output of the network have values between 0-1 on each class but when testing the network after a completed training, the output is 1 (100% prob) for one class. It is not the same class that get 1 as output for different input, but each time the network seems to be really sure on one class. It only gets the correct class on less then 1% of the time.</p>
<p>This seems a bit strange, it should not be like this right? Do you have any idea on what is wrong here?</p>

ptrblck: <p>Besides the issue of the “overconfident” model predicting one class (I don’t know what’s causing this), I would recommend to use <code>F.log_softmax</code> instead of <code>torch.log(torch.softmax())</code> for better numerical stability.<br>
Alternatively, you could also pass the outputs of the last layer (logits) directly to <code>nn.CrossEntropyLoss</code>.</p>
Using two loss functions and taking an averaging them causes an error to come

Harshit_Pandey: <p>The following error arises when I add two loss functions together. On removing one of the loss functions it seems to work fine.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d41706e9644e73fdddd0ed08ea323b399127745.png" data-download-href="https://discuss.pytorch.org/uploads/default/2d41706e9644e73fdddd0ed08ea323b399127745" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d41706e9644e73fdddd0ed08ea323b399127745_2_690x368.png" alt="image" data-base62-sha1="6slGIrRbTjvLSuZFCJ43d9OG6YR" width="690" height="368" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d41706e9644e73fdddd0ed08ea323b399127745_2_690x368.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d41706e9644e73fdddd0ed08ea323b399127745_2_1035x552.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d41706e9644e73fdddd0ed08ea323b399127745_2_1380x736.png 2x" data-dominant-color="30312D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1619×865 62.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/error-pytorch-bigan/113626/4">This post</a> explains the error.<br>
The generator was already updated, while you are trying to calculate the gradients again via <code>d_loss.backward()</code> using stale forward activations.<br>
Detach <code>generated_sample</code> before passing it to the discriminator and it should work.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
PyTorch for audio data like speech enhancement and noise cancellation

Superklez: <p>Hello! So I’m planning on working on a project that deals with speech enhancement and noise cancellation and I just had a few questions in mind:</p>
<ol>
<li>What are the recommended resources that I may read related to this project?</li>
<li>Are RNNs used when dealing with audio data?</li>
<li>What are the common loss functions when dealing with audio data?</li>
</ol>
Pytorch 1.7.0 new torchvision API questions

rafale77: <p>I just upgraded to pytorch 1.7.0 with cuda11.1 and cudnn8.0.5.</p>
<p>I was reading the documentation on the <a href="http://torchvision.io" rel="noopener nofollow ugc">torchvision.io</a> fine grained API and got very interested but have a few questions.</p>
<ol>
<li>Looking at the <code>torchvision.</code> <code>set_video_backend</code> (<em>backend</em> ) function and the options. Is there a way to determine whether the ffmpeg backend will use GPU or CPU? I am looking at using the nvidia GPU based video decoding and it was not clear how to do this.</li>
<li>In this documentation: <a href="https://pytorch.org/docs/stable/torchvision/io.html#video" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/io.html#video</a><br>
Can the video source be an rtsp stream (uri) instead of a file name?</li>
<li>What format would the frame extracted from the video be? I suppose it would be a tensor?</li>
</ol>
<p>My current deployment makes use of video inference and I am using openCV’s video API to create multiple video streams from various cameras, extract a frame (as an opencv formated numpy array), convert to a tensor and then forward into models. I am looking to see if I can replace the opencv API with torchvision and eliminate a few conversion steps… basically extract the frame and keep it in the GPU memory as a tensor and infer directly from it instead of doing video<br>
stream GPU decoding(opencv ffmpeg) -&gt; frame extraction to CPU (opencv) -&gt; numpy array (pytorch) -&gt; CPU conversion to tensor (pytorch)-&gt; GPU inference (pytorch).</p>
<p>Ideally:<br>
Video stream GPU decoding (torchvision) -&gt; frame extraction in GPU (torchvision) -&gt; GPU inference (pytorch)</p>
<p>Is this possible?</p>

Felix_Lessange: <p>I’m interested in exactly the same thing.</p>
<p>However, if this is possible, and you’re in a multiprocessed environment, I’m afraid you’re gonna have big spikes of GPU memory (every process has to load at least 1GB of cuda kernels to use torch GPU, and that’s not even talking about the space to store the frames you’re gonna be manipulating). Ignore if you’re not though.</p>
<p>Anyways, I’m currently looking at this <a href="https://github.com/osai-ai/tensor-stream" class="inline-onebox" rel="noopener nofollow ugc">GitHub - osai-ai/tensor-stream: A library for real-time video stream decoding to CUDA memory</a>. I haven’t tested it yet. If it’s what you’re looking for, please share your results. Or if you found another way, please keep me posted.</p>
<p>PS : I suggest you could change the name of the topic to be more on point with what you are describing. What about “Video loading and preprocessing fully on GPU” or something like that ?</p>

rafale77: <p>Thanks Felix,</p>
<p>I did explore the option of using tensor-stream as well but ended up giving up on the idea as it doesn’t appear to be all that efficient GPU memory wise. The saving made from not moving the frames from GPU to system RAM back and forth actually would be loss in some of the image manipulations I am having to make to them which, it seems are faster on the CPU and more memory consuming.</p>
Pytorch save, is saving more than asked for. Bug?

tueboesen: <p>I’m currently preprocessing a lot of data and I noticed that some of my data files ended up being really really big, which surprised me.<br>
I then realized that when I take a tensor and only select a slice of it, as done in the code below and then save that slice, it does not only save that slice, but actually still saves the entire tensor! Though when I load the tensor it only shows me the expected shape.</p>
<p>The reason why I noticed it was because some of my files ended up being 10GB large, even though they only contain a single integer tensor, with a dimension of 30000 x 326, which should only take up around 78 MB as int64 or around 10MB as int8.</p>
<p>I’m assuming a copy of the part of the tensor can fix this. But is this a known problem or what is the normal workaround for this?</p>
<pre><code class="lang-auto">for i, a2mfile in enumerate(a2mfiles):
    t1 = time.time()
    msas = read_a2m_gz_file(a2mfile, AA_LIST=AA_LIST, unk_idx=unk_idx, max_seq_len=max_seq_len, min_seq_len=min_seq_len,
                            verbose=True)
    if msas is None:
        continue
    else:
        seq = msas[-1,:]
        msas = msas[:-1,:].to(dtype=torch.int8)
        n_msa = msas.shape[0]
        seq_len = msas.shape[1]
        print("{:}/{:} MSAs = {:}, sequence length {:}, time taken: {:2.2f}s,ETA: {:2.2f}h".format(i+1,nfiles,n_msa, seq_len, time.time() - t1,(nfiles-(i+1))*(time.time()-t0)/(i+1)/3600))

        if n_msa &gt; max_samples:
            indices = torch.randperm(n_msa)
            msas_s = msas[indices[:max_samples],:]
        else:
            msas_s = msas

        data = {'seq':seq,
                'msas': msas_s,
                'seq_len': seq_len,
                'n_msas_org': n_msa
                }

        filename = Path(Path(a2mfile).stem).stem

        torch.save(data, "{:}{:}.pt".format(folder_out,filename))
</code></pre>
<p>Here is a minimal example showcasing it more clearly:</p>
<pre><code class="lang-auto">import torch

n = 1000000
l = 300

data = torch.ones((n,l),dtype=torch.int64)
data_small = data[:30000,:] #.to(dtype=torch.int8)
torch.save(data_small,'test')
data_small2 = torch.ones_like(data_small)
torch.save(data_small2,'test2')
</code></pre>
<p>When I run this minimal example, test will have a size of 2.4 GB, while test2 takes 72 MB.</p>

ptrblck: <p>Are you able to reproduce this behavior in the latest <code>1.8.0</code> release?<br>
If so, could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we could track and fix it, please?</p>

tueboesen: <p>Just installed 1.8 now, and yeah the problem still exist there. So I will open an issue on github.</p>
Broadcasting list/tuple of tensors to all processes

KickItLikeShika: <p>Broadcasting one tensor to all processes in PyTorch is straightforward. But i got a tuple/list of tensor, How to broadcast them?</p>
<ul>
<li>
<p>broadcasting the tuple/list fails</p>
</li>
<li>
<p>looping through the tuple/list and broadcast one at a time works fine for the first tensor, then it fails.</p>
</li>
</ul>
<p>Any idea how to broadcast a tuple/list of tensor in PyTorch?</p>

agolynski: <p>Would torch.distributed.broadcast_object_list work for you?</p>
Debugging Noisy Validation Loss

Niranjan_Rao: <p>I am training a multiclass image segmentation model using <strong>U-Net</strong> and <strong>R2U-Net</strong> architectures from <a href="https://github.com/LeeJunHyun/Image_Segmentation/blob/master/network.py" class="inline-onebox" rel="noopener nofollow ugc">Image_Segmentation/network.py at master · LeeJunHyun/Image_Segmentation · GitHub</a></p>
<p>Using Unet I get very good convergence of Validation and Training <strong>Cross Entropy Loss</strong></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07208c20bcd177b61d728c846d9db983f79ea0a3.png" alt="image" data-base62-sha1="1134DedKmQGSacecbkHbWUEWO8X" width="496" height="320"></p>
<p>Whereas R2-UNet shows an erroneous validation loss whereas training loss converges nicely.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17fded01a3cdac671ba0aee6a664d1bc652dcaa6.png" alt="image" data-base62-sha1="3qf0X0CEbxgQBQroDI3R5ZeNSdM" width="423" height="246"></p>
<p>I am training both the models using the same batch size, Learning Rate, epochs. Is this a case of overfitting? How can I overcome this, will regularization/LR scheduling/ SGD/ Gradient Clipping help in convergence.<br>
On a more general note how should I debug such results, is it just by trying out different things and finding out which gives the best result? Would like to know if there is mathematically intuitive approach to debugging.<br>
Thank you in advance!</p>

ayalaa2: <p>I would agree that your R2-UNet model is showing signs of overfitting. We can note that your model is training well considering that your training loss is consistently going down.</p>
<p>The easiest option is to simply save your model after every epoch and deploy early stopping using your validation loss. You can see that around epoch 7 your validation loss is at its best. You can use your model and that point and it’s possible that it performs well enough for your application.</p>
<p>Otherwise you will need to deploy methods to prevent overfitting. There’s many ways to go about this, you could add dropout to your model, you could make your model less complex, you could give it more data to train with, etc. It’s mainly going to be experimenting and see what works best for your model.</p>

Niranjan_Rao: <p>Thank you Alex for your reply!<br>
After some inspection I realized that the R2U-Net has around 6 million parameters compared to vanilla UNet which has 1 million parameters. So in a mathematical sense a more complex model results in a curve which fits the training data points perfectly and does not generalize well to unseen data.</p>
<p>Early stopping would imply I am settling for an inferior model performance. If I want to improve the model performance I would have to opt for regularization and more data. Is this conclusion right?</p>

ayalaa2: <p>The main conclusion would that we would need to combat overfitting. Regularization and more data is definitely a way to go about it, but there’s a handful of techniques out there. But I agree that those would be good places to start.</p>
Building quick and dirty bindings to cuDNN benchmark cache

belval: <p>I am trying to write an extension to get/set the cuDNN cache in <a href="https://github.com/pytorch/pytorch/blob/42e098323011eb8464b6238ffb011d0bb1b9ac2c/aten/src/ATen/native/cudnn/Conv_v7.cpp#L120" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Conv_v7.cpp at 42e098323011eb8464b6238ffb011d0bb1b9ac2c · pytorch/pytorch · GitHub</a>. The idea being that for a homogenous fleet we can save the benchmarking time by simply setting the cache to settings that are known to be optimal (same GPU, same system).</p>
<p>To do so I need to have access to the cache which what I am trying to patch manually into PyTorch. At first I tried to simply add a header file to expose the ATen/native/cudnn/Conv_v7.cpp file, but it seems like nothing from the cuDNN directory makes it to the final include directory. Why?</p>
<p>To keep the changes to a minimum in PyTorch itself, I am trying to use a cpp_extension to call my newly-created header file. Which so far seems to work to access anything in ATen/native, but not in the subdirectories.</p>
<p>/context, below is the TLDR:</p>
<ul>
<li>How to export the header files in ATen/native/cudnn when building PyTorch from source?</li>
<li>Is there a better way to get/set the BenchmarkCache that I haven’t found yet.</li>
</ul>
<p>Thanks</p>
Validation loss not following training loss on same sample

ball: <p>I am trying to overfit my model on 2 samples.<br>
I run the training until training loss is 0 and then I run validation on the same 2 samples.<br>
But the validation loss is higher, as can be seen here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccad4a64a27f00e75ddcf542159f268ccad92514.png" data-download-href="https://discuss.pytorch.org/uploads/default/ccad4a64a27f00e75ddcf542159f268ccad92514" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccad4a64a27f00e75ddcf542159f268ccad92514_2_690x174.png" alt="image" data-base62-sha1="tcEFZNcFfQxvk0Zo4AjGIAZOnZO" width="690" height="174" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccad4a64a27f00e75ddcf542159f268ccad92514_2_690x174.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccad4a64a27f00e75ddcf542159f268ccad92514_2_1035x261.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccad4a64a27f00e75ddcf542159f268ccad92514_2_1380x348.png 2x" data-dominant-color="FDFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1700×429 20.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why is the validation loss not 0 as well and why is it going in the opposite direction ?</p>

crcrpar: <p>Is the output used for validation loss calculation the same as the one for training loss?</p>
<p>I ask this because I wonder whether the model’s behavior is up to the value of <code>model.training</code>.</p>

ball: <p>Yes it’s exactly the same, and yes I do call model.eval() for validation.</p>

crcrpar: <p>It sounds weird. Could you share a minimal reproducible code?</p>
2 rewards for RL (DQN/DDPG/TD3)

vdg: <p>If I have two objectives (E.G. Maximize state x, have minimal control action u) and I define a reward for both, something like:<br>
<code>Reward x  = delta x</code><br>
<code>Reward u = -u^2</code><br>
I can train the network on the reward <code>delta x - u^2</code>. Would it also be possible to define the two rewards separately and then use a (weighted) sum as my intuition tells me this gives the network more “insight” in the dynamics/reward structure and it could come to a policy which maximizes x while keeping a low u.<br>
However I didn’t find any papers discussing multidimensional rewards.</p>
<p>Anyone have an idea if my proposed reward structure would be beneficial?</p>
C++ API for broadcasting on certain axes

josevalim: <p>Hi! <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"></p>
<p>We are writing libtorch bindings <a href="https://github.com/elixir-nx/nx" rel="noopener nofollow ugc">for Elixir’s Nx</a>. The initial design was built on top of XLA and one of the operations XLA provides is <a href="https://www.tensorflow.org/xla/operation_semantics#broadcastindim" rel="noopener nofollow ugc">BroadcastInDim</a>, which accepts an explicit mapping of the axes we want to broadcast.</p>
<p>Here is an example:</p>
<pre><code class="lang-auto">    iex&gt; t = Nx.tensor([1, 2, 3])
    iex&gt; Nx.broadcast(t, {2, 3, 2}, axes: [1])
    #Nx.Tensor&lt;
      s64[2][3][2]
      [
        [
          [1, 1],
          [2, 2],
          [3, 3]
        ],
        [
          [1, 1],
          [2, 2],
          [3, 3] 
        ]
      ]
    &gt;
</code></pre>
<p>In the example above, we are trying to broadcast a tensor of shape (3) to shape (2, 3, 2). Using the default broadcast semantics, which follows Numpy, this would not be possible, because the lower dimensions do not match (2 != 3). However, I can pass the axes option, that maps each axis of the broadcasting tensor to the shape. In this case, I am mapping the only axis of the tensor (axis 0) to the axis 1 of the shape, allowing the tensor to be broadcast to (2, 3, 2).</p>
<p>The custom broadcasting rules come in hand on some of the autograd operations. We can mimic the behaviour using broadcast+transpose but I wonder if there is a preferred way to achieve this (and if we could be missing some optimizations by going the broadcast+transpose route).</p>
<p>Thank you!</p>

gchanan: <aside class="quote no-group" data-username="josevalim" data-post="1" data-topic="113633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/josevalim/48/35343_2.png" class="avatar"> josevalim:</div>
<blockquote>
<p>The custom broadcasting rules come in hand on some of the autograd operations. We can mimic the behaviour using broadcast+transpose but I wonder if there is a preferred way to achieve this (and if we could be missing some optimizations by going the broadcast+transpose route).</p>
</blockquote>
</aside>
<p>The only cost I can think of has to do with the overhead of the extra operations (e.g. transpose [or unsqueeze], plus normal broadcast, plus transpose back)</p>
Relationship between output and hidden in bidirectional GRU

cerebralnetworkdev: <p>From what I understand, when we run a GRU, e.g.</p>
<pre><code class="lang-auto">embed = nn.Embedding(vocab_size, embed_size)
gru = nn.GRU(embed_size, hidden_size, batch_first = True, bidirectional = True)

# src is a bsz x seqlen x vocab_size tensor
embedded_src = embed(src)

# embedded_src is a bsz x seqlen x embed_size tensor
out, hidden = gru(embedded_src) 

# out is a tensor of seqlen x bsz x hidden_size * n_directions
# hidden is a tensor of n_directions * n_layers x seqlen x hidden_size
</code></pre>
<p>I’ve seen people concatenate the hidden state, but is it a misunderstanding to assume that <code>out</code> and <code>hidden</code> should be identical in any way? e.g. for Seq2Seq models I’ve seen this being done commonly to represent the hidden state:</p>
<pre><code class="lang-auto"># this operation doesn't matter if it's batch_first?
# returns a tensor of seqlen x n_directions * hidden_dim
context = torch.cat((hidden[-2, :, :], hidden[-1, :, :]), dim = 1)
</code></pre>
<p>My understanding from the documentation was that</p>
<pre><code class="lang-auto"># in batch_first case for 0th example
# o[0][-1] == context
</code></pre>
<p>It does seem to be the case for the forward part of the GRU output but not the reverse? Have I misunderstood something? Thanks!</p>
Train from multiple csv files each containing many samples

matthewleigh: <p>Hello,</p>
<p>I am trying to setup up my dataloader to sample from multiple csv files within a directory, each of which contains a variable number of samples.</p>
<ul>
<li>Each sample is a row in one of the csv files</li>
<li>Each file is too big to load in as a single batch (~50k)</li>
<li>Each file contains a different number of samples (between 30k-60k)</li>
<li>There are several thousand csv files in the folder</li>
<li>The entire training set is to large too hold in memory (around 200M samples)</li>
</ul>
<p>I have looked at some of the examples on this forum and they include using torchvision.datafolder, however I think that assumes that each csv file contains a single sample, which does not apply to my situation.</p>
[torch.distributions] JIT support?

tjoseph: <p>Hi,<br>
does anyone know whether there is some information somewhere about whether/when torch.Distributions will receive jit support?</p>
<p>Best<br>
Tim</p>

Yanan_Cao: <p>As far as I know, you can use <code>torch.jit.trace</code> on <code>torch.Distributions</code>. And there are tests covering this use case. <code>torch.jit.script</code>, however, doesn’t fully work with <code>torch.Distributions</code> yet. Hope this helps.</p>

googlebot: <p>I can’t directly answer that, but from viewing sources, I would say a complete redesign/rewrite would be needed for TorchScript conformance, unfortunately.</p>
Request of mentorship

zetyquickly: <p>Hello there,</p>
<p>It’s no secret that PyTorch is the one of the most powerful and widely-used backbone of the lots of RnDs all over the world. For me it is also clear that some developers (and I am not an exception) has the time to invest in self-development and career boosting. One may dive into in kaggle-like competitions, writing their own frameworks or pet projects, etc. But as for me I’d love to contribute to PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’ve made a successful PR earlier and it contains improvements necessary for the one of my work projects, in some sense it was obvious what to do. Now I have an assumption on how to move forward with contributing: for example to find an open issue and try to resolve it, but that might not be an optimal way.</p>
<p>So I have an unusual question for this forum, could anyone mentor me in this path?</p>

ptrblck: <p>Congrats on your first PR! <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=9" title=":tada:" class="emoji" alt=":tada:"></p>
<p>I would recommend to check some issue from <a href="https://github.com/pytorch/pytorch/projects/5">here</a>, especially in the “usability/simple-fixes” category. However, looking at these issues, it seems these projects haven’t been updated in a while. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>In that case, in which general area of PyTorch would you like to contribute to (frontend, backend, linalg, CUDA, usability etc.)?</p>

zetyquickly: <p>Thanks for your advice on  “usability/simple-fixes” issues, that might be very good place to start.</p>
<p>Answering your question, I’d say that I am interested in low level things like backend, linalg and CUDA. But I have more experience in Python than in cpp so it may be difficult to work on that</p>

ptrblck: <p>Since the previous link is a bit outdated, have a look at the <a href="https://github.com/pytorch/pytorch/labels/enhancement">enhancement</a> and <a href="https://github.com/pytorch/pytorch/labels/good%20first%20issue">good first issue</a> issues.<br>
Once you find a suitable one, add a comment in the corresponding issue and say that you would like to implement/fix this issue. The code owner or someone else from the community can then support you through the PR. In case you get stuck in this process, feel free to ping me.</p>
I released a c++ open source semantic segmentation library

Allent_Dan: <p>Hello everyone. I am a fan of pytorch and pytorch cpp (or libtorch). I wrote c++ segmentation and detection CNNs based on pytorch cpp. However, It seems not so many people use c++ to train a CNN or there is not so much need for c++ training.</p>
<p>I am new here for suggestions about my new project named <a href="https://github.com/AllentDan/SegmentationCpp" rel="noopener nofollow ugc">libtorch segment</a>. This is my first open source project, but not as popular as I imagined. Could it be the reason that semantic segmentation is only a small part of pytorch cpp user?  Or there must be something wrong with my project.</p>

Racek: <p>I think that the case is that most people use python for training, but I’m sure that your work will be useful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to debug torch.nn.export()

Jean_Da_Rolt: <p>Hi,</p>
<p>I’m trying to convert a Pytorch model to CoreML format. As suggested, I’m aproaching this issue in two steps:</p>
<ul>
<li>Pytorch to ONNX</li>
<li>ONNX to CoreML format</li>
</ul>
<p>However, while converting a simple UNET with ResNet backend to ONNX, I get the following error:<br>
RuntimeError: Failed to export an ONNX attribute, since it’s not constant, please try to make things (e.g., kernel size) static if possible</p>
<p>I took a look at torch.onnx code, but it is not easy to find exactly what line of the code (model) is provoking this error.</p>
<p>So, my question is: how to debug this conversion? There is a way to find what is causing this error? (turning verbose to True do not help)</p>
<p>Regards,</p>

LeviViana: <p>You can put prints everywhere you want to debug. The ONNX serializer traces the model, so it will execute it once, take a look at the <a href="https://pytorch.org/docs/stable/onnx.html" rel="nofollow noopener">documentation</a> for more details.</p>
<p>Here is a snippet showing how you could debug:</p>
<pre><code class="lang-python">import torch
class Conv(torch.nn.Module):
	def __init__(self):
		super(Conv, self).__init__()
		self.conv = torch.nn.Conv2d(3, 7, 5, 1, 2)
	def forward(self, x):
		print("-" * 40)
		print("DEBUG forward BEFORE convolution")
		print("-" * 40)
		x = self.conv(x)
		print("-" * 40)
		print("DEBUG forward AFTER convolution")
		print("-" * 40)
		return x

model = Conv()
dummy_input = torch.rand(16, 3, 100, 100)
torch.onnx.export(model, dummy_input, "debug.onnx", verbose=True)
</code></pre>

Jean_Da_Rolt: <p>Thanks <a class="mention" href="/u/leviviana">@LeviViana</a>,</p>
<p>I did not know that it would work. Looking at the call stack the execution was not anywhere near my model (it was inside torch.onnx in many calls). I took a look at the onnx code and found out where I could print the name of the current operation (even if symbolic), so I did that using gdb as I usually do.</p>
<p>In the end I found out a bug that is already registered at github.</p>
<p>Regards</p>
Methods about multi-class segmentation

zs963048949: <p>I want to know if it is the same for the following both train ways:</p>
<ul>
<li>3-class channels for the mask (0 or 1 for each channel)：
<ul>
<li>sigmoid output → bceLoss  (train)</li>
<li>sigmoid output → ge(0.5) → dice  (val)</li>
</ul>
</li>
<li>one-hot for the mask:
<ul>
<li>softmax output → cross-entropy loss  (train)</li>
<li>softmax output → argmax → dice  (val)</li>
</ul>
</li>
</ul>
<p>Did they cause a large influence on the model training?<br>
Thanks in advance.</p>
Any plans to improve Long tensor arithmetic?

iordanis: <p>In many functions, e.g. <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.7.1 documentation</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html?highlight=embedding#torch.nn.Embedding" class="inline-onebox" rel="noopener nofollow ugc">Embedding — PyTorch 1.7.1 documentation</a></p>
<p>It requires Long tensor as input which has significant larger memory foot-print than a fp16 or fp32 without actually being needed or used. This leads to optimization problems when training larger models.</p>
<ol>
<li>Are there workarounds in not using Long tensors?</li>
<li>Are there any plans to extend support to other int precision types?</li>
<li>Would there be reasons to not extend support to other int precision types?</li>
</ol>

ptrblck: <p>Using <code>float16</code> might be too limiting, as the precision limitations for integer values start at <code>2049</code> as described <a href="https://en.wikipedia.org/wiki/Half-precision_floating-point_format">here</a>.<br>
<code>float32</code> might be suitable, but it also depends on the overall gains.<br>
While it’s true that <code>LongTensors</code> would need more memory, you are often not dealing with huge target tensors, so the memory savings might be small.<br>
E.g. how large is the target tensor for <code>nn.CrossEntropyLoss</code> in your case and how much memory would you save if you could use e.g. <code>int32</code>?</p>
HTTP error on google colab using torchvision.models

Yolkandwhite: <p>I’m trying to use google colab<br>
but when I run the code</p>
<p>torchvision.models.whatever</p>
<p>the HTTP error 403 : forbidden occurs<br>
how can I fix it to use torchvision.models in google colab?<br>
(It works good on my personal computer but not in google server)</p>

kl_divergence: <p>It means torchvision isn’t able to connect to the pytorch servers. It usually happens when permissions are not made available to connect to the internet. Can you try downloading any other thing from it ? I don’t think it’s specific to torchvision.</p>

Yolkandwhite: <p>I just tried several more other models using torchvision it still didn’t work…</p>
<p>and I tried to download other stuffs like image file using urllib.request, it worked good.</p>
<p>I think google server is blocking torchvision somehow</p>

ptrblck: <p><a class="mention" href="/u/yolkandwhite">@Yolkandwhite</a> Thanks for letting us know.<br>
We’ll look into this issue.</p>
<p>EDIT: <a href="https://discuss.pytorch.org/t/loading-pretrained-resnet-on-google-colab-fails-with-http-403-forbidden/69427">related issue</a></p>

smth: <p>This seems to be CloudFlare doing terrible things – for no reason whatsoever with it’s “protection and caching layer”.</p>
<p>I’ve switched it off for now, and in ~4 hours or so, all requests should be direct again.</p>
<p>I’ll keep tabs on this thread to hear back if things are fixed.</p>
<p>Sorry about this whole mess.</p>

Kirty_Vedula: <p>This problem resurfaced again today. It was working fine couple of days ago. Could you please look into it?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/iam-not-able-to-download-dataset-from-torchvision/113643/2">Related post</a> with fix.</p>
SIGKILL when Running Distributed Applications

cotton_candy: <p>I have been trying to the MoCo code from Facebook Research on a machine with 4 GPUs but have been consistently receiving SIGKILLs.</p>
<p>If I run this command (which uses the nccl backend by default),</p>
<pre><code class="lang-auto">python main_moco.py -a resnet50 --lr 0.015 --batch-size 128 --dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 --mlp --moco-t 0.2 --aug-plus --cos
</code></pre>
<p>I get a SIGKILL as follows:</p>
<pre><code class="lang-auto">File "main_moco.py", line 406, in &lt;module&gt;
    main()
  File "main_moco.py", line 130, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "my_path/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "my_path/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "my_path/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 107, in join
    (error_index, name)
Exception: process 2 terminated with signal SIGKILL
</code></pre>
<p>With some simple print statements, I have pinpointed this issue to the init_process_group call <a href="https://github.com/facebookresearch/moco/blob/master/main_moco.py#L155" rel="noopener nofollow ugc">here</a>.</p>
<p>I also tried to run the code with the gloo backend. Now the init_process_group call succeeds, but my code fails with a SIGKILL just a little while later in the call to <a href="https://github.com/facebookresearch/moco/blob/master/main_moco.py#L170" rel="noopener nofollow ugc">model.cuda()</a>.</p>
<p>To narrow in on the issue a little more, I tried running the example code in the Setup section of the PyTorch distributed applications tutorial. This code runs perfectly with a gloo backend, but when I replace it with an nccl backend, the code either hangs on the call to init_process_group or crashes with the following stack trace:</p>
<pre><code class="lang-auto">File "multi_proc_test.py", line 17, in init_process
    dist.init_process_group(backend, rank=rank, world_size=size)
  File "my_path/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "my_path/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: Broken pipe
</code></pre>
<p>If it helps, I have confirmed that torch.distributed.is_nccl_available() returns true.</p>
<p>Any ideas why the code is failing in these places when I use the gloo/nccl backends?</p>
<p>Thank you in advance for your help!</p>

ptrblck: <p>The broken pipe error could be raised if one process died unexpectedly.<br>
You could rerun the minimal example with <code>NCCL_DEBUG=INFO</code> and check, if NCCL detects any errors.</p>
RandomResizedCrop with log aspect ratio

KaiHoo: <p>Could anyone explain why we need log aspect ratio for RandomResizedCrop? <a href="https://github.com/pytorch/vision/blob/7d4154735f421b254c408c16e0980b1ca0dd9b8e/torchvision/transforms/transforms.py#L831" class="inline-onebox" rel="noopener nofollow ugc">vision/transforms.py at 7d4154735f421b254c408c16e0980b1ca0dd9b8e · pytorch/vision · GitHub</a> I could not find any reference for this:</p>
<pre><code class="lang-auto">log_ratio = torch.log(torch.tensor(ratio))
</code></pre>

ptrblck: <p>It seems this sampling was added in <a href="https://github.com/pytorch/vision/pull/3344">this PR</a> to avoid the bias in the uniform sampling from the defined aspect ratios. <a href="https://github.com/pytorch/vision/issues/618#issuecomment-469644732">This comment</a> explains it in more detail.</p>
Output of each layer

swarup: <p>This is the code for my NN i want to know the outputs of each layer i heard about forward_hooks but dont know how to use,anyone help me</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87d6770144eea4083fa5bb53931700a33fce43bc.png" alt="image" data-base62-sha1="jnFWJEAbwSL9ycsRGOIfKPLimuw" width="562" height="442"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/0970d9a68090b6f5fe544a0f4e9a5b2d81d0fb6d.png" alt="image" data-base62-sha1="1lw4OxVn7E37oE05EbW74yYrMDj" width="655" height="467"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b48093e7a0a921770305802c82ede73f4059e48e.png" data-download-href="https://discuss.pytorch.org/uploads/default/b48093e7a0a921770305802c82ede73f4059e48e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b48093e7a0a921770305802c82ede73f4059e48e.png" alt="image" data-base62-sha1="pKNq7gNZ7BfMDKN0xgZ9skD4iZM" width="690" height="266" data-dominant-color="EFF0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">694×268 5.88 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

shivammehta007: <p>I don’t think you need hooks to see the output of each layer, you can just log it in the forward function,<br>
For example you can just print it</p>
<pre><code class="lang-auto">x1 = self.fc_spinal_layer1(x[:, 0:Half_width])
print(x1, x1.shape)
</code></pre>
<p>Or maybe I am misunderstanding something?</p>

Eta_C: <p>Hope <a href="https://www.kaggle.com/sironghuang/understanding-pytorch-hooks" rel="noopener nofollow ugc">this</a> can help you.</p>

swarup: <p>its not working neithers showing any error</p>
Change the "functional.interpolate" layer to "functional.max_unpool2d"

Firas_moh: <p>I have this model which has an encoder-decoder structure. I want to change the “functional.interpolate” layer (which does upsampling layer) to “functional.max_unpool2d”. But as you can see in the following code in the “colab link” <a href="https://colab.research.google.com/drive/1UHjW5tEEG-ZwIEIlPzVuZSgvGnPuKVMI?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1UHjW5tEEG-ZwIEIlPzVuZSgvGnPuKVMI?usp=sharing</a>, the max pooling is inside a Sequential, and I couldn’t get the indices of the tensor of max pooling layer. Are there any ideas on how to change the “functional.interpolate” layer to “functional.max_unpool2d” without changing a lot in the code. Really appreciate any help. Thank you so much in advance.</p>

ptrblck: <p>To use this layer in an <code>nn.Sequential</code> container, you could take a look at <a href="https://discuss.pytorch.org/t/how-can-i-pass-multiple-inputs-to-nn-sequential-layers/50868">this post</a>, which shows some approaches how multiple inputs can be used.</p>

Firas_moh: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>. That is very helpful.<br>
May I ask you, what is the advantage that can I expect from using “functional.max_unpool2d” instead of “functional.interpolate”?. What I have understood from my reading, that will yield a cheaper computation model. Is this correct?</p>

ptrblck: <p>I don’t know the theoretical complexities of both algorithms and how they are implemented.<br>
If you are concerned about the performance of these layers, you could profile them via e.g. the <code>torch.utils.benchmark</code> utilities.</p>
RuntimeError: ONNX export failed: Couldn't export operator aten::feature_dropout

Uday_Jagtap: <p>Trying to export the PyTorch model into the ONNX and I’m getting ‘RuntimeError: ONNX export failed: Couldn’t export operator aten::feature_dropout’ this error and one warning ‘UserWarning: ONNX export failed on feature_dropout because training mode not supported’. I’m new to PyTorch and I’m not able to find out where is the issue?</p>

5had3z: <p>Have you tried model.eval() before exporting? I’m not sure if torch.onnx.export does this for you. That will skip/turn off any training only related operations such as drop out or injected noise etc.</p>
Different batch sizes in each gpu using distributed training

zhenxing_mi: <p>Hi, <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">Distributed Data-Parallel Training</a> is a very useful tool. I am training a network needing different batch_sizes in different gpus. I’d like know if pytorch distributed training supporting different batch sizes in the gpus. Thank you!</p>
Run on whatever gpu whose memory is larger than the max_memory_allocated of my model

zhenxing_mi: <p>Hi, I have a model trained on GeForce RTX 3090. The nvidia-smi shows that 17GB memory is occupied. In my model I use “torch.cuda.max_memory_allocated” and find that 10GB memory is allocated. I know pytorch will cache memory allocator. My question is if my model will run on a gpu whose total memory is lower than 17GB but higher than 10GB?</p>

Dwight_Foster: <p>It would depend on how large your model is and how large your inputs and batch sizes were. That is a good amount of memory so you should be fine unless you are doing some huge vision problem.</p>

zhenxing_mi: <p>Hi, <a href="https://discuss.pytorch.org/u/Dwight_Foster">Dwight_Foster</a>, thank you for your answer. I will test on other gpus later.</p>
UserWarning: different input and target sizes

maxmatical: <p>I’m trying to implement DCGAN on MNIST</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/maxmatical/pytorch-projects/blob/master/generative%20models/DCGAN-MNIST.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/maxmatical/pytorch-projects/blob/master/generative%20models/DCGAN-MNIST.ipynb" target="_blank" rel="nofollow noopener">maxmatical/pytorch-projects/blob/master/generative models/DCGAN-MNIST.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4.1\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "print(torch.__version__)"
   ]
  },
</code></pre>

  This file has been truncated. <a href="https://github.com/maxmatical/pytorch-projects/blob/master/generative%20models/DCGAN-MNIST.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>When I start training the model, I’m getting the following warning:</p>
<pre><code class="lang-auto">/Users/maxtian/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:1594: UserWarning: Using a target size (torch.Size([128])) that is different to the input size (torch.Size([128, 1, 1, 1])) is deprecated. Please ensure they have the same size.
  "Please ensure they have the same size.".format(target.size(), input.size()))
</code></pre>
<p>How can I fix this, I’m thinking it might be due to the torch.ones/torch.zeros</p>

Ranahanocka: <aside class="quote no-group" data-username="maxmatical" data-post="1" data-topic="23541">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/3ec8ea/48.png" class="avatar"> maxmatical:</div>
<blockquote>
<p>Using a target size (torch.Size([128])) that is different to the input size (torch.Size([128, 1, 1, 1])) is deprecated</p>
</blockquote>
</aside>
<p>That’s your answer. You need to change the target size (i.e., the label size) to be 128,1,1,1</p>
<p>So for example,</p>
<p>target = target.view(-1,1,1,1)</p>

Raaj: <p>Adding to the answer by <a class="mention" href="/u/ranahanocka">@Ranahanocka</a>, if you  get a ‘not contiguous error’ use target.contiguous() before that line.</p>

yangjunting: <p>Thanks your answer.  Can I ask you if we can fix the label size ,and change  the input size (torch.Size([128, 1, 1, 1])) to be [128]<br>
for example,<br>
input = input.reshape(-1)<br>
to make the torch.Size([128, 1, 1, 1]) to be [128]<br>
and so solve the question “UserWarning: different input and target sizes”</p>

RAOPT: <p>It’s very useful, thank you all!</p>

Arcadianlee: <p>I think its more appropriate to change the label size here, rather than the input size.</p>