akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why GPU memory allocations are associated with the cuda stream?

username: yitongh
cooked: <p>From <a href="https://github.com/pytorch/pytorch/blob/master/c10/cuda/CUDACachingAllocator.cpp#L31" rel="noopener nofollow ugc">pytorch/c10/cuda/CUDACachingAllocator.cpp</a>, I found this:</p>
<blockquote>
<p>Allocations are associated with a stream. Once freed, blocks can be re-allocated on the same stream, but not on any other stream.</p>
</blockquote>
<p>And in <a href="https://github.com/pytorch/pytorch/blob/master/c10/cuda/CUDACachingAllocator.cpp#L227" rel="noopener nofollow ugc">code</a>, it actually follows this rule.</p>
<p>I can’t figure out why this limitation exists. As far as I know, CUDA doesn’t forbid different cuda streams to access the memory allocated by cudaMalloc. And this limitation will disallow some free blocks allocated by one cuda stream to be reused by other cuda streams.</p>
<p>Example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

a = torch.rand((12, 12)).cuda()   # allocated by default cuda stream
print(torch.cuda.memory_cached())

torch.cuda.synchronize()
s = torch.cuda.Stream()
with torch.cuda.stream(s):
    b = torch.rand((12, 12)).cuda()    # allocated by other cuda stream

torch.cuda.synchronize()
print(torch.cuda.memory_cached())
</code></pre>
<p>output is: 2097152 and 4194304.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

a = torch.rand((12, 12)).cuda()   # allocated by default cuda stream
print(torch.cuda.memory_cached())

b = torch.rand((12, 12)).cuda()   # allocated by default cuda stream
print(torch.cuda.memory_cached())
</code></pre>
<p>output is: 2097152 and 2097152.</p>
<p>Environment: PyTorch 1.3, cuda 10.1</p>

username: albanD
cooked: <p>My understanding was that this reduces the number of necessary synchronizations.<br>
cc <a class="mention" href="/u/colesbury">@colesbury</a> will have a better insight.</p>

username: colesbury
cooked: <p>As <a class="mention" href="/u/alband">@albanD</a> wrote, limiting CUDA allocations to a single stream reduces the number of CPU-GPU synchronizations necessary. CUDA kernels are asynchronous, so when an allocation is “freed” the kernel may not be finished (or may not have even started). Reusing the same allocation in a different stream could cause memory corruption because work in that stream may start before previously launched work in the original stream finishes.</p>
<p>It’s safe to immediately reuse the allocation in the same stream because operations within a stream are ordered sequentially. This is the strategy the caching allocator uses.</p>
<p>The CUDA memory API handles this differently: The cudaFree call synchronize all streams – the CPU waits until all streams finish all outstanding work before the cudaFree call completes. This ensures that subsequent uses of the memory are guaranteed to happen after the previous uses finish. However, this makes cudaFree a relatively expensive call. The primary goal of the caching allocator is to avoid this type of synchronization.</p>
Speed of different batch size

username: silly_monkey
cooked: <p>I have a problem to figure out the influence of batch size when the <strong>input requires grad</strong>.</p>
<p>Here is my code,</p>
<pre><code class="lang-auto">def test_generate_time(model, epoch_num, bs):
    model.eval()
    for p in model.parameters():
        p.require_grad = False
    model = model.cuda()

    imgs = torch.zeros((bs, 3, 224, 224)).float().cuda()
    imgs.requires_grad = True      # want to update inputs
    labels = torch.ones(bs).long().cuda()
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam([imgs], lr=1e-6)

    for epoch in range(epoch_num):
        epoch_begin = time.time()

        forward_start = time.time()
        out = model(imgs)
        loss = criterion(out, labels)
        forward_time = time.time() - forward_start

        backward_start = time.time()
        loss.backward()
        optimizer.step()
        backward_time = time.time() - backward_start
        epoch_time = time.time() - epoch_begin
        print('Bs {} forward time {:.3f} backward time {:.3f} epoch_time {:.3f}'.format(bs, forward_time, backward_time, epoch_time))
</code></pre>
<p>And here is the result of batchsize=1 and batchsize=32<br>
Bs 1 forward time 0.008 backward time 0.017 epoch_time 0.025<br>
Bs 32 forward time 0.021 backward time 0.296 epoch_time 0.317</p>
<p>It seems batch size has a big influence on the speed, which contradicts my knowledge because you know when training a network (instead of inputs), different batch size has little impact of speed. I also run a test of time for the ordinary image classification.<br>
Bs 1 forward time 0.0018 backward time 0.006<br>
Bs 32 forward time 0.0021 backward time 0.006</p>
<p>I really don’t know what’s happening here. Does any one knows why they differ? Or even improve the speed of training inputs with large batch size ?</p>

username: albanD
cooked: <p>It looks like you’re running all your code on cuda but you don’t do any synchronization when timing.<br>
The cuda api is asynchronous, so you need to manually add <code>torch.cuda.synchronize()</code> before calling <code>time.time()</code> if you want to measure the actual runtime.</p>

username: silly_monkey
cooked: <p>Thanks a lot! That’s very helpful.</p>
tf.matrix_band_part equivalent 

username: nima_rafiee
cooked: <p>I need to create upper triangular masking tensor in pytorch. in tensorflow it’s easy using <a href="https://www.tensorflow.org/api_docs/python/tf/linalg/band_part" rel="nofollow noopener">matrix_band_part</a>.  is it any pytorch equivalent for this function?<br>
it’s like numpy triu_indices but I need for tensor, not just matrix.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Isn’t that what <a href="https://pytorch.org/docs/stable/torch.html#torch.triu"><code>torch.triu()</code></a> is doing?</p>

username: nima_rafiee
cooked: <p>thank you for answer but torch.triu() is also only works with matrix.</p>

username: blade
cooked: <p>Did you find any solutions for this?</p>

username: nima_rafiee
cooked: <p>no. I did a for loop for that task at that time. I dont know if they have added any new possibility or not.</p>

username: albanD
cooked: <p>The batch version of <code>.triu()</code> has been added since then <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: blade
cooked: <p>I just tried it yesterday with PyTorch 1.0.0, is that for a newer version? If not, I’ll leave a code to reproduce my error here.</p>

username: albanD
cooked: <p>We’re at 1.3.1 now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I just tried again with latest version and it works:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.rand(2, 4, 4)
&gt;&gt;&gt; a.triu()
tensor([[[0.6312, 0.0578, 0.9598, 0.4663],
         [0.0000, 0.3202, 0.9285, 0.6618],
         [0.0000, 0.0000, 0.4512, 0.0584],
         [0.0000, 0.0000, 0.0000, 0.2116]],

        [[0.1818, 0.8329, 0.9879, 0.6801],
         [0.0000, 0.2870, 0.3073, 0.3876],
         [0.0000, 0.0000, 0.6292, 0.7130],
         [0.0000, 0.0000, 0.0000, 0.7048]]])

</code></pre>

username: blade
cooked: <p>Just upgraded and works as expected. Thanks for the tip!</p>
I am anxious about this 'RuntimeError'

username: Foolish
cooked: <p>Traceback (most recent call last):</p>
<p>File “”, line 1, in <br>
runfile(‘D:/360Downloads/VS/train.py’, wdir=‘D:/360Downloads/VS’)</p>
<p>File “C:\Users\16147\Anaconda3\envs\tensorflow\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 827, in runfile<br>
execfile(filename, namespace)</p>
<p>File “C:\Users\16147\Anaconda3\envs\tensorflow\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 110, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)</p>
<p>File “D:/360Downloads/VS/train.py”, line 135, in <br>
fit(epochs, model, loss_func, opt, train_dl, valid_dl)</p>
<p>File “D:/360Downloads/VS/train.py”, line 47, in fit<br>
losses, nums = zip(*[loss_batch_train(model, loss_func, xb, yb, opt) for xb, yb in train_dl])</p>
<p>File “D:/360Downloads/VS/train.py”, line 47, in <br>
losses, nums = zip(*[loss_batch_train(model, loss_func, xb, yb, opt) for xb, yb in train_dl])</p>
<p>File “D:/360Downloads/VS/train.py”, line 19, in loss_batch_train<br>
loss = loss_func(model(xb), yb)</p>
<p>File “C:\Users\16147\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\functional.py”, line 1970, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)</p>
<p>File “C:\Users\16147\Anaconda3\envs\tensorflow\lib\site-packages\torch\nn\functional.py”, line 1790, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)</p>
<p>RuntimeError: Expected object of scalar type Long but got scalar type Int for argument <span class="hashtag">#2</span> ‘target’</p>
<p>what shulde I do</p>

username: albanD
cooked: <p>From the error, it looks like “nll_loss” " Expected object of scalar type Long but got scalar type Int". So I guess you should change the input from an int Tensor to a long Tensor?</p>
How does PyTorch parallelize across multiple CPU cores?

username: mshuaibi
cooked: <p>We can define the number of cores to be used for CPU training with <code>torch.set_num_threads()</code>. But how exactly does PyTorch parallelize across multiple cores (no batching is involved here)? I’ve seen a custom training loop not in PyTorch that distributes the data across multiple cores with the model on each core, loss functions then evaluated, returned to the main process, optimizer is called, and the updated models then sent to the cores and the process is repeated. How does this compare to PyTorch? Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The low level CPU compute libraries that we use like mkl use multithreading if a single task is big enough.<br>
For each library, this flag is used to select the number of threads that are available.</p>
Jacobian gradient matrix between two images

username: Sam3
cooked: <p>Hallo,</p>
<p>I have two images and i want to calculate the gradient for each pixel in one image according to each pixel in another image , so i want to calculate the jacobian matrix by Pytorch,  can you give me plese any function or code to help me.<br>
Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can find a Jacobian function in this gist: <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7">https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7</a><br>
Note that it will be very slow if you want to full jacobian for any medium/large image.</p>

username: Sam3
cooked: <p>Thank you very much…I know it is very slow…I want to alculate the gradient for each pixel in one image according to each pixel in another image so jacobian matrix is 256 <em>256</em>256*256</p>

username: albanD
cooked: <p>A back of the envelope check for this means that just the Jacobian matrix will be 17GB.<br>
I’m not sure what your usecase is but it might be tricky.</p>
Can't download imagenet datset

username: dato_nefaridze
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/094aed2e955bd058412161cf08de5adc6d56fb58.png" data-download-href="https://discuss.pytorch.org/uploads/default/094aed2e955bd058412161cf08de5adc6d56fb58" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094aed2e955bd058412161cf08de5adc6d56fb58_2_690x387.png" alt="image" data-base62-sha1="1kcPf4MQNWmz0rcRqXKS65C36gU" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094aed2e955bd058412161cf08de5adc6d56fb58_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094aed2e955bd058412161cf08de5adc6d56fb58_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/094aed2e955bd058412161cf08de5adc6d56fb58.png 2x" data-dominant-color="3A3A3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 54.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
does anybody know how to download imagenet dataset?(i know i has branches and subset datasets, suppose i want bird dataset, how can i do that?)</p>

username: albanD
cooked: <p>Hi,</p>
<p>You should update your version of torchvision.<br>
Unfortunately, you cannot download imagenet dataset without logging in anymore. So you will have to manually download it. The newest version of torchvision will explain that issue if you try to download imagenet.</p>

username: dato_nefaridze
cooked: <p>ok but why pytorch documentation has</p>
<pre><code class="lang-auto">imagenet_data = torchvision.datasets.ImageNet('path/to/imagenet_root/')
data_loader = torch.utils.data.DataLoader(imagenet_data,
                                         batch_size=4,
                                         shuffle=True,
                                         num_workers=args.nThreads)
</code></pre>
<p>this code as a guide?</p>

username: albanD
cooked: <p>Yes, there is no <code>download=True</code> in the doc. You need to give the path where it is already downloaded as we cannot do it automatically anymore.</p>

username: dato_nefaridze
cooked: <p>if i download it manually than what’s the point of the datasets module?</p>

username: albanD
cooked: <p>For the publicly available datasets, we do provide them. But this one is not available to freely download online anymore.<br>
That wouldn’t be nice to share our own copy of their dataset.</p>
MutliClass Classification of imbalanced text data

username: sai_m
cooked: <p>any suggestions or tutorial links for multi classification of text data? I am planning to start with RNNs and LSTM. Is this a good start?</p>
<p>Thanks</p>

username: Kalyan_Katikapalli
cooked: <p>This link contains jupyter notebooks having code for  both binary as well as multi classification text classification with clear explanations.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/bentrevett/pytorch-sentiment-analysis" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/8006479?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/bentrevett/pytorch-sentiment-analysis" target="_blank" rel="nofollow noopener">bentrevett/pytorch-sentiment-analysis</a></h3>

<p>Tutorials on getting started with PyTorch and TorchText for sentiment analysis. - bentrevett/pytorch-sentiment-analysis</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: sai_m
cooked: <p>Awesome!! Will go through. Thanks kalyan</p>
PyTorch Distributed Gloo Backend

username: Tabrizian
cooked: <p>Is it a good practice to create a new group on every training iteration?</p>
<p>Using dist.new_group requires every process to pass through the function even if they are not a part of the distributed training process. Sometimes it hangs up on this function for a reason that I am not aware of. I was wondering has anybody come up with a better solution for this?</p>
<p>Thanks!</p>

username: Yanli_Zhao
cooked: <p>iiuc, it is not good practice to create a new group every training iteration, it is non trivial to initialize it, there are communication costs.</p>
<p>why do you need to create new group every iteration?</p>

username: Tabrizian
cooked: <p>Thanks for the answer. Because I need to run allreduce on a subset of nodes in each iteration. Do you have any recommendation about how to do it differently?</p>

username: Yanli_Zhao
cooked: <p>do you want to create these sub groups before training loop, and each iteration just uses corresponding created sub groups? instead of creating the same sub groups repeatedly inside the loop?</p>
<p>also, add time out if it hangs?</p>
<p>lastly, maybe need to figure out why it hangs, starting with debugging using a small number of training iterations?</p>

username: Tabrizian
cooked: <p>Thanks for the suggestion! I originally thought about this approach. But because I would like to create this for large number of machines (around 100) the number of groups can become quite large (combination of 10 out of 100 =17310309456440!!).</p>
<p>I debugged it and it seems like the limit for the number of cgroups exceeds the current user limit. I guess this may be related to the dist.new_group and repeated creation of groups.</p>
<p>The error printed in the kernel messages is printed below:</p>
<pre><code class="lang-auto">cgroup: fork rejected by pids controller in /user.slice/user-1000.slice/session-3.scope
</code></pre>
<p>I wish there was a method to delete the groups in order to avoid this problem.</p>

username: Yanli_Zhao
cooked: <p>you can call dist.destroy_process_group()</p>
Np.vstack using torch in libtorch and pytorch

username: Esmaeil_Farhang
cooked: <p>how can translate this numpy code to both pytorch and libtorch:</p>
<pre><code class="lang-auto">np.vstack([fourier_basis[:cutoff, :], fourier_basis[:cutoff, :] )])
</code></pre>

username: albanD
cooked: <p>I’m not sure what <code>np.vstack</code> is doing but <code>torch.stack</code> sounds like a good candidate <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Esmaeil_Farhang
cooked: <p>thank’s, but   <code>torch.stack</code> and  <code>torch.cat</code>  don’t work.</p>

username: tom
cooked: <p>Yeah, well, it’s equivalent to <code>stack</code> for 1d inputs, to <code>cat</code> for &gt;=2d. In your case <code>cat</code>.</p>

username: mailcorahul
cooked: <p><code>torch.cat((a, b), dim=0)</code> should work.<br>
Essentially vstack concatenates row-wise.</p>

username: Esmaeil_Farhang
cooked: <p>thanks , its ok.<br>
in numpy :</p>
<pre><code class="lang-auto">filter_length = 4  
fourier_basis = np.fft.fft(np.eye(filter_length))
 cutoff = int((filter_length / 2 + 1))
np.vstack([np.real(fourier_basis[:cutoff, :]),np.imag(fourier_basis[:cutoff, :])])
</code></pre>
<p>is equal to pytorch:</p>
<pre><code class="lang-auto">fourier_basis_torch = torch.rfft(torch.eye(filter_length), 1, False) 
real = fourier_basis_torch[:,:,0].transpose(0,1)
imag = fourier_basis_torch[:,:,1].transpose(0,1)
torch.cat( (real[:cutoff, :],imag[:cutoff, :]), dim=0)
</code></pre>
<p>and in libtorch:</p>
<pre><code class="lang-auto">int64_t filterLength = 4;
int cutoff = (int)((filterLength / 2 + 1));
auto npFft = at::rfft(at::eye(filterLength),signal_ndim, onesided);
auto realPart = torch::nn::functional::_narrow_with_range(npFft,2,0,1);
realPart = at::transpose(realPart, 0, 1);    
realPart = realPart.view({realPart.size(0), realPart.size(1)}); 
auto imgPart = torch::nn::functional::_narrow_with_range(npFft,2,1,2);
imgPart =  at::transpose(imgPart, 0, 1);  
imgPart = imgPart.view({imgPart.size(0), imgPart.size(1)});         
auto out = at::cat({realPart, imgPart}, 0);
</code></pre>
How to upgrade torchvision without upgrading torch?

username: dalalaa
cooked: <p>My torch and torchvision version:</p>
<pre><code class="lang-auto">(pytorchfromsource) dai@jqkj-EG820G-G10:~/scripts$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.2.0.dev20190527'
&gt;&gt;&gt; torchvision.__version__
'0.2.1'
</code></pre>
<p>I want to upgrade torchvision to 0.2.1 to use pretrained MobileNetv2, but when I’m upgrading torchvision through pip, torch is also upgrading.</p>
<pre><code class="lang-auto">(pytorchfromsource) dai@jqkj-EG820G-G10:~/scripts$ pip install torchvision==0.4.1
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting torchvision==0.4.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fc/23/d418c9102d4054d19d57ccf0aca18b7c1c1f34cc0a136760b493f78ddb06/torchvision-0.4.1-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)
     |████████████████████████████████| 10.1MB 2.9MB/s
Requirement already satisfied: numpy in /home/jqkj/miniconda3/envs/pytorchfromsource/lib/python3.6/site-packages (from torchvision==0.4.1) (1.16.3)
Requirement already satisfied: six in /home/jqkj/miniconda3/envs/pytorchfromsource/lib/python3.6/site-packages (from torchvision==0.4.1) (1.12.0)
Requirement already satisfied: pillow&gt;=4.1.1 in /home/jqkj/miniconda3/envs/pytorchfromsource/lib/python3.6/site-packages (from torchvision==0.4.1) (6.0.0)
Collecting torch==1.3.0 (from torchvision==0.4.1)
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/05/50a05de5337f7a924bb8bd70c6936230642233e424d6a9747ef1cfbde353/torch-1.3.0-cp36-cp36m-manylinux1_x86_64.whl (773.1MB)
     |█▊                              | 42.6MB 737kB/s eta 0:16:31^C

</code></pre>
<p>How can I upgrade torchvision without upgrading torch?</p>
<p>Looking forward to your replay.</p>

username: dalalaa
cooked: <p>Sorry， I found the solution myself:</p>
<pre><code class="lang-auto">pip install torchvision==0.4.1 --no-deps
</code></pre>

username: alanzhai219
cooked: <p>Just git clone torchvision repo and build what you want. It’s very simple.</p>
Is this the right way to use Cross entropyloss

username: VikasRajashekar
cooked: <p>I have a classifier that has to predict whether a given sentence is positive or negative or Neutral.</p>
<p>This is my forward pass:</p>
<pre><code class="lang-auto">Sentiment_LSTM(
  (embedding): Embedding(19612, 400)
  (lstm): LSTM(400, 512, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=512, out_features=3, bias=True)
)
</code></pre>
<pre><code class="lang-auto">Using the follwing loss function and learning rate:
lr=0.001

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
</code></pre>
<p>Here when I am doing back prop:</p>
<pre><code class="lang-auto"> output = net(inputs)
loss = criterion(output, labels)
</code></pre>
<p>Here my Labels for each example is jus 0 or 1 or 2. and output is the output of linear layer from the network(Size three for each example). Is this the right way to use this loss?</p>
<p>How do I find the best learning rate and optimizer?</p>

username: KFrank
cooked: <p>Hi Vikas!</p>
<aside class="quote no-group quote-modified" data-username="VikasRajashekar" data-post="1" data-topic="63066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vikasrajashekar/48/19580_2.png" class="avatar"> VikasRajashekar:</div>
<blockquote>
<p>I have a classifier that has to predict whether a given sentence is positive or negative or Neutral.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
</code></pre>
<pre><code class="lang-auto">loss = criterion(output, labels)
</code></pre>
<p>Here my Labels for each example is jus 0 or 1 or 2. and output is the output of linear layer from the network(Size three for each example). Is this the right way to use this loss?</p>
</blockquote>
</aside>
<p>Yes.  Your <code>output</code>, the first argument to <code>CrossEntropyLoss</code>, being<br>
the output of your linear layer, are triples of values that range from<br>
-infinity to infinity, and can be understood as <em>logits</em> (in contrast to<br>
probabilities, that would range from zero to one).  Your <code>labels</code> are<br>
single values in {0, 1, 2}, that is, they are integer categorical labels.<br>
This is what <code>CrossEntropyLoss</code> expects.  (Internally, in effect,<br>
<code>CrossEntropyLoss</code> converts the logits you pass it into probabilities<br>
by, in effect, passing them through a softmax function.)</p>
<aside class="quote no-group">
<blockquote>
<p>How do I find the best learning rate and optimizer?</p>
</blockquote>
</aside>
<p>Experimentation.  Too small a learning rate, and your network will<br>
train slowly; too large, it will jump around or become unstable.</p>
<p>The <code>Adam</code> optimizer can be a good choice.  I would also suggest<br>
trying, at least as a baseline, the plain-vanilla <code>SGD</code> optimizer (again<br>
with a variety of learning rates).</p>
<p>Good luck.</p>
<p>K. Frank</p>
Maximizing Gaussian Likelihhood Using Pytorch

username: zabanmail
cooked: <p>Hey I am trying to maximize Gaussian likelihood in PyTorch, Unfortunately getting very unreasonable results.</p>
<h2>Start of the code</h2>
<pre><code class="lang-auto">mu= Variable(torch.rand(1), requires_grad=True)
sigma= Variable(torch.rand(1), requires_grad=True)
learning_rate = 0.00002
for t in range(100000):
    NLL = -torch.sum(torch.log((1/(np.sqrt(2*3.14)*sigma))*torch.exp((x-mu)**2/(2*sigma**2))))
    NLL.backward()

    if t % 1000 == 0:
        print("loglik  =", NLL.data.numpy(),"sigma",sigma.data.numpy(),"mu",mu.data.numpy())
    
        mu.data -= learning_rate * mu.grad.data
        sigma.data -= learning_rate * sigma.grad.data
    
        mu.grad.data.zero_()
        sigma.grad.data.zero_()
        
Not sure where the problem goes wrong? Do you think this can implemented and solved using techniques like adam optimizers?</code></pre>
How to save Pytorch model (graph + weights) in a single file

username: sujay
cooked: <p>I have saved the model using the torch.save(_model, PATH1) function and weights in torch.save(‘model_state_dict’: _model.state_dict(), PATH2).</p>
<p>I want to load the model from another system.</p>
<p>When I load the “_model” I get the error saying that the model architecture is not defined (Can’t get attribute ‘Net’) and had to instantiate the class that creates the model and then load the weights from the checkpoint _model.state_dict in path2.</p>
<p>Is there a way to save both the weights and architecture in a file so that I need not depend on the class definition file, to first create the model object and then load the weights from state dict?</p>
<p>I am looking for a solution similar to what Lua+Torch implementation of torch.save(model, path) does.</p>
<p><a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/apaszke">@apaszke</a> <a class="mention" href="/u/colesbury">@colesbury</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately this is a python limitation. You cannot save the code easily and in a portable way.</p>
<p>You can try to save a torchscript model that you get from the jit if your model can be handled by torchscript.</p>

username: sujay
cooked: <p>thanks for clarifying</p>
Model giving outputs only among some nodes out of all

username: rustagiadi95
cooked: <p>Hello there,</p>
<p>I am training a model of predicting the nationality from the name, so I have 95 countries as labels.<br>
But out of those 95 the model is always giving only among 5 or 6 countries as the output.<br>
I have read some posts for this, <a class="mention" href="/u/ptrblck">@ptrblck</a> says that it can be due to high learning rate(this could be the reason as I have high learning rate), some say that it is due to “batch_normalization” (Although, I haven’t used it in my model).</p>
<p><strong>But I have used dropout(I have 3 layers for unigrams, bigrams and trigrams) and gradient clipping, I am wondering that if these 2 factors are also contributing to this problem or not?</strong></p>
<p>PS :- Also I am decaying the learning rate by a decay rate of 0.99 per epoch, but still a high learning rate is there for about 150 epochs.</p>
<p>Any help is appreciated<br>
Thanks in advance.</p>

username: ptrblck
cooked: <p>Have you tried to play around with the learning rate a bit?<br>
The overfitting could also be due to an imbalanced dataset, but I’m not sure if that’s the case for your use case.</p>
<p>How many samples of each class do you have?</p>

username: rustagiadi95
cooked: <p>Hi,</p>
<p>Thanks for the response, I haven’t got time to check the imbalance in the dataset, I was thinking of this reason as well.<br>
Also I have played it a little bit, decreasing the learning rate does increases the validation accuracy, but there is a research paper I am trying to implement and they started with a high LR, so I did the same just to see the difference. With high LR, the accuracy has decreased but due to slow decay the accuracy is rising but not risen up to the accuracy achieved with smaller learning rate I trained with earlier yet . Upon checking the results of validation and testing, I think it might be a data imbalance but I was wondering if gradient clipping and dropouts is also contributing to this problem or not.<br>
P.S.  I employed gradient clipping only because I was using LSTMs. And yeah, I am doing model.train() and model.eval() switches.</p>

username: rustagiadi95
cooked: <p>You are right, I am having a great data imbalance(far too great). But still I am able to train it well now.<br>
Many Thanks for your suggestions, but still the original question is still unanswered!!</p>
Select values along one dimension

username: Mian
cooked: <p>The question is if I have a 2-dimensional tensor [ [0, 1], [2, 3], [0, 0] ], how to write code without any loop to get the result [ [2], [1, 3] ] ?  The zeros denote invalid positions which we do not select.</p>

username: beaupreda
cooked: <p>Hi,</p>
<p>I suppose that the expected result is [1, 2, 3] and not [[2], [1, 3]]. Here is one way you can do it:</p>
<pre><code class="lang-auto">    x = torch.tensor([[0, 1], [2, 3], [0, 0]], dtype=torch.int32)
    mask = (x != 0)
    y = x[mask]
    print(y)
    # tensor([1, 2, 3], dtype=torch.int32)
</code></pre>
<p>Let me know if it is not what you wanted!</p>

username: Mian
cooked: <p>Thanks for your answer. But it is not what I want. The result I want, [ [2], [1, 3] ], reflects that the element 2 is selected from the first column of [ [0, 1], [2, 3], [0, 0] ], and the elements 1 and 3 are selected from the second column of [ [0, 1], [2, 3], [0, 0] ].</p>

username: beaupreda
cooked: <p>Ok, the result you want is a tensor? Because I am pretty sure that you cannot have a tensor with a varying number of columns. Or do you want your result as Python lists?</p>

username: Mian
cooked: <p>Yes, a python list is ok.</p>

username: Mian
cooked: <p>maybe we should first convert [ [0, 1], [2, 3], [0, 0] ] to python list, because pytorch build-in functions always return tensors which must have the same number of numbers along one dim.</p>

username: beaupreda
cooked: <p>Hmm, the problem with converting to python lists is now that we have to use for loops… I am afraid that I am not able to help you with your problem, the only way I arrive at your expected result involves a for loop.</p>
<p>Sorry, and best of luck!</p>

username: Mian
cooked: <p>Thanks for your patience.</p>
How to set many-to-one network

username: osousa
cooked: <p>I have this network , that i took mainly from this <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="noopener nofollow ugc">tutorial</a>, and i want to have sentences as input (Which is already done)  and just a one line tensor as a result.</p>
<p>From the tutorial,  this sentence “John’s dog likes food” , gets a 1 column tensor returned:</p>
<blockquote>
<p>tensor([[-3.0462, -4.0106, -0.6096],<br>
[-4.8205, -0.0286, -3.9045],<br>
[-3.7876, -4.1355, -0.0394],<br>
[-0.0185, -4.7874, -4.6013]])</p>
</blockquote>
<p>tag_list[ “name”, “verb”, “noun”]</p>
<p>where each line has the probability of a tag being associated with the word. (The first word has [-3.0462, -4.0106, -0.6096] vector where the last element corresponds to the maximum scoring tag)</p>
<p>The tutorial’s dataset looks like this:</p>
<pre><code class="lang-auto">training_data = [
    ("The dog ate the apple".split(), ["DET", "NN", "V", "DET", "NN"]),
    ("Everybody read that book".split(), ["NN", "V", "DET", "NN"])
]
</code></pre>
<p>And i want mine to be of this format:</p>
<pre><code class="lang-auto">training_data = [
    ("one man".split(), ["ONE"]),
    ("two birds".split(), ["TWO"]),
    ("three stones".split(), ["THREE"])
]
</code></pre>
<p>The parameters are defined as:</p>
<pre><code class="lang-auto">class LSTMTagger(nn.Module):
    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim
        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)

    def forward(self, sentence):
        embeds 		= self.word_embeddings(sentence)
        lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
        tag_space	= self.hidden2tag(lstm_out.view(len(sentence), -1))
        tag_scores 	= F.log_softmax(tag_space, dim=1)
        return tag_scores
</code></pre>
<p>As of now, the sizes from input and output are not matching:<br>
<strong>ValueError: Expected input batch_size (2) to match target batch_size (1).</strong></p>

username: osousa
cooked: <p>I solved this issue, by simply getting the hidden states of the last layer from lstm_out</p>
<p>I am sorry for the inconvenience.</p>
Change color in rgb images

username: Majid_Shirazi
cooked: <p>Hi everyone,</p>
<p>I have a image and I want to change where rgb values are [0.4, 0.4, 0.4] to [0.54, 0.27, 0.07].<br>
By doing this I want to change road color from gray to brown. Would you please help me with this?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1d1fd091077b5f0218933bee853bb4c49d7d078.png" alt="image" data-base62-sha1="pn4prvSeG9clY4Q9WAr2S1tS4sw" width="531" height="351"></p>

username: sdv4
cooked: <aside class="quote no-group" data-username="Majid_Shirazi" data-post="1" data-topic="63057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/majid_shirazi/48/13913_2.png" class="avatar"> Majid_Shirazi:</div>
<blockquote>
<p>image</p>
</blockquote>
</aside>
<p>If <code>im</code> is your torch.tensor of shape (L, W, 3), then it would be the same as in numpy:</p>
<pre><code class="lang-auto">im[:, :, 0][im[:, :, 0] == 0.4] = 0.54
im[:, :, 1][im[:, :, 1] == 0.4] = 0.27
im[:, :, 2][im[:, :, 2] == 0.4] = 0.07
</code></pre>
<p>There is probably a nicer way to do it in one line, but this will work and is more clear than the one line solution I am thinking of.</p>
Reproducibility of model using ctc loss

username: karaspd
cooked: <p>It would be great if there would be a full list of operations that might not get reproducible even by enabling cudnn.detereministic flag.<br>
I am using  <code>torch.nn.CTCLoss</code>  with cpu the results are reproducible but with gpu and following setting the results from two experiments slowly get diverging. I just want to make sure there is no way for me to get reproducible training (loss values, parameter values) as long as I am using ctc loss.</p>
<pre><code class="lang-auto">    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    import numpy as np
    np.random.seed(seed)
    import os
    os.environ["PYTHONHASHSEED"] = "0"

    import random
    random.seed(seed)

    cudnn.deterministic = True
    cudnn.benchmark = False
</code></pre>
<p>Thanks</p>

username: tom
cooked: <p>If you use CuDNN, deterministic should work (but you need nightlies or CuDNN &lt; 7.5(!)).<br>
You need to follow the input conventions and probably check for the grad_fn to be the CuDNN one.<br>
Or you could write the deterministic native backward.</p>
<p>Best regards</p>
<p>Thomas</p>

username: karaspd
cooked: <p>Thank you for your response. I have another question, I have installed pytorch with <code>pip install torch==1.2.0 torchvision==0.4.0</code> does it come with cuDNN installation?<br>
<code>torch.backends.cudnn.version()</code> gives me 7602 which make me believe that it has cudnn version 7.6 with cuda version ‘10.0.130’. If that’s the case, how can I  change the cuDNN version?</p>
<p>Thank you for the help.</p>
ModuleNotFoundError: 'pycocotools._mask' during Pytorch tutorial

username: SeanAbnerNash
cooked: <p>Hello I’m new to Pytorch and I’ve been trying to work through this tutorial.<br>
[<a href="https://github.com/pytorch/tutorials/blob/master/intermediate_source/torchvision_tutorial.rst" rel="noopener nofollow ugc">https://github.com/pytorch/tutorials/blob/master/intermediate_source/torchvision_tutorial.rst</a>]</p>
<p>I’m using Visual Studio Code, conda installed the Pytorch. Initially the issue was that it was missing the module engine. But this problem was solved here:</p><aside class="quote" data-post="3" data-topic="59564">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/frank_wang1/48/17471_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/modulenotfounderror-no-module-named-engine/59564/3">ModuleNotFoundError: No module named 'engine'</a> 
  </div>
  <blockquote>
    So how do we get the references/detection/ folders? What should we download and install? I have installed the pytorch, torchvision in my environment, but I could not find those files. Thanks
  </blockquote>
</aside>

<p>With a step I misread.<br>
So I downloaded and copied the pycocotools into the project directory and the vision/detection/ .py files in to it too.<br>
That being done it fixed that issue but the error it is now giving is.</p>
<pre><code class="lang-auto">(base) C:\Users\Sean\Desktop\Project\Test\Tutorial&gt;D:/Anaconda/python.exe c:/Users/Sean/Desktop/Project/Test/Tutorial/tv-training-code.py
Traceback (most recent call last):
  File "c:/Users/Sean/Desktop/Project/Test/Tutorial/tv-training-code.py", line 13, in &lt;module&gt;
    from engine import train_one_epoch, evaluate
  File "c:\Users\Sean\Desktop\Project\Test\Tutorial\engine.py", line 8, in &lt;module&gt;
    from coco_utils import get_coco_api_from_dataset
  File "c:\Users\Sean\Desktop\Project\Test\Tutorial\coco_utils.py", line 9, in &lt;module&gt;
    from pycocotools import mask as coco_mask
  File "c:\Users\Sean\Desktop\Project\Test\Tutorial\pycocotools\mask.py", line 3, in &lt;module&gt;
    import pycocotools._mask as _mask
ModuleNotFoundError: No module named 'pycocotools._mask'
</code></pre>
<p>I’m not quite sure what the issue here is or how to fix it. Given that the _mask.pyx is present and is what I think is being imported. As I said I’m not to sure what the issue is but I would appreciate the help.</p>
<p>Python Version : 3.7.4<br>
Pytorch: 1.2.0 (Cuda 10)</p>

username: ptrblck
cooked: <p>Probably <code>pycocotools</code> is missing, which can be installed via:</p>
<pre><code class="lang-python"># install pycocotools
cd $INSTALL_DIR
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install
</code></pre>
Pad + Random Crop

username: pedro
cooked: <p>I have a series of images with size 178 × 218 and I was wondering how to pad them and randomly crop them in a way that it doesn’t eliminate any of the interesting parts of my image (which are normally at the center) for the task I’m doing.</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>I believe <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomCrop" rel="nofollow noopener">RandomCrop</a> is the transformation you are looking for (part of torchvision). If you are not familiar with torchvisions’s <code>transforms</code>, I suggest that you look at the following <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#transforms" rel="nofollow noopener">tutorial</a>.</p>
How can i get all the gradients of network

username: dato_nefaridze
cooked: <pre><code class="lang-auto">a=torch.tensor([3.0],requires_grad=True)
def sqrt(x):
  return x*x
b=sqrt(a)
print(b)
c=sqrt(b)
c.backward()
print(c.grad)
</code></pre>
<p>how can i see gradient of b tensor with respect c? it should be 2*x, and if b==9, gradient should be 18</p>

username: tom
cooked: <p>You can use <code>b.retain_grad()</code> to instruct PyTorch to store the gradients of intermediates like <code>b</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
Calculate the "backward" for only one Tensor

username: barakb
cooked: <p>Hi, I want to make a backward pass that will calculate gradient update need just for one specific tensor.<br>
I understand that the tensors that are closer to the end of the network will also get a gradient calculation, but my goal is just for the specific tensor.<br>
So how can I stop the loss.backword() operation in some layer? The main reason btw is to save time.</p>
<p>Thanks.</p>

username: tom
cooked: <ul>
<li>You can set parameters not needing gradients to not require grad (using <code>p.requires_grad_(False)</code>). PyTorch only computes gradients required for those values that require gradients.</li>
<li>Even so, backpropagation has it that you use the chain rule from the final result (i.e. loss) back to the first use of the parameter that requires gradients. You cannot escape calculating gradients for the intermediates in between.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: barakb
cooked: <p>As said I’m not trying to escape calculating the layers between the desire tensor and output, but on earlier layers.<br>
One thing that I’m never sure about, does <code>requires_grad_(False)</code>  block the gradient from moving to earlier layers?<br>
lets say I want the gradient just for conv3:</p>
<pre><code class="lang-auto">x = self.conv1(x)
x = self.conv2(x)
x = self.conv3(x)
out = self.conv4(x)
</code></pre>
<p>Should this be enough:</p>
<pre><code class="lang-auto">self.conv2.requires_grad_(False))
</code></pre>
<p>or this is a must:</p>
<pre><code class="lang-auto">self.conv1.requires_grad_(False))
self.conv2.requires_grad_(False))
</code></pre>
<p>Thanks.</p>

username: tom
cooked: <p>The latter. If you picture the autograd tree, <code>self.conv2.requires_grad_(False)</code> only cuts the bits from the “stem” to the leaves in <code>self.conv2</code> (i.e. weight/bias).</p>

username: barakb
cooked: <p>Thanks, Is it possible to do the following:</p>
<pre><code class="lang-auto">                model.module.layer2.requires_grad_(False)
                model.module.layer1.requires_grad_(False)
                out = model(input_var,first=1)
                loss_1 = criterion(out, t)
                optimizer.zero_grad()
                loss_1.backward(retain_graph=True)
</code></pre>
<p>And then:</p>
<pre><code class="lang-auto">                model.module.layer2.requires_grad_(True)
                model.module.layer1.requires_grad_(True)
                loss_2 = criterion(out, t)
                optimizer.zero_grad()
                loss_2.backward()
</code></pre>
<p>Means, does the .requires_grad_(True) option need to be submitted in the forward pass of the network, or it can be also before the forward pass</p>

username: tom
cooked: <p>You need it <em>before</em> the forward pass.<br>
So technically, it’s not a Module but a Parameter/Tensor property and you need to decide before you use the Tensor whether you want to differentiate later so that the autograd can start recording.</p>
<p>Best regards</p>
<p>Thomas</p>
TypeError after assigning nn.parameter to regular tensor

username: smonsays
cooked: <p>I am experiencing a strange behaviour where after assigning a <code>nn.parameter</code> to a tensor, this tensor becomes of type <code>nn.parameter</code> as well which leads to a <code>TypeError</code> in future assignments:</p>
<pre><code class="lang-auto">Cannot assign 'torch.FloatTensor' as parameter 'weight_current' torch.nn.Parameter or None expected)
</code></pre>
<p>This problem appears to be generated by the following code-snippet in one of my modules:</p>
<pre><code class="lang-auto">class MyLayer(torch.nn.Linear):

    def forward(self, input, sample):
        if sample:
	    self.weight_current = self.weight + torch.randn_like(self.weight)
	    self.bias_current = self.bias + torch.randn_like(self.bias)
	else:
	    self.weight_current = self.weight
	    self.bias_current   = self.bias

	return torch.nn.functional.linear(input, self.weight_current, self.bias_current)
</code></pre>
<p>That is, after calling <code>forward</code> with <code>sample=False</code>, I cannot call it with <code>sample=True</code> anymore, as this raises the <code>TypeError</code>.</p>
<p>It seems odd to me that this is happening at all. Can somebody explain why pytorch behaves the way it does and what I can do about it?</p>

username: albanD
cooked: <p>That’s because assigning a Parameter to an attribute of an nn.Module has a special meaning: register new weights.<br>
Do you need to put <code>weight_current</code> in <code>self.</code>? Removing the <code>self.</code> will solve the issue.</p>

username: smonsays
cooked: <p>Okay, that is what I suspected. Unfortunately, I need to keep <code>weight_current</code> as an attribute of the class, so removing <code>self.</code> is not an option. Is there anything else I can do?</p>
<p>Is there a particular reason pytorch decided to implicitly register a new Parameter after assignment of a Parameter to an attribute in a nn.Module derived class? It seems to me that in most cases it would be preferable to do that explicitly.</p>

username: albanD
cooked: <p>Because in the init, you always do <code>self.weight = nn.Parameters(foo)</code>.<br>
A workaound is to reset the current weight at the beginning of the forward with:<br>
<code>del self.weight_current</code> and <code>del self.bias_current</code>.</p>
Memory when storing states in a list

username: Knievel
cooked: <p>From <a href="https://discuss.pytorch.org/t/memory-leak-when-appending-tensors-to-a-list/25937">this thread</a> i learned that I always store the whole computational graph when I append a tensor to a list.</p>
<p>To fix this I could detach the tensor before storing it, however in my case I need them for gradient computation. For each pixel of an image I have to store one state tensor, as soon as all state tensors are computed I can compute the output. If I use more layers I need to store n times the amount of tensors.</p>
<p>Currently I use 64x84 images and with two layers of 256 units I already run out of memory on my universities cluster.</p>
<p>Is there some way of reducing the storage needed or am I just not able to use larger numbers with my architecture?</p>

username: albanD
cooked: <p>If your want to backprop through all these, then you need to have them.</p>
<p>An alternative is to trade memory for compute using the <a href="https://pytorch.org/docs/stable/checkpoint.html?highlight=checkpoint">torch.utils.checkpoint</a> module.</p>
Manually set dropout for each of the samples

username: guitarman
cooked: <p>Let’s say I have a sparse vector of ones and zeros (NxM) and I would like to have it as a dropout filter on the conv2d layer output which is also (NxM). Is it possible to achieve that by just multiplying it with the output?</p>
<p>Furthermore, is it possible to set a different dropout “mask” for each of the X batch samples? Or I would need to have X backprops by 1 batch sample?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You should be able to multiply the conv output with your custom mask.<br>
If you want to use a different mask for each batch element, your mask should have a batch dimension and different masks for each sample in the batch.<br>
Are you seeing any issues with this approach?</p>

username: guitarman
cooked: <p>Thank you, I have implemented it this way.</p>

username: guitarman
cooked: <p>Is there any convenient way of knowing whether the forward is called during training or eval, I would like to apply the dropout mask in between the sequential classes in the forward function.</p>
<p>I guess passing a parameter in forward() is not recommended?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You could check the internal <code>self.training</code> flag inside the module.</p>