akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there any way to add noise to trained weights?

username: 3c06d7576e3434b36c48
cooked: <p>Hello I’ve got a trained network.<br>
Is there any way to add noise(for example gaussian) to trained weights?<br>
thank you</p>

username: ptrblck
cooked: <p>You could use this sample code to add gaussian noise to all parameters:</p>
<pre><code class="lang-python">with torch.no_grad():
    for param in model.parameters():
        param.add_(torch.randn(param.size()) * 0.1)
</code></pre>
<p>If you want to skip certain parameters, you could use <code>model.named_parameters()</code> and filter out all unnecessary parameters.</p>

username: 3c06d7576e3434b36c48
cooked: <p>Thank you very much!</p>

username: sirius
cooked: <p>Dear Ptrblck,</p>
<p>I only want to add the noise to the weights in each epoch, Do you have a more convenient way to do that, instead of filling other parameters one by one?</p>
<p>Moreover, I am not quite sure how to use this piece of code. Adding these codes to the training loop?</p>
<p>Could you provide more specific information how to use them?</p>

username: ptrblck
cooked: <p>You would somehow have to access each parameter, but a custom method and the usage of <code>model.apply</code> might be more convenient:</p>
<pre data-code-wrap="python"><code class="lang-python">def add_noise_to_weights(m):
    with torch.no_grad():
        if hasattr(m, 'weight'):
            m.weight.add_(torch.randn(m.weight.size()) * 0.1)

model.apply(add_noise_to_weights)
</code></pre>
<aside class="quote no-group" data-username="sirius" data-post="4" data-topic="29829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sirius/48/12986_2.png" class="avatar"> sirius:</div>
<blockquote>
<p>Moreover, I am not quite sure how to use this piece of code. Adding these codes to the training loop?</p>
</blockquote>
</aside>
<p>It depends on your use case, so you could call this method after each iteration, in each epoch, etc.</p>

username: sirius
cooked: <p>Thank you very much and I would try this</p>

username: rzhang63
cooked: <p>Hi, I’m wondering why you added ‘torch.no_grad()’ at the top. Is it to prevent the parameters from updating? So if I wish to train the parameters aftering adding the gaussian noise, should I not include the no grad argument?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>This is just a guard to make sure this particular operation is not recorded by Autograd.<br>
It’s most likely not necessary at this point, but I’m used to write it in that way.<br>
The parameters will still be trained. Just the addition using the Gaussian noise won’t be recorded.</p>

username: youdar
cooked: <p>When I try this method I get an error in the back propagation step</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1]], which is output 0 of TBackward,.....
</code></pre>
Issues with importing and using PyTorch on Windows

username: khantroll
cooked: <p>Hi all! I’ve been trying to get ESRGAN to work, but it’s been fruitless so far. I received an error saying the detected driver for my 1050TI was outdated despite having the latest one. After some research, I found a post that recommended downgrading to PyTorch 1.4. Since then, I’ve been getting this error:</p>
<p>\AppData\Roaming\Python\Python38\site-packages\torch_<em>init</em>_.py", line 97, in </p>
<p>from torch._C import *</p>
<p>ImportError: DLL load failed while importing _C: The specified module could not be found.</p>
<p>I’ve uninstalled 1.4 and reinstalled 1.6. I’ve removed everything and reinstalled everything. I’ve removed everything and installed Anaconda in order to start over. None of that has resulted in a change.</p>
<p>I’m grateful for any suggestions.</p>
<p>Thanks!</p>
How to sample uniformly at random from the unit sphere?

username: keyboardAnt
cooked: <p>Hi,</p>
<p>What’s the most PyTorch-ist way to sample uniformly at random from the unit sphere?</p>
<p>Best regards</p>

username: keyboardAnt
cooked: <p>Sampling points <em><strong>on</strong></em> the unit sphere (i.e. points of norm 1) is possible by using BoTorch’s <code>sample_hypersphere</code>. Referece: <a href="https://github.com/pytorch/botorch/pull/369/files" rel="noopener nofollow ugc">https://github.com/pytorch/botorch/pull/369/files</a></p>
How to do dot product of two tensors

username: timbmg
cooked: <p>In numpy I can do a simple matrix multiplication like this:</p>
<pre><code>a = numpy.arange(2*3).reshape(3,2)
b = numpy.arange(2).reshape(2,1)
print(a)
print(b)
print(a.dot(b))
</code></pre>
<p>However, when I am trying this with PyTorch Tensors, this does not work:</p>
<pre><code>a = torch.Tensor([[1, 2, 3], [1, 2, 3]]).view(-1, 2)
b = torch.Tensor([[2, 1]]).view(2, -1)
print(a)
print(a.size())

print(b)
print(b.size())

print(torch.dot(a, b))
</code></pre>
<p>This code throws the following error: RuntimeError: inconsistent tensor size at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:503</p>
<p>Any ideas how a simple dot product can be conducted in PyTorch?</p>

username: SherlockLiao
cooked: <p>torch.dot() means inner product, it needs two tensor 1 D. If you want to do matrix product, you can use <a href="http://torch.mm" rel="nofollow noopener">torch.mm</a>(a, b)</p>

username: Wu_jiang
cooked: <p>when I use</p>
<pre><code class="lang-auto">std::vector&lt;at::Tensor&gt;  dcnv2_forward(
        at::Tensor im,
        at::Tensor weights,
        at::Tensor Offset,
        at::Tensor Mask,
        int64_t num_deformable_group,
        int64_t kernel,
        int64_t pad,
        int64_t stride,
        int64_t dilation,
        bool use_bias,
        at::Tensor bias
        )
{
....
 at::Tensor out = at::mm(weights,
                        col.reshape(col2d_reshape));
...
}
</code></pre>
<p>Bug report :<br>
<code>RuntimeError: /opt/conda/conda-bld/pytorch_1532581333611/work/torch/csrc/autograd/variable.h:127: Variable: Assertion</code> is_variable() || !defined() <code>failed: Tensor that was converted to Variable was not actually a Variable</code></p>
<p>I guess at::mm need Variable input , so which function should use to do multiplication？at::mm or something else?</p>

username: hoangcuong2011
cooked: <p>I guess you can use einsum as well. Here is an example:</p>
<pre><code class="lang-auto">a = torch.rand(2, 3)
b = torch.rand(2, 3)
torch.einsum('ij,ij-&gt;i', a, b)
</code></pre>
<p>I checked and the result is correct. Hope it helps.</p>

username: mathematics
cooked: <p>sometimes <code>@</code> operation come in handy when doing  <code>dot</code> products.<br>
<code>a@b</code></p>
Training an encoder-decoder (sentence generator) using a pre-trained sentence classifier

username: asunayak
cooked: <p>I have a dataset of sentences which belong to 2 categories. I have trained a classifier on this data to classify a given sentence into one of the 2 categories. Next, I want to train a generator (encoder-decoder) model to convert a given sentence from class-1 to class-2 using the pre-trained classifier. So basically it’s a style transfer in NLP. Here is a skeleton of my models:</p>
<p>Encoder:<br>
Embedding layer --&gt; LSTM [outputs= o, h]</p>
<p>Decoder:<br>
Embedding layer --&gt; LSTM --&gt; Linear --&gt; Relu --&gt; log_softmax [output= probability for each word in vocab]</p>
<p>Classifier:<br>
Encoder --&gt; Linear layer1 --&gt; Linear2 --&gt; sigmoid [output = class probability]</p>
<p>Generator:<br>
Encoder --&gt; Decoder --&gt; topK(1) [outputs = token for each word in generated sentence as floats]</p>
<p>What I am trying to do is, train the generator using error signals from pre-trained Classifier model. Whether this model would work or not is a separate question (and I would definitely love to hear feedbacks from more experienced members here). But, the major concern here is that Generator returns an array of sentence tokens (words) as floats which should then be passed to a (freezed) Classifier model which contains Embedding layer as first layer, which only accepts Long datatype. But converting float to long tensor would destroy the gradient history. From what I understand from similar embedding related questions, it’s not possible to retain gradients with type conversion. So what are my options here? Any workarounds? For example, astute readers would have noticed that even “topK/argmax” operation would break the gradients history, for which I am planning to use a Linear layer to find argmax while training. Any similar solutions for “embedding” problem too? I am quite sure people would have tried similar things but I can’t find any resources on seq2seq + classifier.</p>
<p>Note: I am not posting code to keep the post clean for better understanding. If required I can provide related sections.</p>
Checkpoint parameters

username: gnadaf
cooked: <p>What are the important parameters do we need to save in checkpoint?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Usually you save all the info of your model given by <code>model.state_dict()</code>. This contains the Parameters and Buffers.<br>
If your optimizer also has a state that is important to maintain, you can also save the optimizer’s state_dict().</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="97968">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>o save the opti</p>
</blockquote>
</aside>
<p>What about scheduler.state_dict().</p>
<p>Which parameters are the important in order to restart model from the checkpoint?<br>
(model.state_dict(), optimizer.state_dict(), scheduler.state_dict())   all these are important to restart model from checkpoint?</p>

username: albanD
cooked: <p>If you use one sure.</p>
<p>It depends a lot on your code and what you use I’m afraid so there isn’t one single overall answer here.<br>
You will need to save the states of all the components that are important and that you want to save the state of.</p>

username: gnadaf
cooked: <p>I am using tranformer model, (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a>)</p>

username: albanD
cooked: <p>In this case these 3 should be good yes.</p>
Building errors from running libtorch

username: Chen0729
cooked: <p>HI, All</p>
<p>I get a building error on libtorch</p>
<p>torch::Tensor x = x1.view({-1, input_dim});</p>
<p>error: no matching function for call to ‘at::Tensor::view()’<br>
torch::Tensor x = x1.view({-1, input_dim});</p>
<p>Any suggestions of how to modify it please ?</p>
<p>The python code is:<br>
x = x.view(-1, self.input_dim)</p>
<p>Much appreciated.<br>
^</p>

username: Lin_Jia
cooked: <p>Are you trying to reshape the tensor? If yes, there is a reshape API call on Tensor, could you try:<br>
x = x1.reshape({-1, input_dim}) ?</p>

username: Chen0729
cooked: <p>Thanks for suggestions</p>
Illegal memory access when I use GroupNorm

username: tcapelle
cooked: <p>I don’t understand why this small code produces an Illegal memory access:</p>
<pre><code class="lang-auto">class PhyCell_Cell(nn.Module):
    def __init__(self, ch_in, hidden_dim, ks=3, bias=True, group=True):
        super().__init__()
        padding = ks // 2
        bias = bias
        self.f = nn.Sequential(
                 nn.GroupNorm(4, ch_in) if group else nn.BatchNorm2d(ch_in),   
                 nn.Conv2d(ch_in, hidden_dim, ks, padding=padding),
                 nn.Conv2d(hidden_dim, ch_in, kernel_size=(1,1)))

        self.convgate = nn.Conv2d(2*ch_in,
                                  ch_in,
                                  kernel_size=(3,3),
                                  padding=(1,1), 
                                  bias=bias)

    def forward(self, x, hidden=None): 
        "x ~[batch_size, hidden_dim, height, width]"  
        if hidden is None: hidden = self.init_hidden(x)
        hidden_tilde = hidden + self.f(hidden)
        combined = torch.cat([x, hidden_tilde], dim=1)
        combined_conv = self.convgate(combined)
        K = torch.sigmoid(combined_conv)
        next_hidden = hidden_tilde + K * (x - hidden_tilde)
        return next_hidden
    
    def init_hidden(self, x):
        bs, ch, h, w = x.shape
        return one_param(self).new_zeros(bs, ch, h, w)
</code></pre>
<p>small reproductible example on colab:</p>
<p><a href="https://colab.research.google.com/drive/1YCEbnDHLWnZ49kGfg4FX0EBQ2RTbGaat?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1YCEbnDHLWnZ49kGfg4FX0EBQ2RTbGaat?usp=sharing</a><br>
If I replace with BatchNrom (what I am doing) it works.</p>

username: ptrblck
cooked: <p>Thanks for reporting this issue and providing the code snippet!<br>
We could reproduce it and Xiao fixed it in <a href="https://github.com/pytorch/pytorch/pull/44863">this PR</a>.<br>
It should thus be available in the next nightly build.</p>

username: tcapelle
cooked: <p>Thanks!<br>
some older version of pytorch seems to work.</p>
Doing a Linear Transform over a dimension

username: Alex_Sugar
cooked: <p>Hey All, I’m learning how to use pytorch and am wondering what the best way to do the following is:</p>
<p>in a custom module, I have an intermediate tensor that has a shape like [2,8,16], I’d like to apply a linear layer transform, say from 8 -&gt; 4, in the middle dimension and get out a tensor of shape [2,4,16]. The obvious way to do it would be to use a for loop, but I was wondering if I ran this in a large network, with my layer in the GPU, would pytorch automagically parallelize this? Or perhaps I should extract the weights of the linear layer and formulate it as a matrix multiply? What’s the most efficient way to do this?</p>
<p>Thanks for any help</p>

username: ptrblck
cooked: <p>You could permute the dimensions and just apply a linear layer:</p>
<pre><code class="lang-python">x = torch.randn(2, 8, 16)
x = x.permute(0, 2, 1).contiguous()
lin = nn.Linear(8, 4)
output = lin(x)
</code></pre>
<p>The <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Linear">docs</a> explain the behavior of the input and output shapes.</p>

username: Taro_Kiritani
cooked: <p>You could use Conv1d with kernel_size=1.</p>
GPU support for quantization

username: Haxxardoux
cooked: <p>Will quantization be supported for GPUs anytime soon? I have a project where evaluation speed is a very major concern and would love to use quantization to speed it up.</p>
<p>I see the CPU quantization tutorial on the docs was written about 6 months ago, so I am really just curious if this is on the developers’ radar at all and if we can expect this <em>eventually</em> or in the near future.</p>

username: jerryzh168
cooked: <p>We will be considering the GPU option in the second half of the year, but I think it probably won’t be a high priority item.</p>

username: Haxxardoux
cooked: <p>is there a particular reason it is not a high priority? i am still a student but was under the impression that inference with large models was typically done on GPUs, and quantization would be very beneficial</p>

username: jerryzh168
cooked: <p>this depends on our internal customers, we haven’t decided yet.</p>

username: KevinPD66
cooked: <p>I’m not sure if there is a voting process but we (as a company) are using pytorch in our production process and inference speed of our custom BERT model is critical for us. In my opinion to get more adoption of pytorch in production and commercial applications inference speed is going to be critical and this feature would be a huge step forward for that. My two cents.</p>

username: berserkr
cooked: <p>I am looking to contribute some work in the area of quantization for multiple architectures including fpgas and gpus. Any suggested guides on how to get started with contributing to pytorch? Cheers!</p>
How to use torch.utils.bottleneck for profiling a module (vs a script)?

username: Shreeyak
cooked: <p>I’ve used setup.py to create a package of my code. The training script is a module that I invoke with “python -m”. Incidentally, it also uses hydra and so it’s command line args are a diff format:</p>
<pre><code class="lang-auto">python -m train.py arg1=foo arg1.bar=foo
</code></pre>
<p>How do I use torch.utils.bottleneck to profile my code?</p>
<pre><code class="lang-auto">$ python -m torch.utils.bottleneck train.py
Running environment analysis...
Running your script with cProfile
Traceback (most recent call last):
  File "/home/shrek/anaconda3/envs/cloud/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/shrek/anaconda3/envs/cloud/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/shrek/anaconda3/envs/cloud/lib/python3.8/site-packages/torch/utils/bottleneck/__main__.py", line 231, in &lt;module&gt;
    main()
  File "/home/shrek/anaconda3/envs/cloud/lib/python3.8/site-packages/torch/utils/bottleneck/__main__.py", line 209, in main
    cprofile_prof = run_cprofile(code, globs)
  File "/home/shrek/anaconda3/envs/cloud/lib/python3.8/site-packages/torch/utils/bottleneck/__main__.py", line 74, in run_cprofile
    exec(code, globs, None)
  File "train.py", line 17, in &lt;module&gt;
    from .config_parse.train import TrainScriptConfig
ImportError: attempted relative import with no known parent package
</code></pre>
RandomRotation transform

username: Maicol_Polvere
cooked: <p>Good Morning,<br>
I would like to know if there is a way of knowing what random angle is used to perform transformation</p>

username: ptrblck
cooked: <p>You have to define the angle by passing the <code>degrees</code> to <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomRotation"><code>torchvision.transforms.RandomRotation</code></a>.<br>
From the docs:</p>
<blockquote>
<p><strong>degrees</strong>  ( <em>sequence</em>  <em>or</em>  <a href="https://docs.python.org/3/library/functions.html#float"> <em>float</em> </a>  <em>or</em>  <a href="https://docs.python.org/3/library/functions.html#int"> <em>int</em> </a>) – Range of degrees to select from. If degrees is a number instead of sequence like (min, max), the range of degrees will be (-degrees, +degrees).</p>
</blockquote>
Export torch.tensor to csv

username: nicoliKim
cooked: <p>Hello everyone.<br>
Very simple and straightforward question.</p>
<p>I have a model that at the end of every training epoch generates a torch.tensor of dim[6].<br>
[[a,b,c,d,e,f,g]]</p>
<p>I want to append it to a .csv file (at the end of each epoch) in order to the plot how the tensor evolves in time during training.</p>
<p>Is there a simple way to do that? Didn’t find anything online so far.</p>
<p>Thanks in advance. Best, kim</p>

username: rasbt
cooked: <p>I guess you could just use Numpy for that</p>
<pre><code class="lang-auto">import numpy as np        

your_file = open('filename', 'ab')
for epoch in range(num_epochs):
    # ...
    np.savetxt(your_tensor.numpy())
your_file.close()
</code></pre>

username: Saurabh_Kale
cooked: <p>Can you elaborate (‘filename’, ‘ab’). What is ab?</p>

username: ptrblck
cooked: <p><code>ab</code> opens a file for <strong>a</strong>ppending in <strong>b</strong>inary format. You can find the different modes <a href="https://www.tutorialspoint.com/python/python_files_io.htm">here</a>.</p>
Convert 1d tensor to 2d tensor

username: adithya1
cooked: <p>Hi I am new to pytorch , I just wanted to convert 1 dimensional tensor to 2 dimensional tensor but when I use tensor.view() it my code still throws a dimensionality problem.please help</p>

username: ajbrock
cooked: <p>You can’t change the number of elements in the tensor, so you’re likely calculating the dimensions wrong. If you have a 100 element tensor, you can’t view it to x.view(20,10) because that would require you to have 20*10=200 elements. Try just taking a 100 element tensor and running x.view(1,100,1) and you’ll see how you can arbitrarily add dimensions.</p>

username: dhpollack
cooked: <p>If it’s a tensor and not a variable you can also use unsqueeze to add a dummy dimension</p>

username: OilGains
cooked: <p>I have done like this before:</p>
<pre><code>import torch
x_train  = torch.linspace(-1, 1, 101)    # 1D tensor
print(x_train.size())
# torch.Size([101])   

x_train = x_train.view(101, 1)          # convert to 2D tensor
print(x_train.size())
# torch.Size([101, 1])</code></pre>

username: satbek
cooked: <p>If you have a tensor <code>img</code> with a size</p>
<p><code>torch.Size([784])</code></p>
<p>and you want to convert it to a size of</p>
<p><code>torch.Size([1, 784])</code></p>
<p>you can call the <code>resize_</code> method like below:</p>
<p><code>img.resize_(1, 784)</code></p>

username: ptrblck
cooked: <p>I would consider the usage of <code>resize_</code> to be dangerous and applicable for advanced use cases, and would thus recommend to use <code>tensor.view(1, -1)</code> or <code>tensor.unsqueeze(0)</code> for this use case.<br>
From the docs of <code>resize_</code>:</p>
<blockquote>
<p>This is a low-level method. The storage is reinterpreted as C-contiguous, ignoring the current strides (unless the target size equals the current size, in which case the tensor is left unchanged). For most purposes, you will instead want to use <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view"> <code>view()</code> </a>, which checks for contiguity, or <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape"> <code>reshape()</code> </a>, which copies data if needed. To change the size in-place with custom strides, see <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.set_"> <code>set_()</code> </a>.</p>
</blockquote>
How to process each item in batch at the same time?

username: DoubleMuL
cooked: <p>Hi, I’m trying to process each item in batch at the same time.</p>
<p>Ex)<br>
I have a batch (8 x 3 x 224 x 224) where its batch size is 8 and l also have a function whose input is one image (3 x 224 x 224).</p>
<p>Then, I want to repeat the same function on each image. I can do this by iterating all items like:<br>
for i in range(data.shape[0]):<br>
input[i] = function(inputs[i])</p>
<p>But I think this loop is too time-consuming. Is there a more efficient way to process a batch of data at the same time?</p>

username: ptrblck
cooked: <p>It depends on the implementation of your <code>function</code>.<br>
PyTorch methods usually accept batched data, so if you are using internally in <code>function</code> PyTorch methods, it could directly work.</p>
Difference in accuracy when setting model.eval() and not setting it

username: Megh_Bhalerao
cooked: <p>Hi all,<br>
I am working on a <strong>semantic segmentation</strong> problem. I train my model using <strong>UNet</strong> like architecture, and save the model weights. So, now when I try to generate segmentations from this saved model, basically the inference part, I am facing an unusual issue. When I set <code>model.eval()</code> before the inference loop, my <strong>dice score is 0.4ish</strong> and when I don’t set it (I think, in this case it defaults to <code>model.train()</code>), dice score is <strong>around 0.6ish</strong>. This has perplexed me. I know that <code>model.eval()</code> switches off the <code>Batchnorm</code> and <code>Dropout</code> layers, but did not know that there can be such variation in accuracy. Can someone help me with this?<br>
Thanks!</p>

username: ptrblck
cooked: <p>Did you see the same effect during training by observing the training and validation loss (which would correspond to calling <code>model.train()</code> and <code>model.eval()</code>) or are you only seeing this issue after loading the trained model?</p>
CNN code for classification of three classes of image

username: Fatiha
cooked: <p>I am new to CNN and trying to train the images and then test them and then classify the type of image. I am using python and PYQT designer for GUI. My task is to perform classification on three types of Brain Tumor MRI images. All the images are assigned labels.<br>
My GUI is stated as follows:</p>
<ol>
<li>Load Data (complete dataset)  for training by a button named as “Load”.</li>
<li>Automatically split the dataset  into Testing and Training by a button named as “split”.</li>
<li>Train the dataset by a button named as “Training”.</li>
<li>Test the image by a button “Testing”.</li>
<li>The result displays the image with its label.</li>
</ol>
<p>Any help in this regard will be appreciated. Thank you.</p>

username: ptrblck
cooked: <p>The splitting of the workload into these steps sounds reasonable.<br>
Where are you stuck exactly and what have you already implemented?</p>

username: Fatiha
cooked: <p>I am stuck in testing and training split part, although i’m following standard format of piece of code used for splitting.</p>

username: ptrblck
cooked: <p>Could you post your code snippet showing how you are splitting and what is currently not working?</p>

username: Fatiha
cooked: <p>There is a problem in file reading or data loading. Could you please help me that how can I load my custom dataset for classification?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> gives you a good overview on how to create a custom <code>Dataset</code>. If you get stuck, please post some code so that we could have a look at it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Fatiha
cooked: <p>I have my dataset of 3000 images. I want to load them and then split them into testing and training ratio.</p>

username: ptrblck
cooked: <p>You could either use <code>torch.utils.data.random_split</code> and provide the lengths you would like to use for the random splitting or alternatively, if you want to use a stratified split, you could use <code>sklearn</code>'s <code>train_test_split</code> as seen <a href="https://discuss.pytorch.org/t/how-to-use-sklearns-train-test-split-on-pytorchs-dataset/31521/2">here</a>.</p>
How to shut down all processes with 'Ctrl + C' when using PyTorch Distributed DataParallel training?

username: feiyuhuahuo
cooked: <p>I was using Distributed DataParallel to train a model. I ran my code on two processes with two GPUs (one process one GPU). After I pressed ‘Ctrl + C’ in terminal, one process was shut down and the other one remains running. This can be observed by <code>top</code> or <code>nvidia-smi</code> command. So how to shut down all the processes in my terminal?</p>

username: pritamdamania87
cooked: <p>How did you launch the two processes? Did you use <code>torch.distributed.launch</code> or some other mechanism?</p>

username: feiyuhuahuo
cooked: <p>@ pritamdamania87  Yes, I use <code>python -m torch.distributed.launch</code> to run my code. And with <code>Ctrl+C</code> to shut down the training, some processes  are not closed. I must kill them manually.</p>

username: BramVanroy
cooked: <p>This is an issue in many multiprocessing tasks, not just PyTorch. The best that you can do is iteratively calling ctrl+C until all processes are terminated.</p>
Changing the kernel points in CNN

username: COA
cooked: <p>Hi,</p>
<p>I would like to modify the coordinates of points used in the kernel during the convolution operation for an entire CNN. I don’t want to use a regular grid kernel but to adapt the shape of it depending on its location on the “image”.</p>
<p>Would you have tips on how to do that in Pytorch?<br>
Can I simply add functions in my Python main and keep the Pytorch engine intact or do I need to modify the backend in order to keep the GPU/CUDA performances?</p>
<p>Thank you,<br>
CO</p>

username: ptrblck
cooked: <p>I would recommend to not focus on the performance and backend at the beginning as your use case is not trivial.<br>
Would the grid be used in fixed windows or would the window size also be changed depending on the image location?<br>
In the first step, I would recommend to use either a nested loop and make sure your method is working as expected and then use <code>unfold</code> to create the windows and calculate the output via a matrix multiplication, if that fits your use case.</p>

username: COA
cooked: <p>Thanks for your answer.<br>
The grid size will stay the same, I just want to modify the coordinates of the feature points used.</p>
<p>If it helps for understanding, I did it in Caffe by modifying the im2col operation. Instead of taking a regular grid to compute the output of the function, I have just modified the position of the points depending on the location of the kernel center in the feature maps.</p>
<p>And if I have understood correctly the documentation, the <em>unfold</em> pytorch function does exactly the same operation. So it looks like the function I need to modify, isn’it?<br>
So do you think it’s possible to modify the <em>im2col</em> pytorch operation, rebuild pytorch framework and directly use it for already built networks?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="COA" data-post="3" data-topic="97475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coa/48/29169_2.png" class="avatar"> COA:</div>
<blockquote>
<p>So do you think it’s possible to modify the <em>im2col</em> pytorch operation, rebuild pytorch framework and directly use it for already built networks?</p>
</blockquote>
</aside>
<p>That could be possible, but I would rather create a new method (via a custom C++ extension) than manipulating native functions.<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">This tutorial</a> shows how to implement such a method.</p>
Increase my speed with 8 GPUs

username: slavavs
cooked: <p>My model works with a packet size of 128 by 1 GPU. I want to speed up my work and use 8 GPUs. But I cannot increase the packet size. Using a packet size of 128 with 8 GPUs will not increase my speed. Is there a tutorial to speed things up on 8 GPUs without increasing the batch size?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="slavavs" data-post="1" data-topic="97833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>Is there a tutorial to speed things up on 8 GPUs without increasing the batch size?</p>
</blockquote>
</aside>
<p>Why would want to keep the global batch size constant and split the computation to different devices?<br>
Note that GPUs will usually achieve a higher utilization for a larger workload (which would e.g. correspond to the batch size).</p>

username: slavavs
cooked: <p>Sorry, but I didn’t understand your answer. I’ll try to ask again. In my model with one GPU, the packet size is 128. Accordingly, I do loss.backward () in increments of 128. If I have a packet size of 128 * 8 = 1024, then my model cannot learn.</p>

username: ptrblck
cooked: <p>Do you mean the batch size by “packet size”? If so, could you explain what you mean by</p>
<aside class="quote no-group" data-username="slavavs" data-post="3" data-topic="97833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>Accordingly, I do loss.backward () in increments of 128.</p>
</blockquote>
</aside>
<p><code>loss.backward()</code> will calculate the average loss for the input and target batch by default and will thus compute the average gradients for all parameters w.r.t. this loss.</p>
<aside class="quote no-group" data-username="slavavs" data-post="3" data-topic="97833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>If I have a packet size of 128 * 8 = 1024, then my model cannot learn.</p>
</blockquote>
</aside>
<p>If you are increasing the batch size in a data parallel model you might need to adapt some hyperparameters such as the learning rate if your model is sensitive to the increased batch size.</p>
FP16 or FP32 accumulate during AMP forward?

username: Sharpy
cooked: <p>When GPU training with automatic mixed precision, does forward pass use FP16, or FP32 accumulate? I’m asking this, because cheap GPUs have FP32 accumulate tensor core performance intentionally halved and NVidia claims that FP16 accumulate is okay for inference, but not for training, so I’m guessing, good enough for forward, but not backward pass. Would like to better assess  whether to get something like Titan RTX, or like 3090.</p>

username: ptrblck
cooked: <p>FP32 accumulate is used during mixed-precision training.</p>

username: Sharpy
cooked: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"><br>
What about inference?</p>

username: ptrblck
cooked: <p>Inference can also be used with amp and would also apply the FP32 accumulation.<br>
You could try to use FP16 accumulations using custom CUDA extensions and check if you see a speedup.</p>

username: Sharpy
cooked: <p>custom CUDA extensions = ?</p>
<p>Googled for any existing solutions for PyTorch, could only find some mentions of FP16 for inference in TensorFlow. Here <a href="https://blog.tensorflow.org/2019/06/high-performance-inference-with-TensorRT.html" rel="noopener nofollow ugc">https://blog.tensorflow.org/2019/06/high-performance-inference-with-TensorRT.html</a> and <a href="https://www.slideshare.net/VadimSolovey/gan-training-with-tensorflow-and-tensor-cores" rel="noopener nofollow ugc">https://www.slideshare.net/VadimSolovey/gan-training-with-tensorflow-and-tensor-cores</a> for example.</p>
Save loss parameter in checkpoint

username: gnadaf
cooked: <p>Is it necessary to save <strong>loss</strong> while saving checkpoint?<br>
Is it an important parameter?</p>
<p>If I save checkpoint with <strong>model.state_dict(), optimizer.state_dict()</strong> and <strong>scheduler.state_dict()</strong>, when I restart training from checkpoint, I see first epoch has different loss value</p>

username: tom
cooked: <p>loss isn’t a parameter but it is computed, so you don’t need to save it except for informational purposes.<br>
It would be expected that the loss is different after loading and running the model again, just like it would be if you had run the original model right after saving it.</p>
<p>Best regards</p>
<p>Thomas</p>

username: gnadaf
cooked: <p>Thanks <a class="mention" href="/u/tom">@Tom</a> for the detailed answer</p>

username: gnadaf
cooked: <p>What are the important parameters do we need to save in checkpoint?</p>
Weighted Sum of Matrices

username: Ahmed_Abdelaziz
cooked: <p>I have 2 matrices A, B. A has a size of (X, 500, 8) and B has the size of (X, 8, 1). So A basically is X records of shape (500,8). B holds the weights for each x in X. I would like to calculate the weighted sum for each x so that I end up with a vector (500,1) for each x. The final result should be (X, 500).</p>
<p>I would like to think of it as a group of dot products of shapes: (500x8), (8x1). Any idea how to do that in pytorch?</p>

username: Caruso
cooked: <p>Hi,</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.bmm.html?highlight=bmm#torch.bmm" rel="noopener nofollow ugc">Batch MatMul</a> could be what you mean, so each matix gets multiplied with its equivalent.</p>

username: Ahmed_Abdelaziz
cooked: <p>Exactly what I wanted, thank you!</p>