akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Views for Cyclic Permutation or Zero Padding

Florian_Bruckner: <p>Hi, I am wondering whether it is possible to make a view of an array with does cyclic permutation or zero padding. With view I mean that no data should be copied.</p>
<p>This would be very helpful for calculating finite differences with certain boundary conditions.<br>
torch.diff allows to append/prepend some data, but for multidimentional arrays it is very cumbersome, and I think that it also copies the data finally.</p>
<p>Thanks for any advice<br>
best wishes<br>
Florian</p>
CUDA out of memory when back propgation

Omega_Ma: <p>Hi, I have a large-size network that is out of the memory of one GPU. So I put different layers into different GPUs. This has fixed the 'out of memory error when loading the model. However, the error still shows in the step of backpropagation.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/scratch/project_2005641/THz_DNN/THz_Huge.py", line 379, in &lt;module&gt;
    Loss_cache, Lr_list = train_model()
  File "/scratch/project_2005641/THz_DNN/THz_Huge.py", line 127, in train_model
    loss.backward()  # backpropagation
  File "/usr/local/lib64/python3.9/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib64/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 16.00 GiB (GPU 2; 31.75 GiB total capacity; 16.01 GiB already allocated; 14.78 GiB free; 16.02 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>Any solution to fix this?</p>

pentachris: <p>The backward pass locates additional memory on your GPU to store each parameter’s gradient value. Only leaf tensor nodes (model parameters and inputs) get their gradient stored in the grad attribute. Therefore the memory usage is increasing between the inference (forward pass) and the backpropagation. So its no suprise that you get this message if it even had problems only by inference.<br>
Iam no expert in GPU utilization but you could try:</p>
<ol>
<li>Make your Network smaller or try to get a more powerful computer (the obvious ones if its possible :D)</li>
<li>If you have more GPUs left, try to experiment and use everything you have on more different layers</li>
<li>You can go to the documentation for memory management and pytorchs cuda config or watch some toutorials/read through the internet about GPU utilization</li>
</ol>
<p>Best of luck!</p>

Omega_Ma: <p>Thanks for the suggestion.</p>
Aloccating Required Memory for Each GPU in Data Parallelism Automatically

Tm4: <p>In my program, I use three pretrained encoders and for training my model I am using DataParallel. I want to use three Gpus that each of them has different free capacity. I am usung DataParallel in the following scenario:</p>
<pre><code>dev = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nn.DataParallel(model, output_device=[0,1,2], device_ids = [0,1,2])
model = model.to(dev)
</code></pre>
<p>Unfortunately, the program doesn’t distribute the required memory on the three device automatically and it wants to use beyond the capacity of one of the devices though other two devices have required free space. It shows the error that one of the devices doesn’t have enough memory but at the same time most of the memory space of other two gpus are free. How can I fix that?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2be4e6665cc574cde03eb56294a79b603935012a.png" data-download-href="https://discuss.pytorch.org/uploads/default/2be4e6665cc574cde03eb56294a79b603935012a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2be4e6665cc574cde03eb56294a79b603935012a.png" alt="image" data-base62-sha1="6giWJ6aonQw53CnfNL6VrVoG2Ey" width="690" height="40" data-dominant-color="DBCBCA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1253×73 7.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How could I divide list with distributed.scatter

JuyiLin: <p>Currently I have this code:</p>
<pre><code class="lang-python">            if rank == 0:
                x = torch.arange(3.)
                scatter_list = list(torch.tensor_split(x, 3))
            else:
                scatter_list = []
            dist.scatter_object_list(output, scatter_list, src=0)
</code></pre>
<p>Now I have a list have 6 elements, I want to divide them, and each rank obtains 2 elements, can I use scatter to achieve it?<br>
Thank you for your help!</p>
Proper way to access to get the output of the i-th layer of a model

Vendrick17: <p>Hello, I am working with <code>functorch</code> Jacobian, I am using it to get the Jacobian of an intermediate layer <code>i</code> in my model with respect to the input <code>z</code>, however, I have two methods to access the i-th layer, the first is by just taking the model indexes, as follows:</p>
<pre><code class="lang-auto">i_layer_out = G.model[:i+1](z)
</code></pre>
<p>and the second is using recurssion:</p>
<pre><code class="lang-auto">inp = z
for j in range(i+1):
     inp = G.model[j](inp)
i_layer_out = inp   
</code></pre>
<p>The output of the model slightly varies depending on the approach, but the Jacobians are different, so I was wondering what is the best method to get the output of the intermediate layer, thanks for any comment.</p>
What is the spatial conv?

YuhskeHujisaki: <p>What is the spatial conv?<br>
I would like to create the model shown in the figure below. What are spatial and temporal conv? And what kind of pytorch should be structured?<br>
<img src="https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8840768/8851681/8852362/song2-p8-song-small.gif" alt="Fig. 2. - Detailed nework architecture of the representation module." width="405" height="500"></p>

JuanFMontesinos: <p>Most likely it’s just a categorical meaning. You are probably using certain date which represents space and time. The author denotes the convolution as “temporal” meaning the kernel spawns in the temporal dim. Analogously to the spatial one.</p>
<p>PD: it’s just a 2d convolution.</p>
Self define layer not autograd

Hsu_Chih-Shen: <p>I tried to customize a layer to automatically adjust the brightness and contrast by a linear layer to determine the coefficient, but could not backpropagation, my layer is:</p>
<pre><code class="lang-auto">class MyLinear(torch.autograd.Function):
    def __init__(self, in_units, units):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(in_units, units))
        self.bias = nn.Parameter(torch.randn(units,))
        self.reset_parameters()


    def reset_parameters(self):
        stdv = 1. / math.sqrt(self.weight.size(0))
        self.weight.data.uniform_(0, stdv)

    def forward(self, x1, x2):
        x1 = torch.matmul(x1, self.weight.data) + self.bias.data
        print(x1 ** 2)
        x1 = nn.functional.relu(x1)
        print(x1 ** 2)
        print('-' * 50)
        img_new = torch.zeros((x2.shape[0], 3, size, size))
        for i in range(0, x2.shape[0]):
            for j in range(0, x2.shape[1]):
                if j == 0:
                    coef1 = round((x1[0][0] * x1[0][0]).item(), 8)
                    coef2 = round((x1[0][1] * x1[0][1]).item(), 8)

                elif j == 1:
                    coef1 = round((x1[0][2] * x1[0][2]).item(), 8)
                    coef2 = round((x1[0][3] * x1[0][3]).item(), 8)

                elif j == 2:
                    coef1 = round((x1[0][4] * x1[0][4]).item(), 8)
                    coef2 = round((x1[0][5] * x1[0][5]).item(), 8)
                # print(coef1, coef2)
                img = x2[i, j, :, :]
                img = F.adjust_brightness(img, coef1)
                img = img.reshape(-1, size, size)
                img = F.adjust_contrast(img, coef2)
                
                if j == 0:
                    z = img.reshape(1, size, size).cuda()
                else:
                    z = torch.cat((z, img), 0)
            z = z.reshape(1, 3, size, size)
            img_new[i] = z.clone().detach()
        img_new = img_new.clone().detach().requires_grad_(True)
        
        return img_new.cuda()
</code></pre>
<p>And my model is:</p>
<pre><code class="lang-auto">module = torchvision.models.efficientnet_b4(pretrained=True)
class bsEfficientnetb4(nn.Module):
    def __init__(self):
        super(bsEfficientnetb4, self).__init__()
        self.brightcontrast = MyLinear(1, 6)
        self.eff = module.features
        self.avgpool = module.avgpool
        self.classifier = module.classifier
        
        
    def forward(self, x1, x2):
        x = self.brightcontrast(x1, x2)
        x = self.eff(x)
        x = self.avgpool(x)
        x = x.reshape(-1, 1792)
        x = self.classifier(x)
        return x
</code></pre>
<p>My input x1 is:</p>
<pre><code class="lang-auto">x1 = torch.tensor([[1.,]], requires_grad=True).cuda()
</code></pre>
<p>And x2 is image, I’d like to get your suggestions for modifications, thanks!</p>

soulitzer: <p>Doing things like calling .item() or calling .detach() will break the graph. You’ll want to avoid doing those things during your forward if you’d like to backprop through that part of the computation.</p>
Parallelize n-fold validation on k different GPU in a single process

elaine: <p>Hi everyone,</p>
<p>I would like to parallelize the folds training in n-fold validation by splitting the folds on k different GPUs, i.e. data parallelism at the fold level.</p>
<p>I have read some tutorials for module-level parallelism, and am wondering if there is a similar tutorial for parallelism at the fold level.</p>
<p>I have written up a pseudo-code, but I am unsure of how to implement these. I have tried using Python’s concurrent library. Nevertheless, it does not seem to work for multiple GPUs.</p>
<pre><code class="lang-python">for idx_fold, (index_train, index_test) in enumerate(kf.split(data)):
    if idx_fold % 2 == 0:
        # train on the 1st GPU
        # and when the 1st GPU training finished, it will train the next fold allocated to the GPU
    else:
        # train on the 2nd GPU
        # and when the 1st GPU training finished, it will train the next fold allocated to the GPU
</code></pre>
<p>In addition, it would be great if all the distributed training on k GPU can be regarded as <strong>a single job on a slurm GPU server</strong> (as seen by the admin).</p>
<p>Thanks a lot for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
Change learning rate

abolfazl_saheban: <p>hello, I’m going to train a model with an SGD optimizer, and I want to divide the learning rate by a factor of 10 when iteration number reaches a specific number. How should I do that?</p>

srishti-git1110: <p>Hi, please have a look at pytorch learning rate schedulers to select the one that fits your use case.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/abolfazl_saheban">@abolfazl_saheban</a>,</p>
<p>You can do something like this,</p>
<pre><code class="lang-auto">if(epoch == specific_epoch):
  for group in optim.param_groups:
    group['lr'] /= 10
</code></pre>
<p>Although, sharing a minimal reproducible example does help people when debugging problems!</p>

abolfazl_saheban: <p>Hello, I saw them but they talked about decreasing at specific learning rate not iteration</p>

abolfazl_saheban: <p>hello, thanks for your attention<br>
I wrote a similar code for specific iteration but it didn’t work, so I created this topic<br>
are you sure this will be worked for decreasing at specific iteration?</p>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example for your problem?</p>

abolfazl_saheban: <p>Hello, this is my training script and I want change  learning rate during one epoch.</p>
<pre><code>optimizer = optim.SGD(retinanet.parameters(), lr=0.01, momentum=0.9, weight_decay=0.0001)

scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=1, verbose=True)

loss_hist = collections.deque(maxlen=500)

retinanet.train()
retinanet.module.freeze_bn()

print('Num training images: {}'.format(len(dataset_train)))

epochs = 100

for epoch_num in range(epochs):

    retinanet.train()
    retinanet.module.freeze_bn()

    epoch_loss = []



    for iter_num, data in enumerate(dataloader_train):

        
        try:
            optimizer.zero_grad()

            if torch.cuda.is_available():
                classification_loss, regression_loss = retinanet([data['img'].cuda().float(), data['annot']])
            else:
                classification_loss, regression_loss = retinanet([data['img'].float(), data['annot']])
                
            classification_loss = classification_loss.mean()
            regression_loss = regression_loss.mean()

            loss = classification_loss + regression_loss

            if bool(loss == 0):
                continue

            loss.backward()

            torch.nn.utils.clip_grad_norm_(retinanet.parameters(), 0.1)

            optimizer.step()

            loss_hist.append(float(loss))

            epoch_loss.append(float(loss))

            print(
                'Epoch: {} | Iteration: {} | Classification loss: {:1.5f} | Regression loss: {:1.5f} | Running loss: {:1.5f}'.format(
                    epoch_num, iter_num, float(classification_loss), float(regression_loss), np.mean(loss_hist)))

            del classification_loss
            del regression_loss
        except Exception as e:
            print(e)
            continue

    scheduler.step(np.mean(epoch_loss))
</code></pre>
Pros and cons of minibatch

robinho: <p>Through some small scale tests with ~100 epochs, batchsize~1e3, minibatchsize~(1e2,1e3), minibatch training times seem to <strong>drop</strong> monotonically with an <strong>increasing</strong> minibatchsize. Then what’s the benefit of using minibatch at all when using a whole batch minimizes the time?</p>

ptrblck: <p>Larger models usually won’t be able to accept the entire dataset due to memory limitations, so I assume you are working on a small example, which completely fits into your GPU (e.g. MNIST with a small CNN).<br>
If so, then also check the generalization error by calculating the final validation loss, as smaller batches commonly yield a better generalization compared to the full dataset training.</p>

robinho: <p>thanks so much. Then what about the training time and how is it related to the minibatch size?</p>

robinho: <p>thanks so much. Then what about the training time and how is it related to the minibatch size?</p>

pentachris: <p>Smaller batches (~2-32 for image recognition) are usually better because the network parameters are updated every time you run through a batch with an arbitrary batch size. If you have larger batch sizes then the training speed will decrease (because gradient calculation takes a lot of time and the gradients are only calculatet once for each batch). The caviat is that larger batch sizes dont perform that well especially on more difficult data (worse generalization). And the performance of a model is way more important then the time it takes to train a model. So its almost a nobrainer to take smaller batches. You can simply experiment by only changing your batch size (2, 4, 8, 16, 32, 64, … for example) and see how it performs on your evaluation dataset.</p>
Pad should be smaller than or equal to half of kernel size

tsp_t: <p>torch.nn.functional.avg_pool2d make error.</p>
<pre><code class="lang-auto">RuntimeError: pad should be smaller than or equal to half of kernel size, but got padW = 5, padH = 5, kW = 9, kH = 9
</code></pre>
<p>However, this is intended usecase.</p>
<p>I don’t know why pytorch’s implementation chose <code>assert padW &lt;= kW//2 and padH &lt;= kH // 2</code> instead of <code>assert padW &lt; kW and padH &lt; kH</code>.</p>
<p>There is an inconvenience in that padding for input must be performed before avgpool2d.</p>
<p>I’m asking if it’s an inevitable choice for pytorch’s optimization or if it’s a bug.<br>
Or is the problem solved in the latest pytorch version?</p>
How to define a 4D observation space in gym

Majid_Shirazi: <p>Dear all, I am new to reinforcement learning and I wanna create a gym 4D environment, where it is a 468x225x182x54 space. And every location in this space has a unique reward (or penalty). And my agent (e.g. rabbit) can jump anywhere in this space and will be rewarded based on where it is. I thought my action space could be defined as</p>
<pre><code class="lang-auto">class CustomEnv(gym.Env):
    def __init__(self):
           self.action_space = gym.spaces.MultiDiscrete([468, 225, 182, 54])


</code></pre>
<p>For example</p>
<pre><code class="lang-auto"> print(CustomEnv.action_space.sample())
[172 54 101 37]
</code></pre>
<p>so, my agent collects the reward of the location [172 54 101 37].<br>
I want the step function for episodes of the game be like a rabbit make a jump, collects the reward and return the reward and the state of where it jumped in this 4D space.<br>
However, I don’t know how should I define my observation space and I really appreciate your help.</p>

vmoens: <p>The implementation you suggest seems fine, what would be the problem with it? What’s blocking you from using that?</p>
A question about normalisation ranges and their effectiveness

zezimabig: <p>Hello PyTorch community!</p>
<p>I have a question regarding the use of different normalisation ranges and their benefits in training a neural network model.</p>
<p>Currently, I have normalised my inputs to the neural network to be between 1.0 and 10.0 (reinforcement learning), but I was wondering if a normalisation range of between -1.0 and 1.0 or even 0.0 and 1.0 will improve the training speed and stability.</p>
<p>Additional information: The neural network uses the tanh activation function (PPO) with 3 layers of 64 nodes each.</p>
<p>This is the normalisation code block I am using</p>
<pre><code class="lang-auto">from sklearn import preprocessing
def normalisation_of_vector(self, vector, lower_bound = 1, upper_bound = 10):
        '''
        Might change to between 0 and 1 or -1 and 1
        '''
        np_list = np.array(vector).reshape(-1,1)
        normalization_scaler = preprocessing.MinMaxScaler(feature_range=(lower_bound, upper_bound))
        normalized_np_list = normalization_scaler.fit_transform(np_list)
        normalized_literal_list = normalized_np_list.tolist()

        flattened_list = functools.reduce(operator.concat, normalized_literal_list)

        rounded_flattened_list = list(np.around(np.array(flattened_list), 2))

        return rounded_flattened_list
</code></pre>
<p>Any help is greatly appreciated!</p>

vmoens: <p>Inputs cantered on 0 are definitely preferable, and a scale around unity is also a good default choice given the initialisations that are used by default in the various modules provided by pytorch. So I’d rather use (-1, 1) than (1, 10).</p>

zezimabig: <blockquote>
<p>Inputs cantered on 0 are definitely preferable, and a scale around unity is also a good default choice given the initialisations that are used by default in the various modules provided by pytorch. So I’d rather use (-1, 1) than (1, 10).</p>
</blockquote>
<p><a class="mention" href="/u/vmoens">@vmoens</a> thank you for your reply! Could I ask what do you mean by “a scale around unity”?</p>

vmoens: <p>It means that things should be normalised such that their distribution approaches a standard normal distribution (mean 0 and standard deviation of 1)<br>
Eg<br>
x = (x-mean(sample_x))/std(sample_x)</p>
Train model using small subset of image in the folder

Hai_Luu: <p>Hi everyone,<br>
I am trying to train a neuron network, I am using the coco dataset for that, but I don’t wanna use all images because it is too large and I just wanna test the model. How can I just like using 500 images for training? and I am using pre-built dataset and dataloader from PyTorch</p>

Hai_Luu: <p>Here is the solution: use random_split and then feed that splitted dataset to the dataloader</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/886d1e89683292678cd6db725281a8ee110797f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/886d1e89683292678cd6db725281a8ee110797f4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/886d1e89683292678cd6db725281a8ee110797f4_2_690x214.png" alt="image" data-base62-sha1="jsSIP5Nwtv3fAFjiUEoUjdaa0qU" width="690" height="214" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/886d1e89683292678cd6db725281a8ee110797f4_2_690x214.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/886d1e89683292678cd6db725281a8ee110797f4_2_1035x321.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/886d1e89683292678cd6db725281a8ee110797f4.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1057×328 36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch.istft with real input

pepper8362: <p>Hi,</p>
<p>I note this line in Pytorch documentation for <code>torch.istft</code> about the input of this function:</p>
<blockquote>
<p>Deprecated since version 1.8.0: Real input is deprecated, use complex inputs as returned by <code>stft(..., return_complex=True)</code> instead.</p>
</blockquote>
<p>I have some codes which do use real input with <code>torch.istft</code>. Will it produce different results from using this function with complex input after version 1.8.0? Will the backward gradient be different in training?</p>
<p>Thanks</p>
Torch.matmul CUDA OOM

dangjaya: <p>I have been trying to run my transformer codebase on top of a single CPU.</p>
<p>But I hit a wall when the code is trying to run matrix multiplication as part of self-attention :</p>
<pre><code>def forward(self, hidden_states, attention_mask):
    mixed_query_layer   = self.query(hidden_states)
    mixed_key_layer     = self.key(hidden_states)
    mixed_value_layer   = self.value(hidden_states)

    query_layer         = self.transpose_for_scores(mixed_query_layer)
    key_layer           = self.transpose_for_scores(mixed_key_layer)
    value_layer         = self.transpose_for_scores(mixed_value_layer)
    
    print (query_layer.shape, key_layer.transpose(-1,-2).shape ) 
    # torch.Size([1, 8, 10381, 16]) torch.Size([1, 8, 16, 10381])
    
    attention_scores    = torch.matmul(query_layer, key_layer.transpose(-1, -2))
</code></pre>
<p>It always ‘bombed’ out on the last operation ( torch.matmul ) with the following error :</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 8.38 GiB (GPU 0; 15.78 GiB total capacity; 8.75 GiB already allocated; 5.47 GiB free; 8.76 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
<p>I’m surprised that matmul operation takes so much memory so such as small matrix multiplication.</p>
<p>Did anyone experience the same thing ? Any workaround, guys ?</p>
<p>Appreciate any help.</p>
<p>-Dony</p>
Cross entropy ignore _index with class probabilities

Krishnan_Jothi: <p>The <em>ignore_index</em> value of the cross entropy loss is -100. In the documentation it is mentioned that <code>ignore_index</code> is only applicable when the target contains class indices. But I would like to use it with class probabilities.</p>
<p>input = torch.randn(3, 5, requires_grad=True)<br>
target = torch.empty(3, dtype=torch.long).random_(5)<br>
ignore_input = torch.randn(3, 5, requires_grad=True)<br>
ignore_target = torch.Tensor([-100, 2, -100]).to(torch.long)<br>
r = loss(input, target)<br>
r_ignore = loss(ignore_input, ignore_target)<br>
r.backward()<br>
r_ignore.backward()<br>
print(input.grad)<br>
print(ignore_input.grad)</p>
<p>output:<br>
tensor([[ 0.0117, -0.3307,  0.0511,  0.2001,  0.0678],<br>
[ 0.0515,  0.0019,  0.2177,  0.0161, -0.2874],<br>
[-0.2171,  0.0212,  0.0193,  0.1339,  0.0427]])</p>
<p>tensor([[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],<br>
[ 0.0979,  0.1573, -0.8633,  0.5849,  0.0232],<br>
[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000]])</p>
<p>The above code works fine, ignores the target -100, and doesn’t update the weights (because gradient are zero), but the targets here represents the class indices. But I would like to use this approach with class probabilities, i.e when the shape of the input and the target tensors are same. But I am not sure how to do that.</p>
<p>Help me out with this issue <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Thanks,<br>
Krishnan Jothi Ramalingam</p>
Attention layer and softmax

abolfazl_saheban: <p>Hello, I want to implement a attention layer which consist of Query, Key and Value.<br>
So as I understand, I should perform a matrix multiplication between Query and Key then<br>
I should normalize matrix by sqrt of it’s dimension and then pass the result to a softmax.<br>
My question is about softmax and it’s not clear for me to understand on which dimension of result of multiplication between Query and Key should I perform softmax?<br>
dimension of Query : [batch, depth, x, y]<br>
dimension of Key : [batch, depth, x, y]<br>
Query * transpose(Key) → Normalization → softmax</p>
Normalization on width dimension only

mfatih7: <p>Hello</p>
<p>I want to use a custom version of instance normalization which performs normalization only on W dimension of activation maps of size BxCxHxW.</p>
<p>Without reshaping the activation maps <a href="https://pytorch.org/docs/stable/generated/torch.nn.InstanceNorm2d.html#torch.nn.InstanceNorm2d" rel="noopener nofollow ugc">InstanceNorm2d</a> is not proper for this.</p>
<p>Any suggestions?</p>

mfatih7: <pre><code class="lang-auto">def forward(self, x):
	out = self.InstanceNorm2d(x)        
	return out

def forward(self, x):
	out = torch.reshape(x, (x.shape[0], x.shape[1]*x.shape[2], x.shape[3], 1) )
	out = self.InstanceNorm2d(out)
	out = torch.reshape(out, (out.shape[0], self.c, self.n, -1 ) )
	return out
</code></pre>
<p>Adding 2 reshaping operations makes training 2 times slower.</p>

ptrblck: <p>Could you try to manually write the normalization and allow nvFuser to code-gen the code for you similar to <a href="https://discuss.pytorch.org/t/pixel-normalization-through-channels/155895/4">this use case</a>?</p>
How to force DataLoader to use same random crop dimensions for image dataset?

ajwitty: <p>I am currently running a program that resamples data based on pseudo-labels for image patches. I give those pseudo-labels to a sampler to be used for balancing a resampled dataset. However, I want the dataloader to return the same randomly sampled image patch as before resampling, instead of a new random patch. Is there a way to do this within a dataloader? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Why are saved models so large?

sen1: <p>I have implemented a neural network with an LSTM model (see below). After training the model with a hidden size of 512, I saved it by calling <code>torch.save(model).</code></p>
<p>I expected the model size to measure in the low tens of kilobytes, accounting for three layers of LSTM’s hidden parameters. I also enumerated the parameters via <code>model.named_parameters()</code>. I was surprised to find that the actual unzipped size of the parameters is more like 30 MB.</p>
<p>Why is the saved model so large, and what can I do to cut down its size?</p>
<pre><code class="lang-auto">class IRNN(nn.Module):
    def __init__(self, obs_size, hidden_size):
        super(IRNN, self).__init__()

        self.num_layers = 3
        input_size = obs_size
        self.hidden_size = hidden_size
        self.hidden = self.init_hidden()
        self.hiddenc = self.init_hidden()
        self.rnn = nn.LSTM(
            input_size=input_size, hidden_size=hidden_size, num_layers=self.num_layers
        )
        self.i2o = nn.Linear(hidden_size, 256)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="sen1" data-post="1" data-topic="166324">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sen1/48/46234_2.png" class="avatar"> sen1:</div>
<blockquote>
<p>I expected the model size to measure in the low tens of kilobytes, accounting for three layers of LSTM’s hidden parameters. I also enumerated the parameters via <code>model.named_parameters()</code>.</p>
</blockquote>
</aside>
<p>That’s strange as I’m seeing an expected parameter size of 56MB using:</p>
<pre data-code-wrap="python"><code class="lang-python">class IRNN(nn.Module):
    def __init__(self, obs_size, hidden_size):
        super(IRNN, self).__init__()

        self.num_layers = 3
        input_size = obs_size
        self.hidden_size = hidden_size
        self.rnn = nn.LSTM(
            input_size=input_size, hidden_size=hidden_size, num_layers=self.num_layers
        )
        self.i2o = nn.Linear(hidden_size, 256)
        
        
model = IRNN(512, 512)
torch.save(model.state_dict(), "tmp.pt")

size = sum([param.nelement() * param.element_size() for p in model.parameters()])
print("{} parameters".format(sum([param.nelement() for p in model.parameters()])))
# 14680064 parameters
print("size: {}MB".format(size / 1024**2))
# size: 56.0MB
</code></pre>
<p>Note that I had to remove the <code>self.hidden(c)</code> attributes, as <code>self.init_hidden</code> is undefined, but this would even lower my size estimation and yours should show a larger size.</p>
<p>On my system the <code>state_dict</code> is ~26MB when saved to disk, which sounds alright given a ~2x size reduction.</p>
Changing nll_loss to mse_loss but get The size of tensor a (64) must match the size of tensor b (10) at non-singleton dimension 1

clemsoncoder: <p>Here is my code.</p>
<pre><code class="lang-auto">def simplest_loss(output, target):
    #loss = target-output
    loss = torch.mean((output - target)**2)
    return loss


network = Net()
optimizer = optim.SGD(network.parameters(), lr=learning_rate,
					  momentum=momentum)

train_losses = []
train_counter = []
test_losses = []
test_counter = [i*len(train_loader.dataset) for i in range(n_epochs + 1)]

def train(epoch):
	network.train()
	for batch_idx, (data, target) in enumerate(train_loader):
		optimizer.zero_grad()
		output = network(data)
		#loss = F.nll_loss(output, target)

		#loss = simplest_loss(output,target)
		loss = F.mse_loss(target,output);
		#loss = torch.nn.MSELoss(target,output);

		print("output")
		print(output)
		print("target")
		print(target)
		print("loss")
		print(loss)

		loss.backward()
		optimizer.step()
		if batch_idx % log_interval == 0:
			print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
				epoch, batch_idx * len(data), len(train_loader.dataset),
				100. * batch_idx / len(train_loader), loss.item()))
			train_losses.append(loss.item())
			train_counter.append(
				(batch_idx*64) + ((epoch-1)*len(train_loader.dataset)))
			torch.save(network.state_dict(), 'results/model.pth')
			torch.save(optimizer.state_dict(), 'results/optimizer.pth')
</code></pre>

ptrblck: <p>The expected model output and target shapes differ for <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss"><code>nn.NLLLoss</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss"><code>nn.MSELoss</code></a> as explained in their docs.<br>
I don’t know what your actual use case is but the current error in <code>nn.MSELoss</code> is raised as both the model output and target should have the same shape.</p>

clemsoncoder: <p>Turns out that the output had 10 by 64 and the target needed to be transformed into a hot 1 encoding from 64 to 10 by 64.</p>
<p>I’m still not sure how NLLLoss works after reading your link, but I don’t need to know at the moment but might visit this again later.</p>
<p>Thanks anyway.</p>

ptrblck: <aside class="quote no-group" data-username="clemsoncoder" data-post="3" data-topic="166303">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e9a140/48.png" class="avatar"> clemsoncoder:</div>
<blockquote>
<p>Turns out that the output had 10 by 64 and the target needed to be transformed into a hot 1 encoding from 64 to 10 by 64.</p>
</blockquote>
</aside>
<p>Double check these shapes, since and output in the shape <code>[10, 64]</code> should not work with a target of <code>[64]</code> using <code>nn.NLLLoss</code>.</p>
Time series regression and classification

kokusnuss: <p>Hi Everyone,<br>
I am new two time series related problem.<br>
I have some time series data from imu sensors.</p>
<p>The time is given in seconds. I dnt have the date month  year information.<br>
For each time there is a label of activity like what happened or not.</p>
<p>Let say we have some data from sensor X. It would look the following example.</p>
<p>time,  sensor_id, field 1, field2, field3 , … , Event<br>
3.009, uixcd994, 0, 10, 4.5, …, Exercisee<br>
…<br>
and after hundreds for rows<br>
3000.845, uixcd994, 0, 10, 4.5, …, Training<br>
and after thousands of more rows some other event.</p>
<p>Remember that, Order can be anything,<br>
Training → Exercise → Training or<br>
Exercise → Training → Something Else → No_acitivity → Training.<br>
However, we have maximum four possibility. Training, Exercise, Warm_up or No activity.</p>
<p>I have similar data from dozens of sensors. What I want to do is: For a given test data , I want to identify which part of the sensor data is what e,g, training exercise, etc.</p>
<p>However, I could not find a good example how to start and how to process the data .Since the problem sounds like both classification and regression.  Any ideas, example ( for classification, regression or for both) would be really appreciated.</p>
<p>Thanks</p>
Ubuntu 18.4: 'ModuleNotFoundError: No module named 'pycocotools._mask''

karsil: <p>I am trying to use MeanAveragePrecision as described in <a href="https://torchmetrics.readthedocs.io/en/stable/detection/mean_average_precision.html" rel="noopener nofollow ugc">here</a></p>
<p>Minimal example</p>
<pre><code class="lang-auto">conda create -n myenv python=3.9
conda activate myenv
pip install torch torchmetrics torchvision
</code></pre>
<p>Then the following code from the torchmetrics page:</p>
<pre><code class="lang-auto">import torch
from torchmetrics.detection.mean_ap import MeanAveragePrecision
preds = [
    dict(
        boxes=torch.tensor([[258.0, 41.0, 606.0, 285.0]]),
        scores=torch.tensor([0.536]),
        labels=torch.tensor([0]),
    )
]
target = [
    dict(
        boxes=torch.tensor([[214.0, 41.0, 562.0, 285.0]]),
        labels=torch.tensor([0]),
    )
]
metric = MeanAveragePrecision()
metric.update(preds, target)
from pprint import pprint
pprint(metric.compute())
</code></pre>
<p>This is working quite fine on my Mac, but not on Ubuntu 18.04, my main processing machine.<br>
I am recieving the following error:</p>
<pre><code class="lang-auto">  File "/home/karsil/mAP_simple.py", line 2, in &lt;module&gt;
    from torchmetrics.detection.mean_ap import MeanAveragePrecision
  File "/home/karsil/miniconda3/envs/myenv/lib/python3.9/site-packages/torchmetrics/detection/mean_ap.py", line 31, in &lt;module&gt;
    import pycocotools.mask as mask_utils
  File "/home/karsil/models/research/pycocotools/mask.py", line 3, in &lt;module&gt;
    import pycocotools._mask as _mask
ModuleNotFoundError: No module named 'pycocotools._mask'
</code></pre>
<p>I’ve already found <a href="https://discuss.pytorch.org/t/modulenotfounderror-pycocotools-mask-during-pytorch-tutorial/63014">this solution</a> from this forum and applied it with no change.</p>
<p>Glad for any help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to get the intermediate output from torchscript?

carrot0007: <p>How to get the intermediate output from torchscript ?<br>
​​​<br>
​​​​<br>
​​​​</p>

Gamrix: <p>It is possible to do any python things you want like store intermediate values, and even attach a debugger by using <code>torch.jit.ignore</code> to create a Python entrypoint in the middle of a Torchscript function. To get an intermediate variable into Python, just make a function with the decorator <code>@torch.jit.ignore</code> and pass the intermediate value into said function.</p>
<p>Docs:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.jit.ignore.html#torch.jit.ignore" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.jit.ignore.html#torch.jit.ignore</a></p>

tjoseph: <p>Hi,<br>
does this also hold work torch.jit.trace?</p>
<p>Is there some pattern that is used to do logging while tracing? The only thing that comes to my mind is passing around the logs so that the logs are also an output of every module. However, I would rather directly write to a SummaryWriter in a module and keep the interface to/from the module clean.</p>
A view of a leaf Variable that requires grad is being used in an in-place operation

Sergei_Vorobev: <p>I have the following code that works on PyTorch 1.11 and fails on 1.13</p>
<pre><code class="lang-python">import torch

print(torch.__version__)
model = torch.jit.load("foo/ranker.pt")
model = model.eval()
x = torch.jit.load("foo/tensor_all.pt")
inputs = list(x.parameters())
for i in inputs:
    i.requires_grad = False
# first call works fine
model.forward(*inputs)
# second call fails
model.forward(*inputs)
print("ALL GOOD")
</code></pre>
<p>The error that it fails with is</p>
<pre><code class="lang-auto">relu
        return handle_torch_function(relu, (input,), input, inplace=inplace)
    if inplace:
        result = torch.relu_(input)
                 ~~~~~~~~~~~ &lt;--- HERE
    else:
        result = torch.relu(input)
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>I don’t quite understand why would I get this error, if I do the <code>model.eval()</code> in my code? Should not it make all operations not require gradients?</p>

ptrblck: <p>No, <code>model.eval()</code> will recursively change the internal <code>self.training</code> attribute of all registered <code>nn.Modules</code> to <code>False</code> and (some) modules will change their behavior (i.e. if the <code>self.training</code> attribute is used in their <code>forward</code> method). If you want to disable the gradient calculation, wrap your forward pass into a <code>with torch.no_grad()</code> or <code>with torch.inference_mode()</code> guard.<br>
However, I also don’t understand why the error is raised in the first place as your code snippet doesn’t show a backward pass, but just two forward passes.</p>

Sergei_Vorobev: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> awesome, that helped! Thank you!</p>
<p>I have a follow-up question but posted it in <a href="https://discuss.pytorch.org/t/pytorch-1-13-runtimeerror-tensors-used-as-indices-must-be-long-byte-or-bool-tensors/166321" class="inline-onebox">PyTorch 1.13: RuntimeError: tensors used as indices must be long, byte or bool tensors</a></p>
Why the import .module syntax

Algon-33: <p>In <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/tensorboard/__init__.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/__init__.py at master · pytorch/pytorch · GitHub</a> there’s a line where you try to get classes from a file called writer.py with the syntax “from .writer import…”</p>
<p>Why on earth is the . there? I had to remove the dot, and the dot prefix from many other files in the same directory, in order to get things to work.</p>
<p>Side note: Even then, the use of LooseVersion raised a deprecation error, which I had to deal with by using packaging.version.version from setuptools._vendor instead.</p>

ptrblck: <p>The leading dot indicates a relative import as described <a href="https://docs.python.org/3/reference/import.html#package-relative-imports">here</a>:</p>
<blockquote>
<p>Relative imports use leading dots. A single leading dot indicates a relative import, starting with the current package. Two or more leading dots indicate a relative import to the parent(s) of the current package, one level per dot after the first.</p>
</blockquote>
<p>I don’t know what your use case is, but you should not change the source files and also not use the PyTorch source code as your working directory as it could raise import errors.</p>

Algon-33: <p>Thanks for the reply!</p>
<p>Yeah, I figured out the . meant a relative import. It seems like I had another package called tensorboard installed closer than the package I should have been using. That was the cause of those errors.</p>
<p>Also, thanks for the advice about not using the source code as my CWD. Funnily enough, the problem was the opposite of what you advised. I got more import errors when I wasn’t calling code from the within the source code directory. Still, I got it fixed and managed to resolve that error so I could run the code in the “Try it out” section of <a href="https://github.com/joonleesky/train-procgen-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - joonleesky/train-procgen-pytorch: Pytorch implementation on OpenAI's Procgen ppo-baseline, built from scratch.</a> without getting import errors (code still crashes though…)</p>
<p>But I don’t know what to say about changing the source files. Like, the code raises a warning to use classes from packaging.version instead of LooseVersion. Just ignoring it seems like a bad idea, no? If you look at the source for LooseVersion, the author claims they’ve given up on getting the code working right, so that made using it seem like an even worse idea.</p>
Non-Linear regression model

Mohamed_Taha: <p>I would like to create a model that predicts the equation of multiplying, division, or even power or roots the only equation that can be predicted is linear like ( pred = w * input + b )<br>
so how to create that model with two input numbers that were multiplied</p>

Mohamed_Taha: <p>I tried to make a model that can predict the power of the input and I made a dataset with input x and output with y that’s y = x^2 .<br>
I thought that if I separate the inputs 2 times for 2 neurons and get the log of each one I will get that log(x) and log(x).<br>
then I collect the output at one neuron and get the exp(2log(x)) it will gives me x^2<br>
so that I made this network.</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 2)
        self.fc3 = nn.Linear(2, 1)
    def forward(self, x):
        x = self.fc1(x)
        x = torch.log(x)
        x = self.fc2(x)
        x = torch.exp(x)
        x = self.fc3(x)
        return x

model = Network()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)
</code></pre>
<p>and the predicted output is  “tensor([[nan]])”<br>
so how to make that network?</p>
<p>and another question I don’t know how to arrange the tensor for 2 neurons input of network.<br>
how to arrange the shape of input tensors to make it acceptable by network of 2 or whatever number of inputs ?</p>

ma-sadeghi: <p>It’s probably because you’re talking the <code>log</code> of the first layer, whose values could be negative, which leads to <code>nan</code>s.</p>
Time series data prediction

kokusnuss: <p>Hallo Everyone!</p>
<p>Apology in advance  since I am not sure this is the right place to ask such question.<br>
I have some IMU sensor data. These data describes what individual players did e.g.<br>
Jogging, running, long pass, short pass etc.  I want to develop a model using this data.</p>
<p>The target is to predict whether the session was a training one or a real match or something else.<br>
The first idea comes to my mind is to use LSTM or any special version of LSTM which will be able to predict the  Event type (training , match etc.) from given activities(e.g. running, pass, jogging) after the end of the session.</p>
<p>Any idea would be really appreciated.</p>

AbdulsalamBande: <p>Can you describe the data in more detail, or show paste some of it.</p>

kokusnuss: <p>time	label.ball_contact	label.movement_speed	label.movement_distance	event.kick.confidence	event.kick.ball_speed	event.motion.speed.average	event.motion.speed.peak	event.motion.distance	challenge.idx	challenge.id	Event<br>
0	1.372256	0.0	-1.000000	-1.0	0.00	0.000000	0.000	0.000	0.000000	0.0	0.0	No_Event<br>
…<br>
1000	1.472256	0.0	-1.000000	-1.0	0.00	0.000000	0.000	0.000	0.000000	0.0	0.0	Match<br>
…</p>
<p>200	1.572256	0.0	-1.000000	-1.0	0.00	0.000000	0.000	0.000	0.000000	0.0	0.0	Exercise<br>
…</p>
Odd behavior in LSTMCell research

John_Schwarcz: <p>Hello,</p>
<p>I am doing research on LSTMs and have made changes to the LSTMCell in quantizable/'modules/rnn.py.</p>
<p>I have added ‘if’ conditions that should functionally turn the LSTM into an RNN. Visualising the network activity shows everything being clamped to 1 or 0 as intended.</p>
<p>However this ‘handcrafted RNN’ is able to consistently learn an RL task that the official Pytorch RNN consistently fails on.</p>
<p>I am very confused as to what could be different between my ‘handcrafted RNN’ and the official Pytorch RNN. I use the tanh activation function for both and the same number of neurons. Clamping the forget gate activity to 1 indeed prevents learning, so I know my modifications are having an effect.</p>
<p>Any help in what could be going on under the hood would be greatly appreciated.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e729a2a405e7f446670881b220bb1389c367f2be.png" data-download-href="https://discuss.pytorch.org/uploads/default/e729a2a405e7f446670881b220bb1389c367f2be" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e729a2a405e7f446670881b220bb1389c367f2be_2_448x500.png" alt="image" data-base62-sha1="wYXuVRSyzc3UC0D3uLyk7jPMu50" width="448" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e729a2a405e7f446670881b220bb1389c367f2be_2_448x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e729a2a405e7f446670881b220bb1389c367f2be_2_672x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e729a2a405e7f446670881b220bb1389c367f2be_2_896x1000.png 2x" data-dominant-color="212A33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1089×1214 192 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Where is nll_loss_nd::call function implimented?

clemsoncoder: <p>The source code has a file called nll_loss_nd_ops.h that defines nll_loss_nd, but it only has the struct with function headers and now implmentation.</p>
<p>I tried searching for where call function is implimented</p>

ptrblck: <p>I think the operation would call into the <code>nll_loss</code> implementations <a href="https://github.com/pytorch/pytorch/blob/8a9aca7b8d7fba320b4f2a8c2f18a25f572c46b6/aten/src/ATen/native/LossNLL.cpp">here</a>.</p>