akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: The size of tensor a (64) must match the size of tensor b (7) at non-singleton dimension 1

mai1: <blockquote>
<p>I get the following Error</p>
<p>The function:</p>
<p>def test_accuracy(model, test_loader):<br>
model.eval()<br>
acc = 0.0<br>
total = 0.0</p>
<pre><code class="lang-auto">with torch.no_grad():
    for data in test_loader:
        images, labels = data
        images = images.to(torch.device('cuda'))
        labels = labels.to(torch.device('cuda'))
        # run the model on the test set to predict labels
        outputs = model(images)
        # the label with the highest energy will be our prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        acc += (predicted == labels).sum().item()

# compute the accuracy over all test images
acc = (100 * acc / total)
return (acc)
</code></pre>
<p>I would really appreciate your help,<br>
thank you in advance</p>
</blockquote>
<blockquote>
<p>Blockquote</p>
</blockquote>

InnovArul: <p>I would suggest printing <code>tensor.shape</code> in before the operations and see if there is any apparent size mismatch.<br>
Also, it will be helpful if you give the line number of the error.</p>
Not to update part of the parameters with backpropagation

sfr000: <p>I am creating my own <code>nn.Module</code> with its own <code>forward</code> function and matrix of parameters, <code>P</code>. But, I would like to impose that part of the matrix of parameters is set to zero and remains zero. Is there a way to do so without defining the matrix using multiple parameters definition?</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, n_objects):
        super(Model, self).__init__()

        self.n_objects = n_objects
        P = torch.empty(

            (self.n_objects, self.n_objects),
            requires_grad=True)
        nn.init.uniform_(P)

        self.P = torch.nn.Parameter(P)
</code></pre>
<p>Here, for example, I would like that all the elements in the main diagonal of <code>P</code> should not change w.r.t. the original values.<br>
Thanks for the help!</p>
F1 score in QSVM

AP_M: <p>How can I compute the other metrics such as f1 score, sensitivity, specificity in QSVM…Link: “<a href="https://qiskit.org/documentation/stable/0.24/tutorials/machine_learning/01_qsvm_classification.html" class="inline-onebox" rel="noopener nofollow ugc">Quantum-enhanced Support Vector Machine (QSVM) — Qiskit 0.24.1 documentation</a>” taking example of breast cancer dataset.</p>
Computing expectation with information across different tensors

shashank_gupta1: <p>Background: I am working with a Plackett-Luce model in a learning-to-rank setting, which means for a given query q, I have N sampled rankings, where each ranking has a d number of documents. Assuming there is max ‘d’ documents to be ranked.</p>
<p>For each ranking in a given query, I have a probability of the given ranking, i.e. a tensor ranking_prob of dimension [<span class="hashtag-raw">#query</span>, <span class="hashtag-raw">#ranking_samples</span>].</p>
<p>From the model, for each query, I have sampled N rankings, where each ranking is of size d, i.e. I have a tensor sampled_ranking of dimensions: [<span class="hashtag-raw">#query</span>, <span class="hashtag-raw">#ranking_samples</span>, <span class="hashtag-raw">#documents</span>].</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/caa8e95bc18bcff6ff374343397789f1fa830cfa.png" alt="Screenshot 2022-05-24 at 09.37.33" data-base62-sha1="sUOl6nYeFqqzLQxupzlU9ACR2Hg" width="648" height="122"></p>
<p>I want to basically compute this expectation. For each document d, in a ranking y, the alpha value is based on the rank of the document in the ranking y. I have a fixed alpha vector, where each index is the ranking position.</p>
<p>Now with these tensors, I don’t know how to compute this expectation. The information is spread across different tensors, and I don’t know how to aggregate them.</p>
Pytorch lstmcell issue regarding usage of lstmcell instead of into lstm

metaljoe342_metalj: <p>Hi.<br>
I’m trying to refactor a code that uses lstm into lstmcell<br>
it’s sniippet is:</p>
<pre><code>def forward(self, sentence):

    embeds = self.embeddings(sentence)
    #
    out = out.view(self.sequence_len, embeds.size(0), -1)
    lstm_out, _ = self.lstm(sentence)
    None
    #
    # Run through the lstm.
    lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))

    # Activate the ReLU on the linear layer's output.
    x = self.activation(self.linear1(lstm_out[-1].view(1, -1)))

    # For the second linear layer.
    x = self.linear2(x)

    return F.log_softmax(x, dim=-1)
</code></pre>
<p>can u assist?<br>
thanks.</p>

vdw: <p>Well, <code>nn.LSTM</code> processes the whole input sequence, whole <code>nn.LSTMCell</code> only a single time step. So you will need a loop over your sequence. The <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html" rel="noopener nofollow ugc">docs</a> actually provide an example.</p>
<p>Apart from that, I’m rather sure that your use of <code>view()</code> messes up your data; have a look <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">here</a>.</p>
Tell pls your favorite PyTorch couses

Doronin_Mark: <p>Help, me please to choose a course for PyTorch junior. I have a list of courses <a href="https://skillcombo.com/topic/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Top PyTorch Courses (May 2022) - Study Online on SkillCombo</a></p>
How to get percentage of softmac

Max_Tyson: <p>So i was following <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.11.0+cu102 documentation</a> to make my program. The problem I am having is dealing with images with out the target, eg. a blank picture of idk grass. Currently t he program thinks everything is the target class. So i decided to print the probality of it being that class as a percentage.<br>
This is my current code:<br>
outputs = model(inputs)<br>
print(nn.functional.softmax(outputs, dim=1))<br>
The result is this:<br>
tensor([[1.0000e+00, 2.2341e-06],<br>
[1.0000e+00, 2.7773e-06],<br>
[1.0000e+00, 3.9080e-06],<br>
[9.9999e-01, 5.0023e-06]])<br>
My question is: How do i turn theese numebrs into percentages</p>

Matias_Vasquez: <p>These numbers are already in percentages (<code>1.0000e+00 ≈ 1 = 100%</code> / <code>2.2341e-06 ≈ 0 = 0%</code>). If you wanted to see it more clearly, then you could multiply them by <code>100</code>.</p>
<pre><code class="lang-python">torch.set_printoptions(precision=2)
print(nn.functional.softmax(outputs, dim=1) * 100)
</code></pre>
<p>As you mentioned, the model is predicting everything to be the first class with almost <code>100%</code> certainty.</p>
<p>There are some reasons that this might happen:</p>
<ul>
<li>Unbalanced Dataset</li>
<li>Not enough training time</li>
<li>Wrong hyperparameters (e.g. learning rate too high)</li>
<li>Training data not shuffled</li>
<li>“Dying ReLU”</li>
<li>Forgot to preprocesses images</li>
<li>…</li>
</ul>
<hr>
<p>If you are sure that you do <strong>not</strong> have an unbalanced dataset, then you could take a small subset of your dataset and overfit your model. This way you will see if the model is capable of learning on your dataset.</p>
CrossEntropy value changes with batch size at initial validation - is this a bug?

enterthevoidf22: <p>i have two models that differ only by the batch size, meaning they have identical weight initialization and all other parameters.</p>
<p>i perform initial validation, which means before any training to the models i simply run inference.</p>
<p>the cross_entropy loss for batch size = 3 is 0.9632</p>
<p>the cross_entropy loss for batch size = 1 is 0.8576</p>
<p><strong>i’m not using any batch norm layers in my model, so this has no effect.</strong></p>
<p>what else can cause such behavior?</p>

ptrblck: <p>Are you using other normalization layers, which might be size-dependent or e.g. dropout without calling <code>model.eval()</code>?<br>
If not, could you post a minimal, executable code snippet showing this issue, please?</p>

enterthevoidf22: <ul>
<li>
<p>i’m using GroupNorm.</p>
</li>
<li>
<p>using dropout but calling model.eval() so shouldn’t take any effect</p>
</li>
</ul>
<p>unfortunately i can’t share executable code at the moment, was hoping to get a general idea what might cause this behavior</p>
Torchscript executing static graph with different shaped tensors inputs does not trigger recompilation (retracing)

xiacijie: <p>Hello,</p>
<p>I am new to torchscript and playing around with it. I have some experience in TensorFlow and try to compare those two frameworks.</p>
<p>Here is an example I am playing with:</p>
<pre><code class="lang-auto">import torch 

class Test(torch.nn.Module):
    def __init__(self) -&gt; None:
        super(Test, self).__init__()
    
    def forward(self, input):
        res = torch.argwhere(input)
        return res

x = torch.tensor([1, 2, 3])
y = torch.tensor([0, 1, 2, 3, 5])

t = Test()
traced_t = torch.jit.trace(Test(),(x))

print(traced_t(x))
print(traced_t(y))
</code></pre>
<p>Here <code>x</code> and <code>y</code> have different sizes.<br>
I trace Module <code>Test</code> with <code>x</code> as an example input. Then I try to execute the traced graph with example<code> x</code> and<code> y</code>. Because <code>y</code> have a different size than <code>x</code>, I expect the graph to be retraced or recompiled as it will do in TensorFlow’ static graph. However, I did not observe any recompilation happen.</p>
<p>Can anyone points out:</p>
<ol>
<li>When I run the statement <code>traced_t(y)</code>, is the graph recompiled?</li>
<li>If it is not recompiled, how is it achieved (compared with TensorFlow’s static graph) ?</li>
</ol>
<p>Best,<br>
Cijie</p>
How to determine train, test iteration and vocab?

teohsinyee: <p>Hi everyone.</p>
<p>I want to fit this code with my personal dataset. Not the IMDB ones. The load_data_imdb() returns 3 parameters.<br>
To fit my dataset in, I’ve to set the 3 parameters right? But what value should I put?</p>
<p>Reference: <a href="https://classic.d2l.ai/chapter_natural-language-processing/sentiment-analysis.html#put-all-things-together" class="inline-onebox" rel="noopener nofollow ugc">13.8. Text Classification and Data Sets — Dive into Deep Learning 0.7 documentation</a></p>
<pre><code class="lang-auto">import torch
from torch import nn
from d2l import torch as d2l

batch_size = 64

train_iter, test_iter, vocab = d2l.load_data_imdb(batch_size)
</code></pre>
Einsum calculation accuracy ？

sev777: <p>When the matrix result is calculated directly and the two-step calculation, the results are different.<br>
Is it because of the problem of calculation accuracy （torch.einsum）?<br>
I use the GeForce RTX 3090 GPUs</p>
<pre><code class="lang-auto">a=torch.randn(2,3,5)
b=torch.randn(2,3,4)
z1=torch.einsum('ijk,ijb-&gt;bk',a,b)
z2=torch.einsum('ibk-&gt;bk',torch.einsum('ijk,ijb-&gt;ibk',a,b))
z1==z2
</code></pre>

InnovArul: <p>Floating point numbers cannot / should not be compared with <code>==</code> due to precision errors in representing them.<br>
Use <code>torch.allclose()</code> instead for comparing them.</p>
How to copy parameters of one network to another directly in Libtorch C++?

Fitanium: <p>I have two networks with the same structure,</p>
<pre><code class="lang-cpp">class MLP : public torch::nn::Module
{
private:
    torch::nn::Linear input{nullptr};
    torch::nn::Linear hidden{nullptr};
    torch::nn::Linear output{nullptr};

public:
    MLP();
    ~MLP();

    MLP(const int inDim, const int outDim);
    torch::Tensor forward(torch::Tensor x);
};

std::shared_ptr&lt;MLP&gt; NetA;
std::shared_ptr&lt;MLP&gt; NetB;
</code></pre>
<p>and I only train <code>NetA</code> and update its parameters. And at some moment, I want to copy parameters of <code>NetA</code> to <code>NetB</code>.</p>
<p>I want to know if there exist some function/method in LibTorch allows me to do this directly? like assignment operator</p>
<p>Thanks.</p>

Fitanium: <p>saving and loading is ok, but not elegant</p>
<pre data-code-wrap="c++"><code class="lang-nohighlight">std::shared_ptr&lt;MLP&gt; NetA;
std::shared_ptr&lt;MLP&gt; NetB;

// ...

torch::save(NetA, "./model.pt");
torch::load(NetB, "./model.pt");
</code></pre>
<p>REF: <a href="https://discuss.pytorch.org/t/libtorch-how-to-save-model-in-mnist-cpp-example/34234/8">(libtorch) How to save model in MNIST cpp example?</a></p>
MONAI's Compose output filename

fpizzaga: <p>Hi,<br>
I would like to edit the name after defining a <a class="mention" href="/u/monai">@MONAI</a> transform, as below.<br>
I would like for run <code>transform</code> in a for loop. Is it possible to edit the name of the output file, before applying <code>transform?</code> ?<br>
Thank you,<br>
fp</p>
<pre><code class="lang-auto">from monai.transforms import (
    AddChannel,
    Compose,
    EnsureType,
    LoadImage,
    NormalizeIntensity,
    RandRotate90,
    RandSpatialCrop,
    SaveImage,
)

transform = Compose(
    [
        LoadImage(image_only=True),
        NormalizeIntensity(),
        AddChannel(),
        RandSpatialCrop((128, 128, 128), random_size=False),
        RandRotate90(prob=0.5, spatial_axes=(0, 1)),
        EnsureType(),
        SaveImage(
            output_postfix="postfix",
            output_ext=".nii",
            resample=False,
            separate_folder=False,
            print_log=False,
        ),
    ]
)
</code></pre>
Torch fx forward

Meng_Wang: <p>model = torchvision.models.resnet18(pretrained=True)<br>
x = torch.rand(1,3,224,224)<br>
m = torch.fx.symbolic_trace(model)<br>
m.layer1(x)</p>
<p>error:*** NotImplementedError</p>
<p>how to solve it.</p>

MartinZhang: <pre><code class="lang-auto">model = torchvision.models.resnet18(pretrained=True)
x = torch.rand(1,3,224,224)
m = torch.fx.symbolic_trace(model)
m(x)
</code></pre>

Meng_Wang: <p>why not m.layer1(x), sometimes I only use to infer part of network.</p>

benli: <p>This module is used to modify the graph. If you only want layer1(x), you only get it from module rather than fx. If you can combine some parts, you can modify graph and generate a new one code.</p>
Torch.fx.symbolic_trace doesn't preserve module methods

dskhudia: <p>Are module methods other than forward preserved? I expected scale to be there in the traced_mod.</p>
<pre><code class="lang-auto">import torch
import torch.fx

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(512, 512)

    def forward(self, x):
        x = self.linear(x)
        x = torch.relu(x)
        return x

    def scale(self, x):
        return x * 2

mod = MyModule()

traced_mod = torch.fx.symbolic_trace(mod)

x = torch.randn(1, 512)

orig_out = mod.scale(x)

traced_out = traced_mod.scale(x)
"""
Traceback (most recent call last):
  File "fx_preserve_module_methods.py", line 25, in &lt;module&gt;
    traced_out = traced_mod.scale(x)
  File "~/venv_sys/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'MyModule' object has no attribute 'scale'
"""
</code></pre>
<p>PyTorch version:</p>
<pre><code class="lang-auto">python -c "import torch; print(torch.__version__)"
1.10.2+cu113
</code></pre>

benli: <p>Fx only trace the module about forward function. Do not trace your function you write.</p>
Custom Conv2D using nn.Linear

A_Bdeir: <p>I have a project that requires me to define a conv2d operation using a linear layer (will be replaced with a custom layer later). My first though was to check the logic of it and implement it using the vanilla nn.Linear layer.</p>
<pre><code class="lang-auto">
class CustomConv(nn.Module):
    """
    2D convolution over an input of shape (batch x channel x H x W)
    """
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, dilation=1, padding=0, bias=True):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.kernel_size = kernel_size
        self.padding = padding

        if isinstance(stride, int):
            self.stride = (stride, stride)
        else:
            self.stride = stride

        if isinstance(kernel_size, int):
            self.kernel_size = (kernel_size, kernel_size)
        else:
            self.kernel_size = kernel_size

        if isinstance(padding, int):
            self.padding = (padding, padding)
        else:
            self.padding = padding

        if isinstance(dilation, int):
            self.dilation = (dilation, dilation)
        else:
            self.dilation = dilation

        self.linearized_kernel = torch.nn.Linear(self.kernel_size[0] * self.kernel_size[1]*self.in_channels, self.out_channels, bias=bias)
        
        self.unfold = torch.nn.Unfold(kernel_size=(self.kernel_size[0], self.kernel_size[1]), dilation=dilation, padding=padding, stride=stride)

    def forward(self, x):

        bsz = x.shape[0]
        h, w = x.shape[2:4]
        h_out = math.floor((h + 2*self.padding[0] - self.dilation[0]*(self.kernel_size[0] - 1) - 1)/self.stride[0] + 1)
        w_out = math.floor((w + 2*self.padding[1] - self.dilation[1]*(self.kernel_size[1] - 1) - 1)/self.stride[1] + 1)

        patches = self.unfold(x)  
        
        patches = patches.view(bsz, -1, self.kernel_size[0] * self.kernel_size[1]*self.in_channels)

        # perform the matrix multiplication
        patches = self.linearized_kernel(patches)
        
        patches = patches.view(bsz, self.out_channels, h_out, w_out)
        #patches = torch.nn.functional.fold(patches, (h_out, w_out), (1, 1))

        # return in the expected shape
        return patches
</code></pre>
<p>When training a simple CNN model using the custom conv, it learns but only achieves half the accuracy of using a normal conv2d layer. Is there any issue with the logic being used here?</p>

A_Bdeir: <p>So apparently switching from tensor.view to transpose fixed the problem.<br>
Is there any particular reason for this? The problem with this solution is that it halves the training time for some reason.</p>
<p>EDIT: figured it out, can delete the post if wanted</p>
Data of Many Small Numpy Arrays

hadarshavit: <p>Hi,</p>
<p>I have a dataset that consists of many small numpy array. The total size of it is around 300 GB.<br>
When loading, each call to the Dataset’s <strong>getitem</strong>, I read a file and return its content. However, the training takes long time. Each time around 15 batches are processed, and then there is a minute of GPU idle time.<br>
How can I optimize this pipeline?<br>
Thanks</p>

nivek: <p>It seems like you may be running out of memory. Have you been tracking your memory usage?</p>
<ol>
<li>Are you storing one sample per file? If there are multiple samples per file, you may be reading more than you need.</li>
<li>You may also want to use a lower number of workers (if it is not 0 already) because the memory consumption increases with the number of workers being used.</li>
</ol>
GPU memory OOM error when I skip a batch

RH_Jung: <p>Hello, I am training a CLIP (ViT) model.</p>
<p>I would like to skip a mini-batch when I find NANs in the output of the model.</p>
<p>So this is a simplified pseudo code example.</p>
<p>for data in dataloaders:<br>
output = model(data[0])<br>
if torch.isnan(output):<br>
continue<br>
loss = loss_fn(output, data[1])</p>
<pre><code>  loss.backward()
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()
</code></pre>
<p>However, when output contains NAN and if I skip the mini-batch, it faces GPU lack of memory error.<br>
I suspect that something is not properly freed If I skip the mini-batch.<br>
Could I get any answer?<br>
Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="RH_Jung" data-post="1" data-topic="151388">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rh_jung/48/49492_2.png" class="avatar"> RH_Jung:</div>
<blockquote>
<p>I suspect that something is not properly freed If I skip the mini-batch.</p>
</blockquote>
</aside>
<p>Yes I think you are right and since you are not calling <code>backward()</code> if the batch is skipped, the forward activations would be kept alive until the reference is deleted.<br>
You could try to <code>del output</code> in this case and see, if this would free the intermediates.</p>

RH_Jung: <p>Thanks!<br>
The solution was really simple indeed!<br>
Just del output solved everything!</p>
Is there an alterntive to tf.keras.layers.Masking in PyTorch? Or Is there any way to implement it from scratch?

HoundOfBakerSt: <p>I have a time series dataset with a lot of NAs that I need to use with LSTM network. Previously with TensorFlow, I used to initially replace NAs with -1(Which is not present in the data) and use  <code>tf.keras.layers.Masking</code>(<a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Masking" rel="noopener nofollow ugc">Documentation</a>) within the model to stop learning when the model encounters -1 and resume when encountering something else. Since then, I have switched to PyTorch and need to use something similar again. So, my question is this: Is there an alternative to this in PyTorch? Or is there any way to do this while using PyTorch? See the code version of what I do below:</p>
<pre><code class="lang-auto">...
dataset.fillna(-1, inplace = True) # Replace na cells with -1 for masking in LSTM
...
...
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Masking(mask_value=-1,input_shape=(timesteps, features)))
model.add(tf.keras.layers.LSTM(32))
...
...
</code></pre>
How can i extract the features from grayscale images

Sammo1: <p>I need to extract features from medical images using Pytorch but the features I need are before the final layer for the classification … i used like this</p>
<pre><code class="lang-auto"> model = VGG16()
 model = models.Model(inputs=model.inputs, outputs=model.layers[-2].output)
</code></pre>
<p>does that right or the medical needs another way, please ?</p>

kaizhao: <p>PyTorch does not identify specific image modalities, eg medical image or natural images.</p>
<p>Basically, all you need is a 3-channel tensor, no matter what it contains.</p>
<p>But you may finetune on your own data before extracting features I guess.</p>

Michal_Bogacz: <p>Hello,<br>
I believe what you wanted is just the convolutional part of VGG16.<br>
This is the base of style-transfer problem in ML.<br>
<a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html#:~:text=%23%20desired%20depth%20layers,style_losses%2C%20content_losses" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/neural_style_tutorial.html#:~:text=%23%20desired%20depth%20layers,style_losses%2C%20content_losses</a><br>
Under this link you will find code which extracts tensors of features on different lvls of the network. So effectively you just remove the part of the network you don’t need.</p>

Sammo1: <p>Thanks a lot, i will see it now but excuse me can you help in this</p><aside class="quote quote-modified" data-post="1" data-topic="152379">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sammo1/48/46342_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/extract-feature-using-pytorch/152379">Extract feature using pytorch</a> 
  </div>
  <blockquote>
    I need to write a code using PyTorch to extract features like this one 
def extract_features(image):
    feature_extractor = models[config.base_model_name]["model"](
        include_top=False, weights="imagenet"
    )

    print(feature_extractor.summary())
    if config.pooling != "avg":
        feature_extractor = tf.keras.Model(
            feature_extractor.input,
            tf.keras.layers.AveragePooling2D(int(config.pooling[0]))(
                feature_extractor.output
            ),
   …
  </blockquote>
</aside>
Does it affect the output if I reshape or permute a tensor after convolution or graph convolution operation?

tanvir_14: <p>I have an input graph and its adjacency. After performing graph convolution my output shape is a 3D tensor (ex. 256,17,3). Now, for passing this output to Conv2d, I’m reshaping the output to a 4D tensor (ex. 256,1,17,3). So does it affect the output if I reshape or permute a tensor after convolution or graph convolution operation?</p>

ptrblck: <p>In your example you are adding a channel dimension and the conv layer would thus use the last two dimensions as the spatial size.</p>
<aside class="quote no-group" data-username="tanvir_14" data-post="1" data-topic="152361">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tanvir_14/48/30966_2.png" class="avatar"> tanvir_14:</div>
<blockquote>
<p>So does it affect the output if I reshape or permute a tensor after convolution or graph convolution operation?</p>
</blockquote>
</aside>
<p>Generally yes. Reshaping or permuting a tensor will affect the next operations as the layer(s) could use the input in different ways. E.g. in your previous example you could also <code>unsqueeze</code> another dimension such as dim2, which would yield a tensor of <code>[batch_size=256, channels=17, height=1, width=3]</code>, which would then also change the output of the next conv layer.</p>

tanvir_14: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the explanation</p>
Asks for the meaning of MiDaS evaluation mode

Christopher_Jones: <p>The code at <a href="https://pytorch.org/hub/intelisl_midas_v2/" class="inline-onebox" rel="noopener nofollow ugc">MiDaS | PyTorch</a> puts MiDaS into evaluation mode by calling<br>
midas.eval()</p>
<p>What exactly does evaluation mode do and when should you enable it?</p>

ptrblck: <p>During evaluation mode some layers will change their behavior and it’s used during the validation or test run of your model.<br>
E.g. batchnorm layers will use the running stats to normalize the input activations instead of the batch stats (which are then used to update the running stats) as was done during training. Dropout layers will also be disabled during evaluation.</p>
<p>Internally, <code>model.eval()</code> will recursively set the <code>self.training</code> flag of all <code>nn.Module</code>s to <code>False</code> and depending on the layer implementation this flag will be checked in the <code>forward</code> method to switch the behavior.</p>
<p>Note that calling <code>model.eval()</code> does not disable the gradient calculation. This is why your validation code usually also wraps the forward pass into the <code>with torch.no_grad()</code> guard in order to save memory.</p>
Why can't iter dataloader?

Bro.Circle: <p>when  i use the function: prefetch, i found the the code runtime error<br>
The code like this:<br>
class data_prefetch():<br>
def <strong>init</strong>(self, cfg, loader, is_train):<br>
self.loader = loader</p>
<pre><code>def preload(self):
    try:
        self.next_meta = next(self.loader)
   except:
        self.next_meta = None
        return
</code></pre>
<p>the function run and come to the except.</p>
<p>or like this:<br>
class data_prefetch():<br>
def <strong>init</strong>(self, cfg, loader, is_train):<br>
self.loader = iter(loader)</p>
<pre><code>def preload(self):
    try:
        self.next_meta = next(self.loader)
</code></pre>
<p>when i used the second class, the object come to runtime error?</p>
<p>why and how to solve this problem?</p>

Bro.Circle: <p>import torchvision.transforms as transforms<br>
class data_prefetch():<br>
def <strong>init</strong>(self, cfg, loader, is_train):<br>
self.loader = loader<br>
self.is_train = is_train<br>
self.preload()</p>
<pre><code>def preload(self):
    try:
        self.next_meta = next(self.loader)
        print("try is okey")
    except:
        self.next_meta = None

def next(self):
    meta = self.next_meta
    self.preload()
    return meta

def __len__(self):
    return len(self.loader)</code></pre>

ptrblck: <p>Calling <code>next</code> on an <code>iterator</code> is expected to raise a <code>StopIteration</code> which you would need to handle as seen here:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.randn(5))
loader = DataLoader(dataset)
iter_loader = iter(loader)

for _ in range(6):
    x = next(iter_loader)
# StopIteration
</code></pre>

Bro.Circle: <p>Thanks，maybe other question about custum dataset or dataloader.</p>
LinearRegression

MLGuru: <p>I am trying to solve a linear regression problem using deep learning(Pytorch)<br>
For linear regression<br>
-  Do we need to use the activation function?<br>
- How many hidden layers are recommended?</p>

Andrei_Cristea: <p>If it’s a simple linear regression of the form y = b1 * x + b0 then a single layer with no activation function should suffice (its weight should fit to b1 and its bias should fit to b0).</p>
I get better metrics with the dropout activated on inference

DHOUIBM: <p>Hello,<br>
So I’m training an end to end model for information extraction.<br>
Sadly the code is a bit complicated so I can’t put it here.<br>
I noticed that my model performs way better at training than testing.<br>
So I tried testing with model.train() and to my surprise the metrics jumped from 0.5 to 0.92.<br>
To be sure that the dropout is causing it, I made the same test with model.train() while changing all dropout probabilities to 0 and the metrics dropped to the first 0.5.<br>
This make me think that the dropout is deterministic in some way.<br>
I’m using torch.checkpoint.checkpoint for many blocks that contains dropout layers. And the offciel website indicates that the mechanism of the dropout in this case change in a way that the model can have the same dropout repartition  for the second pass. But the dropout repartition should change from one iteration to the next right ?<br>
Any one can give me some insight please ?<br>
i use PyTorch nightly last version</p>

Andre_Amaral_IST: <p>Hey,<br>
The dropout allows reducing the overfitting effect.<br>
If without it the model has overfitting using it will reduce the overfiting thus improving the performance of your model in the test set</p>

DHOUIBM: <p>Hello,<br>
Thanks for your answer.<br>
In fact dropout is always activated in training, it is on inference (testing) where I have problems.<br>
The model gets way better metrics on inference with dropout activated the model.train() line.</p>

Andrei_Cristea: <p>One thought is that perhaps the dropout is compensating for something poorly specified elsewhere in the model. For instance, perhaps a certain layer doesn’t have a bias parameter and the input data has a very positive mean, but the desired target has a closer to zero-mean. The model then might benefit from dropout pulling the means closer to zero.</p>
<p>So two things you can try:</p>
<ul>
<li>If you have max pooling layers in your model, move those to average pooling.</li>
<li>Add normalization steps throughout your model to keep the data somewhat normalized i.e. mean-zero and close to unit variance.</li>
</ul>
<p>These ought to reduce the effect I hypothesized above, where the model is effectively relying on Dropout to get better predictions.</p>

DHOUIBM: <p>Hey thanks for your answer.<br>
Actuallly I have already many normalization layers.<br>
Also if I train the model without dropout I get good results when testing. So I think that somehow the checkpoint.checkpoint is making the same nodes dropping to zero at each step so that only a part of the model is trained.<br>
I will verify this hypothesis and come back to you</p>
Install pytorch for cuda 11.3 with pip

talhaanwarch: <p>Running <code>nvcc --version</code> shows me following,</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0
</code></pre>
<p>I had already install cudnn on ubuntu 20.04 and tensorflow is picking it.</p>
<p>Now i want to install PyTorch with same cuda and cudnn. I dont want PyTorch to install its own cuda toolkit but use existing installed cuda toolkit. I am unable to find correct pip link for this.</p>

ptrblck: <p>The pip wheels and conda binaries ship with their own CUDA runtime as well as cuDNN, NCCL etc. and won’t be using the system CUDA toolkit.<br>
If you don’t want to use the shipped libraries, you could build PyTorch from source using the locally installed CUDA toolkit.</p>

talhaanwarch: <p>Previously i have cuda toolkit 11.0 installed. And i am installing pytorch via <code>pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html </code> it mean that its not using already installed 11.0 version, but using its own cuda things?</p>

talhaanwarch: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> waiting for your kind response</p>

ptrblck: <p>Yes, the pip wheels and conda binaries ship with their own CUDA runtime as mentioned before. The system-wide installed CUDA toolkit will be used if you are building PyTorch from source or a custom CUDA extenstion.</p>
How to implement a loss that averages the coordinates of a heatmap weighted by its values

mbinry: <p>I want to implement a loss that extracts the maximum location of a heatmap. Since argmax is not differentiate, an alternative method is to calculate the average coordinates of the heatmap weighted by its values.</p>
<p>For example, [[0, 1], [3,2]] has a maximum value of 3 and its coordinate is (1,0), which can be approximately calculated as follows:<br>
0/4x(0,0)+1/4x(0,1)+3/4x(1,0)+2/4x(1,1).</p>
<p>The loss takes a heatmap in the shape of BxNxWxHxD as input, outputting the coordinates in the shape of BxNx3.</p>
<p>I am not familiar with PyTorch and do not know how to implement such loss. I will appreciate it if someone can help me.</p>
Efficient way to normalize a tensor by channel

Surayuth_Pintawong: <p>Hello, I want to normalize an input tensor by the maximum value of each sample.<br>
For example, if the input is</p>
<pre><code class="lang-auto">x = torch.rand(4,1,5,5)
</code></pre>
<p>I can do something like</p>
<pre><code class="lang-auto">max_values = x.view(4, -1).max(dim=1, keepdim=True)[0]
</code></pre>
<p>to get the max values of each sample.<br>
Then I can get the normalized tensor by</p>
<pre><code class="lang-auto">norm_x = x / (max_values * torch.ones(4, 25)).view(4,1,5,5)
</code></pre>
<p>But this looks really ugly and not efficient. Could you please suggest a better way to do this?</p>
<p>Thanks.</p>

AlphaBetaGamma96: <p>Hi,</p>
<p>I’m sure there’s a faster way to do this, you can speed up your code by using broadcasting (rather than allocating a new Tensor of ones to match the size of each Tensor).</p>
<pre><code class="lang-auto">def slow_func(x):
  numel_per_sample = x.size()[1:].numel()
  max_values = x.view(x.shape[0], -1).max(dim=-1, keepdim=True)[0]
  norm_x = x / (max_values * torch.ones(x.shape[0], numel_per_sample)).view(x.shape)
  return norm_x
  
def fast_func(x):
  max_values = x.reshape(x.shape[0],-1).max(dim=-1,keepdim=True)[0]
  norm_x = x / max_values.unsqueeze(2).unsqueeze(3)
  return norm_x
</code></pre>
<p>I ran it for a larger batch size (1000) and here’s the results I got.</p>
<pre><code class="lang-auto">x=torch.rand(1000,1,5,5)

%timeit slow_func(x) #37.2 µs ± 168 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit fast_func(x) #24.9 µs ± 113 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
</code></pre>
<p>I’m somewhat surprised <code>torch.max</code> doesn’t work over multiple dimensions</p>

Surayuth_Pintawong: <p>The code looks promising and very clean. Thank you so much!</p>
How to use ResNet for multilabel classification

albusdemens: <p>[PyTorch newbie] I have a dataset of images, where for each image we have 20+ attributes. I need to train a classifier which takes images as input, and returns the predicted attributes as output. As a first step, I would like to fine-tune ResNet. I need to complete the task using PyTorch. How do you recommend to proceed? Thank you!</p>

eqy: <p>If you have an architecture that returns logits (often the case), you wouldn’t need to change the model architecture instead. I would recommend finetuning via a multilabel loss function e.g., <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiLabelSoftMarginLoss.html#torch.nn.MultiLabelSoftMarginLoss" rel="noopener nofollow ugc">MultiLabelSoftMarginLoss — PyTorch 1.11.0 documentation</a> and iterating on the model/data from there.</p>

albusdemens: <p>Thanks for the hint! I’ll check what’s the loss function of ResNet and I’ll try to change is as you suggested.</p>
Estimate mutual information between two tensors

Arsalan: <p>I am trainin a model with pytorch, where I need to calculate the degree of dependence between two tensors (lets say they are the two tensor each containing values very close to zero or one, e.g. v1 = [0.999, 0.998, 0.001, 0.98] and v2 = [0.97, 0.01, 0.997, 0.999]) as a part of my loss function. I am trying to calculate <a href="https://en.wikipedia.org/wiki/Mutual_information" rel="noopener nofollow ugc">mutual information</a>, but I can’t find any mutual information estimation implementation in pytroch. Has such a thing been provided anywhere?</p>

mxahan: <p>I wondered if the v1 and v2 can be considered probability values?</p>

Arsalan: <p>They are the outputs of a gumbel softmax function to approximate sampling categorical data. But I don’t think it really matters here.</p>

mxahan: <p>Well, to calculate the MI we will be requiring joint and their individual distribution. Is there any way you can find this during your batch execution?</p>

ID56: <p>There isn’t any built in MI score implementation unfortunately. Take a look at <a href="https://github.com/connorlee77/pytorch-mutual-information" rel="noopener nofollow ugc">this</a> and see if you can perhaps somehow use it in your problem.</p>
DDP devices_ids and world_size and ranks multi-host setup

spyroot: <p>Hi Folks,</p>
<p>I need a bit of help. I can’t find proper documentation on the logic of how multi-host<br>
setup in DDP works. I see it mainly focusing on multi-GPU on a single host in all cases.</p>
<p>If you have two hosts and each host has one single GPU, what should the correct rank<br>
on a worker node and, what device id should it pass to DDP, and what is the correct<br>
format for device id? ( note set_cuda is integerger)  set_device is “cuda:0” etc), so most of the tutorial<br>
a bit broken.</p>
<p>For example, in this tutorial<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
<p>Notice that it passes rank as id.  (now if you have a single host with nxGPU it does make sense)  since<br>
host 0 rank = 0 cuda:0.<br>
host 0 rank = 1 cuda:1 ( so in DDP devices_ids[0,1]</p>
<p>But if you have two hosts, my understanding is we count world size from <strong>1</strong>, not 0.<br>
So two hosts and a single GPU per host = world_size 2</p>
<p>Hence on master node DDP devis_id[0] output device=0</p>
<p>// in all torch dock they pass rank but rank 1, not a device id.<br>
Worker node rank 1 DDP must initlized devices_id[0] output device=0</p>
<p>So,<br>
a) when model is created on <strong>worker</strong> device it must to(“cuda:0”)<br>
b) DDP created with devices_id[0]</p>
<p>Right now I’m getting some strange CUDA error when the model starts a training loop.<br>
RuntimeError: CUDA error: invalid device ordinal</p>
<p>Thank you,</p>
'list' object has no attribute 'cuda'

yingchaoxue629: <h1><a name="step-through-with-a-reasonable-batch-size-and-build-up-the-output-dataset-1" class="anchor" href="#step-through-with-a-reasonable-batch-size-and-build-up-the-output-dataset-1"></a>Step through with a reasonable batch size and build up the output dataset</h1>
<blockquote>
<p>model.eval()<br>
outputs = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for images in test_loader:<br>
# move tensors to GPU if CUDA is available<br>
if train_on_gpu:</p>
<pre><code>    images = images.cuda()
    #images = torch.tensor(images).cuda()
test_output = model({'images': images })

outputs.extend(list(test_output.cpu().detach().numpy().reshape(len(test_output))))
</code></pre>
<p>img_names = list( test_df.iloc[:, 0].values)<br>
outputs = [round(x,6) for x in outputs]</p>
<p>output_df = pd.DataFrame({‘Id’: img_names, ‘Pawpularity’: outputs})<br>
output_df.head(10)</p>
</blockquote>
<p>#############################################<br>
This is the source code but the error is reported</p>
<blockquote>
<p>7     if train_on_gpu:<br>
8<br>
----&gt; 9         images = images.cuda()<br>
10         <span class="hashtag-raw">#images</span> = torch.tensor(images).cuda()<br>
11     test_output = model({‘images’: images })</p>
<p>AttributeError: ‘list’ object has no attribute ‘cuda’<br>
After that I modified the code：<br>
but  it makes a new error  i dont know how to solve it<br>
<span class="hashtag-raw">#images</span> = images.cuda()<br>
images = torch.tensor(images).cuda()<br>
9         <span class="hashtag-raw">#images</span> = images.cuda()<br>
—&gt; 10         images = torch.tensor(images).cuda()<br>
11     test_output = model({‘images’: images })<br>
12</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>
Traversing Autograd Graph to Find Dependencies (from Python)

agu: <p>Suppose I have an output <code>torch.Tensor</code> that is computed from some <code>nn.Module</code>'s <code>forward()</code>. My goal is to get, for each tensor <code>t</code> computed by a <code>Function</code> in the autograd subgraph for the <code>nn.Module</code>'s <code>forward()</code>, the list of parameters (which should presumably also be from the same <code>nn.Module</code>) that are <strong>directly</strong> needed to compute <code>t</code>'s gradient (as opposed to some intermediate activation tensor that is not one of the <code>nn.Module</code>'s parameters).</p>
<p>Is this possible to do from Python? If not, is this possible to do from C++? What might a solution look like? At a high-level, it seems like we want to traverse the autograd graph starting from the output <code>torch.Tensor</code>'s <code>grad_fn</code>'s <code>AccumulateGrad</code> object and somehow access what each <code>Function</code> passed into <code>save_for_backward()</code>, checking the input against the <code>nn.Module</code>'s parameters.</p>
Dice score with 2 empty masks

mopi47: <p>Currently my understanding of dice score (in that task of semantic segmentation) is that</p>
<pre><code class="lang-auto">we iterate over our input images 
   for each class 
       we calculate the dice score the ground truth mask of that class and the predicted mask 
</code></pre>
<p>now my question is what happen when the ground truth mask and the predicted mask of certain class are both empty, do we skip the calculation for the this class or do we reward the model for not predicting any false positive by setting the score to 1 directly</p>
Sparse_sparse_matmul memory

omer_sahban: <p>how can we calculate the required memory for <code>torch.sparse.matmul</code>?<br>
a dense <code>(m, k) * (k, n)</code> matmul will create an <code>(m, n)</code> shaped output tensor. but I can’t figure out how much memory a sparse-sparse matmul requires. it seems to be some function of <code>tensor._nnz()</code>, but I want to know what it exactly is. Sometimes <code>torch.sparse.matmul</code> tries to allocate an unreasonable amount of memory and results in OOM error.</p>
Loading pickle files with pytorch dataloader

demonvferrari: <p>Hello Everyone,</p>
<p>I am using the intermediate output of a pretrained CNN model as input to my model. The input to the pretrained CNN model is a color image. I don’t want to compute the intermediate output every time. So, I have saved the intermediate output (60x256x45x80) in pickel format(.pt) using toarch.save(intermediate output). Now i get a bunch of pickel files.</p>
<p>Now, I want to directly load these pickel files(total of 230) using the pytorch dataloader and use it as input to my model and train my model. So, i use the code below.<br>
#************************************************************************************#<br>
class pickle_Dataloader(data.Dataset):</p>
<pre><code>def __init__(self,  root = '.', iext = '.pt', transforms=None):
    
    self.transforms = transforms
    self.image_list = []
    arr = os.listdir(root)

    for i in range(len(arr)):
        filename = root + '/' + arr[i]
        with open(filename, 'rb') as fp:
            item = torch.load(fp)
        
        self.image_list += item
    self.size = len(self.image_list)
    
def __getitem__(self, index):

    # index = index % self.size
    images = self.image_list[index]
    

    return images

def __len__(self):
    return self.size 
</code></pre>
<p>train_data_pkl = pickle_Dataloader(root = ‘.’)<br>
Trainloader_pkl = torch.utils.data.DataLoader(train_data_pkl, shuffle = True , batch_size = BATCH_SIZE  , num_workers = 4, drop_last=False)</p>
<p>#************************************************************************************#</p>
<p>Is this the correct way to load the pickel files with dataloader?</p>
<p>But, I get the RuntimeError: “CUDA out of memory. Tried to allocate 212.00 MiB (GPU 0; 10.92 GiB total capacity; 10.09 GiB already allocated; 9.44 MiB free; 10.14 GiB reserved in total by PyTorch)”</p>
<p>I tried varying the batch size. But, i get the same error.</p>
<p>I am using Python3 and torch version 1.9.0+cu102 on linux environment.</p>
<p>Can you please tell me where am i going wrong?</p>
<p>Thank you</p>

arya47: <p>The custom dataset you’ve written its probably better to name it pickle_Dataset as Dataset and DataLoader are two separate things and make the question confusing.</p>
<p>In the dataset class you’ve written notice how’re you’re loading all the data in the constructor.<br>
Try loading a image in the <strong>getiitem</strong> method instead.<br>
( create a list of all files using os.listdir in the <strong>init</strong> method and load in getitem by indexing the name)</p>

jS5t3r: <p>How to avoid opening the pickle file at once within the dataloader? Can it done with a pointer?</p>
<p>How to just load one sample / batch after another from the pickle file?</p>
<p>My second question would be if I can save data differently. Not as pickle. That makes loading more easier?</p>
<p>Similar thread: <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-load-a-large-numpy-array-as-pytorch-dataset/106549" class="inline-onebox">What is the best way to load a large numpy array as PyTorch dataset?</a></p>