akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Update one tensor column from other tensor

saluei: <p>I have tensor ts1 and ts2 , I want to update ts1 last column from value of second column of ts2 on the base of condition ts2[:,1]==ts1[:,0] , but without loop, thankyou , any help</p>
<pre><code class="lang-auto">ts3=torch.tensor([
 [ 24,  24,   8,0 ],
 [ 38 , 100 , 93,0],
 [ 65 , 91 , 73 ,0],
 [ 49 , 26 , 45 ,0],
 [ 78 , 54 , 11 ,0],
 [ 24 , 98 , 12 ,0],
 [ 27 , 55 , 50 ,0],
 [ 11 , 65 , 96 ,0],
 [ 45 , 40 , 16 ,0],
 [ 99 , 54 , 27,0 ]],dtype=torch.int32)

ts2=torch.tensor([[100,10],[26,20],[54,30]],dtype=torch.int32)

the result should be

([
 [ 24,  24,   8,0 ],
 [ 38 , 100 , 93,10],
 [ 65 , 91 , 73 ,0],
 [ 49 , 26 , 45 ,20],
 [ 78 , 54 , 11 ,30],
 [ 24 , 98 , 12 ,0],
 [ 27 , 55 , 50 ,0],
 [ 11 , 65 , 96 ,0],
 [ 45 , 40 , 16 ,0],
 [ 99 , 54 , 27,30 ]])

</code></pre>

callahman: <p>I spent a little time trying to find something similar to a “Join on value”, but couldn’t come up with anything…</p>
<p>This is the best I could come up with:</p>
<pre><code class="lang-auto">import torch

ts3=torch.tensor([
 [ 24 , 24 , 8 ,0 ],
 [ 38 , 100 , 93 ,0],
 [ 65 , 91 , 73 ,0],
 [ 49 , 26 , 45 ,0],
 [ 78 , 54 , 11 ,0],
 [ 24 , 98 , 12 ,0],
 [ 27 , 55 , 50 ,0],
 [ 11 , 65 , 96 ,0],
 [ 45 , 40 , 16 ,0],
 [ 99 , 54 , 27 ,0]],dtype=torch.int32)

ts2=torch.tensor([[100,10],[26,20],[54,30]],dtype=torch.int32)


for x, y in ts2[:]:
    ts3[:, 3] = torch.where(ts3[:, 1] == x, y, ts3[:, 3])
</code></pre>
<p>I know you didn’t want a loop, but this may help cut down on the amount of looping you were originally thinking there would be.</p>
<p>I’m still pretty new to Pytorch, so there is likely still a better way.</p>

saluei: <p>thank you callahman,<br>
I want without loop !,</p>
How to efficiently perform averaging across predefined groups in a tensor

gaosh: <p>Hi,<br>
I am wondering is there a efficient way to perform group averaging in a tensor.<br>
Take 1-D vector as a sample:<br>
vector: [1, 2, 3, 4, 5, 6]<br>
group_label:[1, 2, 1, 2, 3, 3]<br>
results:[2, 3, 5.5 ]<br>
The ideas is that values in a tensor with same group label will be averaged and produce a new tensor.</p>

ptrblck: <p>You could use <code>scatter_add</code> and a small hack to get the unique counts of your indices:</p>
<pre><code class="lang-python">x = torch.arange(1, 7, dtype=torch.float)
idx = torch.tensor([0, 1, 0, 1, 2, 2])
idx_unique = idx.unique(sorted=True)
idx_unique_count = torch.stack([(idx==idx_u).sum() for idx_u in idx_unique])
res = torch.zeros(len(idx_unique)).scatter_add(0, idx, x)
res /= idx_unique_count.float()
</code></pre>
<p>I think it’s time to add a <code>return_counts</code> option to <code>torch.unqiue</code>.</p>

gaosh: <p>Thanks for the help.</p>

GHALI_Ahmed: <p>Thank you very much Sir.</p>

GHALI_Ahmed: <p>thank you for your question!</p>
Multiple output head

sripathisony_sony: <p>how to implement multiple head classification fo single model</p>

ptrblck: <p>You can define multiple heads in the <code>__init__</code> method of your model and use different paths in the <code>forward</code> method. A short example is given e.g. <a href="https://discuss.pytorch.org/t/multiple-output-using-pytorch/48635/6">here</a>, where two classifiers are used.</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks, one doubt I have trained my model with different two datasets. then I want to have two output heads.</p>
<p>I doubt if I give one input how do my model give two different outputs? (won’t be the same output?)<br>
not changing any loss function,</p>

ptrblck: <aside class="quote no-group" data-username="sripathisony_sony" data-post="3" data-topic="111575">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sripathisony_sony/48/33487_2.png" class="avatar"> sripathisony_sony:</div>
<blockquote>
<p>one doubt I have trained my model with different two datasets. then I want to have two output heads.</p>
</blockquote>
</aside>
<p>Is the current model using a single output layer? Are these two datasets using the same number of classes? If so, what would the two heads to in your use case, i.e. would they try to predict a different number of classes?</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
current model using a single output layer. yes, the same number of classes.</p>
<p>what is the advantages of using two head classification(if we have same classes)<br>
another dataset I have different labels.</p>

ptrblck: <p>I don’t know if there would be any advantages, but it depends on your use case and apparently you have some ideas with it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
If both heads are supposed to return the same number of classes (e.g. 2 classes), but different labels, you would have to be able to split the input data, so that the first head would get data from the first dataset and the second one from the second dataset. While this would be possible during training (since you have the labels) it won’t be possible during evaluation/testing.</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  thank you</p>
<p>If I have a number of labels different in two datasets, how I can train my model related to different head</p>

ptrblck: <p>As explained before, you could split the input batches using the targets and forward the data to the corresponding head during training. However, since this won’t work during testing you would have to come up with a strategy how this can be done and why you want to use different heads.</p>
Running part of the model on CPU

milebril: <p>Hi,</p>
<p>I’m using some libraries who don’t have GPU support for the thing I’m trying to do. This forces me to cast my model to cpu using <code>model.to('CPU')</code> and back after I’m done with the CPU part.<br>
My needed tensors are converted using:<br>
<code>radiance_clone = radiance.clone().detach().requires_grad_(True).to('cpu')</code></p>
<p>A simple forward pass works as intended. And results me the same results as if I would run solely on CPU. But for training CPU training is just too slow…<br>
When training I do not get the same results as just training on CPU. My model does some weird stuff and converges really quick.</p>
<p>Are there weird things happening behind the scenes when converting my model back and forth in an epoch to run a specific part on the CPU? Or do I clone my tensors in a wrong way?</p>
<p>Any help is appreciated!</p>
<p>Emil</p>
How to crop weight file to use as pre-trained model?

omeralierdemir: <p>hi everyone,<br>
I have created autoencoder model with use ResNet backbone. Then I trained this model. I want to subtract the weight of the encoder part to load it into my normal ResNet model. But I don’t know how to do.<br>
I tried crop autoencoder’s encoder parts model after load autoencoder weight file then I saved this model. But layers name changed and I could not load weight into normal ResNet model.</p>
<p>How can I do ?</p>

comely420x: <p>By subtract the weights you mean, you want to remove the weights from the autoencoder or you want to take those weights out and place them into your ResNet model?</p>

omeralierdemir: <p>I want to take those weights out and place them into your ResNet model (just encoder part weights)</p>

comely420x: <p>You can take out the model state dict from your encoder part and add then take out the other part from the ResNet model.</p>
<pre><code class="lang-auto">Encoder = encoder.__dict__
</code></pre>
<p>you can then take out the first layer like:</p>
<pre><code class="lang-auto">first_layer = Encoder._modules['layer1']
</code></pre>
<p>and then you can either take out the layers that you need from your resnet or you can just change the layers.</p>
<pre><code class="lang-auto">resnet = torchvision.models.__dict__(resnet50)(pretrained=true)
final_layer = resnet._modules['layer4']
</code></pre>
<p>and then pass combine these two. Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

omeralierdemir: <p>I will try. Thanks for your help</p>
Pretrained Model for 128X128 RGB images

rmokady: <p>Hi All,<br>
I want to use pretrained model for feature extraction (pretrained on Imagenet). I prefer GoogleNet, but I think ResNet, VGG or similar will do.<br>
The problem is that I want to use 128X128 RGB images and I notice that the images in torchvision.models were pretrained on larger images.</p>
<p>Do you familiar with such pretrained model (128X128)?</p>

Sunshine352: <p>Two methods are available:<br>
One is to resize your input images to 229x229, then directly using ResNet or VGG etc.<br>
Two is that You can revise the net-structure of  ResNet, VGG etc.,making ResNet, VGG-variants accept 128x128 RGB images, then load part weights of the pretrained model ResNet or VGG，finally use your own dataset to train/finetune the revised Network.</p>

rmokady: <p>I don’t want to resize as I’m afraid my GPU straggle with large images.</p>
<p>By loading part weights, you mean load the weights to only some of the layers?</p>

ptrblck: <p>If resizing to <code>224x224</code> isn’t an option, note that the <code>torchvision</code> models now were updated to adaptive pooling layers (e.g. <a href="https://github.com/pytorch/vision/blob/6eac421ca2ea1ca52f49e10d7ac191c0d6900df1/torchvision/models/vgg.py#L28" rel="nofollow noopener">vgg16</a>) so that the input size is now more flexible.</p>

111429: <p>thanks.<br>
maybe “where H and W are expected to be at least 224.”<br>
in <a href="https://pytorch.org/docs/stable/torchvision/models.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models — PyTorch 1.7.1 documentation</a><br>
could be removed?</p>

ptrblck: <p>This should be the case for some models, but the minimal size would need to be determined for each model separately. Feel free to create a GitHub issue and let us know, if you would be interested in fixing this. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Accessing different layers in pytorch

faraz_moradi: <p>I want to access my network last layer in a class but I got this error!</p>
<p>import torch<br>
import torch.nn as nn<br>
import numpy as np</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self, number_of_class, ):<br>
super(Net, self).<strong>init</strong>()<br>
self._conv1 = nn.Conv2d(1, 32, kernel_size=(3, 5))<br>
self._pool1 = nn.MaxPool2d(kernel_size=(1, 3))<br>
self._batch_norm1 = nn.BatchNorm2d(32)<br>
self._prelu1 = nn.PReLU(32)<br>
self._dropout1 = nn.Dropout2d(.5)</p>
<pre><code>    self._conv2 = nn.Conv2d(32, 64, kernel_size=(3, 5))
    self._pool2 = nn.MaxPool2d(kernel_size=(1, 3))
    self._batch_norm2 = nn.BatchNorm2d(64)
    self._prelu2 = nn.PReLU(64)
    self._dropout2 = nn.Dropout2d(.5)
    
    self._fc1 = nn.Linear(3072, 500)
    self._batch_norm3 = nn.BatchNorm1d(500)
    self._prelu3 = nn.PReLU(500)
    self._dropout3 = nn.Dropout(.5)
    
    
    self.fc = nn.Linear(500, number_of_class)
    
    self.initialize_weights()
    
    print(self)
    
    print("Number Parameters: ", self.get_n_params())

def get_n_params(self):
    model_parameters = filter(lambda p: p.requires_grad, self.parameters())
    number_params = sum([np.prod(p.size()) for p in model_parameters])
    return number_params

def init_weights(self):
    for m in self.modules():
        torch.nn.init.kaiming_normal(m.weight)
        m.bias.data.zero_()

def initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            torch.nn.init.kaiming_normal_(m.weight)
            m.bias.data.zero_()
        elif isinstance(m, nn.Linear):
            torch.nn.init.kaiming_normal_(m.weight)
            m.bias.data.zero_()

def forward(self, x):
    conv1 = self._dropout1(self._prelu1(self._batch_norm1(self._conv1(x))))
    # print(conv1.shape)
    pool1 = self._pool1(conv1)
    conv2 = self._dropout2(self._prelu2(self._batch_norm2(self._conv2(pool1))))
    # print(conv2.shape)
    pool2 = self._pool2(conv2)
    # print(pool2.shape)
    flatten_tensor = pool2.view(pool2.size(0), -1)
    # print(flatten_tensor.shape)
    fc1 = self._dropout3(self._prelu3(self._batch_norm3(self._fc1(flatten_tensor))))
    # print(fc1.shape)
    Output = self.fc(fc1)
    # print(Output.shape)
    return Output
</code></pre>
<p>in_features = Net.fc.in_features</p>
<p>type object ‘Net’ has no attribute ‘fcPreformatted text’</p>

comely420x: <p>I had to something like this some time ago, and I just returned the output before the linear layer i.e.</p>
<pre><code class="lang-auto">before_fc1 = fc1
output = self.fc(fc1)
return before_fc1, output 
</code></pre>
<p>I am guessing you have to send this feature to your loss function, in that case this may work.</p>
Pytorch model converted to 'DataParallel' object when `CUDA_VISIBLE_DEVICES` not set

kyy: <p>Hi, I am using Catalyst for training my model.  It works great if I explicitly set <code>CUDA_VISIBLE_DEVICES</code>, however if I did not set it, I would have following error:</p>
<pre><code class="lang-auto">ModuleAttributeError: 'DataParallel' object has no attribute '_loss'
</code></pre>
<p>Looks like that the model has been automatically converted to <code>DataParallel</code> object in Catalyst runner while the corresponding <code>_loss</code> method of my model was not assigned to it.  Does anyone know if there is a way to either:</p>
<ol>
<li>correctly assign <code>_loss</code> method of the model to <code>DataParallel</code> object or</li>
<li>disable conversion to <code>DataParallel</code> object in Catalyst runner?</li>
</ol>
<p>Thank you!</p>

ptrblck: <p>CC <a class="mention" href="/u/scitator">@scitator</a> as the dev of Catalyst <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Error when "make" command is run in glow/fmt/build

Kittu28: <p>i’ve got the error as shown in the attachment when i try to run “make” in glow/fmt/build/<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b99b8a964224f5afbfb2b07dade3936a9cf85e6.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b99b8a964224f5afbfb2b07dade3936a9cf85e6" title="Screenshot from 2021-02-10 16-58-29"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b99b8a964224f5afbfb2b07dade3936a9cf85e6.png" alt="Screenshot from 2021-02-10 16-58-29" data-base62-sha1="1ECB74RqOBbvxaBMtFjdTwi1Lwi" width="690" height="287" data-dominant-color="411F35"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-10 16-58-29</span><span class="informations">732×305 47.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jfix: <p>Thanks for reporting this. Unfortunately this is an issue inside the <code>fmt</code> dependence, and not specific to Glow. I imagine this issue occurs due to the compiler you’re using to compile <code>fmt</code>. One option could be to try a different compiler, or alternatively change the code so it won’t complain there (i.e. just add the <code>{}</code> there to fix the problem in the source code)</p>

Kittu28: <p>I changed the compiler and it worked ,thanks…!</p>
Is there a non-blocking LSTM?

asdmjhe2010: <p>As I learned from the document, “By default, GPU operations are asynchronous. When you call a function that uses the GPU, the operations are enqueued to the particular device, but not necessarily executed until later. This allows us to execute more computations in parallel, including operations on CPU or other GPUs.”</p>
<p>However, I find that LSTM doesn’t seem to be asynchronous. I don’t know why. See the examples below.</p>
<p>LSTM:</p>
<pre><code class="lang-auto">a = torch.rand(1000, 20, 10000).to(device)
net = nn.LSTM(10000, 100).to(device)
torch.cuda.synchronize()
t = time.time()
with torch.no_grad():
    for i in range(10):
        c = net(a)
print(time.time() - t)
torch.cuda.synchronize()
print(time.time() - t)
</code></pre>
<pre><code class="lang-auto">0.3161756992340088
0.3302645683288574
</code></pre>
<p>Linear:</p>
<pre><code class="lang-auto">a = torch.rand(1000, 20, 10000).to(device)
net = nn.Linear(10000, 100).to(device)
torch.cuda.synchronize()
t = time.time()
with torch.no_grad():
    for i in range(10):
        c = net(a)
print(time.time() - t)
torch.cuda.synchronize()
print(time.time() - t)
</code></pre>
<pre><code class="lang-auto">0.0007715225219726562
0.02486276626586914
</code></pre>

ptrblck: <p>To get some more information about both workloads, I would recommend to profile them via e.g. Nsight Systems and check the timelines for both workloads. This would allow you to see any synchronizations and how how many kernels are executed etc.</p>
Custom weight intialization

comely420x: <p>I am sorry if this is a very simple question. How do I initialize the weights of my network such that the weights are produced by the softmax function on the learnable log-probabilities?</p>

ptrblck: <p>You could create the desired initial weights using any method and copy them into the model parameter via:</p>
<pre><code class="lang-python">my_weight = ... # your method to create the initial weights
with torch.no_grad():
    model.my_layer.weight.copy_(my_weight)
</code></pre>
Semantic Segmentation Loss Function & Data Format Help

Thummper: <p>Hi there, I was wondering if somebody could help me with semantic segmentation.</p>
<p>I am using the segmentation_models_pytorch library to train a Unet on the VOC2012 dataset.</p>
<p>I have not trained semantic segmentation models before, so I am not sure what form my data should be in. Specifically, I am not sure what loss function to use, and what format my data needs to be in to go into that loss function.</p>
<p>So far:<br>
The input to my network is a bunch of images in the form:<br>
[B, C, H, W]</p>
<p>This is currently [12, 3, 512, 512]</p>
<p>The output from my network is in the form:<br>
[B, NUMCLASSES, H, W]</p>
<p>This is currently [12, 20, 512, 512]</p>
<p>I am loading in my normal images and segmentation truths using PIL.<br>
So my input images have 3 channels, and my segmentation images have 1 channel (in PIL’s P mode).</p>
<p>So my input image goes into the network and outputs a shape of [12, 20, 512, 512]<br>
My ground truth images are in the shape of [12, 1, 512, 512]</p>
<p>My question:<br>
What loss function should I use, and what format / shapes should my data be in?<br>
Is is possible to just shove my outputs into a loss function as they are and calculate a loss, or do I need to reshape them in any way?<br>
Do I have to calculate an output prediction with output.argmax(1) for input into a loss function?</p>
<p>Thanks for your time.</p>

ptrblck: <p>The shapes look almost right. For a multi-class segmentation use case you could use <code>nn.CrossEntropyLoss</code> as the criterion, which expects the model output to contain logits in the shape <code>[batch_size, nb_classes, height, width]</code>. The target should have the shape <code>[batch_size, height, width]</code> (remove <code>dim1</code> in your script via <code>target = target.squeeze(1)</code>) and should contain the class indices in the range <code>[0, nb_classes-1]</code>.</p>
<p>Assuming you are dealing with 20 classes, here is a small code example:</p>
<pre data-code-wrap="python"><code class="lang-python">output = torch.randn(2, 20, 24, 24, requires_grad=True)
target = torch.randint(0, 20, (2, 24, 24))
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
</code></pre>
<aside class="quote no-group" data-username="Thummper" data-post="1" data-topic="111486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/c67d28/48.png" class="avatar"> Thummper:</div>
<blockquote>
<p>Do I have to calculate an output prediction with output.argmax(1) for input into a loss function?</p>
</blockquote>
</aside>
<p>No, <code>nn.CrossEntropyLoss</code> expects the logits for each class. You could use <code>torch.argmax(output, dim=1)</code> to compute the predictions, where each pixel would contain the the predicted class index.</p>
Linking errors when using nightly prebuilt binaries

IgnacioPickering: <p>I’m currently using the prebuilt binary version of pytorch nightly, built with cuda 11.0.<br>
which I installed from <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>, using pip.</p>
<p>I have a custom model that I can normally run on my GPU (Quadro RTX 600) without any issues<br>
(My cuda driver is 460.32.03)</p>
<p>My issue is that if I try to evaluate my model after jit-scripting it, I get a linker error:</p>
<pre><code class="lang-auto">    result = self.forward(*input, **kwargs)
../../../anaconda3/envs/myenv/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:132: in prof_meth_call
    return prof_callable(meth_call, *args, **kwargs)
../../../anaconda3/envs/myenv/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: in prof_callable
    return callable(*args, **kwargs)
E   RuntimeError: Error in dlopen or dlsym: libnvrtc.so.11.0: cannot open shared object file: No such file or directory
</code></pre>
<p>This is very strange to me, since <strong>I believed that if I used the prebuilt binaries all necessary cuda libraries would be correctly linked with pytorch, and pytorch would not be using / needing my systems cuda. Am I mistaken about this?</strong></p>
<p>Furthermore, I actually do have cuda 11.2 installed in my system, for reference the output of nvcc --version is</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0
</code></pre>
<p>Also, this error doesn’t appear at all if I use pytorch 1.7.1</p>
<p>Is it possible that there is actually something wrong with my cuda installation / driver or is this some internal pytorch error?</p>
<p>Other potentially useful info: The linking error appears if I use the default fuser; if I run this using “fuser2” instead (the nvfuser I believe) then I get:<br>
<code>E   RuntimeError: false INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/codegen/cuda/type.cpp":439, please report a bug to PyTorch. No data type found for scalar type.</code></p>
<p>This is frustrating since my custom model is very large, and I’m unable to pinpoint which part exactly is the one that is causing this, so I think it is very unlikely I would be able to post some minimal working example.</p>

ptrblck: <p>Thanks for reporting it. Issue is tracked <a href="https://github.com/pytorch/pytorch/issues/52075">here</a>.</p>
Understanding memory allocation

vb000: <p>Hi,</p>
<p>I’ve a model which accepts a series of inputs and uses previous predictions as inputs for subsequent predictions. So a batch, for instance, is a python list with a series of inputs and each input in the list can have batch dimension. I’ve an evaluation loop like this with memory allocation printed out after each prediction:</p>
<pre><code class="lang-auto">    # compute metrics over the dataset
    for i, batch in enumerate(tqdm(dataloader)):
        if params.cuda:
            batch = net.batch_to_cuda(batch)
        print('memory allocated for model and input batch:', torch.cuda.memory_allocated(0))

        output_batches = []
        targets = []
        losses = []

        # Train on consecutive predictions -- feeding previous predictions as "previous" frames.
        for j, model_in in enumerate(batch):
            # Ground truth
            targets.append(model_in['target'])

            # Replace neigbor_hd with prior predictions
            for k, out in enumerate(reversed(output_batches)):
                if k &lt; len(model_in['previous']):
                    model_in['previous'][k] = out

            # compute model output and loss
            output_batches.append(model(model_in))
            loss = loss_fn(output_batches[j], targets[j])

            losses.append(loss.item())
            print('memory allocated after pred %d:' % j, torch.cuda.memory_allocated(0))
        
        total_output_size = 0
        for out in output_batches:
            total_output_size += out.element_size() * out.nelement()
        print('Total output size:', total_output_size)
</code></pre>
<p>Since each prediction is an independent inference, and noting the fact that I only indent to use the value of the previous prediction, shouldn’t the memory allocated after each prediction should remain the same? But, I’m seeing progressive allocation of more and more memory:</p>
<pre><code class="lang-auto">memory allocated for model and input batch: 198102016
memory allocated after pred 0: 4024618496
memory allocated after pred 1: 7850085888
memory allocated after pred 2: 11675553280
Total output size: 3145728
</code></pre>

googlebot: <p>You have a single autograd graph here, with pending gradient computations for all iterations, hence nothing gets released. loss.backward() inside the loop may help, but you’d have to split graph by .detach() -ing model outputs passed to a next model.</p>

vb000: <p><a class="mention" href="/u/googlebot">@googlebot</a> You’re right, for training loop, I detached the previous model outputs when using them as next inputs, memory allocated remains the same after each prediction. For evaluation though, apparently I just needed to add this to <code>torch.no_grad()</code> context and memory is freed after each prediction.</p>
Variable output shape for adaptivemaxpooling

sajidmahmud14: <p>Hi I would like to create a network, where the last layer will be an adaptive max-pooling layer, and the output shape will vary on the input size of the network. Now in my dataset, the input sizes are not of equal size. Here is my network code:</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self, blocks, block, in_channel, out_channel):

    super(Net, self).__init__()

    layer = []

    layer.append(block(in_channel,out_channel))

    for i in range(0,blocks):

      layer.append(block(out_channel,out_channel))

    self.network = nn.Sequential(*layer)

    self.maxpoolbyrow = nn.AdaptiveMaxPool2d(output_size=(50,50))

def forward(self, x):

    out = self.network(x)

    out = self.maxpoolbyrow(out)

    return out.size()
</code></pre>
<p>Now here in the output_size argument inside nn.AdaptiveMaxPool2d, I want it to be (L x 50) instead of a fixed (50x50), where the L is the Height of the input image. Considering the image sizes are variable, how can I extract the height of the input in the forward function and add it as the output_size parameter in AdaptiveMaxPool2d?</p>

ptrblck: <p>You could get the shape of the input via: <code>x.size()</code> or <code>x.shape</code> and use the functional API via <code>out = F.adaptive_max_pool2d(out, output_size=...)</code>.</p>
Early stopping Criteria on Valid set or Training set?

sgaur: <p>I just need to know how to decide early stopping criteria do I use early stopping on Validation set or training set.</p>
<p>Like:-<br>
Training set is not converging but validation set still converging or Vice versa ??</p>

jmandivarapu1: <aside class="quote no-group" data-username="sgaur" data-post="1" data-topic="75133">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/3da27b/48.png" class="avatar"> sgaur:</div>
<blockquote>
<p>Training set is not converging but validation set still converging or Vice versa ??</p>
</blockquote>
</aside>
<p>Usually you stop when the training loss remains constant over the time.</p>

sgaur: <p>So my training loss became constant for last 4-5 epochs but my validation set still converging.I can stop it.?</p>

jmandivarapu1: <p>Yeah you should stop it. Because validation set performance should be checked once you stopped the training.</p>

TinfoilHat0: <p>That doesn’t seem correct to me. You measure both validation, and training performance during the training. Ideally, you should stop once validation loss converges, i.e., doesn’t decrease for a few epochs.<br>
Then, you measure the final performance on a test set.</p>
DP and Transfer Learning

Andreas_Kopp: <p>Hi Opacus community,<br>
I am looking for experiences / best practices for using DP with transfer learning. Let’s say a hospital decides to build a DP image classification model based on patient data.</p>
<p>3 scenarios com to mind:</p>
<ol>
<li>baseline network pre-trained on public dataset (from a similar domain, e.g. x-rays)</li>
<li>baseline network pre-trained on existing data of the hospital (classic, non private way)</li>
<li>baseline network pre-trained with DP on existing data of the hospital</li>
</ol>
<p>I would assume that generally all approaches could make sense because probably less epochs are required compared to training from scratch (hence, spending less privacy budget).</p>
<p>I would appreciate learning your experiences / thoughts about these scenarios.</p>
<p>Thanks<br>
Andreas</p>

karthikprasad: <p>Hello <a class="mention" href="/u/andreas_kopp">@Andreas_Kopp</a> !<br>
With transfer learning, there are two separate datasets to consider: one used for training the baseline model and another for fine-tuning/training your final model.</p>
<p>In scenarios 1, the privacy of the public dataset is not preserved, but the privacy of the final model is preserved if you train with DP.</p>
<p>In scenario 2, just as in scenario 1, the privacy of the dataset used to train your baseline model is NOT preserved, but only that of the data used to train your final model is. If preserving the data used to train your baseline model is not important for your use case, this might result in better accuracy than that of scenario 1 (depending, ofcourse, on your task as well as the size and distribution of this data compared the public dataset)</p>
<p>In scenario 3, the privacy of all the data is preserved, but the accuracy will be very low.</p>
<p>Hope this adds some clarity.</p>
Nll loss and crossentropy loss not decreasing

Rukan_Mahfuz: <p>Apologies for the long post. I am very new to pytorch. And really confused about crossentropy loss and nll loss in pytorch.<br>
I am trying to use softmax on a sequence to sequence problem. I have targets of two types. First target has number of tokens of 13 and second one has 100. The model has two outputs and i am computing loss for each type and sum them. I have the model output shape <code>[seq_length, number of tokens, embedding dim]</code> and output shape as <code>[seq_length, number of tokens]</code>. For nll_loss i am using log softmax in output dimension 1 then passing output to loss function by transposing last two dimensions like <code>[seq_length, number of tokens, embedding dim]</code>. Also in crossentropy i am passing raw logits of shape <code>[seq_length, number of tokens]</code>. But each case loss just fluctuates and does not decrease. What I am doing wrong here.</p>
<p>A short description of full process is as follows.<br>
Inputs are a sequence of tuples like <code>[(1,15), (2,27), (13,10)]</code>. First i one hot encoded each token in each tuple. Then embed them so that first elements of each tuple have shape <code>[10, embed dim]</code>, then add the two tensors of each tuple to create one embedding for each tuple. finally stack these to create the input of shape <code>[3, 110, embed dim]</code> And the two output logits have shape <code>[3, 10, embedding dim]</code> and <code>[3, 100, embedding dim]</code>. And if the target is <code>([2,27), (12,10), (5, 90)]</code>. They are onehot encoded separately so the target 1 has shape <code>[3, 10]</code> and target shape is <code>[3,100]'</code></p>
Layer wise count of tensor using torch.count_nonzero()

cbd: <p>Lets say “a=torch.rand(1,4,2,2)” and “count_a=torch.zeros(4)”.</p>
<p>I want count of non zero of a(0,0,:,: ) to store in count_a(0), a(0,1,:,: ) to store in count_a(1) etc</p>
<p>If i used below code it stores total non zero count of “a” in to “count_a”.</p>
<p>Code:</p>
<p>import torch<br>
count_a=torch.ones(4)<br>
a=torch.rand(1,4,2,2)<br>
count_a=torch.count_nonzero(a)<br>
print("a value is ",a)<br>
print("count is ",count_a)</p>
<p>Output:</p>
<p>a value is tensor([[[[0.8222, 0.4329],<br>
[0.4672, 0.2243]],</p>
<pre><code>     [[0.3086, 0.6668],
      [0.2699, 0.3848]],

     [[0.0251, 0.9675],
      [0.7624, 0.8663]],

     [[0.4905, 0.8086],
      [0.0784, 0.4926]]]])
</code></pre>
<p>count is tensor(16)</p>
<p>I want output to be tensor[4 4 4 4] as each layer has 4 non zero value instead of tesnor(16).</p>

ptrblck: <p>I don’t quite understand the expected output shape.<br>
Your input tensor <code>a</code> has 16 values (<code>1*4*2*2=16</code>), so <code>count_nonzero</code> can return a value in <code>[0, 16]</code>, while the expected output shape would contain more values.<br>
Could you explain your use case a bit more?</p>

cbd: <p>If “a=torch.rand(1,4,2,2)” then think of 4 layer having 2x2  value. If “a=torch.rand(1,8,2,2)” then think 8 layer having 2x2  value.</p>
<p>I want layer wise non zero count value. so for “a=torch.rand(1,<strong>4</strong>,2,2)” if all <strong>4</strong> layer 2x2 value is non zero then i want output to be “count_a=[4 4 4 4]”. Sorry i forgot to mention that count_a is of size “torch.Size([4])” and i want output to be tensor([4., 4., 4., 4.])</p>
<p>If “a=torch.rand(1,<strong>8</strong>,2,2)”,  if all the <strong>8</strong> layers 2x2 values are non zero then i want output to be tensor([4., 4., 4., 4., 4., 4., 4., 4. ]).</p>

ptrblck: <aside class="quote no-group" data-username="cbd" data-post="3" data-topic="111286">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>I want layer wise non zero count value. so for “a=torch.rand(1,<strong>4</strong>,2,2)” if all <strong>4</strong> layer 2x2 value is non zero then i want output to be “count_a=[4 4 4 4]”. Sorry i forgot to mention that count_a is of size “torch.Size([4])” and i want output to be tensor([4., 4., 4., 4.])</p>
</blockquote>
</aside>
<p>Thanks for the clarification!<br>
I misunderstood the <code>[4, 4, 4, 4]</code> output as the shapes, not the values.<br>
In that case, you could flatten the last two dimensions and use the <code>dim</code> argument in <code>count_nonzero</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.rand(1,4,2,2)
count_a = torch.count_nonzero(a.view(1, 4, -1), 2)
print("a value is ",a)
print("count is ",count_a)
&gt; count is  tensor([[4, 4, 4, 4]])
</code></pre>

cbd: <p>Thanks. That’s exactly i want.</p>
Difference between SelectBackward and MaxBackward1

anon: <p>In PyTorch, if I do <code>x = torch.tensor([1., 2., 3.], requires_grad=True)</code>, then <code>max(x)</code> returns <code>tensor(3., grad_fn=&lt;SelectBackward&gt;)</code>, while <code>torch.max(x)</code> returns <code>tensor(3., grad_fn=&lt;MaxBackward1&gt;)</code>. What is the difference between these two gradients? When calling <code>backward()</code> they both give the same answer at least in this small example.</p>

soulitzer: <p>The two backward functions behave differently when you try an input where multiple indices are tied for maximum. SelectBackward routes the gradient to the first maximum value while MaxBackward evenly distributes.</p>
<pre><code class="lang-auto">x = torch.tensor([1., 1.], requires_grad=True)
print(torch.autograd.grad(torch.max(x), (x,))[0]) # tensor([0.5000, 0.5000])
print(torch.autograd.grad(max(x), (x,))[0]) # tensor([1., 0.])
</code></pre>

anon: <p>Thank you, very interesting. Is there any page that documents all grads and their behaviors? I couldn’t find it by googling.</p>

soulitzer: <p>In general we don’t consider what backward functions we use part of the public api, so unfortunately they aren’t super well documented. They are subject to change, so try not to rely on that behavior.</p>
<p>For example when you call <code>max(tensor)</code> in versions&gt;=1.7, the grad_fn is now UnbindBackward instead of SelectBackward because max is a python builtin that relies on<code>Tensor.__iter__</code>.<br>
The behavior for <code>torch.max</code> to evenly distribute gradients is also new in 1.7 and that is documented in the release notes here [0].</p>
<p>[0] <a href="https://github.com/pytorch/pytorch/releases/tag/v1.7.0" class="inline-onebox" rel="noopener nofollow ugc">Release PyTorch 1.7 released w/ CUDA 11, New APIs for FFTs, Windows support for Distributed training and more · pytorch/pytorch · GitHub</a></p>
Quantized model weights

yayapa: <p>Hello everyone,<br>
I am quantizing the retinanet using standard methods of Pytorch, namely PTQ and QAT and got a great results. The model size has been reduced from 139MB to 39MB and Inference time on cpu from 90min to 20min for a big valid dataset by accuracy loss smaller that 1%. Thus, although the results are great, I tried to check the weights of the quantized network and found out, if I use</p>
<p>print(model.head.cls_subnet[0].conv.weight().<strong>int_repr()</strong>)</p>
<p>I get a really quantized integer tensor like<br>
[[-33,   6, -56],<br>
[-36,  47,  24],<br>
[ 12,   1,  25]],</p>
<pre><code>     [[-22,  18,  22],
      [-45,  43, -55],
      [  4,   1, -58]],



     [[ 19,  27,  10],
      [-73,   9, -53],
      [  2, -38, -24]]]], dtype=torch.int8)
</code></pre>
<p>But if I access a weight without <strong>int_repr()</strong></p>
<p>print(model.head.cls_subnet[0].conv.weight())</p>
<p>I get a tensor like<br>
[[-0.0096,  0.0017, -0.0163],<br>
[-0.0105,  0.0137,  0.0070],<br>
[ 0.0035,  0.0003,  0.0073]],</p>
<pre><code>     [[-0.0064,  0.0052,  0.0064],
      [-0.0131,  0.0125, -0.0160],
      [ 0.0012,  0.0003, -0.0169]],



     [[ 0.0055,  0.0079,  0.0029],
      [-0.0212,  0.0026, -0.0154],
      [ 0.0006, -0.0111, -0.0070]]]], size=(256, 256, 3, 3),
   dtype=torch.qint8, quantization_scheme=torch.per_channel_affine,
   scale=tensor([0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003,
</code></pre>
<p>So the question is: Was the quantitation done correctly or I still use the full-precision weights? Why does it look like? Is it an intern representations of weights?</p>
<p>Thank you in advance.<br>
Best regards,<br>
yayapa</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="yayapa" data-post="1" data-topic="111397">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yayapa/48/32522_2.png" class="avatar"> yayapa:</div>
<blockquote>
<p>So the question is: Was the quantitation done correctly or I still use the full-precision weights? Why does it look like? Is it an intern representations of weights?</p>
</blockquote>
</aside>
<p>If you print a quantized tensor, it’s expected to see the floating point values, a scale and a zero point.  The internal representation is stored in integers, and you can see that with <code>int_repr()</code>.  You can use the equation <code>fp = clamp(std::nearbyint(q - zp) * scale, qmin, qmax)</code> to convert from int + scale + zp to float.</p>
Why doesn't matrix multiplication speed depend on row/column major ordering?

pytorchkiw: <p>Given two matrices A and B, both square, my naive assessment would be that the multiplication A * B should be much slower than A * B^T, since in the first case the sum</p>
<pre><code class="lang-auto">C_ij = sum_j A_ij * B_ji
</code></pre>
<p>does not respect the row-major ordering of <code>B</code> whereas when I am taking the transpose of <code>B</code> it does. Nevertheless I cannot measure a significant difference:</p>
<pre><code class="lang-auto">a = torch.empty(5000, 5000).normal_()
b = torch.empty(5000, 5000).normal_()

start = timer()
torch.matmul(a, b)
end = timer()
print(end - start) 

start = timer()
torch.matmul(a, b.T)
end = timer()
print(end - start)
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">0.7230800580000505
0.6886540420236997
</code></pre>
<p>Why is that so?</p>
FP16 Is there a plan to implement missing methods for half tensor in CPU

xkunglu: <p>I noticed that HalfTensor methods are only partially implemented.<br>
Is there a plan to complete this implementation?<br>
torch.<strong>version</strong> ‘1.0.1.post2’</p>
<p>I can create a float16 numpy array and convert it to torch tensor, but I cannot run .max() on the result unless I send it to gpu.</p>
<p>I can create a float16 cuda tensor but I cannot create the same tensor in cpu.</p>
<p>U understand that half tensor methods are specifically useful for GPU training, but I would have expected to be able to do CPU operatons on them as well. no?</p>
<p>Examples</p>
<pre><code class="lang-python">f16 = np.random.random((1,3,24,24)).astype(np.float16)
tf16 = torch.from_numpy(f16)
print(tf16.dtype, tf16.shape) # OK
print(tf16.max()) 
# FAILS:  RuntimeError: _th_max is not implemented for type torch.HalfTensor
</code></pre>
<pre><code class="lang-python">ft16 = torch.zeros((1,3,16,16), dtype=torch.float16, device="cuda" )
print(ft16.dtype) # OK
</code></pre>
<pre><code class="lang-python">ft16cpu = torch.zeros((1,3,16,16), dtype=torch.float16)
print(ft16cpu.dtype) 
#FAILS: RuntimeError: _th_zero_ is not implemented for type torch.HalfTensor
</code></pre>

Pierre-Bartet: <p>After two years I guess the answer is no.</p>

jwdink: <p>Best I could find was this comment <a href="https://github.com/pytorch/pytorch/issues/48245#issuecomment-730714723" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: "threshold_cpu" not implemented for 'Half' · Issue #48245 · pytorch/pytorch · GitHub</a></p>
Add three tensors of different sizes

Rukan_Mahfuz: <p>i have three tensors. First one, lets say ‘a’ has a dimension of <code>(15, 128)</code>. Second one, ‘b’ has dimension <code>(2, 128)</code> and third one ‘c’ has dimension <code>(417, 128)</code>.<br>
Now i want to add them like,</p>
<pre><code>d = a+b+c
</code></pre>
<p>I do not want to concatenate or stack them. I just want to add them because i want<br>
d should have dimension of <code>(417,128)</code>. How can i do this in pytorch?</p>
<p>This is the error i get:</p>
<pre><code>a = torch.randn(2,128)
b = torch.randn(15,128)
c = torch.randn(417,128)

d = a+b+c


RuntimeError: The size of tensor a (2) must match the size of tensor b (15) at non-singleton dimension.</code></pre>

Dwight_Foster: <p>I don’t know if this is what you mean but you could do something like this</p>
<pre><code class="lang-auto">c[:15] += b
c[:2] += a
</code></pre>

Rukan_Mahfuz: <p>Thanks for your answer. This might work. I will try this and let you know if it worked.</p>
PyTorch nn.Conv2d output comptation

grid_world: <p>I am using Python 3.8 and PyTorch 1.7.1. I saw a code which defines a Conv2d layer as follows:</p>
<pre><code class="lang-auto">Conv2d(3, 6, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
</code></pre>
<p>The input ‘X’ being passed to it is a 4D tensor-</p>
<pre><code class="lang-auto">X.shape
# torch.Size([4, 3, 6, 6])
</code></pre>
<p>The output volume for this conv layer is:</p>
<pre><code class="lang-auto">c1(X).shape
# torch.Size([4, 6, 3, 3])
</code></pre>
<p>I am trying to use the formula to compute output spatial dimensions for any conv layer: O = ((W - K + 2P)/S) + 1, where W = spatial dimension of image, K = filter/kernel size, P = zero padding &amp; S = stride.</p>
<p>For ‘c1’ conv layer, we get, W = 6, K = 3, S = 2 &amp; P = 1. Using the formula, you get O = ((6 - 3 + (2 x 1)) / 2) + 1 = 5/2 + 1 = 3.5.</p>
<p>The output volume: (4, 6, 3, 3) since number of filters used = 6. How is the spatial output from ‘c1’ then (3, 3)? What am I not getting?</p>
<p>Thanks!</p>

ptrblck: <p>The formula to calculate the output shape is given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">docs</a>.<br>
Your posted formula is missing the dilation and the subtraction of the constant <code>1</code>s.<br>
Also, note that the floor operation is used as indicated by the brackets.</p>

Codefresher: <p>Actually, after the operation of divide is a ceil function to make sure the dim of each shape is a integer when computing in our CPU/GPU… so it’s not 5/2 + 1=3.5, but ceil(5/2)+1=3…the same as the output</p>
ValueError: Target size (torch.Size([16, 19])) must be the same as input size (torch.Size([16, 432]))

aryan1: <p>I am taking part in a Multi-Label Classification problem I have created my complete training and validation pipleline but I can’t train because of this error:<br>
<code>ValueError: Target size (torch.Size([16, 19])) must be the same as input size (torch.Size([16, 432]))</code><br>
I am using the <code>nn.BCEWithLogitsLoss()</code> loss function.</p>
<p>Here is my dataset class:</p>
<pre><code class="lang-auto">class HPADataset(Dataset):
    def __init__(self, train_df, train_dir, transform=None):
        self.df = train_df
        self.train_path = train_dir
        self.transform = transform
        self.filter = ['blue', 'yellow', 'red']
        
    def __len__(self):
        return len(self.df)
    
    def __getitem__(self, item):
        name = self.df.ID[item]
        img = [cv2.imread(os.path.join(self.train_path, f'{name}_{filter_}.png'), cv2.IMREAD_GRAYSCALE) for filter_ in self.filter]
        img = np.stack(img, axis=-1)
        target = self.df.Label[item]
        target = encode_label(target)
        
        if self.transform:
            img = self.transform(image=img)
            img = img["image"]
#             img = np.transpose(img, )
            
        return {
            "image": torch.tensor(img, dtype=torch.float32),
            "target": target
        }
</code></pre>
<p>The data looks like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d919f1ebaffa3376ad13502ca9dbdc58bfbbd0ea.png" alt="image" data-base62-sha1="uYzch6Ixwgnut8zF1KwLd6Bz3Yu" width="470" height="221"></p>
<p>and I have encoded the labels to look like this:<br>
<code>tensor([1., 0., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])</code> (Essentially a one-hot encoding)<br>
It would be great if someone could help me.<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Dwight_Foster: <p>What model are you using?</p>

aryan1: <p>I tried resnet18 and efficientnetb0</p>

Dwight_Foster: <p>Did you change the output classes to match the number of labels?</p>

aryan1: <p>Yes, I did that as well.</p>

Dwight_Foster: <p>Ok can you send you training loop and print out the shape of your model output?</p>

aryan1: <p>Ohhh, I just checked. I was not giving the correct number of classes to the model. Now it’s working.<br>
Thanks a lot!</p>

Dwight_Foster: <p>No problem glad I could help.</p>