akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
transforms.Resize() misidentified as attribute

JohnPolo: <p>I’m trying to resize my input images. They are pretty big (5312x2988) and I’m trying to shrink them.</p>
<p>This is my code. It’s based on the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">code in a tutorial</a>:</p>
<pre><code class="lang-auto">import transforms as T

def get_transform(train):
    transforms = []
    # converts the image, a PIL image, into a PyTorch Tensor
    transforms.append(T.ToTensor())
    transforms.append(T.Resize((400*5312/2988,400)))  # *&lt;-- this is where I added T.Resize()*
    if train:
        # during training, randomly flip the training images
        # and ground-truth for data augmentation
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)
</code></pre>
<p>This is the error message:</p>
<blockquote>
<p>AttributeError                            Traceback (most recent call last)</p>
<p> in ()<br>
1 model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)<br>
----&gt; 2 dataset = four_chs(root = ‘/content/drive/MyDrive/four_chambers’, transforms = get_transform(train=True))<br>
3 data_loader = torch.utils.data.DataLoader(<br>
4  dataset, batch_size=1, shuffle=True, num_workers=2,<br>
5  collate_fn=utils.collate_fn)</p>
<p> in get_transform(train)<br>
8     # converts the image, a PIL image, into a PyTorch Tensor<br>
9     transforms.append(T.ToTensor())<br>
—&gt; 10     transforms.append(T.Resize((360,360)))<br>
11     if train:<br>
12         # during training, randomly flip the training images</p>
<p>AttributeError: module ‘transforms’ has no attribute ‘Resize’</p>
</blockquote>
<p>I suppose that I’ve coded transforms.Resize incorrectly if it’s being read as an attribute? I wrote that line the way I did because it seems to be the same format as T.ToTensor(). I don’t see the difference between applying ToTensor and Resize. Please tell me how should I code the transforms chunk to apply Resize to the tensors?</p>

bsridatta: <p>Could you try <code>from torchvision import transforms</code> ?</p>

JohnPolo: <p>It looks like that worked. I have a different error now. lol. I’ll probably post about that in a few minutes.</p>
<p>Thanks for your help.</p>
Standardize data on the fly?

jw3126: <p>I have a large data set of 3D images that does not fit into memory. The channels of this dataset have very different scale and location. E.g. there is one channels with values in [0,1] and another one with values in [900,1200]. I would like to standardize each channel to have mean 0 and variance one.</p>
<p>I would prefer not to safe an extra copy of the dataset, but do the normalization on the fly. Something like batch normalzation, but where the mean/std over all previously seen examples is used instead of the current batch. How to do this?</p>

bsridatta: <p>we normalize general RGB images using mean and std of imagenet data.<br>
Similarly precompute mean and std of each channel across the whole dataset - just once - dont need to save the normalize data, just save the mean and std.<br>
And in your dataset’s <code>__getitem__</code> sub and divide the current sample by precomputed mean and std.</p>
<p>I hope this helps</p>
Getting "TypeError: module, class, method, function, traceback, frame, or code object was expected" when jit compiling

Prashant: <p>Hi I’m trying to debug an unrelated issue and I was trying to construct a very basic test case.</p>
<p>However, I’m getting a different, error when I am running my basic test case that makes me question if I understand the basics of jit compilation.</p>
<p>I defined the following two dummy classes</p>
<pre><code class="lang-auto">class Dummy1():
    def __init__(self):
        self.__name__ = "dummy"
    def forward(self, x:Tensor)-&gt; Tuple[Tensor, Optional[Tensor]]:
        return (x,x)

class Dummy2(Dummy1):
    def __init__(self):
        super().__init__()
    def forward(self, x:Tensor)-&gt;Tuple[Tensor, Optional[Tensor]]:
        return super.forward(x)
</code></pre>
<p>And I get the following error when trying to torch.jit.script an object of Dummy2</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.8.1'
&gt;&gt;&gt; import multihead_attention_window
&gt;&gt;&gt; dummy = multihead_attention_window.Dummy2()
&gt;&gt;&gt; torch.jit.script(dummy)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Users/psridhar/.pyenv/versions/py3/lib/python3.8/site-packages/torch/jit/_script.py", line 986, in script
    ast = get_jit_def(obj, obj.__name__)
  File "/Users/psridhar/.pyenv/versions/py3/lib/python3.8/site-packages/torch/jit/frontend.py", line 240, in get_jit_def
    sourcelines, file_lineno, filename = get_source_lines_and_file(fn, torch._C.ErrorReport.call_stack())
  File "/Users/psridhar/.pyenv/versions/py3/lib/python3.8/site-packages/torch/_utils_internal.py", line 54, in get_source_lines_and_file
    filename = inspect.getsourcefile(obj)
  File "/Users/psridhar/.pyenv/versions/3.8.1/lib/python3.8/inspect.py", line 696, in getsourcefile
    filename = getfile(object)
  File "/Users/psridhar/.pyenv/versions/3.8.1/lib/python3.8/inspect.py", line 676, in getfile
    raise TypeError('module, class, method, function, traceback, frame, or '
TypeError: module, class, method, function, traceback, frame, or code object was expected, got Dummy2
</code></pre>
<p>I googled around and noticed this might be because of a torch version issue? But I’m on torch 1.8.1. What am I missing here?</p>

ptrblck: <p>The error is raised since you are using Python classes instead of <code>nn.Module</code>s for the scripting.<br>
After deriving from <code>nn.Module</code> this error would be gone. However, TorchScript doesn’t support inheritance as described <a href="https://github.com/pytorch/pytorch/issues/42885#issuecomment-678385108">here</a>, so you might run into other issues with this approach.</p>

Prashant: <p>Ah, thanks for parsing out that error message for me!</p>
<p>And thanks for that inheritance post. That was exactly what I was trying to debug.</p>
How to apply torchvision transform to a single tensor

phantom90: <p>Hi there,</p>
<p>Is there a clean way to apply torchvision transform to a single tensor (shape [3, 224, 224], has normalized to [0, 1]).</p>
<p>Thanks!</p>

bsridatta: <p>I guess you dont want to use all these values then?<br>
<code>transform = transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) </code><br>
<code>normalized = transform(original)</code></p>
<p>If you just want in [0,1] why not just divide by 255? Not a bad idea.</p>
Torch.autograd.grad - calculate grad for each example in batch seperatly

barakb: <p>First I’ll explain my goal,<br>
Input: [b,*]<br>
Now I want to calculate the gradients for a certain layer for each of the examples in the batch <strong>separately</strong><br>
Let’s say my model is a simple FC:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.w = nn.Parameter(torch.rand(4, 2),requires_grad=True)
        self.b = nn.Parameter(torch.rand(2),requires_grad=True)
    def forward(self, x):
        out = torch.mm(x,self.w)
        out = out+self.b
        return out
</code></pre>
<p>Calling the model and ciretrion pay attention that reduction is none:</p>
<pre><code class="lang-auto">resnet = Model()
criterion = nn.CrossEntropyLoss(reduction='none')
</code></pre>
<p>My input is a random 2x4 matrix, where 2 represent the batch:</p>
<pre><code class="lang-auto">rand = torch.rand(2,4)
</code></pre>
<p>Running:</p>
<pre><code class="lang-auto">output = resnet(rand)
</code></pre>
<p>Output shape is (2,2), first dimension is batch.<br>
Declaring (random) target, criterion is ce:</p>
<pre><code class="lang-auto">target = torch.LongTensor([1,1])
loss = criterion(output,target)
optimizer.zero_grad()
</code></pre>
<p>Now, I want to take my losses that holds the shape of:[2], and calc the grad w.r.t of the bias layer.</p>
<pre><code class="lang-auto">a = torch.autograd.grad(loss,resnet.b)
</code></pre>
<p>This returned an error, so I tried:</p>
<pre><code class="lang-auto">a = torch.autograd.grad(loss,resnet.b,grad_outputs=[torch.ones_like(loss), torch.ones_like(loss)])
</code></pre>
<p>This accumulated the gradients…<br>
I expect the return shape would be (2,2), means the batch size, and the layer shape.</p>
<p>Just to recap, I have the loss for each example in the batch seperatly using: “reduction =‘none’”<br>
Now I want the gradients for each of the examples w.r.t to the bias layer, the same as if my batch was <strong>1</strong>.<br>
So why I’m not running with batch=1, for efficency and speed of course.</p>
<p>Any idea how to make it work using:“torch.autograd.grad”?</p>

albanD: <p>Hi,</p>
<p>Automatic differentiation can only do vector jacobian product. So the grad_outputs that you give are multiplied by the jacobian. Since you give a vector full on ones, it sums up the gradients.</p>
<p>The two main ways to be able to get a gradient for each of your loss are:</p>
<ul>
<li>Do one backward for each of them and store the gradients.</li>
<li>Expand your weights to the right batch size before the forward so that the regular backward will give you what you want.</li>
</ul>

chenwydj: <p>Hi <a class="mention" href="/u/alband">@albanD</a></p>
<p>How to expand weights (inside the model) to the batch size? Is there any example?</p>
<p>Thanks!</p>

albanD: <p>I’m not sure we have examples no.</p>
<p>But maybe you can check repos like <a href="https://github.com/cybertronai/autograd-hacks" class="inline-onebox">GitHub - cybertronai/autograd-hacks</a> that provide such features.</p>
How to binarize only model architecture without weights?

a1302z: <p>I would like to binarize the computational flow of a model (basically the model class) without weights to send it over the network and instantiate the same network architecture remotely. I’ve looked into torch.onnx but it seems as if this is not the right tool as it also includes weights and input format needs to be fixed. Any suggestions how this could be achieved?</p>
Windowed tensor duplication

marcpaga: <p>Hi all,</p>
<p>I am trying to reshape and duplicate a tensor based on a window that is applied to one of the<br>
Here is a code example:</p>
<pre><code class="lang-auto">x = torch.rand([1, 10, 60, 256]) # [batch, subsamples, timepoints, channels]
window_size = 3

st = torch.arange(0, x.shape[1] - window_size + 1, 1)
nd = st + window_size

t_list = list()
for s, n in zip(st, nd):
    t_list.append(x[:, s:n, :, :].clone())

new_x = torch.stack(t_list).squeeze(1)
new_x.shape() # torch.Size([8, 3, 60, 256])
</code></pre>
<p>I was wondering if there is a tensor function that does this, or if there is a more elegant manner to do so, perhaps without using a for loop.</p>
<p>Thanks!</p>

ptrblck: <p>You could use <code>tensor.unfold</code> for it:</p>
<pre><code class="lang-python">out = x.unfold(1, window_size, 1)
out = out.squeeze(0).permute(0, 3, 1, 2)
print(out.shape) # torch.Size([8, 3, 60, 256])
print((out == new_x).all()) # tensor(True)
</code></pre>

marcpaga: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! This is exactly what I was looking for.</p>
Error while creating the dataset

lima: <p>I have a train input of size <code>[3082086, 7, 30]</code>, that is 3082086 frames, a window of 7 frames, each single frame contains 30 numbers.<br>
The train labels is of size <code>[3082086, 1]</code>, that is one label for every window of 7 frames.<br>
I tried to stack them to create a train set as follow:<br>
<code>trainset = torch.hstack(train_input, train_labels)</code><br>
…but I got an error <code>RuntimeError: Sizes of tensors must match except in dimension 1. Got 30 and 1 in dimension 2 (The offending index is 1)</code></p>
<p>Any way I can create the dataset with the above dimensions?</p>

bsridatta: <p>You dont want to mix inputs and labels into the same tensor, it doesnt make sense to merge them. Instead create a dataset, have separate tensors <code>self.inputs</code>, <code>self.labels</code> in the datasets <code>__init__</code> and In the <code>__getitem__(idx)</code>, just return <code>self.inputs[idx]</code>, <code>self.labels[idx]</code></p>

lima: <p><a class="mention" href="/u/bsridatta">@bsridatta</a>  That did it. Thanks mate.</p>
DataLoader changes type of variables based on CPU/GPU existance

aviasd: <p>I am getting a batch of strings from the dataloader.<br>
The way I get it is setting batch_size=1 and creating buckets in the Dataset.<strong>init</strong>(), so every batch is only “one file” for the dataloader, but it is 8 strings in a list.<br>
When using CPU, the dataloader puts every string in a tuple like this (“some string”, ).<br>
So we get list of 8 tuples.<br>
When using GPU, the dataloader puts every string in a list like this [“some string”].<br>
So we get list of 8 lists.<br>
Is that something that is known to happen?</p>

Isaac_Kargar: <p>can you pls put a sample code?</p>

ghery: <p>With this code:</p>
<pre><code class="lang-python">import torch

class MyDataset(Dataset):
    def __init__(self):
        self.data = 100*[['a', 'b']]
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
loader = DataLoader(
    dataset,
    batch_size=5,
    num_workers=2,
    shuffle=True
)

for data in loader:
    print(data)
</code></pre>
<p>I get</p>
<pre><code class="lang-python">[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
[('a', 'a', 'a', 'a', 'a'), ('b', 'b', 'b', 'b', 'b')]
</code></pre>
<p>Any clue <a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a> ?</p>
DataParallel on single GPU

papyrus: <p>Hi everyone,</p>
<p>Is it possible to split a GPU in half, and apply DataParallel ? It seems like my model doesn’t use the full GPU capacity, and I’ve read that increasing the batch size (which would give more work to the GPU) changes the learning and could make it worse.</p>
<p>Thanks</p>

bsridatta: <p>To my knowledge, the effect would be the same. Since dataparallel makes the copy of the model on different gpus and merge the results. The backward gradient is also passed across all the gpus and collectively updated. That means, all these copies update at once. So would there be a difference? It shouldnt.</p>
<p>So how to use you gpu more efficiently? run multiple experiments at once!</p>

papyrus: <p>Ok, thanks for the answer!</p>
How to update item_embeddings in every epoch and user_embeddings in each batch in an epoch? RuntimeError: Trying to backward through the graph a second time

icmp_no_request: <p>I was working on a recommendation algorithm and I need <code>user_embeddings</code> and <code>item_embeddings</code> to calculate the score. In the algorithm, <code>user_encoder</code> aims to produce <code>user_embeddings</code> and <code>item_encoder</code> for <code>item_embeddings.</code></p>
<p>The <code>item_embeddings</code> should be updated in every epoch and <code>user_embeddings</code> should be updated in every batch in an epoch. I was wondering how to complete? The part of code I want is as following:</p>
<pre><code class="lang-auto">user_model = UserModel()
item_model = ItemModel()
criterion = nn.BCEWithLogitsLoss().to(device)
optimizer = optim.Adam(list(user_model.parameters()) + list(item_model.parameters()), lr=LEARNING_RATE)

for epoch in range(num_epochs):
    user_model.train()
    item_model.train()
    
    item_embeddings = item_model(input)
    # the size of item_embeddings = [ITEMS NUM, item_embed_dim]
    
    for idx, batch in enumerate(train_dataloader):
        data = batch["data"].to(device)
        label = batch["label"].to(device)
        optimizer.zero_grad()
        user_embeddings = user_model(data, item_embeddings)
        # the size of user_embeddings = [batch size, item_embed_dim]
       
        score = torch.matmul(user_embeddings, item_embeddings.T)
        loss = criterion(score, label)
        loss.backward()
        optimizer.step()
</code></pre>
<p>I met <code>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.</code><br>
Any ideas to fix it? Thanks in advance.</p>
RuntimeError: Number of dimensions of repeat dims can not be smaller than number of dimensions of tensor

DHYEY_PATEL: <p>the error occur in torch.nn.functional.instancenorm()<br>
with mean, variance , weight, and bias having same size<br>
I am not able to figure out what is the problem.</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce this issue (including the used shapes) so that we could have a look at it, please?</p>

wei61547-jp: <p>hi ~<br>
I found that  running_mean/var’s shape is C, not N * C<br>
according to the <a href="https://arxiv.org/abs/1607.08022" class="inline-onebox" rel="noopener nofollow ugc">[1607.08022] Instance Normalization: The Missing Ingredient for Fast Stylization</a>, the shape of mean/var is N *C , but for torch.nn.instancenorm ,<br>
at::alias(running_mean).copy_(running_mean_.view({ b, c }).mean(0, false)); }<br>
i don’t know the operation of “mean(0,false)” is use for ?<br>
So  mean, variance , weight, and bias has has same size is C .</p>
Instance norm 2d running mean and running var

ooodragon: <p>hello</p>
<p>trying to use this instancenorm 2d from pytorch</p>
<p>I now know that it only outputs statistics only if it was mentioned by user ( track_running_stats = True)<br>
why is this different from batchnorm where it always holds on to statistics while training?</p>
<p>one more question</p>
<p>I learned that instancenorm 2d is a normalization to each picture within a batch. However, when I ask for running mean and running var, it will only give me a tensor of shape (mini_batch_size,)</p>
<p>This is the same size to that of batchnorm. However, since instancenorm 2d is doing normalization to each images whereas batchnorm is doing it to whole minibatch, instancenorm 2d should have more statistics than that of batchnorm. Why is it not the case here? Thank you</p>

wei61547-jp: <aside class="quote no-group" data-username="ooodragon" data-post="1" data-topic="105349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ooodragon/48/24206_2.png" class="avatar"> ooodragon:</div>
<blockquote>
<p>cen</p>
</blockquote>
</aside>
<p>hi ~<br>
I find similar questions with you, which the running_mean/var’s shape is C but not N<em>C .<br>
From the page <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Normalization.cpp#L524" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Normalization.cpp#L524</a>, the code shows “at::alias(running_mean).copy_(running_mean_.view({ b, c }).mean(0, false));”. I don’t understand the operation of “mean(0,flase)” is use for ? (B</em>C-&gt;C)</p>
Why InstanceNorm2d doesn't follow the formula?

dsp: <p>In a simple test, using the original formula for InstanceNorm2d in <a href="https://pytorch.org/docs/stable/nn.html?highlight=instancenorm2d#torch.nn.InstanceNorm2d" rel="noopener nofollow ugc">here</a>:</p>
<blockquote>
<p>eps = 1e-5<br>
N = nn.InstanceNorm2d(1, eps=eps)<br>
x = torch.arange(25).view(1, 1, 5, 5).float()<br>
y1 = N(x)<br>
y2 = (x - x.mean())/torch.sqrt(x.var() + eps)</p>
</blockquote>
<p>The error is quite big:</p>
<blockquote>
<p>y2-y1<br>
tensor([[[[ 0.0336,  0.0308,  0.0280,  0.0252,  0.0224],<br>
[ 0.0196,  0.0168,  0.0140,  0.0112,  0.0084],<br>
[ 0.0056,  0.0028,  0.0000, -0.0028, -0.0056],<br>
[-0.0084, -0.0112, -0.0140, -0.0168, -0.0196],<br>
[-0.0224, -0.0252, -0.0280, -0.0308, -0.0336]]]])</p>
</blockquote>
<p>But the result is consistent. I get the same in CPU and GPU.<br>
What is the right formula for InstanceNorm2d?</p>

tom: <p>If you’re into finding out yourself: Can you spot the pattern in the deviation you see? Did you notice they both have mean 0? Look at y1/y2.</p>
<p>It does use the same formula but the formula does not say which variance estimate to use, so you picked the one that’s not the one <code>*Norm</code> uses: you have <code>unbiased=True</code>, <code>*Norm</code> uses <code>unbiased=False</code>. If you do fix this, you’ll get the same up to numerical precision.</p>
<p>Best regards</p>
<p>Thomas</p>

wei61547-jp: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="42674">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>unbiased</p>
</blockquote>
</aside>
<p>I find that running_mean/var’s shape is C, not N * C in torch.nn.instancenorm, but according to the <a href="https://arxiv.org/pdf/1607.08022.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1607.08022.pdf</a> , the shape of mean/var is N * C .<br>
And from the page in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Normalization.cpp#L581" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Normalization.cpp#L581</a>,<br>
the code “at::alias(running_mean).copy_(running_mean_.view({ b, c }).mean(0, false)); }”<br>
i don’t know the operation of “mean(0,false)” is use for and why?</p>
Optimizer agnostic initialisation

Joe_Harrison: <p>Excuse the terminology. What I’m trying to do is initialise an empty optimizer so that I can later assign one. I’ll outline what I mean with some (pseudo-) code:</p>
<pre><code class="lang-auto">void func(string optimizer_type){

    torch::optim::Optimizer opt;

    if(optimizer_type="SGD"){
        opt = torch::optim::SGD(params, 0.01);
    }
   else{
       opt = torch::optim::Adam(params);
   }
   // Use opt
}
</code></pre>
<p>But optim::Optimizer is an abstract class and I can’t assign it this way.</p>
<p>Maybe more of an OOP question, but how can I get this to work in Libtorch without changing parts of the library?</p>

Matej_Kompanek: <p>I would use shared pointer, something like this</p>
<pre><code class="lang-auto">void func(std::string optimizer_type) 
{
	std::shared_ptr&lt;torch::optim::Optimizer&gt; opt = nullptr;
	if (optimizer_type == "SGD") 
	{
		opt = std::make_shared&lt;torch::optim::SGD&gt;(model.parameters(), 0.01);
	}
	else 
	{
		opt = std::make_shared&lt;torch::optim::Adam&gt;(model.parameters());
	}
}
</code></pre>
Pytorch 1.9 still using cuDNN 8.0.5

sbsky: <p>Hello, I noticed that Pytorch 1.9 still uses cuDNN 8.0.5, same as 1.8 even though cuDNN 8.2.1 has been released. Is there a reason for this?</p>
<p>I read in the cuDNN release notes that there were some performance regressions for some legacy APIs since 8.0.5.</p>

ptrblck: <p>Before updating a cudnn version (or any other 3rd party library) we are usually running it for a couple of weeks in the nightlies to get visibility into potential issues.<br>
In the <code>1.9.0</code> release we were working on fixing the <a href="https://github.com/pytorch/pytorch/issues/56747">library split issue</a>, the <a href="https://github.com/pytorch/pytorch/issues/50153">missing cutlass/xmma kernels</a> etc.<br>
While cudnn8.2.1 is already released, the risk was too high to fix the outstanding bugs and update cudnn without the visibility of the nightlies, which is why it’s delayed.</p>

sbsky: <p>Gotcha, thanks for the update! I’ll keep my eye on the nightlies</p>
Do loss.backward and optimizer.step with different frquency

Hyeonuk_Woo: <p>Hi I’m new to pytorch.</p>
<p>If I do <code>loss.backward</code> for every 0.1% of my training set and do <code>optimizer.step</code> for every 1% of  my training set,  what could be the problem?</p>
<p>Due to characteristic of my training data, I write my code as below</p>
<pre><code class="lang-auto">BATCH_SIZE = parameter from user
for epoch in range(1,10001):
     i=0
     (..........)
     for ..... in training_generator:
            (.......)
            loss.backward()
            i+=1
            if i % BATCH_SIZE == 0 :
                  optimizer.step()
                  optimizer.zero_grad()
</code></pre>
<p>As far as I know, <code>loss.backward </code> accumulate gradient in summation. This kind of accumulation of gradient is okay? Is there any way to divide gradient by <code>BATCH_SIZE</code>?</p>
<p>In my case, I use Adam optimizer.</p>

ptrblck: <p>Based on your description I understand that you are calling <code>optimizer.step()</code> more often (1 out of 100 steps) and calculate the gradients only 1 out of 1000 steps.<br>
In this case, the general problem could be that the <code>optimizer</code> updates the parameters with “old” gradients, which might not work.</p>
<p>To change the gradients, you could either scale the loss itself (divide with a constant) or use hooks to manipulate the <code>.grad</code> attributes of all parameters.</p>

Hyeonuk_Woo: <p>Sorry for my English. I write sentence make you confused. Acutally, I want to say  calling <code>loss.backward</code> more often than <code>optimizer.step</code>.</p>
<p>Although, you give the answer what I want to know! Thanks!</p>
Keypoints Detection

sab3rclaw: <p>I am trying to detect keypoints in grayscale images. The problem that I have is that my keypoints are not consistent for each image. So image1 has 10 keypoints and image 2 has 20 keypoints and so on.<br>
In total I have 32 keypoints max but as I said each image has varied number of keypoints.</p>
<p>So how  can I modify my model to accomodate varied number of keypoints?</p>
<p><code>self.fc3 = nn.Linear(400, 32*2)</code></p>
<p>Please guide me</p>

sab3rclaw: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please help</p>
Pytorch Dataset class __getitem__() not being called

avalon1511: <p>Really confused about why the <code>__getitem__()</code> method is not being called here:</p>
<pre><code class="lang-auto">class MaleFacesDataset(Dataset):
    
    def __init__(self, csv_file, root_dir):
        
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)
    

    def __getitem__(self, idx):

        print("GET!")

        if torch.is_tensor(idx):
            idx = idx.tolist()


        img_name = os.path.join(self.root_dir,
                                self.landmarks_frame.iloc[idx, 0])

        image = io.imread(img_name)
        sample = image

        sample = Image.fromarray(np.uint8(sample)).convert('RGB')
        sample = Image.fromarray(sample.astype('uint8'), 'RGB')

        if self.transform:
            sample = self.transform(sample)

        return sample
</code></pre>
<p>Have tried rewriting the whole class a few times but still the same problem - any input is appreciated!</p>

eqy: <p>Can you show an example of how you are using the dataset class?</p>

avalon1511: <p>Yeah, here is how I am instantiating it:</p>
<pre><code class="lang-auto">male_dataset = MaleFacesDataset(csv_file = './attribute_dir', 
                                root_dir= './img_align_celeba', transform= transform)
</code></pre>

Ali1234: <p>But, there is no transform parameter in your <code>__init__()</code> constructor</p>

avalon1511: <p>Oh you’re right, i think that is the issue!</p>
Pytorch 1.8 and rtx 3070

yoad.tewel: <p>Hey,<br>
Should we except any performance changes from pytorch 1.8 Cuda 11.1 in comparison to pytorch 1.7.1 with Cuda 11?</p>
<p>And if not, what version of pytorch should give the most out of the new cards?</p>
<p>Thanks!</p>

ptrblck: <p>You shouldn’t expect a lot of changes for CNNs, as both versions ship with cudnn8.0.5.<br>
Due to <a href="https://github.com/pytorch/pytorch/issues/50153">this issue</a> we couldn’t use the cudnn8.1 release and are working on it. For the best performance on your 3070 you could build from source using the latest cudnn release.</p>

yoad.tewel: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Now that 1.8.1 is out, is the answer different?<br>
Does it use cudnn 8.1?</p>

ptrblck: <p>No, it’s still using cudnn8.0.5, as <code>1.8.1</code> was a hot-fix release mainly for Kineto and we’ll revisit the issue with the now released cudnn8.2 version.</p>

yoad.tewel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hey,<br>
Now that pytorch 1.9, does it come with cudnn8.2? Will it use rtx 3070 full speed?</p>

ptrblck: <p>The binaries will not use it as described <a href="https://discuss.pytorch.org/t/pytorch-1-9-still-using-cudnn-8-0-5/124204/2">here</a>, so you would have to build it from source.</p>
Will the same model input data twice retain the gradient information of the first input data?

Yongjie_Shi: <pre><code class="lang-auto">input1 = #sth
input2 = # sth
label1 = # sth
label2 = # sth
output1 = model(input1)
output2 = model(input2)
loss1 = criterion(output1, label1)
loss2 = criterion(output2, label2)
total = loss1+loss2
total_loss.backward()
optimizer.step()
</code></pre>
<p>I wonder will the backward() function consider the effect of input1?</p>

bsridatta: <p>Yes, it should (if total.backward()). Try to print and see if they are different?<br>
Since the backward is on total i.e loss1+loss2, the computation graph would include both 1, 2 inputs.</p>
<p>You could also refer the GAN tutorial where something similar is done</p>
Autoencoder different mode

Arthur_Zakirov: <p>Hello everyone,</p>
<p>what is the term for an autoencoder, that is trained not to reconstruct X, but to create another output y instead. I’ve heard the term somewhere, but can’t remember it.<br>
For example the autoencoder the autoencoder is trained on the MNIST dataset not to reconstruct the given number X_hat, but to create an image of another number Y.</p>

bsridatta: <p>U-Net, conditional autoencoders? They are usually still called an autoencoder. For example, image colorization auto encoders.</p>
Dataset function that is called on every epoch

Mika_S: <p>Is there a function in dataset that is called on every epoch by the dataloader? I want to be able to reset some variables that i am using to maintain statistics.</p>

bsridatta: <p>Since DataLoader takes an instance of Dataset, I think it is not possible to do programmatically manipulate the dataset. Recreating the Dataset should be the way to go.</p>
<p>To make it a bit efficient, I would create an instance of the processed content that can be directly used by a new dataset that you create every epoch. And have your logic of manipulating the data in the <code>get_item</code> method. This way there shouldn’t be much overhead.</p>
Dataloader not showing images

aleemsidra: <p>I am trying to see the images being loaded in the data loader. I used PIL in a custom loader to load the images. However, when I am using the following code to display images, I face the error:</p>
<pre><code class="lang-auto">images, labels = next(iter(test_loader))
plt.imshow( images )
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-35-d5057d4c4d54&gt; in &lt;module&gt;()
      1 images, labels = next(iter(test_loader))
----&gt; 2 plt.imshow( images )

5 frames
/usr/local/lib/python3.7/dist-packages/matplotlib/image.py in set_data(self, A)
    697                 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]):
    698             raise TypeError("Invalid shape {} for image data"
--&gt; 699                             .format(self._A.shape))
    700 
    701         if self._A.ndim == 3:

TypeError: Invalid shape (4, 3, 224, 224) for image data
</code></pre>
<p>What’s the reason behind this error?</p>

ptrblck: <p><code>matplotlib,pyplot.imshow</code> expects a numpy array in a valid image shape (e.g. <code>[height, width, channels]</code> or <code>[height, width]</code>), while you are passing tensors in the shape <code>[batch_size, channels, height, width]</code> to it.<br>
You would thus have to index each image (via <code>image = images[0]</code>) and <code>permute</code> the axes to match the expected shape:</p>
<pre><code class="lang-python">image = images[0].permute(1, 2, 0).numpy()
plt.imshow(image)
</code></pre>

aleemsidra: <p>I am trying to display the image as:</p>
<pre><code class="lang-auto">figure = plt.figure()
num_of_images = 60
for index in range(1, num_of_images + 1):
    plt.subplot(6, 10, index)
    plt.axis('off')
    plt.imshow(images[index].numpy().transpose(1,2,0), cmap='gray_r')
</code></pre>
<p>I am facing the error:</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-35-f492e5cf63ef&gt; in &lt;module&gt;()
      4     plt.subplot(6, 10, index)
      5     plt.axis('off')
----&gt; 6     plt.imshow(images[index].numpy().transpose(1,2,0), cmap='gray_r')

5 frames
/usr/local/lib/python3.7/dist-packages/matplotlib/image.py in set_data(self, A)
    697                 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]):
    698             raise TypeError("Invalid shape {} for image data"
--&gt; 699                             .format(self._A.shape))
    700 
    701         if self._A.ndim == 3:

TypeError: Invalid shape (28, 28, 1) for image data
</code></pre>
<p>The above format <code>(28,28,1)</code> is compliant with plt.show, then why I am facing this error?</p>

ptrblck: <p>Maybe <code>matplotlib</code> used more strict checks and doesn’t like the additional channel dimension in your version, so you could remove this dimension, if needed.<br>
It also works in my setup using <code>matplotlib==3.4.2</code>:</p>
<pre><code class="lang-python">plt.imshow(np.random.randint(0, 255, (24, 24, 1)).astype(np.uint8))
</code></pre>
VAE prior z conditioned on x

Arthur_Zakirov: <p>Hello everyone,</p>
<p>I’m trying to understand a CVAE implemented in a paper and it just does not make sense to me, that the prior for z in the ELBO depends on X. Isn’t the prior supposed to be an assumption before you have seen the data?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39bf68566dbc432c265388e75f2f7a027c909e80.png" data-download-href="https://discuss.pytorch.org/uploads/default/39bf68566dbc432c265388e75f2f7a027c909e80" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39bf68566dbc432c265388e75f2f7a027c909e80_2_690x130.png" alt="image" data-base62-sha1="8eRiT2loXrbkLzjYtzJaLLH2ACY" width="690" height="130" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39bf68566dbc432c265388e75f2f7a027c909e80_2_690x130.png, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39bf68566dbc432c265388e75f2f7a027c909e80.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39bf68566dbc432c265388e75f2f7a027c909e80.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">925×175 50.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Unlike typical autoencoders this one does not aim to reconstruct x with x_hat, but to produce a different output named y. So my hypothesis is. that the idea is not to approximate p with q but actually the other way around, because during inference the labels won’t be available. So we will probably use the prior during inference. The KL term makes sure, that p produces a smiliar z to q, without knowing y, while q does know y. So the neural network of p captures the relationship between x and y and makes y obsolete as an input. If that makes any sense…</p>
<p>My assumption is</p>
Contiguous grad for non contiguous tensor

saar-eliad: <p>Given a non-contiguous.tensor <code>a</code><br>
If I calculate</p>
<pre><code class="lang-auto">torch.autograd.bakwards([a], [contiguous_grad_of_a])
</code></pre>
<p>Is this OK?</p>
<p>This also makes me wonder: in case it does a clone, is calling <code>tensor.contiguous()</code> blocking?</p>

albanD: <p>Yes this is ok to do. The Tensor and its gradient doesn’t need to have the same contiguity.</p>
<blockquote>
<p>is calling <code>tensor.contiguous()</code> blocking?</p>
</blockquote>
<p>What do you mean by blocking?<br>
It is a differentiable op, so gradients will flow back through it as expected.</p>

saar-eliad: <p>A command is non-blocking - if commands following it can run before it finishes.</p>
<p>(I solved my problem, but still interested to know)</p>

albanD: <p>Ops in general are blocking.<br>
Only when you use cuda that the underlying API is actually asynchronous.</p>
What is the expected image i got from tensor?

samm: <p>i need to plot the images of tensorflow</p>
<p>i tried to implement</p>
<pre><code class="lang-auto">&gt; import torch
&gt; import numpy as np
&gt; from matplotlib import pyplot as plt 
&gt; 
&gt; x = torch.load('tensor-pytorch.pt')
&gt; arr = np.ndarray((x))
&gt; arr_ = np.squeeze(arr) 
&gt; plt.imshow(arr_)
&gt; plt.show()
</code></pre>
<p>but i got</p>
<pre><code class="lang-auto">ValueError: maximum supported dimension for an ndarray is 32, found 100
</code></pre>
<p>i tried to change the line to</p>
<pre><code class="lang-auto">arr = np.array((np.array(x)))
</code></pre>
<p>and got<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82d050fe201a9626e68c8610bd6008410d76e7bb.png" alt="Figure_1" data-base62-sha1="iFenZoq6NdeSdQdbUfNnzhAWVu3" width="640" height="480"><br>
Does that right ?</p>

Saurav_Gupta1: <p>can you paste x and shape of x here?</p>
How to do profiling on pytorch data loader worker process?

BadTimeManagement: <p>I use a simple profiling code to profile my training process.</p>
<pre><code class="lang-auto">	import cProfile, pstats
	cProfile.run("main()", "{}.profile".format(__file__))
	s = pstats.Stats("{}.profile".format(__file__))
	s.strip_dirs()
	s.sort_stats("time").print_stats(10)
</code></pre>
<p>and got something like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d93061eb0d634863dfd8540be984088c0671365.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d93061eb0d634863dfd8540be984088c0671365" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d93061eb0d634863dfd8540be984088c0671365_2_690x206.png" alt="image" data-base62-sha1="fDl2zHbIsCa7iaEVxwPvvd6i2ot" width="690" height="206" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d93061eb0d634863dfd8540be984088c0671365_2_690x206.png, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d93061eb0d634863dfd8540be984088c0671365.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d93061eb0d634863dfd8540be984088c0671365.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">902×270 75.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I did some reading on {method ‘acquire’ of ‘_thread.lock’ objects}, apparently this just shows what the parent process is calling and not the child process. So is it possible to do profiling on the data loader worker process?</p>

Minh_Nguyen: <p>Hello, did you find the solution to your question? I’m having the same problem about pytorch data loader worker process profiling like you.</p>
Difference between Pytorch version 0.2 and 1.7?

UmEr_ANsari: <p>I am trying to transform a code written in PyTorch 0.2 to PyTorch version 1.7.<br>
Somehow I managed to run the code in PyTorch 1.7 but the results are different from the original code.<br>
I believe the loss computed in these two versions is different.</p>
<p>The link of the code is written in PyTorch 0.2:</p><aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/cai-lw/KBGAN/blob/master/base_model.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/cai-lw/KBGAN/blob/master/base_model.py" target="_blank" rel="noopener nofollow ugc">cai-lw/KBGAN/blob/master/base_model.py</a></h4>


    <pre><code class="lang-py">import torch
import torch.nn as nn
import torch.nn.functional as nnf
from config import config
from torch.autograd import Variable
from torch.optim import Adam
from metrics import mrr_mr_hitk
from data_utils import batch_by_size
import logging


class BaseModule(nn.Module):
    def __init__(self):
        super(BaseModule, self).__init__()

    def score(self, src, rel, dst):
        raise NotImplementedError

    def dist(self, src, rel, dst):
        raise NotImplementedError
</code></pre>


  This file has been truncated. <a href="https://github.com/cai-lw/KBGAN/blob/master/base_model.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Can anyone suggest changes in gen_step() and dis_step()  in the base_model.py file in order to transform it to 1.7?</p>
<p>Thanks</p>
Parallelizing for loop iterations on GPU

Manvi_Gupta: <p>Hi!<br>
I want to parallelize a simple <code>for loop</code> computation that iterates over a list of pairs (stored as PyTorch tensor) to run over a <strong>GPU</strong>. The computation inside the loop doesn’t seem to be the bottleneck, time is consumed because of the huge input size.</p>
<p><strong>What it does?</strong><br>
Just applies a  function to each pair and updates its value. Then increments a matrix according to the received pair as the index.<br>
All <em>the iterations of the loop are independent of each other</em>, hence can be parallelized, however, I’m not getting a way to do that in PyTorch so that efficiency can be improved with CUDA. Any kind of help would be appreciated (I’m open to vectorizing, multiprocessing, or change in input style).<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<pre><code class="lang-auto">for i in range(0, batch_size):
    r_Pairs[i] = torch.floor(random.uniform(0.0, 1.0) * r_Pairs[i] * F) % F
    matrix[r_Pairs[i][0]][r_Pairs[i][1]] += 1
</code></pre>

googlebot: <p>line 2 is trivially convertable to vectorized form, just with torch RNG</p>
<p>I don’t remember if line 3 if easily vectorizable - there is scatter_add_, but you may need to convert to 1d view/indexes (i1*size1+i2). Or do that scattering loop on cpu, that will be faster.</p>