akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchvision.transforms has no attribute of ColorJitter?

username: Mylinda
cooked: <p>Hey guys,</p>
<p>I used torchvision.transforms.ColorJitter to do data augmentation. The codes are as below:<br>
data_transform = transforms.Compose([<br>
transforms.ColorJitter(brightness=0.1,constrast=0.1,saturation=0.1,hue=0.1),<br>
transforms.ToTensor(),<br>
transforms.Normalize([pixel_mean, pixel_mean, pixel_mean], [pixel_std, pixel_std, pixel_std])<br>
])<br>
But it gave the error of 'AttributeError: ‘module’ object has no attribute ‘ColorJitter’'<br>
My Pytorch version is 0.2.0_4.  Any help? Thanks in advance</p>

username: SimonW
cooked: <p>The method is not available in 0.2 <a href="http://pytorch.org/docs/0.2.0/torchvision/transforms.html">http://pytorch.org/docs/0.2.0/torchvision/transforms.html</a>. (note the version selector on top left of doc page.) It should be in 0.3 and master according to the doc.</p>

username: Mylinda
cooked: <p>Thanks <a class="mention" href="/u/simonw">@SimonW</a>. It works after upgrading Pytorch to 0.3</p>
How to map tensors of different sizes?

username: John_Price
cooked: <p>I have a tensor X with dimesions E x p and a second tensor Y with dimenisons C x p. Then I calculated the distance matrix with cdist() which leads me to a tensor Z with dimension E x C. Now, I am searching for an operation in pytorch that maps matrices of size (E,C) and (C, p) to a matrix of size (E, p). What is the best to do this in pytoch?</p>
Conver a list to tuple

username: aryan_461
cooked: <p>I have a tuple made like this:</p>
<pre><code class="lang-python">mytuple = (0, 5) + tuple(1 for _ in range(len(some_tuple) - 2))
</code></pre>
<p>I tried to make a script of my class and I got:</p>
<pre><code class="lang-auto">torch.jit.frontend.UnsupportedNodeError: GeneratorExp aren't supported
mytuple = (0, 5) + tuple(1 for _ in range(len(some_tuple) - 2))
                        ~ &lt;--- HERE
</code></pre>
<p>So I change the code to:</p>
<pre><code class="lang-python">mytuple = [0, 5]
for _ in range(len(some_tuple) - 2):
    mytuple.append(1)
mytuple = tuple(mytuple)
</code></pre>
<p>but I gotthis error:</p>
<pre><code class="lang-python">cannot statically infer the expected size of a list in this context:
mytuple = tuple(mytuple)
          ~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>I change the last line to:</p>
<pre><code class="lang-python">mytuple = (mytuple,)
</code></pre>
<p>but in the end I figured out that is not a tuple of integers and it is type ‘Tuple[List[int]]’.</p>
<p>my last try was to unpack the list:</p>
<pre><code class="lang-python">mytuple = tuple(*mytuple)
</code></pre>
<p>I faced this error:</p>
<pre><code class="lang-auto">Unexpected starred expansion. File a bug report:
mytuple = tuple(*mytuple)
                 ~~~~~~~ &lt;--- HERE
</code></pre>
<p>How can I have tuple of integers with dynamic length that is scriptable?</p>
Need help to understand the forward method

username: Shushma_Kulkarni
cooked: <p>I am writing a MLP in pytorch using sequential model, but I am not understanding if the model is actually updating weights when I call :<br>
optimizer.zero_grad()<br>
scores = model(data)<br>
loss = criterion(scores, targets)<br>
# backward<br>
loss.backward()<br>
# gradient descent or adam step<br>
optimizer.step()</p>
<p>My model is as below:<br>
def <strong>init</strong>(self, input_size, out_size):<br>
super(Feedforward, self).<strong>init</strong>()<br>
self.layer1 = nn.Sequential()<br>
self.layer1.add_module(“fc1”, torch.nn.Linear(input_size, 65))<br>
self.layer1.add_module(“bn1”, nn.BatchNorm1d(num_features=65, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))<br>
self.layer1.add_module(“Relu1”, torch.nn.ReLU())<br>
self.layer1.add_module(“dropout”,nn.Dropout(p=0.2))<br>
self.layer1.add_module(“fc2”, torch.nn.Linear(65, 60))<br>
self.layer1.add_module(“bn2”, nn.BatchNorm1d(num_features=60, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))<br>
self.layer1.add_module(“Relu2”, torch.nn.ReLU())<br>
self.layer1.add_module(“dropout2”,nn.Dropout(p=0.2))<br>
self.layer1.add_module(“fc4”, torch.nn.Linear(60, out_size))<br>
self.layer1.add_module(“Softmax”,torch.nn.Softmax(dim=1))</p>
<pre><code>    def forward(self, x):
        x = self.layer1(x)
        return self.fc.forward(x)

    def initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.weight, 1)
            elif isinstance(m, nn.Linear):
                nn.init. xavier_normal_(m.weight)</code></pre>

username: ptrblck
cooked: <p>The code should update the model parameters, if you’ve previously passed them to the optimizer.<br>
You can print a specific parameter before and after the <code>optimizer.step()</code> operation and compare the values to make sure it’s working as intended.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier.</p>

username: Shushma_Kulkarni
cooked: <p>Well, the issue is I am not sure if its working. I have a keras program with same number of layers and other hyperparameters, it gives 92% accuracy. But the pytorch model gives 20% accuracy on the same data.  Can you please explain what is the difference between return x and return self.fc.forward(x) in the forward function.</p>

username: Alexey_Demyanchuk
cooked: <p>I am sorry for interruption. I don’t see in your code you are initializing <code>self.fc</code> anywhere.<br>
With regard to the last question: return x is doing <code>self.layer1(x)</code> will do forward pass of your data through the whole <code>layer1</code> and returning result. If you are returning <code>self.fc.forward(x)</code> it is doing forward pass through it with previously achieved result from <code>layer1</code>.</p>
<p>It is also worth mention, you have to make sure the loss function you are using is awaiting as in input a ‘softmaxed’ version or raw logits (output of the last linear layer).</p>
SWA training question

username: victorc25
cooked: <p>Hello!</p>
<p>I’m trying to use the SWA code in Pytorch 1.6 (<a href="https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/" rel="noopener nofollow ugc">https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/</a>) and I’m following the sample structure similar to the one in the blog:</p>
<pre><code class="lang-auto">from torch.optim.swa_utils import AveragedModel, SWALR
from torch.optim.lr_scheduler import CosineAnnealingLR

loader, optimizer, model, loss_fn = ...
swa_model = AveragedModel(model)
scheduler = CosineAnnealingLR(optimizer, T_max=100)
swa_start = 5
swa_scheduler = SWALR(optimizer, swa_lr=0.05)

for epoch in range(100):
      for input, target in loader:
          optimizer.zero_grad()
          loss_fn(model(input), target).backward()
          optimizer.step()
      if epoch &gt; swa_start:
          swa_model.update_parameters(model)
          swa_scheduler.step()
      else:
          scheduler.step()

# Update bn statistics for the swa_model at the end
torch.optim.swa_utils.update_bn(loader, swa_model)
</code></pre>
<p>However, I’m not certain about a couple of things. After entering the SWA regime, if the SWA scheduler learning rate is the default (0.05), the model in training becomes unstable quickly (NaN), if I lower it (to 0.001), it appears to work (at least no NaN).</p>
<p>Should this be the case that the SWA copy of the model is affecting the model in training? (I had understood that the SWA copy should be updated separatedly from the model in training).</p>

username: ptrblck
cooked: <p>I would also assume that the <code>SWALR</code> object is not interfering with the standard training routine and it seems an additional <code>swa_lr</code> entry is created in the <code>para_groups</code> as seen <a href="https://github.com/pytorch/pytorch/blob/2fe382e931ec5a31715c247fea2b292f7d72cb66/torch/optim/swa_utils.py#L213-L214">here</a>.<br>
However, I’m currently unsure where this <code>swa_lr</code> is used, since the model update doesn’t seem to use <a href="https://github.com/pytorch/pytorch/blob/2fe382e931ec5a31715c247fea2b292f7d72cb66/torch/optim/swa_utils.py#L95-L97">here</a>.</p>

username: victorc25
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>! I’m still unsure about the logic, but I’ll continue testing. Cheers!</p>
What is Forward/backward pass size?

username: swp
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a114a8fbe4ceb2e88d5752bde136a2ea59eb8512.jpeg" alt="ccz" data-base62-sha1="mYZ8tOzu0Tfj42UgmQF2fDvRdvQ" width="469" height="180"></p>
<p>I use the pytorch’s summary library to summarize the size of your deep learning model.</p>
<p>There is a model with a small number of parameters, but a forward pass size is a large model. The forward pass size seems to be related to the speed of the model.</p>
<p>Probably I think that  forward pass size means computed size. Is it the same as FLOPs?</p>
<p>I also wonder if the size of the model we are talking commonly about is the sum of both, or just the number of parameters. I am looking forward to your answer.</p>

username: albanD
cooked: <p>Hi,</p>
<p>All these numbers in MB correspond to the expected memory needed to run the model.</p>

username: chongkai_Lu
cooked: <p>Not sure about its exact meaning. But I tried to change the input_shape and the statistics of “Params”  STAY the same and the “Forward/backward pass size” changed. So guess it’s about FLOPs.</p>
What can one infer from these two graphs?

username: Mridul_kumar
cooked: <p>I was working on two model that achieve similar accuracy. Can someone help me know what can one infer from graph1 and graph2 loss functions plotted as below ?? I will be posting the other graph in replies as forum allows meh to post only one picture</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc79ac4a038a6a4f1e384e00a660da8a1ee8d105.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc79ac4a038a6a4f1e384e00a660da8a1ee8d105" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc79ac4a038a6a4f1e384e00a660da8a1ee8d105_2_690x397.png" alt="image" data-base62-sha1="taS5p7IBk3TTiMKlJenqhf9Wxi5" width="690" height="397" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc79ac4a038a6a4f1e384e00a660da8a1ee8d105_2_690x397.png, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc79ac4a038a6a4f1e384e00a660da8a1ee8d105.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc79ac4a038a6a4f1e384e00a660da8a1ee8d105.png 2x" data-dominant-color="FAFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">727×419 37.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Mridul_kumar
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aac0a24b121261cdc724486c512baaf165a0c0d.png" alt="image" data-base62-sha1="fdFcIfcOnwbW3hAuApGTWsjFPfT" width="626" height="407"></p>
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'embedding' (torch.nn.Parameter or None expected)

username: supervan
cooked: <p>Hi, my model has two parameters, <code>embedding</code> and <code>embedding2</code>. I would like to put embedding2 on gpu and embedding on cpu. Is that possible?</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp


class Reason(torch.nn.Module):
    def __init__(self):
        super(Reason, self).__init__()
        self.embedding = torch.nn.Parameter(torch.zeros(100, 100))
        self.embedding2 = torch.nn.Parameter(torch.zeros(100, 100))

def cuda_mp(model, gpu_id):
    # model.cuda(gpu_id) # this works, but it will put both parameters on gpu
    model.embedding = model.embedding.cuda(gpu_id) # this does not work
    print (gpu_id, model.embedding.device, model.embedding2.device)

if __name__ == '__main__':
    torch.multiprocessing.set_start_method('spawn')

    r = Reason()
    r.share_memory()
    print (r.embedding.device, r.embedding2.device)
    procs = []
    for i in range(3):
        p = mp.Process(target=cuda_mp, args=(r, i))
        procs.append(p)
        p.start()
    for p in procs:
        p.join()
</code></pre>
Validation Loss Diverging - Custom Resnet

username: sauravbanka
cooked: <p>I’ve tried converting my Tensorflow code (a modified resnet) to Pytorch code, but my validation loss in pytorch diverges after the 3rd or 4th epoch. Here’s my model code -</p>
<pre><code class="lang-auto">class ModifiedRes(nn.Module):
    def __init__(self, input_channels = 3, activation="relu"):
        super().__init__()
        self.conv0 = nn.Conv2d(in_channels=input_channels, out_channels=32, kernel_size=7)
        self.activation = nonlinearity(activation)
        self.bn0 = nn.BatchNorm2d(num_features = 32)

        self.bl1 = ResBlock1(32, 16, 32, activation)

        self.bl2_1 = ResBlock2(32, 32, 64, activation)
        self.bl2_2 = ResBlock1(64, 32, 64, activation)

        self.bl3_1 = ResBlock2(64, 64, 128, activation)
        self.bl3_2 = ResBlock1(128, 64, 128, activation)

        self.bl4_1 = ResBlock2(128, 128, 256, activation)
        self.bl4_2 = ResBlock1(256, 128, 256, activation)

        self.bl5_1 = ResBlock2(256, 256, 512, activation)
        self.bl5_2 = ResBlock1(512, 256, 512, activation)

        self.dense = nn.Linear(in_features = 512, out_features = 1)

    def forward(self, x):
        out = self.conv0(x)
        out = self.activation(out)
        out = self.bn0(out)

        out = self.bl1(out)
        out = self.bl1(out)
        out = self.bl1(out)

        out = self.bl2_1(out)
        out = self.bl2_2(out)
        out = self.bl2_2(out)

        out = self.bl3_1(out)
        out = self.bl3_2(out)
        out = self.bl3_2(out)

        out = self.bl4_1(out)
        out = self.bl4_2(out)
        out = self.bl4_2(out)

        out = self.bl5_1(out)
        out = self.bl5_2(out)
        out = self.bl5_2(out)

        out = torch.mean(out, (2, 3))

        out = self.dense(out)

        return out

#loss is BCEWithLogitsLoss
</code></pre>
<p>The rest of my hyperparameters are the same between my Tensorflow Code and Pytorch code. As a sanity check, I checked to see if the number of trainable parameters match up across both implementations. In addition, to check the correctness of my data loader and training loop, I trained a Resnet50 from Torchvision (in which the validation loss decreased as expected.)</p>
<p>I’m stuck on what’s going wrong. Any help is appreciated!</p>

username: ptrblck
cooked: <p>You could check the default parameter initialization in both frameworks and make sure your ported PyTorch model uses the same as your TF model.<br>
If the model is still diverging, you could verify the architecture by loading the TF parameters into the PyTorch model (or vice versa) and check the output for a fixed input tensor.</p>
Linear interpolation between checkpoints in pytorch - Batch Normalization

username: RahimEntezari
cooked: <p>I want to linearly interpolate between two PyTorch trained model checkpoints. For all layers except the batch normalization, I load the stated dict and simply do the linear inteporaltion as follow:</p>
<pre><code class="lang-auto">def interpolate_state_dicts(state_dict_1, state_dict_2, weight):
return {key: (1 - weight) * state_dict_1[key] + weight * state_dict_2[key]
        for key in state_dict_1.keys()}
</code></pre>
<p>I do not know if we can simply do the same for BN layer parameters (Weight, Bias, Running mean, running std) or not? I guess it is not that simple, as mean and std are calculated for a specific batch.</p>

username: ptrblck
cooked: <p>The running stats were updated using all training batches, so if you assume that an interpolation of the parameters works fine, it might also work on the running stats.</p>
<p>EDIT: your use case might also be similar to <a href="https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/">Stochastic Weight Averaging</a>, so you could take a look how the parameters are averaged and how batchnorm layers are treated.</p>
Weighted cross entropy loss

username: dlfma_tjd
cooked: <p>I got training dataset 0 : 1 = 545 : 63 and validation dataset 11: 58.<br>
if i do nn.CrossEntropy(weights=tensor([0.1036, 0.8964], device=‘cuda:0’)), is it okay? I want to do classification with deep learning… but I don’t know how to do with these dataset…please help me…</p>

username: ptrblck
cooked: <p>If your model is overfitting to the majority class, you could pass the <code>weights</code> as e.g. the inverse of the class counts. I’m not sure how the current weights were calculated, but you could check if it’s improving the training by checking the training and validation losses.</p>
RuntimeError: Trying to backward through the graph a second time,

username: Andrew_Musgrave
cooked: <p>Hi,</p>
<p>I am just getting started with pytorch.</p>
<p>I am trying to run the below listed code and get the error “RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time”.</p>
<p>I have tried to replace  1 with 2 and it doesn’t resolved the the issues.</p>
<p>Please could you advise? Thanks in advance.</p>
<p>for epoch in range(20):</p>
<pre><code>opt_Adam.zero_grad() 

output = model(x_train) 

loss = loss_func(output, y_train)
</code></pre>
<ol>
<li>loss.backward()<br>
2.loss.backward(retain_graph=True)</li>
</ol>
<pre><code>opt_Adam.step() 

print(loss.item())</code></pre>

username: ptrblck
cooked: <p>Could you post an executable code snippet to reproduce this issue, please?<br>
You can use random input data (instead of a real dataset), which would make debugging easier.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to convert a list whose element is Tensor to a Tensor?

username: Silvester98
cooked: <p>I have a list like <code>[x1, x2, ..., xn]</code>, and <code>xi</code> is a Tensor with size <code>torch.Size([2, 3])</code>, how to convert this list a Tensor with size <code>torch.Size([n, 2, 3])</code>. I am looking for an elegant method. Thanks for your reply.</p>

username: ptrblck
cooked: <p>You could use <code>torch.stack</code>:</p>
<pre><code class="lang-python">n = 10
l = [torch.randn(2, 3) for _ in range(n)]
t = torch.stack(l)
print(t.shape)
&gt; torch.Size([10, 2, 3])
</code></pre>
Validation loss is neither increasing or decreasing

username: aritra_2506
cooked: <p>Usually when a model overfits, validation loss goes up and training loss goes down from the point of overfitting. But for my case, training loss still goes down but validation loss stays at same level. Hence validation accuracy also stays at same level but training accuracy goes up. I am trying to reconstruct a 2D image from a 3D volume using UNet. Same is the behavior when I am trying to reconstruct 3D volume from 2D image but at higher loss and lower accuracy. Can someone explain the curve that why validation loss is not going down from the point of overfitting?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6f70b85e15961835c0a4e157dd3fd4bd100c168.png" data-download-href="https://discuss.pytorch.org/uploads/default/e6f70b85e15961835c0a4e157dd3fd4bd100c168" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6f70b85e15961835c0a4e157dd3fd4bd100c168_2_690x258.png" alt="image" data-base62-sha1="wXd6NGtImEPTjxVHnv6nXclKWIU" width="690" height="258" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6f70b85e15961835c0a4e157dd3fd4bd100c168_2_690x258.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6f70b85e15961835c0a4e157dd3fd4bd100c168_2_1035x387.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6f70b85e15961835c0a4e157dd3fd4bd100c168.png 2x" data-dominant-color="FAFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1280×480 47.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>It might depend how the model performance and loss are calculated. E.g. if your model outputs the same class for all pixels during validation without significantly changing its confidence, the loss and performance might just get stuck.</p>
Error in BatchNorm1d documentation

username: randbit
cooked: <p>I think there’s an error in the documentation for <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" rel="noopener nofollow ugc">BatchNorm1d</a>. It currently says that num_features is “L from input of size (N,L)”. Shouldn’t it say “C from input of size (N,C)”, as the input shape specification correctly states?</p>
<p>Also, should BatchNorm0d not have its own function? The current API seems confusing since it doesn’t fit the expected pattern of</p>
<ul>
<li>BatchNorm3d for inputs of shape of shape [C,D,H,W]</li>
<li>BatchNorm2d for inputs of shape [C,H,W]</li>
<li>BatchNorm1d for inputs of shape [C,L]</li>
<li>BatchNorm0d for inputs of shape [C]</li>
</ul>

username: ptrblck
cooked: <aside class="quote no-group" data-username="randbit" data-post="1" data-topic="104483">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3bc359/48.png" class="avatar"> randbit:</div>
<blockquote>
<p>It currently says that num_features is “L from input of size (N,L)”. Shouldn’t it say “C from input of size (N,C)”, as the input shape specification correctly states?</p>
</blockquote>
</aside>
<p>Yeah, it might make sense to use the same naming for the <code>num_features</code> and the <code>input</code> documentation. Would you be interested in fixing the docs? If so, feel free to create a GitHub issue with your description of the issue and create the PR after an initial discussion there.</p>
<aside class="quote no-group" data-username="randbit" data-post="1" data-topic="104483">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3bc359/48.png" class="avatar"> randbit:</div>
<blockquote>
<p>Also, should BatchNorm0d not have its own function?</p>
</blockquote>
</aside>
<p>I don’t think it’ll improve the usability of this modules, but let’s see what others think about it.</p>
Prevent crash when using a large tensor

username: anto-nyxius
cooked: <p>Hello, i’m using a pretrained bert model to do text embedding. I’ve applied a tokeniser on my data and when i call the model with a tensor of dimension 2000x64, i get an error: end of process. It works when i use fewer data (1000x64):</p>
<p>with torch.no_grad():<br>
outputs = bertweet(inputs, attention_mask=attention_mask)</p>
<p>How can i use my model with a bigger tensor ?</p>
<p>PS: I’m working locally on my computer on ubuntu 20</p>

username: ptrblck
cooked: <p>Are you using the CPU or a GPU?<br>
Could you check, if you might be running our of system RAM and thus your script just crashes (if your GPU is running out of memory, you should get a CUDA error)?</p>
How to optimally implement pairwise custom kernel operation?

username: paganpasta
cooked: <p>Hi,</p>
<p>Given 1D differentiable vectors <code>A=[Nx1]</code> and <code>B=[Mx1]</code>, I am looking to compute pairwise kernel operation .</p>
<pre><code class="lang-auto">def kernel(a, b):
    a*b*torch.exp(-torch.abs(a-b)/0.4)
</code></pre>
<p>Is there a way to avoid looping over individual items? I need to perform the kernel operation for pairwise entries in <code>{A, A}</code>, <code>{A, B}</code> and <code>{B, B}</code> which might make it computationally heavy if done iteratively.</p>

username: ptrblck
cooked: <p>You could try to use broadcasting as seen here:</p>
<pre><code class="lang-python">a = torch.arange(4).float().view(4, 1)
b = torch.arange(4).float().view(4, 1)

# element-wise
print(a - b)

# pair-wise
print(a.unsqueeze(1) - b)
</code></pre>
<p>which would result in a higher memory footprint, but might be faster than your sequential approach.</p>
Load trained yolov5 model out side the detect.py

username: dulanga_heshan
cooked: <p>I’m going to develop a flask web application using yolov5 trained model. so as described in the doc, it works fine with the command-line argument, what I tried was I tried to apply the oop concept and create a model object for use with every single frame.</p>
<pre><code class="lang-auto">class Model(object):


    def __init__(self, weights, save_img=False):

        self.view_img = True,
        self.save_txt = False,
        self.imgsz = 640
        self.device = select_device()
        print(self.device, "llll")
        self.output = "output"
        if os.path.exists(self.output):
            shutil.rmtree(self.output)  # delete output folder
        os.makedirs(self.output)  # make new output folder
        self.half = self.device.type != 'cpu'  # half precision only supported on CUDA

        # Load model
        self.model = torch.load(weights, map_location=self.device)['model'].float()  # load to FP32
        self.model.to(self.device).eval()
        if self.half:
            self.model.half()  # to FP16
</code></pre>
<p>In side Camera.py i tried to create model using below line<br>
<code>model = Model('weights/best.pt')</code></p>
<p>but now I’m getting below error in the stack</p>
<pre><code class="lang-auto"> File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\flask_app\app.py", line 4, in &lt;module&gt;
    from camera import Camera
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\flask_app\camera.py", line 8, in &lt;module&gt;
    from detect_image import detect_image
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\flask_app\detect_image.py", line 12, in &lt;module&gt;
    model = Model("weights/best.pt")
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\flask_app\model.py", line 27, in __init__
    self.model = torch.load(weights, map_location=self.device)['model'].float()  # load to FP32
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\serialization.py", line 853, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'models'
</code></pre>
<p>ass describe in Yolo I tried to execute below line in the same manner with the same weight parameters in both cases (command line and <code> model = Model("weights/best.pt")</code>)</p>
<p><code>  self.model = torch.load(weights, map_location=self.device)['model'].float()</code></p>
<p>any suggestions or solutions</p>
<p>thank you</p>

username: ptrblck
cooked: <p>It seems you are trying to load the model directly, which is not the recommended way since it may break in various ways, if you don’t keep the file structure equal.<br>
The recommended way would be to save and load the <code>state_dict</code> as described <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">here</a>, which would avoid these import errors.</p>
Why Pytorch library is occupying 13GB storage on my GPU?

username: ruchitpatel
cooked: <p>I’m getting following problem, which I’m not able to solve.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 598.00 MiB (GPU 0; 14.73 GiB total capacity; 13.46 GiB already allocated; 337.88 MiB free; 13.46 GiB reserved in total by PyTorch)
</code></pre>
<p>This is happening when I’m running my model on test images to produce results.</p>
<pre><code class="lang-auto">def buildG(UPSCALE_FACTOR=4):
    netG = Generator(UPSCALE_FACTOR)
    netG.train()
    netG.load_state_dict(torch.load(G_weights_load))
    netG.cuda()

    return netG

netG = buildG()
def test_on_single_image(path='/content/data1/lrtest.jpg',UPSCALE_FACTOR=4):
    img = Image.open(path)
    layer = ToTensor()
    img1 = layer(img)
    sh = img1.shape
    img1 =img1.reshape((1, sh[0], sh[1], sh[2]))
    img2 = netG(img1.cuda(0))
    utils.save_image(img2, imgs_save)
    files.download('/content/temp1.jpg')
</code></pre>
<p>My model is runnig good on the training loop (on the GPU). Then why is it giving such GPU memory full error? And I’m not getting why pytorch is occupying memory? There is no clear indication on what is occupying more memory! Weights of netG are just 3MB in size.</p>

username: ptrblck
cooked: <p>You might be accidentally storing tensor, which are still attached to the computation graph, which should be visible by an increasing usage of the device memory.<br>
Your current code snippet looks fine, so you might have the problematic line of code in another function.<br>
Wrap your code into a <code>with torch.no_grad()</code> block during the validation to avoid storing intermediate tensors and the computation graph.</p>
Best practices when reading a large number of files every dataloader iteration

username: himat
cooked: <p>I’m training a CNN that classifies small regions (7x7) from 40 large input layers that are all used to do the classification. So the CNN takes in a (40, 7, 7) input to be classified.<br>
But the 40 input layers themselves though are very large (over 10k x 10k pixels), and so I can’t read all 40 files in the dataset constructor because that goes beyond the amount of RAM I have.</p>
<p>So instead, I’m forced to read all 40 files during every <code>__getitem__</code> call of the dataset loader, and just read the desired 7x7 location inside the input layers for this iteration.</p>
<p>But this has made my training very slow as I think it’s just taking a long time to open and read the windows from all 40 files every iteration.</p>
<p>I am already using multiple workers</p>
<pre><code class="lang-auto">torch.utils.data.DataLoader(dset, batch_size=32, num_workers=4)
</code></pre>
<pre><code class="lang-python">class MyDataset(Dataset):
  def __init__(self):
    self.all_input_layers = ["file1", "file2", ..., "file40"]
  def __getitem__(self, idx):
    all_layer_data = []
    for (file in self.all_input_layers):
      curr_window = read_file_window(file) # returns 7x7 np array
      all_layer_data.append(curr_window)

    data = np.concatenate(all_layer_data, axis=0) # 40 x 7 x 7 data that will be inputted into CNN

    return data
</code></pre>
<p>What are the other things I should try to speed this up?</p>
<ul>
<li>Should I use <code>torch.multiprocessing.set_sharing_strategy('file_system')</code>?
<ul>
<li>I don’t have any issues with too many file descriptors being open, which is the case that using this ‘file_system’ strategy seems to be recommended for, so not sure if it would help me.</li>
</ul>
</li>
<li>What else?</li>
</ul>

username: ptrblck
cooked: <p>You could profile the method to read the data as well as create the windows from these tensors and check where most of the time is spent.<br>
Once you’ve isolated it, you could try to accelerate the code (e.g. with a 3rd party library if possible) or think about changing the overall data loading (e.g. would it be possible to store the data in another format and only load the desired window instead of the whole data array).<br>
For a more general advice have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>.</p>
Lazy initialization of layers and parameters: "Class Linear does not have an __init__ function defined"

username: Ark-kun
cooked: <p>My high-level goal is to compactly serialize model architectures without parameter values. See <a href="https://discuss.pytorch.org/t/save-model-architecture-only/104558" class="inline-onebox">Save model architecture only</a></p>
<p>To accomplish this goal I’ve tried to lazily initialize layers and parameters in the <code>forward()</code> method.</p>
<pre><code class="lang-python">class Net4(nn.Module):

    def forward(self, x):
        if not hasattr(self, 'fc1'):
            self.fc1 = nn.Linear(16 * 5 * 5, 120)
        if not hasattr(self, 'fc3'):
            self.fc2 = nn.Linear(120, 84)
        if not hasattr(self, 'fc3'):
            self.fc3 = nn.Linear(84, 10)

        x = F.sigmoid(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        x = self.fc3(x)
        return x

script_module4 = torch.jit.script(Net4())
script_module4.save('pytorch_model.Net4.TorchScriptModule')
</code></pre>
<p>I’m getting the following error:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.7/site-packages/torch/jit/_recursive.py in create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
    302     property_rcbs = [p.resolution_callback for p in property_stubs]
    303 
--&gt; 304     concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
    305 
    306 

RuntimeError: 
Class Linear does not have an __init__ function defined:
  File "&lt;ipython-input-23-bda237bd2209&gt;", line 8
    def forward(self, x):
        if not hasattr(self, 'fc1'):
            self.fc1 = nn.Linear(16 * 5 * 5, 120)
                       ~~~~~~~~~ &lt;--- HERE
        if not hasattr(self, 'fc3'):
            self.fc2 = nn.Linear(120, 84)
</code></pre>
<p>How can I work around this error?</p>
Save model architecture only

username: Ark-kun
cooked: <p>I want to have a system where the model authoring and training are separate steps. The model needs to be serialized to disk between those stages.<br>
Keras has “model config” which describes the model architecture in a compact JSON-friendly way: <a href="https://www.tensorflow.org/api_docs/python/tf/keras/models/model_from_config" rel="noopener nofollow ugc">https://www.tensorflow.org/api_docs/python/tf/keras/models/model_from_config</a></p>
<p>I want to have something similar for PyTorch. This problem is almost solved by</p>
<pre><code class="lang-python">torch.jit.script(MyNet()).save('my_net.PyTorchScriptModule')
</code></pre>
<p>But I see that the result is pretty big and binary due to the fact that the weights are included.<br>
Is there a way to not save parameters? Or to save them in un-initialized state so that they’re lazily initialized after loading?</p>
<p>TorchScript seems so powerful. Maybe there is a way to save the whole Module, not just the forward method + initialized constants?</p>
Does Captum support Graph Neural Networks (GNN)?

username: milad123
cooked: <p>Hello everybody. I have recently started using Captum for model interpretability and I’ve found it really interesting. My main question is: Does Captum support Graph Neural Networks too? Thanks a lot.</p>

username: joaquincabezas
cooked: <p>It looks so! There is a Google Colab from <a href="https://github.com/m30m" rel="noopener nofollow ugc">Amin</a> at the list of Pytorch Geometric that uses Captum:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/colabs.html" target="_blank" rel="noopener nofollow ugc">pytorch-geometric.readthedocs.io</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/colabs.html" target="_blank" rel="noopener nofollow ugc">Colab Notebooks — pytorch_geometric 1.6.2 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Regards,<br>
Joaquín.</p>
torch.distributions.categorical.Categorical not producing tensor output of same dimension

username: granth_jain
cooked: <p>Hi,</p>
<p>I have a 3 dimension tensor “prob: tensor([[[0.2793, 0.3314, 0.3893]]], grad_fn=)”<br>
and I am sampling an action by Categorical distribution.</p>
<pre><code>                    print("prob:",prob)
                    action = Categorical(prob).sample().detach()
                    print("action:",action)
                    log_prob_a = log_prob.gather(1, action)
</code></pre>
<p>This is the output I am getting.</p>
<p>prob: tensor([[[0.3201, 0.3268, 0.3531]]], grad_fn=)<br>
action: tensor([[2]])<br>
Traceback (most recent call last):<br>
File “D:\get_data\ind_a3c_lstm.py”, line 99, in train<br>
log_prob_a = log_prob.gather(1, action)<br>
RuntimeError: Index tensor must have the same number of dimensions as input tensor</p>
<p>My, Question is why action is not have same number of dimensions as prob tensor.</p>
About conversion of Variable(x, volatile = True) to C++

username: Chen0729
cooked: <p>Hi, All,</p>
<p>I have a conversion issue of the following from python to C++</p>
<p>self.D(Variable(x, volatile = True), cuda=self.args.cuda, feature = True)[0]</p>
<p>I am wondering how shall I convert Variable(x, volatile = True) to C++ as:<br>
self.D(Variable(x, true), args.cuda, true)[0] has triggered an error in Variable</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p><code>Variable</code>s are deprecated since PyTorch <code>0.4</code> so you can use tensors now in the Python and C++ API.</p>

username: Chen0729
cooked: <p>Thanks for the comments. I appreciate that.</p>
Summed batch mAP score equal to epoch mAP score?

username: cookiereg
cooked: <p>Hi,</p>
<p>I want to calculate mAP score for each epoch. However, keeping the each prediction in memory for an entire epoch is not possible on my machine. So I was wondering if I could simply compute the mAP for the predicitions of each batch and sum them up at the end of the epoch.<br>
Is that possible or will the result be wrong if I do that?</p>
<p>Thanks in advance!</p>
How to make Colormaps for 2d feature maps

username: spadel
cooked: <p>I am trying to reconstruct a neural net for shape and appearance disentanglement. To track progress, I want to produce some color maps that look as presented in the paper:</p>
<p><a href="https://i.stack.imgur.com/F59Lm.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/F59Lm.png" alt="enter image description here" width="148" height="268"></a></p>
<p>The maps have the following shape: [batch_size, n_maps, 64, 64].</p>
<p><code>n_maps</code> equals 16, so I basically want to plot 16 different color maps. Is there an elegant way to do that?</p>
Skip connections in ResNet and Densenet

username: edshkim98
cooked: <p>Hi, this might be a simple question but when I would like to implement skip connections from ResNet or Densenet,<br>
is it the number of channels or the dimension of the image that should be same for previous and current layer to do skip connection?</p>
<p>Thank you!</p>

username: snowe
cooked: <p>Hi <a class="mention" href="/u/edshkim98">@edshkim98</a></p>
<p>I am not 100% sure but the way I understand it (at least for ResNets), is that the skip connections handle different numbers of channels and the stride is handling the different dimensions of the image. If you look at a ResNet34 for example, after the first layer, you have 64 out_channels and in the second block of layer two you expect 128 in_channels. Since the skip connection skips the first block of layer two, you have to transform the 64 to 128 by e.g. applying 128 1x1x64 convolutions to match the number of channels. At the same time, since every new layer starts with a stride of 2, the dimensions of the images are halved at the beginning of every layer (note, I use the term layer as defined in the original paper: <a href="https://arxiv.org/abs/1512.03385" rel="noopener nofollow ugc">https://arxiv.org/abs/1512.03385</a>).</p>
<p>If this is correct, I find the term ‘downsampling’ used in the implementation (<a href="https://pytorch.org/docs/stable/_modules/torchvision/models/resnet.html#resnet34" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torchvision/models/resnet.html#resnet34</a>)  to be a bit confusing, since technically you ‘upsample’ the number of channels…</p>
<p>Hope this helps!</p>
<p>All the best<br>
snowe</p>
Saving and Loading Model Checkpoint - Epoch & Validation

username: moreshud
cooked: <p>I have a large dataset to train and short of cloud RAM and disk space (memory). I think one of the approaches to training all the dataset is by creating a checkpoint to save the best model parameter based on validation and likely the last epoch. I will be glad for guidance on implementing this i.e ensuring training continues from the last epoch with the best-saved model parameter from the previous trainig session</p>

username: snowe
cooked: <p>Hi <a class="mention" href="/u/moreshud">@moreshud</a></p>
<p>You could do something like:</p>
<pre><code class="lang-auto">if accuracy_val &gt; max_accuracy_val:

            checkpoint = {
                'epoch': epoch,
                'model_state': model.state_dict(),
                'optimizer_state': optimizer.state_dict(),
            }
            torch.save(checkpoint, 'path/to/folder/filename.pth')

            max_accuracy_val = accuracy_val
</code></pre>
<p>That way you save the state of the model whenever you have reached a new maximum accuracy on the validation set.</p>
<p>When you want to continue training you can do:</p>
<pre><code class="lang-auto">loaded_checkpoint = torch.load('path/to/folder/filename.pth')
loaded_epoch = loaded_checkpoint['epoch']
loaded_model = model() # instantiate your model
loaded_model.load_state_dict(loaded_checkpoint['model_state'])
loaded_optimizer = torch.optim.SGD(loaded_model.parameters(), lr=0, momentum=0) # or whatever optimizer you use
loaded_optimizer.load_state_dict(loaded_checkpoint['optimizer_state'])
</code></pre>
<p>You can then continue training with the loaded model and loaded optimizer.</p>
<p>DISCLAIMER: I am relatively new to PyTorch myself. This approach works for me but I cannot guarantee that there are no better options <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Anyways, I hope it helps!</p>
<p>All the best<br>
snowe</p>

username: moreshud
cooked: <p>Thanks for the contribution.</p>
TCPStore permission denied

username: Shirley_Han
cooked: <p>While trying to start a multi-gpu process;</p>
<pre><code class="lang-auto"> store = TCPStore(result.hostname, result.port, world_size, start_daemon)
RuntimeError: Permission denied
</code></pre>
<p>Has anyone met with this before?</p>

username: mrshenli
cooked: <p>What port number you are using? and does it work if using sudo?</p>

username: linzhk3
cooked: <p>I have met this problem minutes ago and i slove this problem after i change this code from <code>os.environ['MASTER_PORT'] = '80'</code> to  <code>os.environ['MASTER_PORT'] = '9901'</code></p>
Plotting intermediate data tensors - access data or detach?

username: Manuel_Kurz
cooked: <p>Hello,</p>
<p>I’m still confused with detach(), altough i searched and read a lot… When I’m plotting tensors in each epoch, like input images or decode one hot encoded output images and plot them, is it correct to access this tensors by .data.cpu() or do I need .detach().cpu() ? Im running my model on a gpu, it is basically a VAE.</p>
<p><strong>1st example:</strong></p>
<pre><code class="lang-auto">self.create_output_image_grids(img_data.detach().cpu(), recon_data.detach().cpu())
</code></pre>
<p>In this function I basically use torch.cat() to create a grid of images…</p>
<p><strong>2nd example</strong><br>
If I keep track of latent var’s (mu, logvar) to make some plots e.g. every 5 epochs:</p>
<pre><code class="lang-auto">recon_data, mu, logvar, z = model(img_data)
latent_mu.append(mu.detach().cpu().squeeze())
</code></pre>
<p>Big thanks in advance!</p>

username: googlebot
cooked: <p>It is the same, though it has been suggested that .data is internal/private, it is used too often to break it.</p>
How to train Encoder network in GAN?

username: Jay_Super
cooked: <p>Hi, I am trying to implement <a href="https://arxiv.org/pdf/1711.07461.pdf" rel="noopener nofollow ugc">BiCOGAN</a> which has a structure like below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9a73aa5f5e0761cea4f29296627b4462405ba12.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9a73aa5f5e0761cea4f29296627b4462405ba12" title="스크린샷 2020-11-29 오후 9.55.39"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9a73aa5f5e0761cea4f29296627b4462405ba12_2_690x332.png" alt="스크린샷 2020-11-29 오후 9.55.39" data-base62-sha1="xkZxjn98X8ZIGjjWkVeY2mHfvUe" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9a73aa5f5e0761cea4f29296627b4462405ba12_2_690x332.png, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9a73aa5f5e0761cea4f29296627b4462405ba12.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9a73aa5f5e0761cea4f29296627b4462405ba12.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2020-11-29 오후 9.55.39</span><span class="informations">910×438 57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The author says that the Encoder is trained jointly with generator and discriminator. Can anyone give me some advice on how I can implement this in pytorch? How should the update be implemented? and Should I create a separate optimizer just for Encoder? Any advice would be helpful. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to deal with autoencoder (or other) encodings for similarity inference and clustering

username: vishak_bharadwaj
cooked: <p>I trained an autoencoder that accepts an image and produces an encoding from the encoder. The encoding is 64x64x64 so flattening it into a row vector of size 262144.</p>
<p>I have 5000 images, and given a test image( which goes through the encoder) , I need to find n similar encodings and corresponding images, and maybe later cluster the whole dataset.</p>
<p>one GitHub repo owner seems to have done it by concatenating the image encoding into a 5000,262144 matrix, and then running a knn on it with the test encoding.</p>
<p>I can’t do this on Colab cuz the instance crashes when RAM fills up.</p>
<p>Tried to use a regular for loop to convert each picture and save it as a .npy, but that was extremely painful since Colab just freezes when the loop is more than 500 big.</p>
<p>Even if I do have the .npy files, not sure how to classify, or cluster since I can’t put the whole thing on the RAM.</p>