akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
transforms.Normalize

jasg: <p>please, explain me about transforms.Normalize; I put in my code this transforms.Normalize([0.4161,],[0.1688,]) to image type gray and image type RGB transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])</p>
<p>and<br>
transforms.Normalize(<br>
mean=[-0.485/0.229, -0.456/0.224, -0.406/0.225],             std=[1/0.229, 1/0.224, 1/0.225]</p>
<p>thanks</p>

ptrblck: <p><code>transform.Normalize</code> normalizes the input tensor by subtracting the specified <code>mean</code> and dividing by the <code>std</code> to create a tensor with zero mean and unit variance.<br>
The <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Normalize">docs</a> also give the formula for it.</p>

jasg: <p>please explain how to get ([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
thanks</p>

ptrblck: <p>The posted values are the mean and std of ImageNet, which can often be reused.<br>
You could also calculate these values for your training dataset as e.g. explained <a href="https://discuss.pytorch.org/t/efficient-way-of-calculating-mean-std-etc-of-subset-dataset/84890">here</a>.</p>
ValueError: `Dataloader` returned 0 length. Please make sure that your Dataloader at least returns 1 batch

turian: <p>I am trying to train on a sample number of examples, for quick testing. I do an 80/10/10 train/val/test split. I am using pytorch lightning.</p>
<p>With 10000 examples, and batchsize 4, I am fine.</p>
<p>However, with 1000 examples, I get this error:</p>
<pre><code class="lang-auto">ValueError: `Dataloader` returned 0 length. Please make sure that your Dataloader at least returns 1 batch
</code></pre>
<p>Why? I have verified that I have a Dataset of size 800 for train, 100 for val, and 100 for test. This is in the trainer</p>
<pre><code class="lang-auto">/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py in _has_len(dataloader)
     47         # try getting the length
     48         if len(dataloader) == 0:
---&gt; 49             raise ValueError('`Dataloader` returned 0 length.'
     50                              ' Please make sure that your Dataloader at least returns 1 batch')
     51         return True

ValueError: `Dataloader` returned 0 length. Please make sure that your Dataloader at least returns 1 batch
</code></pre>

cskarthik7: <p>Please share your code so that we can see the error and rectify it.</p>
Onehot matrix generation using scatter_

Prajin_Khadka: <pre><code class="lang-auto">import numpy
import torch
wine_path = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv"
wineq_numpy = numpy.loadtxt(wine_path, dtype=numpy.float32, delimiter=";",skiprows=1)

wineq=torch.from_numpy(wineq_numpy)


target=wineq[:,-1]
target = target.unsqueeze(-1)
print(target.shape)
print(target_onehot.shape)

target_onehot = torch.FloatTensor(target.shape[0], 10)
target_onehot.zero_()
target_onehot.scatter_(1, target, 1)

</code></pre>
<p>Gives  error</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)

&lt;ipython-input-192-2622c1acffeb&gt; in &lt;module&gt;()
      1 target_onehot = torch.FloatTensor(target.shape[0], 10)
      2 target_onehot.zero_()
----&gt; 3 target_onehot.scatter_(1, target, 1)

RuntimeError: index 4668431376536567808 is out of bounds for dimension 1 with size 10


</code></pre>
<p>I don’t get it why its throwing this error ?</p>
<p>I reckon this is the way ?<br>
i tried in doing by generating random values it works.</p>
<pre><code class="lang-auto">import torch

batch_size = 5
nb_digits = 10
# Dummy input that HAS to be 2D for the scatter (you can use view(-1,1) if needed)
y = torch.LongTensor(batch_size,1).random_() % nb_digits
# One hot encoding buffer that you create out of the loop and just keep reusing
y_onehot = torch.FloatTensor(batch_size, nb_digits)

# In your for loop
y_onehot.zero_()
y_onehot.scatter_(1, y, 1)

print(y)
</code></pre>

Nikronic: <p>Hi,<br>
<code>label</code> or <code>target</code> has to be <code>LongTensor</code> not <code>Float</code>.</p>
<aside class="quote no-group" data-username="Prajin_Khadka" data-post="1" data-topic="85646">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prajin_khadka/48/25650_2.png" class="avatar"> Prajin_Khadka:</div>
<blockquote>
<p><code>y = torch.LongTensor(batch_size,1).random_() % nb_digits</code></p>
</blockquote>
</aside>
<p>Here, <code>target</code> is float and you need to convert it to long using <code>.long()</code></p>
<aside class="quote no-group quote-modified" data-username="Prajin_Khadka" data-post="1" data-topic="85646">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prajin_khadka/48/25650_2.png" class="avatar"> Prajin_Khadka:</div>
<blockquote>
<p><code>target = target.unsqueeze(-1)</code> -------&gt; <code>target = target.unsqueeze(-1).long()</code></p>
</blockquote>
</aside>
<p>PS. it would be great if you could use more descriptive title such as <em>onehot matrix generation using scatter_</em>, it help people to interact with questions easier.</p>
<p>Bests</p>

Prajin_Khadka: <p>Oh got it. Thank you so much</p>
What is the official release build options?

Weichao_Luo: <p>I was wondering what is the build options for release package? it seems different than what is said on GitHub build from source section. At least for _C.cpython-36m-x86_64-linux-gnu.so, the official build seems static linking cudnn/nccl etc. but what I got from custom build / docker build, it always dynamic linking cudnn / nccl. I known there’s USE_STATIC_CUDNN / USE_STATIC_NCCL which may works for static linking, but still wondering the exactly build options for release build.</p>

seanprime7: <p>I am also interested in this information.<br>
Can anyone share the official release build options?</p>
<p>Thanks.</p>
Issue with Quantization

Ashish_Gupta1: <p>I have input image of shape <code>32*1*32*128</code> where 32 is my batch size(index 0).<br>
I want to quantize my model. When I call my evaluate function then it shows this error</p>
<pre><code class="lang-auto"> File "/media/ai/ashish/OCR/Text_Recognition/modules/transformation.py", line 158, in build_P_prime
    batch_size, 3, 2).float().to(device)), dim=1)  # batch_size x F+3 x 2
</code></pre>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::_cat' with arguments from the 'QuantizedCPUTensorId' backend. 'aten::_cat' is only available for these backends: [CUDATensorId, CPUTensorId, VariableTensorId].
</code></pre>

xieydd: <p>I meet same question, have you solve the question?</p>

jerryzh168: <p>did you use FloatFunctional(<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/functional_modules.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/functional_modules.py</a>) to replace the call to <code>torch.cat</code>?</p>
Diagonal matrix of parameters

mahsa: <p>Hi,<br>
I want to define my layer matrix. In my defined layer, I have a diagonal matrix which its parameters are on its diameter. I tried the following way, but it didn’t work. Is there any way I can define something like this?</p>
<pre><code class="lang-auto">self.weight = Parameter(torch.diag(torch.FloatTensor(1, nparam)))
</code></pre>
<p>Thanks</p>

ptrblck: <p><code>torch.diag</code> would return a 1D tensor with the diagonal elements of your input, since you are passing a 2D input tensor to this method.<br>
If you would like to create a 2D diagonal matrix, your input tensor should be a 1D tensor.</p>
<p>That being said, note that <code>torch.FloatTensor</code> will use uninitialized memory, so you would need to initialize the tensor beforehand.</p>

mahsa: <p>Thanks a lot lot for your help, I didn’t notice that.</p>
Question about unused 'registerer' variable

Yellowfang: <p>Hello,</p>
<p>I’m reading the PyTorch source code after that the PyTorch package is installed in python.</p>
<p>There is unused(I guess) variable named ‘registerer’ in torch/csrc/autograd/generated/VariableType_{0, 1, 2, 3}.cpp. But i think this variable is important for registering tensor operations.</p>
<p>Does anyone know where the ‘registerer’ variable work?</p>
<p>Or is there any effective debugging method for PyTorch c++ source code?</p>
<p>Thanks!</p>
How to handle for loops with loop count being matrix Tensor

Zorobay: <p>Hi!</p>
<p>I know that the title might not make much sense, so let me explain. I’m writing a differentiable renderer in PyTorch, much like PyTorch3D. I want to model how OpenGL and GLSL renders shaders, which I did first using nested for loops for pixel positions, and then calling a shader function on each pixel. This was ridiculously slow in python, so I switched over to a matrix based approach, where all my pixel positions are pre-computed. This works very well and is about 1000 times faster. However, it now gives me a problem when I’m writing a cloud noise shader. I’m using the concept of <em>fractal brownian motion</em> which adds octaves of noise in a for loop. I want the user to be able to control how many octaves are added through a paramter called <code>detail</code>. The problem is that, since switching to a matrix based renderering, the function is supplied with parameters that are matrices and contains the parameters for each pixel. Let me show the code:</p>
<pre><code class="lang-python">def shade_mat(self, scale: Tensor, detail: Tensor) -&gt; Tensor:
    w, h = Shader.width, Shader.height
    uv = Shader.frag_pos[:, :, :2]  # Pre-calculated fragment coordinates (WxHx3)
    color = torch.tensor((0., 0., 0.)).repeat(w, h, 1)
    color = color + noise.fractalBrownianMotion(uv * scale, detail)

    return color  # This is the full rendered image
</code></pre>
<pre><code class="lang-python">def fractalBrownianMotion(p: Tensor, detail: Tensor) -&gt; Tensor:
    """
    Adds octaves of smooth noise to create "fractal brownian motion" or "fractal noise".
    
    :param p: 2D Tensor of coordinates
    :param detail: Integer 2D tensor dictating the number of octaves to add
    :return: A pseudo-random 2D noise Tensor
    """
    w, h = p.shape[0], p.shape[1]

    # Initial values
    value = torch.tensor((0.0)).repeat(w, h, 1)
    amplitude = torch.tensor((0.5)).repeat(w, h, 1)

    # Loop and add octaves of more and more detailed noise
    for i in range(4):  # detail is supposed to be used here but is a matrix...
        value = value + (amplitude * smoothNoise2D(p))
        p = p * 2.0
        amplitude = amplitude * 0.5

    return value
</code></pre>
<p>I need to use the <code>detail</code> matrix parameter in the for loop so that for pixel <code>(0,0)</code> i use <code>detail[0,0,:]</code>. I could do this if PyTorch had some sort of <code>map</code> function. Any idea how I could solve this?</p>
Cannot load trace file using libtorch 1.5 in C++

sercan: <p>I created a trace file in Python side. When I try to load it in C++ side, It gave me this error message.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55c741216ac9355424fb18a6a4f648161f3694af.png" data-download-href="https://discuss.pytorch.org/uploads/default/55c741216ac9355424fb18a6a4f648161f3694af" title="libtorch_1_5_load_error_message"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55c741216ac9355424fb18a6a4f648161f3694af.png" alt="libtorch_1_5_load_error_message" data-base62-sha1="cePu6FuujsI6wE2QWGZ66bzuws7" width="690" height="260" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">libtorch_1_5_load_error_message</span><span class="informations">1032×390 30.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I couldn’t load this trace file. Could you help me?</p>
Chunking a tensor with overlap

iiSeymour: <p>What is the best way to chunk up a tensor into fixed size chunks with a specified overlap?</p>
<p>So given a tensor T with a shape (for example) W=10,254, target chunksize of C=500 and overlap O=100 create a tensor Q with shape (B, C) where Q[0], Q[1], … is T[0:500], T[400:900], and Q[-1] is padded with a specified value.</p>

ptrblck: <p>You could use <code>F.pad</code> to pad a specific tensor in the specified dimensions as described in the <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=pad#torch.nn.functional.pad">docs</a>.</p>
<p><code>tensor.unfold</code> or <code>F.unfold</code> can be used to create overlapping patches of your tensor.</p>
How to train a generative chatbot with new data without making it inaccurate for the previous data

Fahad_Maqsood_Qazi: <p>Hi, I followed the PyTorch tutorial on creating a chatbot. I trained it on Cornell Movie-Dialogs Corpus. But what I need to do now is to append new data to the chatbot without losing previous weights. how can I do that?</p>
<p>here is the notebook for training the model (from a checkpoint): <a href="https://colab.research.google.com/drive/1oy0UjFpQdIjh2T5dLA3x4o3euVCV3ZaE?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1oy0UjFpQdIjh2T5dLA3x4o3euVCV3ZaE?usp=sharing</a></p>
<p>And here is the notebook where I am loading the previous model and trying to add new data:<br>
<a href="https://colab.research.google.com/drive/1U1JLbjye1d9jqMAw0R4UzCkqy1pbl30Y?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1U1JLbjye1d9jqMAw0R4UzCkqy1pbl30Y?usp=sharing</a></p>
Bug in QAT for classification?

xieydd: <p>When i use a pretrained QAT model, my task is face detection, the face landmark and the bbox is almost same like float training, but the classification score from the result of softmax will be wrong, because it will be greater than 1.It think it is a bug in scale and point from softmax?</p>

xieydd: <p>Anybody help me? <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

xieydd: <p>My code fault, sorry.</p>
Optimizing one of the model output w.r.t input

Neel_S: <p>I have a model with following definition:</p>
<p>class model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(model, self).<strong>init</strong>()<br>
self.linear1 = nn.Linear(2, 512)<br>
self.linear2 = nn.Linear(512, 64)<br>
self.linear3 = nn.Linear(64, 1)<br>
self.linear4 = nn.Linear(512, 1)</p>
<p>def forward(self, x):<br>
x = self.linear1(x)<br>
y = x<br>
x = self.linear4(x)</p>
<pre><code>y = self.linear2(y)
y = self.linear3(y)

return x, y
</code></pre>
<p>After optimizing the model on a particular dataset, I want to find the input tensor that gives the maximum value of y. The summary of the problem is as follows:<br>
i) I have optimized the model on a dataset.<br>
ii) Now, I want to find out what input to the model gives maximum value of y. Here I do not care about the value of x that is being returned by the model.</p>
<p>I know that I have to use gradient ascent for this approach, but I have trouble figuring out how to find the gradient of y w.r.t. the input while still keeping the optimized parameter values of the model.</p>
Is Pytorch REINFORCE implementation correct?

cruzas: <p>Hi everyone,</p>
<p>Perhaps I am very much misunderstanding some of the semantics of <code>loss.backward()</code> and <code>optimizer.step()</code>. In the <a href="https://github.com/pytorch/examples/blob/59caa16a29476f290d2e4d38162a89372e5ccb15/reinforcement_learning/reinforce.py#L62" rel="nofollow noopener">Pytorch example implementation of the REINFORCE algorithm</a>, we have the following excerpt from the <code>finish_episode()</code> function</p>
<pre><code class="lang-auto">for log_prob, R in zip(policy.saved_log_probs, returns):
    policy_loss.append(-log_prob * R)
optimizer.zero_grad()
policy_loss = torch.cat(policy_loss).sum()
policy_loss.backward()
optimizer.step()
</code></pre>
<p>In the REINFORCE algorithm described in the Reinforcement Learning book by Richard S. Sutton and Andrew G. Barto, the reinforce update for the parameter vector, <code>\theta</code>, is done by</p>
<pre><code class="lang-auto">\theta_{t+1} = \theta_t + \alpha G_t (\nabla\pi(A_t|S_t,\theta_t) / \pi(A_t|S_t,\theta_t)),
</code></pre>
<p>i.e., the parameter vector is updated in <em>every</em> step through gradient ascent.</p>
<p>Maybe I am wrong, but <code>policy_loss.backward()</code> appears to compute the gradient of all the arguments in the loss with respect to a <em>single</em> parameter vector <code>\theta</code>, and then <code>optimizer.step()</code> essentially adds these gradients assuming that <code>\theta_t</code> is the same for all t values, which does not seem to be equivalent to the theoretical implementation.</p>
<p>Is there something that I am missing or not seeing clearly here?</p>
<p>Thank you for your help.</p>

iffiX: <p>Actually in real applications, whether to update your actor&amp;critic per step or update them after collecting several episodes does not quite affect the performance of your algorithm.</p>
<p>However, it could be very inefficient to compute backward pass per step, as you cannot fill up your cpu ALU units or CUDA cores on your gpu, the computation price would be way too expensive. Therefore, batching is an economical choice., although it does not conform to the “theoretical model” of the REINFORCE algorithm.</p>
The right way to write submodule and access attribute from submodule

udo: <p>I have to compose a Parent Module with a  child sub module, for OO compatibility, I have to let the Parent object access child properties directly. Below is a reproducible example where I comment the output inside the code.</p>
<p>I have two things to ask:</p>
<ol>
<li>
<p>I found that it seems simply defining submodules inside a module’s initializer also works, by “works” i mean both the parent module and the submodule all get trained.  So my first question is simplying adding Submodule as a normal member is equivalent compared to<br>
<code>self.add_module("child", SubModule())</code> in terms of training?  In this post, it is not explained: <a href="https://discuss.pytorch.org/t/when-to-use-add-module-function/10534/2" class="inline-onebox">When to use add_module function?</a></p>
</li>
<li>
<p>I tried the following code, but it all failed to access child module’s attribute from the parent object (failure message in comment below ), what is the right way to do that?</p>
</li>
</ol>
<pre><code class="lang-auto">import torch.nn as nn

class SubModule(nn.Module):
    def __init__(self):
        super(SubModule, self).__init__()
        self.h = 3

class Parent(nn.Module):
    def __init__(self):
        super(Parent, self).__init__()
        self.child = SubModule()
        self.m = 1

class ParentControl(nn.Module):
    def __init__(self):
        super(ParentControl, self).__init__()
        self.child = SubModule()
        self.m = 1

    def __getattr__(self, name):
        try:
            return self.__dict__[name]
        except KeyError:
            child = self.child
            return child.__getattr__(name)
    


class ParentControl2(nn.Module):
    def __init__(self):
        super(ParentControl2, self).__init__()
        self.child = SubModule()
        self.m = 1

    def __getattr__(self, name):
        try:
            return self.__getattr___[name]
        except AttributeError:
            child = self.child
            return child.__getattr__(name)

pp = Parent()
"m" in pp.__dict__.keys()  #True
"child" in pp.__dict__.keys()  # False
pp.child  # SubModule()
pp.m # 1
pp.h # AttributeError: 'Parent' object has no attribute 'h'

pp1 = ParentControl()                                                                                                    
pp1.child  # RecursionError: maximum recursion depth exceeded while calling a Python object


pp2 = ParentControl2()
pp2.__getattr__("child") # RecursionError: maximum recursion depth exceeded while calling a Python object
pp2.h  # RecursionError: maximum recursion depth exceeded while calling a Python object
</code></pre>
How to add new Relu & fc layer in pretrained Resnet-152 model

aatiibutt: <p>Hy! I have loaded Resnet-152 pre-trained model, and modified the out features of the fc layer through following model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/130d44e364df81e0fd8ea1e29289e6f4dc20e694.png" data-download-href="https://discuss.pytorch.org/uploads/default/130d44e364df81e0fd8ea1e29289e6f4dc20e694" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/130d44e364df81e0fd8ea1e29289e6f4dc20e694.png" alt="image" data-base62-sha1="2Ixv3eTwIZXhkKKB2QGaTVE1KrW" width="690" height="85" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1024×127 3.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de4d2b3279a3f46add87da1411d2ca440d642b6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/de4d2b3279a3f46add87da1411d2ca440d642b6c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de4d2b3279a3f46add87da1411d2ca440d642b6c.png" alt="image" data-base62-sha1="vIzkz3tXEcG5e3bqncYOWmANWoY" width="690" height="229" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1117×372 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now i want to apply relu on the modified fc layer, and then add a new fc layer which take out_features of previous fc layer, and out_feature = 10, like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e31bf70ed299967ea429363bb1ee16834e0671d2.png" alt="image" data-base62-sha1="wp6iDNY4k31oy6wLda4kGt9vFzY" width="658" height="34"></p>
<p>Any Help?</p>

ptrblck: <p>You could assign an <code>nn.Sequential</code> container to <code>model.fc</code> instead of a single linear layer via:</p>
<pre><code class="lang-python">model.fc = nn.Sequential(
    nn.Linear(num_ftrs, 1024),
    nn.ReLU(),
    nn.Linear(1024, 10)
)
</code></pre>

aatiibutt: <p>Thanks Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> ! It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Backward of self-defined loss function is very slow

yf_zhang: <p>I define a loss function by myself, but the duration of loss.backward() is very long.<br>
The code is pasted below. The training speed is apprently slower than tensorflow implementation.Is the reason that I define the loss function in not proper method.</p>
<pre><code class="lang-auto">        def func(logits, target):
            log_likelihood = -F.log_softmax(logits, dim=1)
            batch = logits.shape[0]
            loss = torch.sum(torch.mul(log_likelihood, target)) / batch
            return loss
</code></pre>

albanD: <p>Hi,</p>
<p>This code should be as efficient as it can be.<br>
How do you measure the runtime?</p>

yf_zhang: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="85498">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>This code should be as efficient as it can be.</p>
</blockquote>
</aside>
<p>Oh, I was wrong. The loss function is not reason of slow training speed.</p>
Load Modified Resnet model from github

aatiibutt: <p>I modified the fc layer of resnet model available on github. Now i want to load it into my notebook. But i don’t know about how to load modified model from github and use in notebook.<br>
Any Help?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bcd6b8c4474c60aa953e05033d34657dbc54b370.png" data-download-href="https://discuss.pytorch.org/uploads/default/bcd6b8c4474c60aa953e05033d34657dbc54b370" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bcd6b8c4474c60aa953e05033d34657dbc54b370.png" alt="image" data-base62-sha1="qWxNPh05VgJsUIWqmmPcW0Ecc6I" width="690" height="352" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1021×522 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I assume you’ve forked the <code>torchvision</code> repository and modified the resnet in your local branch?<br>
If that’s the case, you could uninstall the official <code>torchvision</code> package, <code>git clone</code> your repository, and install it from source.<br>
This would use your changes including your modified resnet implementation.</p>
<p>Also, a simpler way would be to copy-paste the resnet definition into your notebook and just use it locally.</p>
Image segmentation with dynamic image resolution?

tueboesen: <p>I’m trying to do “image” segmentation on gene sequences, where I convert a gene sequence into an image using a onehot encoding, which results in my “image” having a dimension (CxWxH)=(1x?x25), where ? changes all the time due to it being dependent on the protein I’m currently training on.</p>
<p>Now I have heard that U-nets are generally considered good for image segmentation problems so I was thinking of such a network, but I’m running into trouble with the pooling, and these dynamic image resolutions, and I’m wondering what people normally do about that issue?</p>

ptrblck: <p>Often the UNet implementation might be a bit flexible when it comes to different input shapes.<br>
Usually resolutions using powers of 2 work the best, as the conv and pooling kernels are often adapted to these shapes.<br>
However, if your input shapes vary a lot, you could try to use adaptive pooling layers, which will yield a specified output shape, or manually pad/crop the activations to the desired shapes.</p>
Shifting tensor over dimension

Ameen_Ali: <p>Hello,<br>
Lets say I have a tensor of A of shape [batch_size X length X 1024]<br>
I want to do the following :<br>
for the i element in the batch i want to shift the (length) elements embedding by their position .<br>
for example the vector A[0 , 0 , : ] should stay the same, and A[0 , 1 , :] should be shifted (or rolled) by 1 , and A[0 , 15 , :] should be shifted by 15.<br>
this is for all the elements in the batch.<br>
so far i did it with for loops, but its not efficient ,<br>
thanks</p>

ptrblck: <p>Could you post your current approach with loops, as I’m not completely understanding the use case.<br>
If you shift <code>A[0, 1]</code> by 1 in dim1, would <code>A[0, 2]</code> be overwritten or also shifted?</p>

Ameen_Ali: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<pre><code class="lang-auto">A = .. # A is of shape [batch_size x len , embedding]
target_embeddings = []
len = A.shape[1]
for i in range(batch_size):
 for index in range(len):
  tmp = A[i , index , :]
  target_embeddings.append(torch.roll(tmp , index , 0))

#do torch.stack thing to get a tensor like object back
</code></pre>
How can I profile the Backward pass of model?

Navdeep_Kumar: <p>I need to profile the backward pass of a model running on a GPU. I need to see how much time each layer’s gradient computation took along with achived TFLOPs during the operation. The problem is, If I use a profiler such as nsight systems then I cannot simply differentiate which kernel ran for which layer just because I cannot annotate the backward pass using nvtx. Is there some way in which the backward pass can be profiled.</p>

ptrblck: <p><a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.profiler.profile"><code>autograd.profiler</code></a> should give your the runtime for the backward functions. If you spot a bottleneck, you could run nsight systems in isolation on this particular <code>backward</code> call.</p>

torch_fresh: <p>Hi, I have a same issue for profiling backward pass of each layer. Can you give me some hints for solving this problem? Thanks for any code or suggestion.</p>

ptrblck: <p>If you just want to profile the backward layer and get the current runtime, this code snippet might be helpful:</p>
<pre><code class="lang-python">def profile(module, input):
    # Warmup
    for _ in range(50):
        output = module(input)

    g0 = torch.rand_like(output)
    for _ in range(50):
        output = module(input)
        output.backward(g0)


    nb_iters = 100
    torch.cuda.synchronize()
    start = time.time()
    for _ in range(nb_iters):
        output = module(input)
    torch.cuda.synchronize()
    end = time.time()
    fwd_time = (end - start) / nb_iters

    # Profile backward pass
    torch.cuda.synchronize()
    start = time.time()
    for _ in range(nb_iters):
        output = module(input)
        module.weight.grad = None
        output.backward(g0)

    torch.cuda.synchronize()
    end = time.time()
    all_time = (end - start) / nb_iters
    bwd_time = all_time - fwd_time
</code></pre>
Train and validation split on IterableDataset

Frederik_Semmel: <p>What are the best options to split an IterableDataset into a train and validation set?</p>
<p>I am using an IterableDataset because the data is stored in multiple tfrecord files, which are easier or faster to read sequentially with generators.</p>
<p>The options I see are:</p>
<ul>
<li>Split the data files (tfrecords) into training files and validation files.<br>
This requires a lot of scripting and extra disk space.</li>
<li>Keep the IterableDataset the same, and define which batches should be used for validation, e.g. batch 1000-1100, 2000-2100, … is validation and the rest is training.<br>
With this approach it is very difficult to have always the same validation step, as the IterableDataset is shuffled (with a buffer) and loaded with multiple workers.</li>
</ul>
<p>Both don’t sound like a good solution. Has anyone solved a similar problem?</p>
<p>Thanks!</p>
Hyperparameter tuning in GANs

OzanGenc: <p>Hi all,</p>
<p>I am using GANs for generating synthetic MR images. I want to optimize some hyperparameters such as learning rate and batch size. I heard that performance of GANs are highly affected by hyperparameters. I think to optimize these hyperparameters using Bayesian Optimization in order to obtain best possible hyperparameter values in a relatively shorter time than using random or grid search. However, I couldn’t find an example using Bayesian Optimization for hyperparameter tuning for GANs. Do you think this is a bad idea? Which other methods do you use for hyperparameter optimization for GANs?</p>
<p>Thanks.</p>
About update generator when training GAN

dmkr001: <p>I am the beginner of Pytorch. And now I want to train a GAN. I have checked many different implement of GAN. I found that when update the discriminator will use the detach function to avoid update the generator. But why when update the generator not use the detach? Example: <a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/context_encoder/context_encoder.py" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/context_encoder/context_encoder.py</a></p>

ptrblck: <p>When you update the discriminator with the fake sample, you are training the discriminator to detect this fake sample as a fake sample. The calculated loss should not create gradients for the generator, as these gradients would update the generator in such a way that the next generated fake samples would be easier to detect as such.</p>
<p><code>.detach()</code> will detach the <code>fake</code> output and will stop Autograd from computing the gradients at this point.</p>
<p>On the other hand, when you are training the generator, you are using the fake output of the generator and pass the “real targets” to the discriminator in order to calculate the gradients for the generator.<br>
The update step will thus update the generator in such a way that the next generated fake output should be more likely to be classified as “real” by the discriminator.</p>
SourceChangeWarning: source code of class

drdlfy: <p>I updated the pytorch version, a lot of this warning, how to solve it? Thank you for your help</p>
<p>SourceChangeWarning: source code of class ‘torch.nn.modules.batchnorm.BatchNorm2d’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set <code>torch.nn.Module.dump_patches = True</code> and use the patch tool to revert the changes.</p>

smth: <p>you can ignore these warnings.</p>

drdlfy: <p>Okay, thank you very much</p>

jike_luo: <p>If I ignore these warnings, can I run successfully?</p>
Please consider padding the input to a length of

Ernst_Oberortner: <p>Hi, I’m using a BucketIterator with a batch size of 64 to iterate over my training and test dataset.</p>
<blockquote>
<p>train_iterator, test_iterator = BucketIterator.splits(<br>
(train_data, test_data),<br>
batch_size=64,<br>
repeat=False,<br>
sort=True, sort_key=lambda x: len(x.prot_seq),<br>
device=device)</p>
</blockquote>
<p>The datasets are not a multiple of 64 though.<br>
When training, I’m getting the following error message:</p>
<blockquote>
<p>If training, sequence Length 35 has to be a multiple of least common multiple chunk_length 64. Please consider padding the input to a length of 64</p>
</blockquote>
<p>Shouldn’t the BucketIterator take care of always giving batches of 64 sequences?<br>
Or is this somehow configurable? The docs are not so helpful in this regard.</p>
<p>Btw. I’m using a Reformer model</p>
<blockquote>
<p>from transformers import ReformerConfig, ReformerModelWithLMHead</p>
</blockquote>
<p>Thanks for your help in advance!</p>
Subset object has no attribute data

patrick126: <p>Hi everyone, I’m new to pytorch.<br>
I’m trying to split the original MNIST test set into a validation and a test set of the same size.<br>
When i try</p>
<pre><code class="lang-auto">dataset_train = torchvision.datasets.MNIST("./mnist", train=True, download=True)
valid_and_test_set = torchvision.datasets.MNIST("./mnist", train=False, download=True)
dataset_valid, dataset_test = torch.utils.data.random_split(valid_and_test_set,[5000, 5000])

mu = dataset_train.data.float().mean()
mu_valid = dataset_valid.data.float().mean()
</code></pre>
<p>On the last line I get the error</p>
<pre><code class="lang-auto">AttributeError: 'Subset' object has no attribute 'data'
</code></pre>
<p>How can I handle this?<br>
Thank you in advance.</p>

harsha_g: <aside class="quote no-group" data-username="patrick126" data-post="1" data-topic="85574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/838e76/48.png" class="avatar"> patrick126:</div>
<blockquote>
<pre><code class="lang-auto">dataset_train = torchvision.datasets.MNIST("./mnist", train=True, download=True)
valid_and_test_set = torchvision.datasets.MNIST("./mnist", train=False, download=True)
dataset_valid, dataset_test = torch.utils.data.random_split(valid_and_test_set,[5000, 5000])
</code></pre>
</blockquote>
</aside>
<p><code>dataset_valid.dataset.data.float().mean()</code> should do.</p>
Dynamic + Post-training static quantization at the same time

felicitywang: <p>Hi,<br>
I have a CNN+linear model that I want to quantize. To make the most of PyTorch’s quantization, is there any way to perform static quantization on the CNN and dynamic on linear?</p>
<p>Thank you.</p>

jerryzh168: <aside class="quote no-group" data-username="felicitywang" data-post="1" data-topic="85469">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/919ad9/48.png" class="avatar"> felicitywang:</div>
<blockquote>
<p>Thank you.</p>
</blockquote>
</aside>
<p>if you are doing static quantization, why don’t do static quantization on both conv and linear?</p>
<p>If you really need to do this, you can first apply static quantization, and then dynamic quantization.</p>
Different outputs even calling model.eval()

ptorres: <p>Hello, when doing prediction with a saved model the output is different when i run the program again.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">		image_transforms = {
			'predict': transforms.Compose([
				transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),
				transforms.CenterCrop(size=224), 
				transforms.ToTensor(),
				transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) 
				]),
		}

		dataset = {'predict' : ImageFolderWithPaths('./det-results', image_transforms['predict'])}
		dataloader = {'predict': DataLoader(dataset['predict'], batch_size = 16, shuffle=False)}

		model.load_state_dict(torch.load(opt.save_file_name))

		with torch.no_grad():
			model.cuda()
			model.eval()

			for inputs, _, paths in dataloader['predict']:
				inputs = inputs.cuda()
				output = model(inputs)
				output = output.cuda()

				_, index = torch.max(output, dim=1)
				print(index)
</code></pre>
<p>Here is the output from the first time i run:</p>
<pre><code class="lang-auto">tensor([1, 2, 2, 2, 2, 0, 3, 3, 3, 3, 1, 2, 2, 2, 2, 2], device='cuda:0')
tensor([2, 2, 1, 1, 1, 1, 0, 1, 0, 3, 2, 3, 1, 2, 2, 2], device='cuda:0')
tensor([1, 1, 3, 2, 1, 1, 1, 1, 1, 1, 1, 3, **2**, 1, 2, 3], device='cuda:0')
tensor([0, 1, 0, 1, 1, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2], device='cuda:0')
</code></pre>
<p>and here for second:</p>
<pre><code class="lang-auto">tensor([1, 2, 2, 2, 2, 0, 3, 3, 3, 3, 1, 2, 2, 2, 2, 2], device='cuda:0')
tensor([2, 2, 1, 1, 1, 1, 0, 1, 0, 3, 2, 3, 1, 2, 2, 2], device='cuda:0')
tensor([1, 1, 3, 2, 1, 1, 1, 1, 1, 1, 1, 3, **1**, 1, 2, 3], device='cuda:0')
tensor([0, 1, 0, 1, 1, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2], device='cuda:0')
</code></pre>
<p>I put double ** in the output that is different (third line), each line corresponds a one batch (size=16)</p>
<p>Anyone can guide me and show what i’m doing wrong?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Nikronic: <p>Hi,</p>
<aside class="quote no-group" data-username="ptorres" data-post="1" data-topic="85568">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/3be4f8/48.png" class="avatar"> ptorres:</div>
<blockquote>
<p><code>transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0))</code></p>
</blockquote>
</aside>
<p>You are cropping image <em>randomly</em> so for every image in each batch, you get different image rather than original image. Although it should not affect model too much as we can see only one difference is happening.</p>
<p>Let me know about the output by removing this line.</p>
<p>Bests</p>

ptorres: <p>I let this detail pass by totally. After remove, the outputs are the same.</p>
<p>Thank you!</p>