akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to increase performance for custom CNN using torchscript

Midhilesh: <p>Hello,<br>
I tried to visualize the runtime performance improvement made by convolution layer which I implemented from scratch Vs torchscript version of convolution layer Vs torch.nn.conv2d() module for 100 iterations with input (128,3,28,28), out_channel =64, kernel size=3.</p>
<p>Convolution layer from scratch in CUDA -&gt; 9.366 seconds<br>
torchscript convolution layer from scratch in CUDA -&gt; 6.636 seconds<br>
torch.nn.conv2d() -&gt; 475.614 milliseconds.</p>
<p><strong>Is there any problem in my approach? and how to optimize even more?</strong> . I request you to help me with this problem.</p>
<p>My code</p>
<pre><code class="lang-auto">class conv2D(nn.Module):
  def __init__(self, in_channel, out_channel, kernel_size):
    super(conv2D,self).__init__()
    self.weight = torch.nn.Parameter(torch.ones(out_channel,in_channel,kernel_size, kernel_size))
    self.bias = torch.nn.Parameter(torch.zeros(out_channel))
    self.kernel_size = kernel_size
    self.in_channel = in_channel
    self.out_channel = out_channel

  def forward(self, image):
    img_height = image.shape[3]
    img_width = image.shape[2]
    batch_size = image.shape[0]
    out_height = img_height-self.kernel_size+1
    out_width = img_width-self.kernel_size+1

    output = torch.zeros(batch_size,self.out_channel,out_width,out_height)
    for k in range(batch_size):
      for i in range(out_height):
        for j in range(out_width):
          temp = torch.sum(image[k,:,j:j+self.kernel_size,i:i+self.kernel_size]*self.weight,dim=(1,2,3))
          output[k,:,i,j]=torch.add(temp,self.bias)
      return output
</code></pre>
<pre><code class="lang-auto">x = torch.ones(128,3,28,28).to("cuda")
c = conv2D(3,64,3).to("cuda")
c_s = torch.jit.script(c).to("cuda")
c_s(x)
</code></pre>
<p>Scripting the model and running with a sample input to get an optimized graph</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True) as prof:
  with torch.no_grad():
    for i in range(100):
      c(x)
print(prof.table())
</code></pre>
<p>Profiling both the scripted and normal method.</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True) as prof:
  with torch.no_grad():
    for i in range(100):
      c_s(x)
print(prof.table())
</code></pre>

ptrblck: <p>Convolutions are not easy to optimize and I’m not sure if there is any JIT compiler in the wild, which is currently able to optimize these layers to a competitive performance (please let me know, if you find it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
Scheduler position in the training loop

Mohamed_Adel: <p>Hello everyone,</p>
<p>I’m facing a strange thing that I would like to know what is behind it.</p>
<p>in my model training loop</p>
<pre><code class="lang-auto">            if phase == 'train':
                scheduler.step()   # &lt;-------
                model.train()
                     ...
                optimizer.step()

</code></pre>
<p>when I place the  scheduler before the optimizer I get an error</p>
<blockquote>
<p>UserWarning: Detected call of <code>lr_scheduler.step()</code> before <code>optimizer.step()</code>. In PyTorch 1.1.0 and later, you should call them in the opposite order: <code>optimizer.step()</code> before <code>lr_scheduler.step()</code>.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.</p>
</blockquote>
<p>but the error rate is much better which means the scheduler is working fine,</p>
<p>when I fix it and put the scheduler behind the optimizer as given in the error message, the error is far bigger and it doesn’t decrease.</p>
<pre><code class="lang-auto">            if phase == 'train':
                model.train()
                     ...
                optimizer.step()
                scheduler.step()   # &lt;-------

</code></pre>
<p>Is there is an explanation for why following the error message recommendation is bad for the training?</p>

ptrblck: <p>I guess the initial learning rate of the optimizer is too high and you might lower it.<br>
The warning is raised, since it’s a new design decision to call the scheduler after the first optimization, such that the first step will use the user-defined learning rate from the optimizer.</p>
Uniform initialization of complete resnet network

ML0401: <p>Hello folks,</p>
<p>I am a little bit confused about the weight initializations. As I am training my network (resnet18) from scratch, I like to initialize the weights by an uniform distribution, with the following function:</p>
<pre><code class="lang-auto">def init_weights(m):
    if isinstance(m, nn.Conv2d):
        nn.init.kaiming_uniform_(m.weight, nonlinearity='relu')
    elif isinstance(m, nn.Linear):
        nn.init.kaiming_uniform_(m.weight, nonlinearity='relu')
    elif isinstance(m, nn.BatchNorm2d):
        nn.init.kaiming_uniform_(m.weight, nonlinearity='relu')
    elif isinstance(m, nn.MaxPool2d):
        nn.init.kaiming_uniform_(m.weight, nonlinearity='relu')
    elif isinstance(m, nn.AdaptiveAvgPool2d):
        nn.init.kaiming_uniform_(m.weight, nonlinearity='relu')
</code></pre>
<p>Do I need to initialize every layer like this or is there an easier way? And is it necessary to zero the bias? When I do it with the function above, I get a VaulueError. Am I missing something?</p>
<p>ValueError: Fan in and fan out can not be computed for tensor with fewer than 2 dimensions</p>
<p>EDIT: Obviously the BatchNorm2d layers have problems and can not be computed with this initialization. Do I have to do another initialization for these kind of layers? What about the pooling layers, do I need to initialize them at all?</p>

ptrblck: <p>Some layers such as batchnorms and the <code>bias</code> parameter of other layers cannot be used in init methods, which require a multi-dimensional parameter.</p>
<p>The <code>weight</code> of <code>nn.BatchNorm2d</code> is by default now initialized with ones, while the bias with zeros, which seems to be the recommended way now.</p>
<p>Pooling layers don’t have any parameters, so you don’t need to initialize them.</p>
How much we can speedup this KNN operation with C++ API in CPU?

zuoxingdong: <p>I have the following KNN operation, it takes a 3D tensor with shape (B, N, D) as the input, where B is the batch size, N is the number of points, and D is the dimension of each point. What it does is basically computing the pairwise distance across the third dimension and returns a 4D tensor consisting of k-nearest neighbors. The entire operation should be differentiable to enable autograd.</p>
<p>A code snippet is attached as follows:</p>
<pre><code class="lang-python">def knn(x, k):
    B, N, D = x.shape

    cdist = torch.cdist(x, x)
    _, indices = cdist.topk(k, dim=-1, largest=False, sorted=False)

    indices = indices + N * torch.arange(B).view(-1, 1, 1).expand_as(indices)
    x = x.view(B * N, D)[indices]  # shape: [B, N, k, D]
    return x
</code></pre>
<p>I am wondering what can be the efficiency of this implementation. How much speedup we may expect if I re-implement this KNN operation at a lower level with C++ API?</p>

ptrblck: <p>I don’t think you will see a significant speedup, if you use <code>libtorch</code>, but please update this thread in case you decide to try it out.<br>
Since you are mostly using PyTorch methods in your code, there shouldn’t be a lot of Python overhead.</p>
Why do I get a segmentation violation (SIGSEGV) when calling a forward pass?

sophia_k: <p>Hi there,</p>
<p>I have been getting a segmentation violation error, also called SIGSEGV when calling a forward pass in my graph convolutional neural network:</p>
<pre><code class="lang-auto">Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
</code></pre>
<p>This error comes after I have created an instance of my graph convolutional neural network and am calling the forward method on it for the first time, thereby passing the data.</p>
<p>This is part of my script:</p>
<pre><code class="lang-auto"># create model
model: LinkPredictHetero = LinkPredictHetero(
    g=deephet.graph,
    h_dim=h_dim,
    out_dim=h_out,
    num_hidden_layers=n_hidden_layers,
)
    
# call forward pass
embed = model(deephet.graph)
</code></pre>
<p>The <code>graph</code> is an instance of the <a href="https://docs.dgl.ai/en/0.4.x/generated/dgl.heterograph.html" rel="nofollow noopener">dgl.heterograph</a>.</p>
<p>The <code>forward()</code>method of the model links to the forward method of the base layer class, defined as follows:</p>
<pre><code class="lang-auto">def forward(self, graph):
    graph = graph.local_var()
    hs = {}
    hs = self.embed_layer(graph)
    for layer in self.layers:
        hs = layer(g, inputs = hs)
    return hs
</code></pre>
<p>Related to this forward method,<br>
<code>self.embed_layer</code> is a <code>torch.nn.ModuleDict()</code>:</p>
<pre><code class="lang-auto">self.embed_layer = torch.nn.ModuleDict()
for node_type in self.graph.node_types:
    in_size = list(self.graph.nodes[node_type].data['feature'].size())[1]
    self.embed_layer[node_type] = torch.nn.Linear(in_size, self.embed_size)
</code></pre>
<p>and <code>self.layers</code> is a <code>torch.nn.ModuleList()</code> to which different layer implementations are appended.</p>
<p>So when in the script I call <code>model(deephet.graph)</code>, which should call the <code>forward()</code> method of the layer, the segmentation violation ends my program, and actually also terminates python.</p>
<p>Could you please help me to find out why this happens? I would really appreciate it.<br>
Thank you, dear community.</p>

111368: <p>What is your pytorch version?</p>

sophia_k: <p>I’ve got this version: torch==1.5.0</p>
How to build a net like this structure in Pytorch?

skyraker: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74a22c07cf93fa4df81e20a50fea240a8204ab0b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/74a22c07cf93fa4df81e20a50fea240a8204ab0b" title="netstructure"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74a22c07cf93fa4df81e20a50fea240a8204ab0b_2_690x453.jpeg" alt="netstructure" data-base62-sha1="gDMPZ4d3hVJ04ARK76516XZDVAT" width="690" height="453" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74a22c07cf93fa4df81e20a50fea240a8204ab0b_2_690x453.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74a22c07cf93fa4df81e20a50fea240a8204ab0b.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74a22c07cf93fa4df81e20a50fea240a8204ab0b.jpeg 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">netstructure</span><span class="informations">736×484 23.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I want to build a network like this structure, but I just can build unidirectional net. How to deal with this complex item? <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>

skyraker: <p>RB has been defined.</p>

Hmrishav_Bandyopadhy: <p>You can define a module in this structure</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc0=nn.Linear(...)
        self.fc1=nn.Linear(...)
        self.fc2=nn.Linear(...)
        self.fc3=nn.Linear(...)
        self.Rb0= ...
        self.Rb1= ...
        self.Rb2= ...
        self.Rb3= ...

   def forward(self,x):
        x_1=self.fc0(x)
        x_2=self.fc1(x_1)
        x_3=self.fc2(x_2)
        x_4=self.fc3(x_3)
        
        x_out_1=self.Rb0(x_1)
        x_out_2=self.Rb1(x_2)
        x_out_3=self.Rb2(x_3)
        x_out_4=self.Rb3(x_4)
        ......
        return final_block
</code></pre>
<p>Later on you can call this module like this:</p>
<pre><code class="lang-auto">model=Net()
</code></pre>
<p>Does that solve your problem?</p>

skyraker: <p>Got it. Thx . <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
How can I exclude some parameters in optimizer during training?

LeeDoYup: <p>Hello.<br>
I want to exclude some parameters in the optimizer <strong>during training</strong>.</p>
<p>Specifically, i want to freeze the updates of some layers during training.<br>
(however, requires_grad=False is not working.)</p>
<p>Before starting the training, i construct an optimizer</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(net.parameters(), lr, momentum=momentum, weight_decay=decay, nesterov=True)

scheduler = torch.optim.lr_scheduler.LambdaLR(
    optimizer,
    lr_lambda=lambda step: cosine_annealing(
        step,
        schedule_epoch * len(train_loader),
        1,  # since lr_lambda computes multiplicative factor
        1e-6 / 0.1))

for epoch in range(total_epoch):
  ...
  # update the models
</code></pre>
<p>In this scenario. i want to freeze some layers (exactly, front layers) after some epochs.<br>
How can i exclude the updated parameters in above optimizer?</p>
<p>Thank you.</p>

Hmrishav_Bandyopadhy: <p>How are you freezing the layers? Freezing is the only way in which you can exclude parameters during training. In your example I see that you have defined your optimizer as checking out all params. While freezing, this is the way to set up your optimizer:</p>
<pre><code class="lang-auto"> optim = torch.optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr, momentum=momentum, weight_decay=decay, nesterov=True) 
</code></pre>
<p>The filter doesn’t offer so much of a change in a simple optimizer with a learning rate but since you are using momentum and weight decay, the params whose <code>requires_grad</code> is set to False will be updating on the basis of these.</p>

LeeDoYup: <p>If i use the lambda function above,<br>
do not i need to exclude manually the freezing params “during training”?</p>
<p>Or, after i change some parameters’ <code>requires_grad=False</code>,<br>
do i construct above optimizer again?</p>

Hmrishav_Bandyopadhy: <p>You freeze the parameters manually <i>before training</i>. For example, if i want to freeze the first 3 layers of a Resnet encoder, I use–</p>
<pre><code class="lang-auto">for i,child in enumerate(model.children()):
   for k,child_0 in enumerate(child.children()):
      if k&lt;=3:
         for params in child_0.parameters():
            params.requires_grad=False
               print("Frozen {} layer {}".format(k,child_0))
   break
</code></pre>
<p>This is a small snippet which i built based on my encoder. You have to come up with your own freezing loop to freeze custom layers. Note that all this happens before training begins.</p>
<p>A very good idea would be to put it just after you have defined the model. After this, you define the optimizer as</p>
<pre><code class="lang-auto">optim = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr, momentum=momentum, weight_decay=decay, nesterov=True) 
</code></pre>
<p>and you are good to go ! You can use this model in the training loop and the frozen layers will not have their weights updated.</p>
Error about custom build. When runing quanted model on android

KevinZ1992: <p>My quanted model “quanted.pt” runing well on android without custom build.<br>
Use torch.jit.export_opnames() to get  SELECTED_OP_LIST,but there is no Relu op such as<br>
“- aten::relu”.Try Relu6 meet the same question.</p>
<p>torch version:1.5.0<br>
Here is the RuntimeError:</p>
<pre><code class="lang-auto">java.lang.RuntimeException: The following operation failed in the TorchScript interpreter.
    Traceback of TorchScript, serialized code (most recent call last):
      File "code/__torch__/torch/nn/quantized/functional.py", line 9, in forward
        pass
      if inplace:
        _1 = torch.relu_(input)
             ~~~~~~~~~~~ &lt;--- HERE
      else:
        _1 = torch.relu(input)
   
    Traceback of TorchScript, original code (most recent call last):
      File "###/python3.7/site-packages/torch/nn/quantized/functional.py", line 322, in forward
            raise ValueError("Input to 'quantized.relu' must be quantized!")
        if inplace:
            return torch.relu_(input)
                   ~~~~~~~~~~~ &lt;--- HERE
        else:
            return torch.relu(input)
    RuntimeError: Operator has been stripped in the custom build.
</code></pre>
<p>Someone can help,thank you!</p>

KevinZ1992: <p>i changed the relu inplace=False，it will work</p>
How to save wrong prediction results of a CNN on MNIST

copyrightly: <p>I am using <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" rel="nofollow noopener">this code</a> to do experiments on MNIST. I am wondering how to save the images receiving wrong predictions and the wrong predicted results (like a 7 predicted as 1). Thank you!</p>

ptrblck: <p>You could adapt the test code to check all wrong predictions and store the passed images as shown here:</p>
<pre><code class="lang-python">import torchvision.transforms.functional as TF

def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += F.nll_loss(output, target, reduction='sum').item()  # sum up batch loss
            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

            # Store wrongly predicted images
            wrong_idx = (pred != target.view_as(pred)).nonzero()[:, 0]
            wrong_samples = data[wrong_idx]
            wrong_preds = pred[wrong_idx]
            actual_preds = target.view_as(pred)[wrong_idx]

            for i in range(wrong_idx):
                sample = wrong_samples[i]
                wrong_pred = wrong_preds[i]
                actual_pred = actual_preds[i]
                # Undo normalization
                sample = sample * 0.3081
                sample = sample + 0.1307
                sample = sample * 255.
                sample = sample.byte()
                img = TF.to_pil_image(sample)
                img.save('wrong_idx{}_pred{}_actual{}.png'.format(
                    wrong_idx[i], wrong_pred.item(), actual_pred.item()))
</code></pre>

copyrightly: <p>Thank you! Can I ask a further question, what if I want to save the indices of the wrong predicted samples in the whole test loader. Actually I want to compare the performance of two networks, i.e. which samples predicted wrong by one network are correctly predicted by the other network.</p>

ptrblck: <p>In that case you could write a custom <code>Dataset</code>, deriving from the <code>MNIST</code> dataset, and return the sample index with the actual data and target sample.<br>
In the provided code snippet, you would get the additional class index and could store it for further processing.</p>

rishabhramteke: <p>I am getting an error in the line for i in range(wrong_idx):</p>
<p>It says : TypeError: only integer scalar arrays can be converted to a scalar index</p>

ptrblck: <p>Try to use <code>for i in range(len(wrong_idx)):</code> instead.</p>
Why OpenCL is missing in PyTorch?

blackbirdbarber: <blockquote>
<p>Implementation of data structures and kernels on other GPU technologies, such as<br>
OpenCL, are not planned at the time of this writing.</p>
</blockquote>
<p>Form the book “Deep Learning with PyTorch” page 63,</p>
<p>Can someone clarify what would be the gain to support OpenCL?</p>
What does “export CUDA_VISIBLE_DEVICES=1” really do?

111357: <p>In a multi-GPU computer(Ubuntu 16),<br>
I want to use <code>GPU1</code> and do the following setting in the shell:</p>
<pre><code class="lang-auto">$ export CUDA_VISIBLE_DEVICES=1
$ python
</code></pre>
<p>But, I got an error when I set the device of the tensor:</p>
<pre><code class="lang-auto">Python 3.8.3 (default, May 19 2020, 18:47:26) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.Tensor(1).to("cuda:0")
&gt;&gt;&gt; a = torch.Tensor(1).to("cuda:1")
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: invalid device ordinal
</code></pre>
<p>Why I can’t specify the device of Tensor to <code>GPU1</code> by <code>to</code>.</p>

111357: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Could you please give me some advice? <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>

ebarsoum: <p>Setting CUDA_VISIBLE_DEVICES=1 mean your script will only see one GPU which is GPU1. However, inside your script it will be cuda:0 and not cuda:1. Because it only see one GPU and its index start at 0.</p>
<p>For example if you do: CUDA_VISIBLE_DEVICES=2,4,5, your script will see 3 GPUs with index 0, 1 and 2.</p>

111357: <p>Got it, thanks a lot!</p>
Load and freeze one model and train others

111357: <p>I have a model A that including three submodels model1, model2, model3.<br>
the model flow: model1 --&gt; model2 --&gt; model3<br>
I have trained model1 in an independent project.<br>
The question is how to use the pre-trained model1 when training the model A?</p>
<p>Now, I try to implement this as follow:<br>
I load the checkpoint of model1 by <code>model1.load_state_dict(torch.load(model1.pth))</code> and then set the <code>requires_grad</code> of the model1’s parameters as False?<br>
Is it right?</p>

Usama_Hasan: <p>That would do it as far as the freezing and loading model1 is concerned.</p>

111357: <p>got it, thanks a lot.</p>
Toutorial for ignite

Aman_Gupta: <p>Can anyone suggest me good toutorial for Pytorch ignite.<br>
I searched but can’t find anything.</p>

vfdev-5: <p><a class="mention" href="/u/aman_gupta">@Aman_Gupta</a> how about</p>
<ul>
<li><a href="https://pytorch.org/ignite/quickstart.html" rel="nofollow noopener">https://pytorch.org/ignite/quickstart.html</a></li>
<li><a href="https://pytorch.org/ignite/concepts.html" rel="nofollow noopener">https://pytorch.org/ignite/concepts.html</a></li>
</ul>
<p>We have also various tutorials with Jupyter Notebooks:</p>
<ul>
<li><a href="https://pytorch.org/ignite/examples.html#notebooks" rel="nofollow noopener">https://pytorch.org/ignite/examples.html#notebooks</a></li>
</ul>
<p>Let me know if it does not help you to start. Thanks</p>

Aman_Gupta: <p>Thanks for sharing<br>
I think jupyter notebook may help</p>
How to ignore and initialize Missing key(s) in state_dict

Neabfi: <p>Hello,<br>
My saved state_dict does not contain all the layers that are in my model. How can I ignore the Missing key(s) in state_dict error and initialize the remaining weights?<br>
Thank you!</p>

tom: <p>Pass <code>strict=False</code> to <code>load_state_dict</code>.<br>
Then you can iterate <code>for n, p in model.named_parameters()</code> and do whatever you want for <code>n not in sd</code> or so.</p>
<p>Best regards</p>
<p>Thomas</p>

Neabfi: <p>It’s exactly what I was looking for, thank you so much!</p>
Pytorch distributions and the equivalent of `dbinom()` and `pbinom()` from R

krishnab: <p>I am new to pytorch and was trying to convert some code from R to Pytorch. Now in R there are functions<br>
to estimate the score or likelihood under some probability distribution. For example, if I have a Binomial distribution with 10 trials and a probability of success of 0.5, then I could evaluate the probability of getting 4 successes out of 10 trials with:</p>
<pre><code> dbinom(4, size=10, prob=0.5)  
</code></pre>
<p>If I wanted to find the probability of getting 4 or less successes in 10 trials, I would use:</p>
<pre><code> pbinom(4, size=10, prob=0.5)
</code></pre>
<p>What would the equivalent code be in Pytorch? I was thinking it would be <code>prob()</code> but I was not sure if that was dbinom or pbinom, or if there was a better way to code this? Thanks.</p>
<p>Point of fact, the actual lines of R code I am trying to replicate come from McElreath’s book,<br>
<em>Statistical Rethinking</em>, and the R code is:</p>
<pre><code class="lang-auto">p_grid &lt;- seq( from=0 , to=1 , length.out=20 ) 
likelihood &lt;- dbinom( 6 , size=9 , prob=p_grid )
</code></pre>

googlebot: <p>There is Distribution.cdf abstract method, but it is not implemented for most distributions. scipy.stats is generally better, if you’re doing interactive work and don’t need gradients (see also: torch.from_numpy(), Tensor.numpy()).</p>
<p>As for density, Binomial(total_count=n,probs=p).log_prob(x).exp() would correspond to dbinom(x,n,p), I think.</p>
Torch.cuda.device_count() with no CUDA compiled / available

vadimkantorov: <p>Would it return 0? Or would it throw? Docs don’t clarify this.</p>
<p>A question is whether I should check <code>torch.cuda.is_available()</code> prior to querying <code>torch.cuda.device_count()</code></p>
<p>Sorry for the lazy question <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> , I don’t have a CPU-only install of pytorch on hands for now to check</p>

vfdev-5: <aside class="quote no-group" data-username="vadimkantorov" data-post="1" data-topic="90328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vadimkantorov/48/365_2.png" class="avatar"> vadimkantorov:</div>
<blockquote>
<p>device_count</p>
</blockquote>
</aside>
<p>On MacOSX (without CUDA support) it is</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
False
&gt;&gt;&gt; torch.cuda.device_count()
0
</code></pre>
<p>either you can check it with <code>CUDA_VISIBLE_DEVICES=""</code>:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES="" python -c "import torch; print(torch.cuda.is_available() + torch.cuda.device_count())"
0
</code></pre>

vadimkantorov: <p>If PyTorch is compiled without CUDA at all (pytorch-cpu), would it also return 0?</p>

vfdev-5: <p>Yes, I would say so</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.5.0'
&gt;&gt;&gt; torch.cuda.is_available()
False
&gt;&gt;&gt; torch.cuda.device_count()
0
&gt;&gt;&gt; 
# conda list | grep torch
cpuonly                   1.0                           0    pytorch-nightly
pytorch                   1.5.0               py3.6_cpu_0  [cpuonly]  pytorch
</code></pre>
L2 loss in PyTorch

Shani_Gamrian: <p>Is there an implementation in PyTorch for L2 loss? could only find L1Loss.</p>

albanD: <p>Hi,</p>
<p>L2 loss is called mean square error, you can find it <a href="https://pytorch.org/docs/master/generated/torch.nn.MSELoss.html?highlight=mse%20loss#torch.nn.MSELoss">here</a>.</p>

Shani_Gamrian: <p>That’s why it felt so familiar <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thanks!</p>

Brando_Miranda: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="13684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>mean square error</p>
</blockquote>
</aside>
<p>from albanD link, going to this link saves some mouse clicks and looking around:</p>
<p><a href="https://pytorch.org/docs/master/generated/torch.nn.MSELoss.html#mseloss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.MSELoss.html#mseloss</a></p>
<pre><code class="lang-auto">MSELOSS
CLASStorch.nn.MSELoss(size_average=None, reduce=None, reduction: str = 'mean')
</code></pre>
Creating a dataloader without target values

AlJazari: <p>Hi,</p>
<p>I am trying to create a dataloader that will return batches of input data that doesn’t have target data.<br>
Here’s what I am doing:</p>
<pre><code class="lang-auto">torch_input = torch.from_numpy(x_train)
torch_target = torch.from_numpy(y_train)

ds_x = torch.utils.data.TensorDataset(torch_input)
ds_y = torch.utils.data.TensorDataset(torch_target)

train_loader = torch.utils.data.DataLoader(ds_x, batch_size=128, shuffle=False)
target_loader = torch.utils.data.DataLoader(ds_y, batch_size=128, shuffle=False)

</code></pre>
<p>I do this because my case study assumes that the train data (x_train) and their corresponding labels (y_train) do not exist on the same device.</p>
<p>However, the above <em>loaders</em> both return lists, not Tensors, and therefore the training always fails.</p>
<p>For illustration purposes: grabing a single batch and printing its type returns a list:</p>
<pre><code class="lang-auto">x= next(iter(train_loader))
print(type(x))
</code></pre>
<p>I found out that I can index x, as x[0] to access the batch as Tensors. However, I would like to know how to do this correctly (i.e., create dataloaders from x variables, without their target).</p>

Hmrishav_Bandyopadhy: <p>Hi,</p>
<p>It would be simple enough if you would create a Dataset class:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class Dset(Dataset):

    def __init__( self , x_train , transform=True):
        
        self.transform = transform
        self.x=x_train


    def __getitem__(self , index):
         if self.trainsform:
                self.x=torch.from_numpy(self.x)
         return self.x
        

    def __len__(self):
        return len(self.x)
</code></pre>
<p>And then use dataloaders like:</p>
<pre><code class="lang-auto">train = Dset(x_train)
target = Dset(y_train)
batch_size=128
train_loader = DataLoader(train, batch_size=batch_size,shuffle=False)
target_loader = DataLoader(target, batch_size=batch_size,shuffle=False)
</code></pre>
<p>While calling them in for training, try:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
       for x_ in train_loader:
           print(type(x_))
           ...
</code></pre>

AlJazari: <p>Thank you! This seems to have solved my main issue, but using the above code made the loader very slow. Actually, if I use it inside a loop:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
       for x_ in train_loader:
           print(type(x_))
</code></pre>
<p>This takes five seconds to print the first time, and then I get an error message that the RAM is crushed for using all available memory!</p>

Hmrishav_Bandyopadhy: <p>Try setting <code>num_loaders=0</code> in dataloader and also reducing batch size-- if you are having GPU issues. I see you are using torch.from_numpy as the transform. If it is slow, try to follow <a href="https://github.com/pytorch/pytorch/issues/5426" rel="nofollow noopener">this</a> thread to fasten that up.</p>
Real Time Recurrent Learning

Stefano_Francesco_Pi: <p>Hello everyone,</p>
<p>I’m new to PyTorch and I have trouble in understanding how to implement a RTRL algorithm for training a RNN on a time serie data with a single scalar as input and a single scalar as output for every time step. Can someone give me some help or maybe the link to some tutorial/post/ repos?</p>

Daniel_Salami: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/petered/uoro-demo" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/1148799?s=400&amp;v=4" class="thumbnail onebox-avatar" width="204" height="204">

<h3><a href="https://github.com/petered/uoro-demo" target="_blank" rel="nofollow noopener">petered/uoro-demo</a></h3>

<p>Contribute to petered/uoro-demo development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/Spraitazz/pytorch-RTRL" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/10031471?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/Spraitazz/pytorch-RTRL" target="_blank" rel="nofollow noopener">Spraitazz/pytorch-RTRL</a></h3>

<p>basic working implementation of Real Time Recurrent Learning ("forward propagation") in Pytorch. Incredibly slow, DO NOT USE! - Spraitazz/pytorch-RTRL</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I found these which is something similar but looks complicated to implement

Stefano_Francesco_Pi: <p>Thanks a lot. I already knew the first repo but I found it quite complicated as a starting point. The second one seems great!</p>

Daniel_Salami: <p>Well if you implement it link it here then</p>
Reset the parameters of a model

Charmander: <p>Is there any method to reset the parameters of the model?<br>
Or I have to save the state_dict of a new model and load it when I want to retrain the model?</p>

ptrblck: <p>It depends on your use case.<br>
If you need exactly the same parameters for the new model in order to recreate some experiment, I would save and reload the <code>state_dict</code> as this would probably be the easiest method.</p>
<p>However, if you just want to train from scratch using a new model, you could just instantiate a new model, which will reset all parameters by default or use a method to initialize your parameters:</p>
<pre><code class="lang-python">def weight_init(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):
        nn.init.xavier_uniform_(m.weight, gain=nn.init.calculate_gain('relu'))
        nn.init.zeros_(m.bias)

model.apply(weight_init)
</code></pre>

Brando_Miranda: <p>does this work for you?</p>
<hr>
<p>This probably works too:</p>
<hr>
<p>This is not a robust solution and wont work for anything except core torch.nn layers, but this works:</p>
<pre><code class="lang-auto">for layer in model.children():
   if hasattr(layer, 'reset_parameters'):
       layer.reset_parameters()
</code></pre>
<hr>
<p>credit: <a href="https://discuss.pytorch.org/t/reinitializing-the-weights-after-each-cross-validation-fold/11034/2" class="inline-onebox">Reinitializing the weights after each cross validation fold</a></p>
CopyBackward between devices

almeetb: <p>If there is some tensor on GPU that requires_grad but is later copied to CPU the grad_fn for the CopyBackwards would set the src_device to be GPU so during the backwards pass would this result in the tensor being copied back to the GPU?</p>
<p>If so, is there a way to disable the second copy back to GPU?</p>
<p>Thanks!</p>

albanD: <p>Yes it will.<br>
The reason is that a Tensor and its gradient are always on the same device.</p>

almeetb: <pre><code class="lang-auto">class Dummy(nn.Module):
    def __init__(self, input_shape):
        super(Dummy, self).__init__()
        self.parameter = nn.Parameter(torch.randn(*input_shape), requires_grad=True)

    def forward(self, x):
        return x + self.parameter
        

model = Dummy((1, 2, 3)).to('cuda:0')
input = torch.randn((1, 2, 3))
input = input.to('cuda:0')
output = model(input)
output = output.to('cpu')

model.to('cpu')
loss_fn = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)

loss = loss_fn(output, golden_output)

optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>I know the above example is pretty useless but since the module has no intermediate outputs, when the output + parameters are moved to cpu wouldn’t the backwards pass be able to run on CPU.</p>
<p>In this way, if this was a larger module with more parameters and gradient outputs that are moved to a different device, wouldn’t the backwards pass be able to run there?</p>
<p>From my understanding of <a href="https://discuss.pytorch.org/t/loss-backward-throws-an-error-with-multi-gpus/34098/9" class="inline-onebox">Loss.backward() throws an error with multi gpus</a> as long as the tensors are moved to the appropriate device the operations should be able to run.</p>

albanD: <p>In general, it is very complex to say if it can or cannot work.<br>
The basic assumption we make is that since the backward pass is very similar to the forward pass, running the backward where the forward happened is a good idea.<br>
And this is why all the backward pass of the op that goes from gpu -&gt; cpu is actually a function that goes from cpu -&gt; gpu.</p>

almeetb: <p>I see, thanks for the clarification!</p>
Python binding for c++

Kenzo: <p>I’m trying to write a simple function in c++ and then call in python but when I do that I’m getting the error that <code>incompatible function arguments. The following argument types are supported: 1. (ag:0 at::Tensor)  -&gt;  at::Tensor</code>. What I am doing wrong?</p>
<p>This is my function:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.
#include &lt;pybind11/pybind11.h&gt;
#include &lt;ATen/ATen.h&gt;
at::Tensor test( at::Tensor&amp; img){
    return img;
}
PYBIND11_MODULE(example, m) {
    m.doc() = "test function!";
    m.def("test", &amp;test, "CudaTorch");
#ifdef VERSION_INFO
    m.attr("__version__") = VERSION_INFO;
#else
    m.attr("__version__") = "dev";
#endif
}
</code></pre>
<p>This is output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8258946b73812dcb7d919937d3a97af710068ee5.png" data-download-href="https://discuss.pytorch.org/uploads/default/8258946b73812dcb7d919937d3a97af710068ee5" title="Screenshot from 2020-07-22 18-20-5a2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8258946b73812dcb7d919937d3a97af710068ee5.png" alt="Screenshot from 2020-07-22 18-20-5a2" data-base62-sha1="iB5QQJD8EbgASBPTXftiMb2a8yp" width="690" height="413" data-dominant-color="36142B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-07-22 18-20-5a2</span><span class="informations">733×439 43.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

LeviViana: <p>I don’t know if it will solve, but you should maybe cast the input and output in the declaration of pybind11, like it is done here: <a href="https://github.com/LeviViana/torch_sampling/blob/master/csrc/sampling.cpp#L44" rel="nofollow noopener">https://github.com/LeviViana/torch_sampling/blob/master/csrc/sampling.cpp#L44</a></p>
<p>BTW, this repo contains an implementation of a pytorch extension, it can serve as tutorial as well.</p>

tom: <p>I would use <code>torch::Tensor</code> even if they’re an alias these days.</p>
<p>All arguments should either be <code>const torch::Tensor&amp;</code> (if not modified) or <code>torch::Tensor</code> (no reference).<br>
(I think there is a brief mention of this in Chapter 15 of our book.)</p>
<p>Best regards</p>
<p>Thomas</p>
Learning weighted sums of network output for multiple inputs

Lionek: <p>Hello, I need to learn network with two different transformations on input data and than weighted sum of outputs, I want to learn weights w of the summing as well:</p>
<pre><code class="lang-auto">w = torch.tensor([.5, .5], requires_grad=True)
optimizer.add_param_group({'params': w})
learning loop of one input data:
    output[0] = transform_1(data)
    output[1] = transform_2(data)
    
    output = torch.matmul(w, input)
    loss = criterion(output, label)
    loss.backward()
    optimizer.step()
</code></pre>
<p>I am working with images and transforms include resizing, so <code>transform_1(data).shape != transform_2(data).shape </code> therefore I cannot stack them together along new dimension.</p>
<p>Using this loop I am getting</p>
<pre><code class="lang-auto">Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5, 2]], which is output 0 of PermuteBackward, is at version 10; expected version 8 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>If I put whole training loop into <code>with torch.autograd.set_detect_anomaly(True):</code> I got error</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5, 2]], which is output 0 of PermuteBackward, is at version 10; expected version 8 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>Do you have any idea how to train network with learning weighted sum of outputs? Thank you very much</p>
Is torch.logsumexp accurate for very negative inputs?

jwillette: <p>I have some numerical issues and I am trying to find out if there is a bug in my calculation or not. I am using a model which samples weights and therefore I am using logsumexp over the sample dimension such as this if we assume I have a tensor of <code>(sample_n, instances)</code>…</p>
<pre><code class="lang-auto">ll = torch.logsumexp(log_lik.sum(dim=1), dim=0)
ll = ll - np.log(sample_n)
ll = ll / instances
</code></pre>
<p>My predictions are extremely low probability because they are out of distribution samples so the log probability value can be something like <code>-100,000</code>. When I try to do the logsumexp operation without using the torch function then <code>exp(-100000)</code> just gives 0.</p>
<p>In this case I would expect that the output of logsumexp would then be <code>-inf</code> since it is taking the log of <code>0</code>, but it always comes up with some answer. How can this be?</p>

albanD: <p>Hi,</p>
<p>It is actually the whole point of having a single function that does log + sum + exp, it is much more precise than doing them one after the other.<br>
The trick is to remove the maximum value from the values in the exp. You can move it out of the sum and do <code>log(exp(max_val)) = max_val</code>. And the only part that you compute the exp/sum/log for are the differential wrt to that max value.  But these differentials are smaller and will lead to smaller numerical error.</p>
Accuracy drops to 50% when lstm stacked layers are added

Chiara: <p>Hello. I’m using the function nn.LSTM for my analysis. My LSTM has 2 layers and the accuracy reaches 90%. When i change the number of layers to 3 the accuracy goes down to 50%. Why does this happen?</p>

Chiara: <p>if anyone is interested in this, I figured it out! To stack more LSTM I decreased the learning rate by a factor of 10. Training is slower but the accuracy doesn’t decrease.</p>
How to handle variable length of sequences when using RNNCell

ChangGao: <p>Hi! I am considering using the RNNCell to handle a sequence task. I know that batch processing with padding and packing can accelerate the calculation. However, it is not feasible in my task because the data needs preprocessing at each time step.</p>
<p>Now I am considering to output the hidden state of the RNN at each time step using RNNCell. But I couldn’t figure out how to handle the variable length of sequences. I don’t know if masking the output would work.</p>
<p>Has anbody had this problem? Thanks for your help!</p>

meng_lin: <p>You can find the answer to this question by reading any work that requires teacher forcing on github. It may not be the most efficient solution but it works.<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/poojahira/image-captioning-with-attention/blob/master/models.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/poojahira/image-captioning-with-attention/blob/master/models.py" target="_blank" rel="nofollow noopener">poojahira/image-captioning-with-attention/blob/master/models.py</a></h4>
<pre><code class="lang-py">import torch
from torch import nn
import torchvision

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


class Encoder(nn.Module):
    """
    Encoder.
    """

    def __init__(self, encoded_image_size=14):
        super(Encoder, self).__init__()
        self.enc_image_size = encoded_image_size

        resnet = torchvision.models.resnet101(pretrained=True)  # pretrained ImageNet ResNet-101

        # Remove linear and pool layers (since we're not doing classification)
        modules = list(resnet.children())[:-2]
</code></pre>

  This file has been truncated. <a href="https://github.com/poojahira/image-captioning-with-attention/blob/master/models.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Here is one example, just read the forward function for DecoderWithAttention class.
Correct use of WeightedRandomSampler?

localh: <p>I am wondering what is the right way to use a sampler like <code>WeightedRandomSampler</code> for imbalanced classification problems.</p>
<p>Specifically, I am unclear as to whether I only use sampling during:</p>
<ol>
<li>Training</li>
<li>Training + Validation</li>
<li>Training + Validation + Testing (whereby each gets its own sampler to capture the distribution in its respective data set)</li>
</ol>
<p>I’ve poked through several threads and noticed people using it for training only and training + validation.</p>
<p>Thanks for your time and help!</p>

ptrblck: <p>I think the common use case would be to use it during the training only.<br>
The validation dataset should act as a good proxy for the final model performance on the unseen test data. So I would try to keep the data distribution of the validation and test datasets as close as possible.<br>
Since your test dataset is also imbalanced, using weighted sampling on it might not give you a proper signal how your model would perform on real world data (which should also be imbalanced).</p>
<p>However, that’s my biased opinion so let’s wait for some other opinions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

localh: <p>Thank you for always responding. This makes a lot of sense!</p>