akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is PyTorch using a C++ compiler or not?

AxelK: <p>Hello Everybody,</p>
<p>I used Keras but started recently with PyTorch 0.4.1 under Windows10 on a laptop without GPU. Everything works fine, but I’m surprised that the TaskManager shows me that all the CPU consumption is by python.exe itself. Keras compiles behind the scenes C++ code, which is then executed for speed.<br>
How does that work for PyTorch?<br>
Surely, all the number crunching can not be done in pure Python ??</p>

albanD: <p>Hi,</p>
<p>The c++ code is compiled as python extensions. So from the task manager point of view, it is python that uses these ressources.<br>
Note that in pytorch, all the compilation is done ahead of time, not at runtime.</p>
LinearTransformation does not accept mean_vector

ttoosi: <p>Hi,<br>
I am trying to perform ZCA on my dataset and I am using LinearTransformation <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.LinearTransformation" rel="nofollow noopener">as instructed in the documentation</a>.<br>
But it seams like LinearTransformation does not accpet a second argument (mean_vector).</p>
<p>transforms.LinearTransformation(transformation_matrix=transformation_matrix, mean_vector=mean_vector)<br>
TypeError: <strong>init</strong>() got an unexpected keyword argument ‘mean_vector’</p>
<p>I would appreciate if someone tells me what I am missing here.<br>
Thanks,<br>
T</p>

ptrblck: <p>Which <code>torchvision</code> version are you using?<br>
It looks like <a href="https://github.com/pytorch/vision/pull/793" rel="nofollow noopener">this PR</a> introduced the <code>mean_vector</code> argument followed by other PRs, which added it back to <code>LinearTransformation</code> instead of <code>AffineTransformation</code>.<br>
Could you update <code>torchvision</code> and check it again?</p>

ttoosi: <p>Thank you for your response, I am using ‘1.1.0.dev20190513’.  Is it because I am using the nightly version?</p>

ptrblck: <p>The <code>LinearTransformation</code> is part of <code>torchvision</code> not PyTorch, so your PyTorch version should be fine.<br>
Try to update <code>torchvision</code> to the latest version.</p>

ttoosi: <p>updated torchvision to 0.3 and now LinearTransformation works as expected. Thanks!</p>
Convolution on tensor slice without memory realloc

ezorita: <p>I have a 1D tensor: a, to which I want to apply multiple 1D convolutions.</p>
<p>Each of these convolutions has to be applied to a diminishing slice of the same vector: a[:-i], i.e. the next convolution is applied after dropping the last element of a.</p>
<p>Slicing is very inefficient in this case because it reallocs the whole remainder of a, so the overall procedure does not fit in memory. Is there any way to .resize_() a without affecting the autograder or, alternatively, to specify the convolution limits in each convolution layer? (each convolution is done with a different layer)</p>
<p>Thanks,</p>

ptrblck: <aside class="quote no-group" data-username="ezorita" data-post="1" data-topic="52320">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ezorita/48/14671_2.png" class="avatar"> ezorita:</div>
<blockquote>
<p>Slicing is very inefficient in this case because it reallocs the whole remainder of a, so the overall procedure does not fit in memory.</p>
</blockquote>
</aside>
<p>That shouldn’t be the case, if I understand your use case correctly.<br>
The output tensor will of course allocate new memory, but since the input is just sliced, it should not trigger a copy.<br>
Do you have a code snippet so that we can have a look?</p>

ezorita: <p>Hi, thanks for your answer. Here is the relevant code:</p>
<pre><code class="lang-auto">for i in np.arange(self.resolution-1,-1,-1):
  # Dilated convolutions
  z = self.sliconv[i](m[:,:,:(i+1)])
</code></pre>
<p>where <code>m</code> is a 1D vector and <code>self.sliconv</code> is defined as:</p>
<pre><code class="lang-auto">      # Filters for sliding convolutions
      self.sliconv = nn.ModuleList()
      self.sliconv.append(
         nn.Sequential(
            nn.Conv1d(in_channels=kernel_size*in_channels, out_channels=out_channels, kernel_size=1, stride=1),
            nn.LeakyReLU(LR_ALPHA)
         ).to(self.device)
      )
      for d in np.arange(self.resolution-1)+1:
         self.sliconv.append(
            nn.Sequential(
               nn.Conv1d(in_channels=kernel_size*in_channels, out_channels=out_channels, kernel_size=2, stride=1, dilation=d),
               nn.LeakyReLU(LR_ALPHA)
            ).to(self.device)
         )
</code></pre>
What are torch.nn.Parameter's in torch.nn.LSTM?

andreiliphd: <p>Hello to everyone!<br>
I am struggling to understand what are torch.nn.Parameter in torch.nn.LSTM?<br>
I have build a toy model and it seems to have 4 torch.nn.Paramater’s.<br>
What are they in the following formulas?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/aa93e6a0b37550ad6f203b2f795d930f0c4fada6.png" alt="lstm" data-base62-sha1="ol02RtOFo5TYUOEMMq53zenL4Wi" width="448" height="185"></p>
<p>Andrei</p>

ptrblck: <p>Have a look at <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM">docs</a> under the <code>Variables</code> paragraph.</p>
<p>Posting it here will mess up the format, but you’ll find some information about which internal LSTM parameters contain which part of the weight and bias matrices:</p>
<blockquote>
<p><strong>LSTM.weight_ih_l[k]</strong>  – the learnable input-hidden weights …</p>
</blockquote>
<p>You can access these parameters directly via:</p>
<pre data-code-wrap="python"><code class="lang-python">lstm = nn.LSTM(1, 1)
for name, _ in lstm.named_parameters():
    print(name)

&gt; weight_ih_l0
weight_hh_l0
bias_ih_l0
bias_hh_l0

lstm.weight_ih_l0
</code></pre>

andreiliphd: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> it is really helpful. Long life PyTorch.</p>
<p>Andrei</p>
Convert c10::IValue to tensor or tuple failed

k3833653: <p>I’m runing  a pytorch script_model in c++ , the model output is a  List[Tensor] , when I prepare to parse it to c++ data structure, either run  output.toTuple() or output.toTensor() is failed,  and get similar error info like this “what():  isTuple() ASSERT FAILED at /data/home/…/libtorch/include/ATen/core/ivalue.h:246”. However, I can print the output directly, the format is [2<br>
[ Variable[CPUType]{} ], 1<br>
[ Variable[CPUType]{} ], 0<br>
[ Variable[CPUType]{} ], 0<br>
[ Variable[CPUType]{} ], 3<br>
[ Variable[CPUType]{} ]]. How can convert the output to tensor or other c++ data structor?</p>

yf225: <p>Have you tried <code>output.toTensorList()</code> and then get the Tensors by indexing the returned list?</p>

k3833653: <p>yes, this is helpful</p>
ConvTranspose2d Weight Shape

ahkarami: <p>I have a question about <strong>ConvTranspose2d.weight</strong> shape. As mentioned in the <a href="https://pytorch.org/docs/stable/nn.html#convtranspose2d" rel="nofollow noopener">PyTorch documentation</a> the shape of <strong>ConvTranspose2d.weight</strong> tensor is as follows:<br>
<code>(in_channels, out_channels​, kernel_size[0], kernel_size[1])</code><br>
Note that I have omitted the <code>groups</code> parameter in the above shape.</p>
<p>Now, my question is that why not its shape as follows?<br>
<code>(out_channels, in_channels, kernel_size[0], kernel_size[1])</code><br>
My problem related to the position of <strong>(in_channels, out_channels)</strong>. As in some resources the Transposed Convolution is implemented via the regular convolution, so the kernel must apply on <code>input_channels</code>. as a result I don’t understand why the shape of the filter (i.e., weight tensor), is mentioned as above equation.<br>
Maybe it relates to the fact that the Transposed Convolution is a kind of inverse (or transposed version) of the regular convolution, so its filters shape is transposed, and as a result, its shape follows the above relationship.<br>
Am I correct?</p>

ahkarami: <p><a class="mention" href="/u/fmassa">@fmassa</a> would you please answer the above question?</p>
Worse performance using pin memory in dataloader

rtrobin: <p>Hi there,</p>
<p>I am running pytorch on our 8-card cloud machine. So I use docker to run it with only required resources. Here is the docker parameters I used to launch it: <code>docker run -it --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES 0 --cpus=4 --shm-size=1G --memory=10240m *image_name* /bin/bash</code></p>
<p>Based on the running command above, I find the training speed is dramatically slower when <em>pin_memory</em> setting True in DataLoader. I write a test benchmark code based on horovod benchmark code. I have post it to my github <a href="https://github.com/rtrobin/codeMixed/tree/master/python/pytorch/benchmark" rel="nofollow noopener">repo</a>.</p>
<p>The benchmark result shows, without data loading, the card could process 300 images/second. Adding data loading process, with <em>pin_memory=False</em>, the card could process 292 images/second, which is reasonable. However, with <em>pin_memory=True</em>, the card could only process 46 images/second.</p>
<p>I know pin memory is used to accelerate tensor async transfer between host memory and device memory. But the performance here is very strange. Could anyone illustrate me why it is and how to fix it? Thanks.</p>

ptrblck: <p>Are you seeing the swap being used a lot?<br>
As far as I know, this might cause the slow down.</p>

rtrobin: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the reply. I have found the trick. I’m not a professional in CPU field. It seems that something with swap does slow the tensor transfer speed.</p>
<p>Our cloud machine has two NUMA nodes. The docker parameter <code>--cpus=4</code> doesn’t allocate 4 cpu cores to the job, but load every cores balanced under cpu rate <strong>4/<code>cpu_core_nums</code></strong>. To run the training job correctly, the docker parameter <code>--cpuset-cpus</code> should be used.</p>
Simultaneously change parameters on all the GPUs

yangzhh: <p>during training the network on 8 GPUs in parallel, I am going to manually change the parameters in the network by the following code,</p>
<pre><code class="lang-python">for param in model.parameters():
  param.data.fill_(other parameter)
</code></pre>
<p>I am wondering if this would change all the parameters on different GPU or just GPU:0?</p>

pietern: <p>It depends what you’re using for parallelism.</p>
<p>If you use <code>nn.DataParallel</code> you should be able to do this, as the model is replicated to the other GPUs in every iteration. This means you only need to modify the parameters of the root module. This is also where you’d run the optimizer, for example.</p>

yangzhh: <p>The model is replicated to the other GPUs in every iteration means the state_dicts are copied to other GPUs every iteration? So the mean and var in BN are also copied to other GPUs from the GPU:0?</p>
<p>Is there any document explain this process elaborately? I am really curious about the parallel mechanism utilized in PyTorch, for I always conduct experiments on multi-gpu environment.</p>

pietern: <p>Yes, that’s correct. The documentation covers this (the replication bit), see <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel" rel="nofollow noopener">torch.nn.DataParallel</a>. Note that this is <strong>not</strong> how the distributed version works. There, every process runs forward/backward/optimizer against a single copy of the model, so its parameters are equivalent already. Not by replicating the values, but by executing the exact same optimizer step.</p>

yangzhh: <p>Thanks a lot! I know what you mean. So, in summarize, the multi-gpu environment works like following:</p>
<ol>
<li>Scatter the model and the state dict from GPU:0 to all the GPUs.</li>
<li>Split the data, and seperately forward them on different GPUs.</li>
<li>Gather output from GPUs to GPU:0</li>
<li>Calculate Loss by using outputs and targets on GPU:0</li>
<li>Backward Loss to GPUs and seperately calculate gradients</li>
<li>Gather gradients from GPUs to GPU:0</li>
<li>Update parameters on GPU:0</li>
<li>GoTo Step1.</li>
</ol>
<p>So, the only thing that not fully synchronized is the mean and var of BN, because it does not gather to GPU:0 during backward. All the other parameters are fully synchronized because of the gather-scatter mechanism.</p>
Custom loss function with large matrix multiplication

WeiQin_Chuah: <p>Hi,</p>
<p>I am working with depth estimation from stereo images and am trying to use a custom loss function.</p>
<p>In the network, images are cropped into size [256 512] and the maximum disparity is set to 192. This will then give me an output. The loss function includes cross product([bs<em>256</em>512  192]  [192  bs<em>256</em>512]), which will generate a massive matrix and requires a lot of memory.</p>
<p>Is there a way to get around this?</p>
How to modify activation in GRU

111125: <p>We know in keras,  <code>Bidirectional(GRU(128, activation='linear', return_sequences=True))(a1) # (240,256)</code>，that is to say, we can choose activation.But in torch,there’s no para to choose.<code>nn.GRU(n_in, n_hidden, bidirectional=True, dropout=droupout, batch_first=True, num_layers=num_layers)</code><br>
I want to know how  to modify activation in GRU in torch</p>
What libraries or frameworks do you wish existed for Pytorch?

Santosh-Gupta: <p>What libraries or frameworks do you wish existed for Pytorch?</p>
<p>An example of this might be a set of operations you watch batched, or a set of functions that do not exist yet.</p>
<p>My team and I were thinking of making a sparse training library for Pytorch, but would like to survey users to see if there’s anything more pressing.</p>

Oli: <p>Hi! I’d say a data vizualisation library that can hook into Visdom (&amp; tensorboardX I guess…). Visdom is great and all in that it gives you the freedom to plot anything but I believe that features on top of it would be great for convenience. It could help with gradient vizualisation, model explainability, model architecture, deeper look into losses which consists of different summations etc.</p>
<p>Another one that I feel is missing is some sort of metric library. Precision, recall, AP &amp; mAP for bounding boxes, segmentation metrics and god knows what.</p>
<p>Edit: I’m not entirely sure about what content exists other frameworks such as ignite &amp; fastai. Do they already cover this?</p>

vfdev-5: <p>Yes, I agree on easily integrated model explainability will be interesting.</p>
<p><a class="mention" href="/u/oli">@Oli</a> training viz is something can be more or less simply inserted into ignite, there are already provided tensorboard, visdom etc loggers and we can plot out-of-box losses, scalar/vector metrics, model weights norms, grads etc: <a href="https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.tensorboard_logger.TensorboardLogger" class="inline-onebox" rel="noopener nofollow ugc">ignite.contrib.handlers — PyTorch-Ignite v0.4.13 Documentation</a></p>
<p>Metric library for all cases (as you site detection tasks) will be awesome. Today in ignite, we have a lot of basic metrics for classification, segmentation tasks and a bunch of regression metrics. Detection mAP, AP will be a of next steps…</p>
<blockquote>
<p>Edit: I’m not entirely sure about what content exists other frameworks such as ignite &amp; fastai. Do they already cover this?</p>
</blockquote>
<p>Hope to answer the question from ignite side.</p>
Semantic segmentation NOT overfitting

Nishanth_Sasankan: <p>I am having some problem with a semantic segmentation implementation. I have 2 classes, a background, and an organ. No matter what I do my model only predicts background. Even if I increase the epoch and use training data for a prediction I only get one class as my output. Did anyone see this kind of problem before? Any and all help is appreciated.</p>
<p>Thanks<br>
Nishanth</p>

ptrblck: <p>Was this issue answered <a href="https://discuss.pytorch.org/t/semantic-segmentation-using-deeplabv3-resnet101-from-torchvision-models/52353/9">here</a> and can be considered solved?</p>

Nishanth_Sasankan: <p>Yes, It was solved. Thanks for your help</p>
Modify forward of pretrained model

muon: <p>I am trying to rewrite inference part so as  to also return hidden layer activations like embeddings.</p>
<p>is this correct implementation, any potential issues with this ?</p>
<pre><code class="lang-auto">class FeatureExtration(torch.nn.Module):
    def __init__(self, pretrained_model):
        super().__init__()
        self.__dict__ = pretrained_model.__dict__.copy()
        
    def forward(self,x):
        
        #... copy original forward ...
     
        return proba, embedings, some_other_layer


model2 = FeatureExtration(pretrained_model)
</code></pre>

ptrblck: <p>If you would like to return additional values in the <code>forward</code> method, but keep the model at it was otherwise, you could directly derive your class from the base model:</p>
<pre><code class="lang-python">class FeatureExtractedModel(BaseModel)
</code></pre>
<p>This would make the implementation cleaner in my opinion.</p>
Problem of backpropogation in RNN

tere3927: <p>Hi,<br>
I am dealing with the problem of “Trying to backward through the graph a second time.”<br>
I think the following two codes are the same, but actually they are not. What’s the difference?</p>
<p>code 1</p>
<pre><code class="lang-auto">import torch
x = torch.tensor(0.1)
w = torch.tensor(0.1,requires_grad = True)
y=x*w
z=y*w
y.backward()
z.backward()
#c = y+z
#c.backward()
</code></pre>
<p>code2</p>
<pre><code class="lang-auto">import torch
x = torch.tensor(0.1)
w = torch.tensor(0.1,requires_grad = True)
y=x*w
z=y*w
#y.backward()
#z.backward()
c = y+z
c.backward()
</code></pre>
<p>What is the reason that the first writing style will cause “Trying to backward through the graph a second time.” ?</p>
<p>Thanks.</p>

ptrblck: <p>The <code>backward</code> call will free the intermediate tensors, which were needed for the gradient computation, so save memory. If you need to call <code>backward</code> multiple times, you would need to set <code>retain_graph=True</code> in the <code>backward</code> call:</p>
<pre><code class="lang-python">x = torch.tensor(0.1)
w = torch.tensor(0.1,requires_grad = True)
y=x*w
z=y*w
y.backward(retain_graph=True)
z.backward(retain_graph=True)
c = y+z
c.backward()
</code></pre>
How to improve performance of tensor plus constant array

Nick_8229: <p>I am moving from CUDA C to PyTorch to achieve high performance parallel computing.<br>
If I need add a constant variable to a tensor, the following way is certainly working.</p>
<pre><code>t = torch.tensor.ones(10, 10000, 1000)
k_a = range(10)
for i in range(10):
   t[i] = t[i] + k_a[i]
</code></pre>
<p>But for achieve better performance,<br>
Do I need copy k_a to GPU memory first?<br>
Can I copy k_a to GPU constant memory?<br>
Or any thing i can do to improve it?</p>
<p>Thanks,</p>

ptrblck: <p>I would suggest to create <code>k_a</code> using <code>torch.arange(10).float()</code> instead of the Python <code>range</code>.</p>
<p>Loops are generally slower than vectorized code, so you could unsqueeze <code>k_a</code> in dim1 and dim2 and just add it in a single call:</p>
<pre><code class="lang-python">device = 'cuda'
t = torch.ones(10, 10000, 1000, deive=device)
k_a = torch.arange(10, device=device).float()
ret = t + k_a.view(-1, 1, 1)
</code></pre>
<p>If you set <code>device='cuda'</code>, this operation will automatically be executed on the GPU.</p>
NotImplemented Error after dropping layer on pretrained network

brookisme: <p>Hi All.  I have a trained Xception model that I want to use as a backbone in another network. In particular I want to drop all the layers starting with the global average pooling.  In the code below this means dropping the last block (GAPClassifier).</p>
<p>The problem is that after doing so I get a <code>NotImplementedError</code>.</p>
<p>I’m assuming that the issue is how I’m dropping the last block, namely</p>
<pre><code class="lang-auto">model=torch.nn.Sequential(*(list(model.children())[:-1]))
</code></pre>
<p>Are there situations where dropping a block is more complicated? Whats the best way to handle this?</p>
<p>Here is a run through of what is happening (unfortunately I can’t include the entire model in this post). Thanks!</p>
<pre><code class="lang-auto">test_batch=torch.rand((3,14,SIZE,SIZE)).cuda()
model(test_batch) #&lt;== executes successfully

def block_names(model,end=None,start=None):
    layer_list=list(model.children())
    for i,layer in enumerate(layer_list[start:end]):
        print(i,layer._get_name())

block_names(model)
"""OUTPUT
0 EntryBlock
1 ModuleList
2 SeparableStack
3 XBlock
4 SeparableStack
5 GAPClassifier
"""

model=torch.nn.Sequential(*(list(model.children())[:-1]))

block_names(model)
"""OUTPUT
0 EntryBlock
1 ModuleList
2 SeparableStack
3 XBlock
4 SeparableStack
"""
model(test_batch) # throws NotImplementedError
</code></pre>

ptrblck: <p>By wrapping submodules in an <code>nn.Sequential</code> block you are assuming all children are applied sequentially in the original model.<br>
While this might be often the case, some models are a bit more complicated so that this simple container won’t restore the original work flow anymore or will throw an error.<br>
E.g. <a href="https://github.com/Cadene/pretrained-models.pytorch/blob/021d97897c9aa76ec759deff43d341c4fd45d7ba/pretrainedmodels/models/xception.py#L206" rel="nofollow noopener">this <code>.view()</code> operation</a> will be missing, if you are extracting all child modules from this model.</p>
<p>I’m not sure, which module throws the <code>NotImplementedError</code> and it’s a bit strange, as I would expect some shape mismatch etc.</p>
<p>If you just want to remove the last layer, you could replace it with an <code>nn.Identity</code> module.</p>

brookisme: <p>That makes sense. Thanks!</p>
How to autograd `x = torch.round(x / self.scale) * self.scale`

dingyongchao: <pre><code class="lang-auto">class Conv2dQuant(nn.Module):
    def __init__(self):
        super(Conv2dQuant, self).__init__()
        self.conv_weight = 5
        self.conv_bias = 5
        self.scale = 3
    def forward(self, x):
        x = torch.round(x / self.scale) * self.scale
        x = x * self.conv_weight + self.conv_bias
        return x
</code></pre>
<p>If i define a module above, I know how to forward, However, When it comes to backward, I do not know the process of round function, and I wonder how to autograd this functions <code>x = torch.round(x / self.scale) * self.scale</code></p>

dingyongchao: <p>I try to train this function, as weight and bias a autogrand = True, Then I got</p>
<pre><code class="lang-auto">      7     def forward(self, x):
----&gt; 8         x = torch.round(x / self.scale) * self.scale
      9         x = x * self.conv_weight + self.conv_bias
     10         return x

RuntimeError: round_vml_cpu not implemented for 'Long'

</code></pre>

ptrblck: <p>I’m not sure, if I understood the use case correctly, but if you would like to train <code>self.conv_weight</code> and <code>self.conv_bias</code>, you should define them as <code>nn.Parameter</code>s (containing float values):</p>
<pre><code class="lang-python">class Conv2dQuant(nn.Module):
    def __init__(self):
        super(Conv2dQuant, self).__init__()
        self.conv_weight = nn.Parameter(torch.tensor([5.]))
        self.conv_bias = nn.Parameter(torch.tensor([5.]))
        self.scale = 3
    def forward(self, x):
        x = torch.round(x / self.scale) * self.scale
        x = x * self.conv_weight + self.conv_bias
        return x

model = Conv2dQuant()
x = torch.randn(1, 1)
output = model(x)
output.backward()
print(model.conv_weight.grad)
tensor([3.])
</code></pre>
Differential programming for RL

smu226: <p>Hello! I am trying to implement something similar to <a href="https://fluxml.ai/2019/03/05/dp-vs-rl.html" rel="nofollow noopener">this</a> (the CartPole example), using Pytorch. Basically I want to build a NN that predicts an action given the state of the system, computes the next state given the action, then backpropagates through everything. They have a code implemented in Julia <a href="https://github.com/FluxML/model-zoo/blob/cdda5cad3e87b216fa67069a5ca84a3016f2a604/games/differentiable-programming/cartpole/DiffRL.jl" rel="nofollow noopener">here</a> and here is the beginning of my code for Pytorch:</p>
<pre><code class="lang-auto">import gym
import argparse
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

env = gym.make('CartPole-v0')

class NN_cart(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_len, 24),               
            nn.ReLU(True),
            nn.Linear(24, 48),
            nn.ReLU(True),
            nn.Linear(48, 1),
            nn.Tanh(),
        )
            
    def forward(self, x):
        x = self.net(x)
        x = (torch.sign(x)+1)/2
        return x

model = NN_cart().cuda()

optimizer = optim.Adam(model.parameters(), lr = 1e-3)

done = False
env.reset()
state = torch.from_numpy(env.state).float().cuda()
while not done:
    model.train()
    optimizer.zero_grad()
    
    action = int(model(state)[0].cpu().data.numpy())
    state, reward, done, info = env.step(action)
    state = torch.from_numpy(state).float().cuda()
    state.requires_grad = True
    
    loss = state[2]**2
    
    loss.backward()
    optimizer.step()
</code></pre>
<p>I get no error, but the NN doesn’t learn. So one thing is the gradient of the sign function. In the Julia implementation they define their own gradient but I am not sure how to do it. Also, is my code the way it is right now able to propagate through everything (including the .step() function)? Thank you!</p>

ptrblck: <p>If I’m not mistaken, the gradient of the sign method should be zero everywhere.<br>
Since the authors implemented a custom backward method, you could have a look at <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="nofollow noopener">this tutorial</a> to see, how to do the same in PyTorch.</p>
<p>I’m not exactly sure, what <code>env.step()</code> is doing, but generally if you leave PyTorch and use e.g. numpy, you will detach the operations from the computation graph, which seems to be the case here.</p>
How is dropout implemented?

coincheung: <p>Hi,</p>
<p>I am wondering where is dropout exactly added. For fc layers of <code>y=Wx+b</code>, dropout randomly drops the parameters of in <code>W</code> matrix rather than the feature <code>x</code>, and for conv layers, dropout randomly drops the parameters in the convolution kernels rather than the feature maps. Is this what exactly happens in the dropout layers?</p>

ptrblck: <p><code>nn.Dropout</code> will randomly zero out some elements of the input, not the weights.<br>
The same applies for 4-dimensional tensors (e.g. conv outputs) using <code>nn.Dropout</code>.<br>
If you want to zero out complete channels, you should use <code>nn.Dropout2d</code> (<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout2d" rel="nofollow noopener">docs</a>).</p>
Define Function, and transfer into scriptModule, Error

dingyongchao: <pre><code class="lang-auto">class MultiOp(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        multiNum = 2
        ctx.save_for_backward(multiNum)
        result = multiNum * input
        return result
    @staticmethod
    def backward(ctx, grad_output):
        multiNum = ctx.saved_variables
        result = grad_output / multiNum
        return result
</code></pre>
<pre><code class="lang-auto">multiOp = MultiOp.apply

class Net(torch.jit.ScriptModule):
    def __init__(self):
        super(Net, self).__init__()
    @torch.jit.script_method
    def forward(self, x):
        x = multiOp(x)
        return x
</code></pre>
<pre><code class="lang-auto">scriptNet = Net()
torch.jit.save(scriptNet, "sample.pt")
</code></pre>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-55-40f7e60ca520&gt; in &lt;module&gt;
----&gt; 1 torch.jit.save(scriptNet, "sample.pt")

~/miniconda3/envs/dingyongchao/lib/python3.7/site-packages/torch/jit/__init__.py in save(m, f, _extra_files)
    196             (sys.version_info[0] == 2 and isinstance(f, unicode)) or \
    197             (sys.version_info[0] == 3 and isinstance(f, pathlib.Path)):
--&gt; 198         m.save(f, _extra_files=_extra_files)
    199     else:
    200         ret = m.save_to_buffer(_extra_files=_extra_files)

~/miniconda3/envs/dingyongchao/lib/python3.7/site-packages/torch/jit/__init__.py in save(self, *args, **kwargs)
   1203 
   1204         def save(self, *args, **kwargs):
-&gt; 1205             return self._c.save(*args, **kwargs)
   1206 
   1207         def save_to_buffer(self, *args, **kwargs):

RuntimeError: 
could not export python function call MultiOp. Remove calls to Python functions before export. Did you forget add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__.:
@torch.jit.script_method
def forward(self, x):
    x = multiOp(x)
        ~~~~~~~ &lt;--- HERE
    return x
</code></pre>
<p>So I want to how to define a Function ?</p>

ptrblck: <p>This might be related to <a class="mention" href="/u/tom">@tom</a>’s <a href="https://github.com/pytorch/pytorch/pull/22582" rel="nofollow noopener">PR</a>.</p>
Binary Classification batch_y VS y_pred

arjung: <p>I am working on a binary classification problem. My <code>batch_y</code> and <code>y_pred</code> do not have the same shape, yet I get no warnings/errors when I compute the loss with <code>nn.BCELoss(y_pred, batch_y)</code>:</p>
<pre><code class="lang-auto">batch_y.shape:  torch.Size([10, 1, 1])
tensor([[[0.]],

        [[1.]],

        [[0.]],

        [[0.]],

        [[0.]],

        [[1.]],

        [[1.]],

        [[1.]],

        [[1.]],

        [[0.]]], device='cuda:0')

y_pred.shape:  torch.Size([10, 1])
tensor([[0.5170],
        [0.5114],
        [0.5103],
        [0.4971],
        [0.4974],
        [0.5024],
        [0.5008],
        [0.4954],
        [0.5035],
        [0.4987]], device='cuda:0', grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>
<p>A more extreme version of this occurs when I do classification with three classes instead of two. Here again, I get no warnings/errors when I compute the loss with <code>nn.CrossEntropyLoss(y_pred, batch_y)</code>:</p>
<pre><code class="lang-auto">batch_y.shape:  torch.Size([3])
tensor([1, 2, 0], device='cuda:0')

y_pred.shape:  torch.Size([3, 3])
tensor([[-0.0718, -0.1237, -0.1143],
        [-0.0757, -0.1294, -0.1150],
        [-0.0792, -0.1128, -0.1106]],
       device='cuda:0', grad_fn=&lt;ThAddmmBackward&gt;)
</code></pre>
<p>Any ideas why this might be the case? Could this potentially (negatively) impact training?</p>

ptrblck: <p>Which PyTorch version are you using?<br>
This code:</p>
<pre><code class="lang-python">criterion = nn.BCELoss()
output = torch.sigmoid(torch.randn(10, 1, 1))
target = torch.randint(0, 1, (10, 1)).float()
loss = criterion(output, target)
</code></pre>
<p>raises a warning:</p>
<pre><code class="lang-auto">UserWarning: Using a target size (torch.Size([10, 1])) that is different to the input size (torch.Size([10, 1, 1])) is deprecated. Please ensure they have the same size.
</code></pre>
<p>in a slightly old nightly build <code>1.2.0.dev20190718</code>.</p>
<p>Regarding the <code>nn.CrossEntropyLoss</code>, the shapes are correct.<br>
This loss function expects a model output containing the class logits in the shape <code>[batch_size, nb_classes, *]</code> and a target containing the class indices in the shape <code>[batch_size, *]</code>, where the asterisk denotes additional dimensions.</p>
Problem with dimension size LSTM

hagsmand: <p>I get this error size mismatch, m1: [50 x 52480], m2: [256 x 2] and try to fix it by change linear dim to [52480 x 2] it can train but always get 0 loss</p>
<p>Here is my code.</p>
<pre><code class="lang-auto">class LSTMtagger(nn.Module):

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, num_layers):
        super(LSTMtagger, self).__init__()

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers, batch_first=True, dropout = 0.5)

        self.dropout = nn.Dropout(p=0.3)
        # The linear layer that maps from hidden state space to tag space
        self.fc = nn.Linear(hidden_dim, output_size)
        
        self.sigmoid = nn.Sigmoid()

    def forward(self, sentence):
        print(sentence.size())
        embeds = self.word_embeddings(sentence)
        print(embeds.size())
        lstm_out, _ = self.lstm(embeds.view(len(sentence), 205, -1))
        print(lstm_out.size())
        d_out = self.dropout(lstm_out)
       
        
#         d_out = d_out.view(50*256,205,1)
#         print(d_out.size())
        tag_space = self.fc(d_out.view(len(sentence), -1))
        tag_scores = F.sigmoid(tag_space)
        print(tag_scores)
        return tag_scores
</code></pre>
<pre><code class="lang-auto">vocab_size = len(vocab_to_int) + 1 # +1 for the 0 padding
output_size = 2
embedding_dim = 205
hidden_dim = 256
num_layers = 2

net = LSTMtagger(vocab_size, output_size, embedding_dim, hidden_dim, num_layers)
print(net)
</code></pre>
<pre><code class="lang-auto">model = LSTMtagger(vocab_size, output_size, embedding_dim, hidden_dim, num_layers)
loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

</code></pre>

ptrblck: <p>Your current code throws an error, if I pass an input of shape <code>[50, 256]</code> to it in the <code>self.lstm</code> layer.<br>
For the <code>batch_first=True</code> argument, the input should be <code>[batch_size, seq, feature]</code>, so just passing <code>embeds</code> without the <code>view</code> gets rid of this error.</p>
<p>If you would like to use the output features of all time samples, setting <code>in_features=256*256</code> should work, as you already tried.</p>
<p>That being said, if you are using <code>nn.NLLLoss</code>, you should apply <code>F.log_softmax</code> on your output instead of <code>F.sigmoid</code>.<br>
Could you change that and try to train your model again?</p>
Error while using Functional Module in PyTorch C++

krshrimali: <p>I’m facing the following error while trying to reproduce results from <a href="https://github.com/pytorch/examples/blob/master/cpp/custom-dataset/custom-dataset.cpp" rel="nofollow noopener">PyTorch Official Examples</a>.</p>
<p>The error seems to be at the line: <code>push_back(Functional(torch::log_softmax, 1, torch::nullopt));</code>.</p>
<p>Things I’ve changed: I’m using my own Dataset class and dataset. The network is exactly the same.</p>
<pre><code class="lang-auto">no matching function for call to ‘torch::nn::Functional::Functional(&lt;unresolved overloaded function type&gt;, int, const c10::nullopt_t&amp;)’
     push_back(Functional(torch::log_softmax, 1, torch::nullopt));
</code></pre>
<p>Any idea why this could happen?</p>

tom: <p>One thing to keep in mind is that you likely need to use a very new version of PyTorch (i.e. compiled from the master git branch or nightly releases).<br>
The example you linked was added just a week ago, so it might need features from very recent PyTorch. The background is that some bits of the C++ API still have some catch-up to do to achieve parity with “classic” PyTorch.</p>
<p>Best regards</p>
<p>Thomas</p>

vvarma: <p>In such a case a static cast to the right variant of the overloaded function works, you can try something like</p>
<pre><code class="lang-auto">torch::nn::Functional(static_cast&lt;torch::Tensor(*)(const torch::Tensor &amp;, int64_t)&gt;(torch::log_softmax), 0)
</code></pre>
(cross) validation of CNN models - best practices & strategies?

johannes.br: <p>Hi folks,</p>
<p>I am interested in your view or best practices on <strong>training, validation &amp; test strategies</strong>!</p>
<p>For a research project, I am working on using a CNN for defect detection on images of weld beads, using a dataset containing about 500 images. To do so, I am currently testing different models (e.g. ResNet18-50), as well as data augmentation and transfer learning techniques.</p>
<p>After having experimented for a while, I am wondering which way of training/validation/testing is best suited to provide an <strong>accurate measurement of performance</strong> while at the same time keeping the <strong>computational expensiveness</strong> at a reasonable level, considering I want to test as many models etc.as possible.</p>
<p>I guess it would be reasonable performing some sort of <strong>cross validation</strong> (cv), especially since the dataset is so small. However, after conducting some research, I could not find a clear best way to apply this and it seems to me that different people follow different strategies.</p>
<p>E.g.:<br>
Do you use the validation set only to find the best performing epoch/weights while training and then directly test every model on the test set (and repeat this k times as a k-fold-cv)?<br>
Or do you find the best performing model by comparing the average validation set accuracies of all models (with k-runs per model) and then check its accuracy on the testset? If so, which exact weigths do you load for testing or would one perform another cv for the best model to determine the final test accuracy?<br>
Would it be an option to perform multiple consecutive training-validation-test runs for each model, while before each run shuffling the dataset and splitting it in “new” training-, validation- &amp; testsets to determine an average test accuracy (like monte-carlo-cv)?</p>
<p>Since I could not find some sort of scientific consensus on this topic, I would be very happy to hear some of your opinions or experiences on this.</p>
<p>Thanks a lot!</p>

ptrblck: <p>I always have a look at <a class="mention" href="/u/rasbt">@rasbt</a>’s <a href="https://sebastianraschka.com/blog/2016/model-evaluation-selection-part3.html" rel="nofollow noopener">blog post about CV</a>, which explains the model selection in CV very well.</p>

johannes.br: <p>Thanks a lot, this was exactly what I needed!</p>
How to change the tensor dimension in my net?

ahhhhah: <p>Hi,</p>
<p>I want to know how to change the tensor’s dimension in net.</p>
<pre><code class="lang-auto">self.net.add_module("out1", nn.softmax(dim=1)
</code></pre>
<p>and I got a tensor[ 1681, 1, 1 ] from output.But I want to  change it into tensor[ 41, 41, 1, 1 ].<br>
Can someone please give me some advice.</p>
<p>Thanks.</p>

Oli: <p>Hi, you can change its shape with the view command. <code>new_tensor = tensor.view(41, 41, 1, 1)</code>. If you want to do this within the network, do that in the forward function</p>
Skip/freeze/omit learnable parameters

cherepanovic: <p>Hello community,</p>
<p>in order to freeze some parameter I could set the feature param.requires_grad to False. However my question is about optimizer, for example:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam([
    {'params': model.feature_extractor.parameters()},
    {'params': model.covar_module.parameters()},
    {'params': model.mean_module.parameters()},
], lr=0.01)
</code></pre>
<p>If I leave out my model parameters in the optimizer definition and leaved out parameters are not set to param.requires_grad = False, will these parameters be changed?</p>
<p>Thank a lot!</p>

tom: <p>No, but they will accumulate gradients and using computation and memory to do so, which may or not be significant. You might be in for surprises when you do add them later.</p>
<p>Best regards</p>
<p>Thomas</p>

cherepanovic: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="52500">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>You might be in for surprises when you do add them later</p>
</blockquote>
</aside>
<p>why?</p>
<p>What is the best way to leave out the parameters?  With means of param.requires_grad = False or leaving out in optimizer definition?</p>
<p>Thanks!</p>

tom: <p>To permanently avoid gradients/updates doing both would be cleanest because you don’t rely in context (à la “we calculate grads, but they are not in the optimizer” or “we pretend to optimizer but don’t calculate grads”) - the Python Zen says explicit is better than implicit. If for some reason (distributed?) no grad becomes grad zero later, you don’t have regularisation messing with your params etc.</p>
<p>Best regards</p>
<p>Thomas</p>
Difference of Module and Function definition of a sublayer

nclk: <p>Hello to everyone,<br>
I’m a newcomer to the Pytorch realm. I am faced with a question like this:</p>
<p>Is it necessary to define the sub-layers that we will use within the main Net module as a class outside the module, or can it be defined as a function?</p>
<p>Herein: <a href="https://github.com/FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict/blob/master/notebook.ipynb" rel="nofollow noopener">https://github.com/FrancescoSaverioZuppichini/Pytorch-how-and-when-to-use-Module-Sequential-ModuleList-and-ModuleDict/blob/master/notebook.ipynb</a>, it’s defined as function, but in <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py</a>, inception is created by using class…</p>
<p>Ex: conv - BN - ReLU</p>
<p>def myConv(x):<br>
x = conv2d(x)<br>
x = bnorm(x)<br>
return ReLU(x)</p>
<p>or</p>
<p>class myConv(nn.Module) …</p>
<p>and also what is the difference?</p>
<p>Thanks…</p>

ptrblck: <p>Both approaches will work the same, if you are using a pure functional approach.<br>
A custom <code>nn.Module</code> class is useful in case your custom module needs to store parameters, buffers etc.</p>
<p>E.g. <code>nn.Conv2d</code> itself is defined as an <code>nn.Module</code>, since it stores the <code>weight</code>, <code>bias</code> parameters as well as some internal attributes, e.g. <code>padding</code>, <code>stride</code>, etc.</p>
<p>In your current example you are just applying some modules, so you can easily define it as a function.</p>

nclk: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> !!  <img src="https://discuss.pytorch.org/images/emoji/apple/+1/3.png?v=9" title=":+1:t3:" class="emoji" alt=":+1:t3:"></p>
Why does convolution's output change even with `deterministic=True`?

0xFFFFFFFF: <h2>Background</h2>
<p>From my understanding <code>manual_seed</code> sets seeds for RNG, and <code>torch.backends.cudnn.deterministic = True</code> makes cuDNN’s output deterministic – fixed when given the same inputs and outputs.</p>
<p>Since there is no RNG factor in convolution (cuDNN documentation on convolution forward doesn’t mention anything about taking a seed as input or RNG of convolution), setting <code>manual_seed</code> should make no difference. However, when we test 2 settings, the results are not as expected:</p>
<ol>
<li>
<p><strong>setting 1:</strong><br>
Only <code>torch.backends.cudnn.deterministic = True</code> the output is NOT fixed on every iteration. It deviates on every iteration.</p>
</li>
<li>
<p><strong>setting 2:</strong><br>
<code>torch.backends.cudnn.deterministic = True</code> and <code>manual_seed</code> set, the output is fixed on every iteration</p>
</li>
</ol>
<h2>Experiment</h2>
<p>Given the same input &amp; weight (yes, we manually gave weight), and with <code>torch.backends.cudnn.deterministic = True</code> turned on, the output of</p>
<pre><code class="lang-python">weight = # some code that reads weight file

conv = nn.Conv1D(...)
conv.weight.data = weight
</code></pre>
<p>changes everytime it is called:</p>
<p><strong>example outputs:</strong></p>
<ol>
<li>iteration 1</li>
</ol>
<pre><code class="lang-auto">[-3.0552e+00, -5.3343e+00, -6.5944e-01,  1.1911e+00,  4.3999e+00,
           1.4698e+00, -3.8650e+00,  1.4742e+00,  1.2590e+00,  5.3744e+00,
          -1.1283e+01,  1.1128e+01, -1.3646e+01,  1.2124e+00, -9.6420e-01,
           7.5311e+00, -5.4766e+00,  2.8123e+00, -9.1796e+00,  6.2736e+00, ...
</code></pre>
<ol start="2">
<li>iteration 2</li>
</ol>
<pre><code class="lang-auto">-2.9771e+00, -5.2562e+00, -5.8136e-01,  1.2692e+00,  4.4779e+00,
           1.5479e+00, -3.7869e+00,  1.5522e+00,  1.3371e+00,  5.4525e+00,
          -1.1205e+01,  1.1206e+01, -1.3568e+01,  1.2905e+00, -8.8612e-01,
           7.6092e+00, -5.3985e+00,  2.8903e+00, -9.1016e+00,  6.3517e+00, ...
</code></pre>
<p>You can see that the outputs are different.</p>
<p>However, when we set the random seed with: <code>torch.manual_seed(0)</code>, then the output becomes identical on every iteration.</p>
<ol>
<li>
<p>why does the output differ given the same inputs and weights, and with <code>torch.backends.cudnn.deterministic = True</code>?</p>
</li>
<li>
<p>Why does <code>torch.manual_seed(0)</code> make the outputs identical?</p>
</li>
</ol>

tom: <p>When you use Modules from nn, you use</p>
<ul>
<li>the functions,</li>
<li>the parameter-holding of the modules,</li>
<li>the default initialization of the parameters.</li>
</ul>
<p>The latter uses random numbers.</p>
<p>Best regards</p>
<p>Thomas</p>
Need help with Autograd in a graph network

peterkaloyannis: <p>Hi there,</p>
<p>I’m building a convolutional graph network for a research project and I have been running into this bug for the past few hours and I think I’m just making an implementation mistake. I basically compute a recursive polynomial to filter my graph and here is how it is currently implemented.</p>
<pre><code class="lang-auto">    def chebychev(self, K, x_0, Fout, weights, bias = None, biasEnable = False):
        N, Fin = x_0.shape #N x Fin
        x = x_0.t().expand(K,-1,-1) # K x Fin x N
#         print(x.shape)
        for i in range(Fin):
            if K &gt; 1:
#                 print(x_0[:,i])
                x1 = t.sparse.mm(self.L, x_0[:,i].clone().expand(1,-1).t())
                x[1,i] = x1.clone().t()
                x0 = x_0[:,i].clone().expand(1,-1).t()
            for k in range(2, K):
                x2 = 2 * t.sparse.mm(self.L, x1) - x0 # 1 x N
                x[k,i] = x2.clone().t()
                x0, x1 = x1, x2
            x = x.transpose(2,1).reshape((N, Fin*K))#x.transpose(0,1) # N x Fin*k
#         print(x.shape)
        GConv = x@weights #here weights is Fin*k x Fout so GConv is N x Fout

        #add bias as necessary
        if biasEnable == True: return GConv + bias #add bias where necessary
        else: return GConv
</code></pre>
<p>Here K is the order of the polynomial, Fin is the input features, Fout is the output features and N is the order of the input data. When I run this I get a really scrambled image even for Fin = Fout = 1. I found that the following slight modification to the code produces the correct result:</p>
<pre><code class="lang-auto">    def chebychev(self, K, x_0, Fout, weights, bias = None, biasEnable = False):
        N, Fin = x_0.shape #N x Fin
        x = x_0.t().expand(K,-1) # K x Fin x N
#         print(x.shape)
        for i in range(Fin):
            if K &gt; 1:
#                 print(x_0[:,i])
                x1 = t.sparse.mm(self.L, x_0[:,i].clone().expand(1,-1).t())
                x[1] = x1.clone().t()
                x0 = x_0[:,i].clone().expand(1,-1).t()
            for k in range(2, K):
                x2 = 2 * t.sparse.mm(self.L, x1) - x0 # 1 x N
                x[k] = x2.clone().t()
                x0, x1 = x1, x2
            x = x.transpose(0,1).reshape((N, Fin*K))#x.transpose(0,1) # N x Fin*k
#         print(x.shape)
        GConv = x@weights #here weights is Fin*k x Fout so GConv is N x Fout

        #add bias as necessary
        if biasEnable == True: return GConv + bias #add bias where necessary
        else: return GConv
</code></pre>
<p>Which should return the same result as the code above with Fin=Fout=1. Can someone please help me look through this or point out the issue? It is something related to adding the extra dimension to the x tensor.</p>