akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to save the tensor in the forward pass of customer autograd function

username: IapCaL
cooked: <p>Hi,</p>
<p>Is there have any method to save a tensor for statistics?</p>
<p>Example</p>
<pre><code class="lang-python">class Exp(Function):
    @staticmethod
     def forward(ctx, i):
         i_max = i.max()
         return i

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output
</code></pre>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you want to save it for the backward pass, then you can save it within the context as <code>ctx.i_max = i.max()</code>.<br>
If you want to save it for external use, then you can use any pythonic way you want: use a global variable, save it in a list, have a stats object where you can record what happens etc.</p>
Insert dropout, is it right?

username: Giuseppe
cooked: <p>Hy guys, I want to inset a dropout layer, I have some doubts.<br>
Is forward right?</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x
</code></pre>
<pre><code class="lang-auto">class Compound_Model_final(nn.Module):

    def __init__(self):
        super().__init__()

        #Load the models scratch for the first machine
        self.model1 = fcn_resnet50(num_classes=9)
        self.model2= _resnet_shallow(pretrained=False)


        #Load the resnet50 for the second machine, pretrained
        self.model3 = models.resnet50(pretrained=True)
        #self.model3.fc = nn.Linear(self.model3.fc.in_features, 4)
        #self.model3.load_state_dict(torch.load("Regressione_ResNet50_cartesian_angle_M95.pth", map_location='cpu'))



        #Get the number of parameters
        n_features_x = self.model2.fc.in_features
        n_features_y = self.model3.fc.in_features
        # Dropout Layer
        self.dropout = nn.Dropout(p=0.5)

        #Layers to utilize for remove FC
        self.fc = nn.Linear((n_features_x + n_features_y), 4)

        #Remove FC
        self.model2.fc = Identity()
        self.model3.fc = Identity()


    def forward(self, x):


        #The first machine is executed
        dist = self.model1(x)['out']
        model2_out = self.model2(dist)


        #The second machine too
        model3_out = self.model3(x)




        #Concatenate the tensors
        z = torch.cat([model2_out, model3_out], 1)




        #I use the linear level
        xyuv = self.fc(z)



        #Return
        return dist, xyuv
</code></pre>

username: JuanFMontesinos
cooked: <p>You have to call dropout in the forward pass.<br>
Whatever you don’t call in the forward pass won’t be computed</p>

username: Giuseppe
cooked: <p>yes I add, before the linear, z= self.dropout(z)</p>
Register_forward_hook is not supported on ScriptModules

username: xzhu1900
cooked: <p>Hi torch experts,</p>
<p>I am trying to add a forward hook to my model. But I got the error message indicating the hook won’t work with jit module.</p>
<p>Is there a particular reason why hook won’t work with jit modules? How can I work around this while still keep my module as a script module?</p>
<p>Thanks in advance!</p>

username: driazati
cooked: <p><code>register_forward_hooks</code> (and <code>register_backward_hooks</code>) currently aren’t supported in TorchScript. If you’d like to see them added, please file a <a href="https://github.com/pytorch/pytorch/issues/new?template=feature-request.md" rel="nofollow noopener">feature request on GitHub</a>.</p>
<p>In your particular case it sounds like you’re inheriting from <code>ScriptModule</code> as the way to access the TorchScript compiler. An API change in v1.2.0 lets you compile <code>nn.Module</code>s without making them inherit from <code>ScriptModule</code> directly, see <a href="https://pytorch.org/docs/master/jit.html#migrating-to-pytorch-1-2-recursive-scripting-api" rel="nofollow noopener">these docs</a> for details:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(5, 5, 2)

    def forward(self, x):
        return self.conv(x) + 10


def my_hook(self, *args):
    print("Hello from my_hook")


m = M()
m.conv.register_forward_hook(my_hook)

# `my_hook` will be called
m(torch.randn(5, 5, 2, 2))

a_scripted_module = torch.jit.script(m)

# `my_hook` will NOT be called, forward hooks are lost
# when an `nn.Module` is compiled
a_scripted_module(torch.randn(5, 5, 2, 2))
</code></pre>
Segmentation fault when using the custom activation function

username: sirius
cooked: <p>I am trying to implement a custom activation function (the codes attached below). Before using the custom activation function, everything works well. However, as long as it is used, the server would throw the error:</p>
<p>Segmentation fault</p>
<p>The error always appears at the first epoch.</p>
<p>I am using</p>
<ol>
<li>Pytorch 1.1.0</li>
<li>Cuda compilation tools, release 9.2, V9.2.148</li>
</ol>
<p>the codes</p>
<p>def mg(x):</p>
<pre><code>c = 1.33
b = 0.4
p = 6.88
input_size = x.shape
num = torch.numel(x) # the element number of the input tensor
x = x.view(num)

out = torch.zeros(len(x))

for i in range(len(x)):
    if x[i] &lt; 0:
        out[i] = 0
    else:
        out[i] = (c * x[i]) / (1 + torch.mul(b * p, torch.pow(x[i], p)))

out = out.view(input_size[0], input_size[1], input_size[2], input_size[3])
return out</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Could you give a full code that I can run to reproduce the issue please?</p>

username: sirius
cooked: <p>Hi AlbanD,</p>
<p>Thanks a lot for your reply. The issue has been solved by modifying the version of PyTorch. Thanks</p>
How to initialize registered parameters seperately?

username: HAORAN_LI
cooked: <p>Hi ! I was trying to add new parameters a,b in every layer of my network as follows:</p>
<pre><code class="lang-auto">self.a = nn.Parameter(torch.ones([1,1],dtype=torch.double), requires_grad=True)       ###
self.b = nn.Parameter(torch.ones([1,1],dtype=torch.double)*torch.sqrt(torch.Tensor([self.dim]).double()), requires_grad=True)       ###
self.register_parameter('aa',self.a)                                                                 ###
self.register_parameter('bb',self.b) 
</code></pre>
<p>I want to initialize them separately, say self.a=1, self.b = sqrt(dim). But as it turns out self.a and self.b are initialized with other parameters in my network: self.a = -1.74 self.b=0.79. Do anybody know how to<br>
initialize self.a, self.b separately, avoid initialization with other parameters in the network?<br>
Also, I find that self.aa, self.bb have different data types from self.a, self.b when using GPU. How to solve this?</p>

username: ptrblck
cooked: <p>Your code works, if I just wrap it in a custom <code>nn.Module</code>:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.a = nn.Parameter(torch.ones([1,1],dtype=torch.double), requires_grad=True)       ###
        self.b = nn.Parameter(torch.ones([1,1],dtype=torch.double)*torch.sqrt(torch.Tensor([4.]).double()), requires_grad=True)       ###
        self.register_parameter('aa',self.a)                                                                 ###
        self.register_parameter('bb',self.b) 
        
    def forward(self, x):
        x = x * self.aa * self.bb
        return x

model = MyModel()
print(model.a)
&gt; Parameter containing:
tensor([[1.]], dtype=torch.float64, requires_grad=True)
print(model.b)
&gt; Parameter containing:
tensor([[2.]], dtype=torch.float64, requires_grad=True)
print(model.aa)
&gt; Parameter containing:
tensor([[1.]], dtype=torch.float64, requires_grad=True)
print(model.bb)
&gt; Parameter containing:
tensor([[2.]], dtype=torch.float64, requires_grad=True)
</code></pre>
<p>As you see, you’ll get the expected values as well as types.</p>
ReduceLROnPlateau keeps reducing lr even when not needed

username: JohnCwok
cooked: <p>Hello,</p>
<p>I am currently trying to incorporate a scheduler in my training loop, using the loss itself as validation measurement. I noticed that the scheduler always reduces the learning rate, independantly of the value of the loss.Setting the <code>patience</code> to 20 for example, I get outputs like :</p>
<pre><code class="lang-auto">0m 6s (5 0%) 3838.608
0m 8s (10 0%) 68974.180
0m 11s (15 1%) 50144.387
0m 14s (20 1%) 46037.418
Epoch    21: reducing learning rate of group 0 to 5.0000e-03.
0m 16s (25 1%) 43242.031
0m 19s (30 2%) 48685.941
0m 21s (35 2%) 51668.488
0m 24s (40 2%) 48751.176
Epoch    42: reducing learning rate of group 0 to 2.5000e-03.
0m 26s (45 3%) 48641.238
0m 28s (50 3%) 52856.855
0m 31s (55 3%) 55549.047
0m 33s (60 4%) 49968.438
Epoch    63: reducing learning rate of group 0 to 1.2500e-03.
0m 35s (65 4%) 40710.168
0m 38s (70 4%) 40792.227
0m 40s (75 5%) 38300.871
0m 43s (80 5%) 44388.047
Epoch    84: reducing learning rate of group 0 to 6.2500e-04.
0m 45s (85 5%) 76917.484
0m 48s (90 6%) 62983.559
0m 50s (95 6%) 45617.375
0m 53s (100 6%) 39899.105
0m 55s (105 7%) 94773.094
Epoch   105: reducing learning rate of group 0 to 3.1250e-04.
</code></pre>
<p>Where at the beginning it makes sense to reduce the LR, but then it also happens when the loss starts decreasing until the LR reaches the minimum allowed value …<br>
I initialized the scheduler like this :</p>
<pre><code class="lang-auto">scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=20, verbose=True)
</code></pre>
<p>taking a step with :</p>
<pre><code class="lang-auto">scheduler.step(loss)
</code></pre>
<p>Did I forgot something ?</p>

username: JuanFMontesinos
cooked: <p>You didn’t set minimum value…<br>
<a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau</a></p>
<pre><code class="lang-auto">min_lr (float or list) – A scalar or a list of scalars. A lower bound on the learning rate of all param groups or each group respectively. Default: 0.
</code></pre>
Gradient of an arbitrary function

username: Mrad
cooked: <p>Hi all, I am a new Pytorch user. I am working on a network structure which does not use the input value directly but first will calculate a function of the input value.</p>
<p>I am doing the simplest case when I give the input to a function and ask PyTorch to calculate the derivative of the function wrt the input. Here is the code,</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>def get_torch_G2(Rij, Rs, eta):<br>
return Variable(0.5 * torch.exp(-eta * (Rij - Rs).pow(2)) * (torch.cos(np.pi * Rij / 12)+1), requires_grad=True)</p>
<p>dd = torch.tensor([2.2], requires_grad=True)<br>
bb = get_torch_G2(dd, 0, 1)<br>
bb.backward()<br>
dd.grad.clone()</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>It gives me the error “‘NoneType’ object has no attribute ‘clone’”. Can anyone help me with my problem please?</p>

username: JuanFMontesinos
cooked: <p>Jus remove Variable and it will work</p>
<pre><code class="lang-auto">import torch
def get_torch_G2(Rij, Rs, eta):
    return 0.5 * torch.exp(-eta * (Rij - Rs).pow(2)) * (torch.cos(3.1416 * Rij / 12)+1)
dd = torch.tensor([2.2], requires_grad=True)
bb = get_torch_G2(dd, 0, 1)
bb.backward()
dd.grad.clone()
</code></pre>
RuntimeError: expected a non-empty list of Tensors

username: Mohammed_Daaniyaal
cooked: <p>I am trying to develop a gesture recognition system using deep learning in pytorch.<br>
I get the following error .</p>
<p>data = torch.cat(imgs,dim=1)<br>
RuntimeError: expected a non-empty list of Tensors</p>
<p>Could anyone help on this?<br>
Thanks!</p>

username: richard
cooked: <p>Is <code>imgs</code> an empty list? Print it out with <code>print(imgs)</code></p>

username: Mohammed_Daaniyaal
cooked: <p>Yes <code>imgs</code> is an empty list.</p>
<p>So what can be done?</p>

username: richard
cooked: <p>What are you trying to do? What should <code>data</code> be?</p>

username: Oli
cooked: <p>Ran into this myself today. I guess id like it to create an empty tensor, but maybe that’s problematic. Will just do an if else I guess. Anyone got a prettier solution than</p>
<pre><code class="lang-auto">if list_of_tensors:
    t = torch.stack(list_of_tensors)
else
    t = torch.tensor([])
</code></pre>
IMDB split datasets

username: Niki
cooked: <p>Hi,</p>
<p>In Torchtext,</p>
<pre><code class="lang-auto">train, test = datasets.IMDB.splits(TEXT, LABEL)
</code></pre>
<p>divides ratio between train and test 50:50, is there any ways that we change this ratio to 80:20?</p>

username: ptrblck
cooked: <p>The ratio for splitting the IMDB dataset originates from the data itself, as 25,000 reviews are provided for training and 25,000 for testing.<br>
From the <a href="http://ai.stanford.edu/~amaas/data/sentiment/">dataset website</a>:</p>
<blockquote>
<p>We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing.</p>
</blockquote>

username: Niki
cooked: <p>Understood. Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

username: Rohan_Kumar
cooked: <p>personally i dont like torchtext all that much, i would recommend you use custom dataloader, and divide the data as you want. I was working on this very dataset, i will upload the code on my github by sunday, you can take a look if you want, i will link it once its up.</p>

username: Niki
cooked: <p>That will help a lot, thank you very much <a class="mention" href="/u/rohan_kumar">@Rohan_Kumar</a>.</p>

username: Niki
cooked: <p>May I ask about Test accuracy for SST dataset, <code>train_data, valid_data, test_data = datasets.SST.splits(TEXT, LABEL)</code>  with LSTM? It is around 60%?</p>
nibabel.filebasedimages.ImageFileError

username: Elahe_Heydari
cooked: <p>Hello,<br>
I want to segment my .nrrd data. I use the following code , but I have this<br>
error:<br>
nibabel.filebasedimages.ImageFileError: Cannot work out file type of …</p>
<p>ROOT_DIR_GMCHALLENGE = “/media/elahe/d/data/dataset”<br>
mri_input_filename = os.path.join(ROOT_DIR_GMCHALLENGE,<br>
‘train’)<br>
mri_gt_filename = os.path.join(ROOT_DIR_GMCHALLENGE,<br>
‘labels’)<br>
pair = mt_datasets.SegmentationPair2D(mri_input_filename, mri_gt_filename)</p>
<p>I would be appreciated if you help me.<br>
Thanks</p>
Get the “sensitivity” of each unit with respect to perturbations of the bias

username: shirui-japina
cooked: <p>As the paper <a href="http://cogprints.org/5869/1/cnn_tutorial.pdf" rel="nofollow noopener">“Notes on Convolutional Neural Networks”</a> tells, I just guessed that the <em><strong>“sensitivity”</strong></em> is a very important factor in deep learning.<br>
My question is how can we get the “sensitivity”: δ = ∂E / ∂b = ∂E / ∂u in PyTorch?<br>
Is there some efficient function? Maybe it’s something like <a href="https://discuss.pytorch.org/t/sensitivity-of-output-y-with-respect-to-input-x/50479">this one</a>.</p>
Troubleshooting dead ReLUs (or something else)

username: bongbang
cooked: <p>I’m implementing a Chinese handwriting recognition model mentioned in this <a href="http://yuhao.im/files/Zhang_CNNChar.pdf" rel="nofollow noopener">paper</a>. If I purposely overfit a single batch of 2 or 4 samples, training loss does drop to 0 as expected. However, when I train on 40000 samples (composing 200 classes) in batches of 4 and validate on about 8000 samples, the training and validation losses hardly budge at all from epoch to epoch.</p>
<p>A quick web search indicates that this is a symptom of “dead ReLUs.”  I doubt the choice of the ReLU is the problem, however, since someone else implemented a <a href="https://github.com/integeruser/CASIA-HWDB1.1-cnn" rel="nofollow noopener">slightly simpler model</a> from the same paper on TensorFlow with expected results. I must be doing something wrong.</p>
<p>Could you tell me what’s wrong with my model below or, if not, suggest a way to troubleshoot, please? Thank you.</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    # Zhang. M7-1
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 64, 3, padding=1)
        self.conv2 = nn.Conv2d(64, 128, 3, padding=1)
        self.conv3 = nn.Conv2d(128, 256, 3, padding=1)
        self.conv4 = nn.Conv2d(256, 512, 3, padding=1)
        self.conv5 = nn.Conv2d(512, 512, 3, padding=1)
        self.fc1 = nn.Linear(512 * 4 * 4, 1024) # 4 = 64/(2^4)
        self.fc2 = nn.Linear(1024, 200)

    def forward(self, X):
        X = F.max_pool2d(F.relu(self.conv1(X)), (2, 2), (2, 2))
        X = F.max_pool2d(F.relu(self.conv2(X)), (2, 2), (2, 2))
        X = F.max_pool2d(F.relu(self.conv3(X)), (2, 2), (2, 2))
        X = F.relu(self.conv4(X))
        X = F.max_pool2d(F.relu(self.conv5(X)), (2, 2), (2, 2))
        
        X = F.dropout(torch.flatten(X, start_dim=1))
        X = F.dropout(F.relu(self.fc1(X)))
        X = self.fc2(X)
        return F.log_softmax(X, dim=1) # log-probabilities

model = CNN()
</code></pre>
Multidimensional Slice in C++

username: markl
cooked: <p>Is there a multidimensional slice in the C++ API?</p>
<p>I think there is a single dimensional one<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/3ad1bbe16a3c1d6bb9566f09229afd63022a82df/aten/src/ATen/native/TensorShape.cpp#L655" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/3ad1bbe16a3c1d6bb9566f09229afd63022a82df/aten/src/ATen/native/TensorShape.cpp#L655" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/3ad1bbe16a3c1d6bb9566f09229afd63022a82df/aten/src/ATen/native/TensorShape.cpp#L655</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="645" style="counter-reset: li-counter 644 ;">
<li>    auto new_values = at::empty(vsize, values.options());</li>
<li>    for (int64_t k=0; k &lt; nnz; k++) {</li>
<li>      new_values[k] = values[k].index_select(dim - sparse_dim, index);</li>
<li>    }</li>
<li>    return _sparse_coo_tensor_with_dims_and_tensors(</li>
<li>        sparse_dim, dense_dim, new_sizes, indices, new_values, self.options());</li>
<li>
</li>
<li>  }</li>
<li>}</li>
<li>
</li>
<li class="selected">Tensor slice(const Tensor&amp; self, int64_t dim, int64_t start, int64_t end, int64_t step) {</li>
<li>  int64_t ndim = self.dim();</li>
<li>  if (ndim == 0) {</li>
<li>    AT_INDEX_ERROR("slice() cannot be applied to a 0-dim tensor.");</li>
<li>  }</li>
<li>  dim = maybe_wrap_dim(dim, ndim);</li>
<li>  auto sizes = self.sizes().vec();</li>
<li>  auto strides = self.strides().vec();</li>
<li>  // TODO: support negative strides</li>
<li>  TORCH_CHECK(step &gt; 0, "slice step must be positive");</li>
<li>  if (start &lt; 0) {</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: yf225
cooked: <p>We don’t have it now, but we will add it (aka. numpy-style indexing) by the end of this year. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
1d conv filter on a 2d dataset

username: pedram1
cooked: <p>Hi everyone<br>
I am trying to apply a 1d conv filter on my torch.Size([65536, 94]) data, however I faced with this error.</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 z = m(y)</p>
<p>1 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)<br>
198                             _single(0), self.dilation, self.groups)<br>
199         return F.conv1d(input, self.weight, self.bias, self.stride,<br>
–&gt; 200                         self.padding, self.dilation, self.groups)<br>
201<br>
202</p>
<p>RuntimeError: Expected 3-dimensional input for 3-dimensional weight 10 1, but got 2-dimensional input of size [65536, 94] instead</p>
<p>Here is my code:<br>
m = nn.Conv1d(1,10,5)<br>
z=m(y)</p>
<p>Dose anyone could help me ?</p>

username: ptrblck
cooked: <p>Based on the definition of your conv layer, it seems you are dealing with a single channel input data.<br>
If that’s the case, make sure to add the channel dimension in dim1 to your input:</p>
<pre><code class="lang-python">y = y.unsqueeze(1)
z = m(y)
</code></pre>
<p>This would create a tensor with the dimensions <code>[batch_size=65535, channels=1, seq_length=94]</code>.</p>

username: pedram1
cooked: <p>That worked.<br>
Thank you, ptrblck</p>
How does torchvision.transforms.Normalize work

username: kenneth_maanika
cooked: <p>Hi.<br>
Whenever I Normalize my input images with its mean and std, and when i plot the images to visualize i get this message along with plots of images which are distorted in a way.<br>
Clipping input data to the valid range for imshow with RGB data ([0…1] for floats or [0…255] for integers).<br>
Clipping input data to the valid range for imshow with RGB data ([0…1] for floats or [0…255] for integers).<br>
If anyone could help me understand why this is happening it will be great.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a6d298b5e84c714db18984e851f0a9967710cfe.png" alt="download" data-base62-sha1="aCpd8DO096boG8PgM12X8bPpODY" width="372" height="226"></p>

username: ptrblck
cooked: <p><code>torchvision.transforms.Normalize</code> will use the <code>mean</code> and <code>std</code> to standardize the inputs, so that they would have a zero mean and unit variance.</p>
<p>Your current library to show these images (probably <code>matplotlib</code>) will clip the values of these float image arrays to <code>[0, 1]</code>, which will distort them.</p>

username: kenneth_maanika
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a> . Yes i am using matplotlib to show the images. Getting distorted images , does it mean that I am doing something wrong ? , If not is there a way i could plot the images without them being distorted ?</p>

username: ptrblck
cooked: <p>The easiest way would be to plot them before normalizing.<br>
However, if that’s not possible, you could also undo the normalization:</p>
<pre><code class="lang-python">x = torch.empty(3, 224, 224).uniform_(0, 1)
mean = (0.5, 0.5, 0.5)
std = (0.5, 0.5, 0.5)
norm = transforms.Normalize(mean, std)
x_norm = norm(x) 

x_restore = x_norm * torch.tensor(std).view(3, 1, 1) + torch.tensor(mean).view(3, 1, 1)

print((x_restore - x).abs().max())
&gt; tensor(0.)
</code></pre>

username: kenneth_maanika
cooked: <p>Thank your very much <a class="mention" href="/u/ptrblck">@ptrblck</a>. I will give this a try <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Export from TorchScript to ONNX

username: racoiaws
cooked: <p>TorchScript -&gt; ONNX conversion of this simple module fails (<a href="https://pastebin.com/DH8RgyQT" rel="nofollow noopener">pastebin</a>). Am I doing something wrong?</p>
<p>If one doesn’t jit-compile the model, everything works.</p>
<pre><code class="lang-auto">from tempfile import TemporaryFile

import torch
import torch.onnx
import torch.jit
from torch import nn, Tensor

print(f"PyTorch version is {torch.__version__}")

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.module = nn.Linear(
            in_features=8, out_features=4)
        self.module2 = nn.Linear(
            in_features=4, out_features=2)
       
    def forward(self, x: Tensor) -&gt; Tensor:
        preout = self.module(x)
        out = self.module2(preout)
        return out


model = Model()
model = torch.jit.script(model)

dummy_input = torch.randn(3, 8)
dummy_output = model(dummy_input)

with TemporaryFile() as temp:
    torch.onnx.export(model=model, 
                      args=dummy_input, 
                      example_outputs=dummy_output,
                      f=temp, 
                      verbose=True)
</code></pre>

username: driazati
cooked: <p>When the ONNX exporter sees an <code>nn.Module</code>, it uses the TorchScript tracer to graph a graph, then converts that graph to an ONNX graph. The TorchScript compiler (<code>torch.jit.script</code>) should be functionally equivalent, so it sound like this is a bug. Could you file an <a href="https://github.com/pytorch/pytorch/issues/new?template=bug-report.md" rel="nofollow noopener">issue on GitHub</a> so we can track this? Thanks!</p>

username: racoiaws
cooked: <p>Sure, I’ll file an issue on GitHub!</p>
<p>Edit: <a href="https://github.com/pytorch/pytorch/issues/27569" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/27569</a></p>
Python functions to create nodes / modify graph

username: flynntax
cooked: <p>I’d like to modify the graph by inserting new inputs (hopefully by using addInput in the graph). Is there a way to do this from Python after getting a torch._C.Graph object? Couldn’t seem to find an constructor/create function when doing dir(torch._C.Node).</p>

username: driazati
cooked: <p>This is purposefully not exposed, the only way to get a graph is by compiling code with <code>torch.jit.script</code> or tracing with <code>torch.jit.trace</code>. Potentially you could extract the code you are compiling initially with the Python <code>inspect</code> module and edit it that way, so it would be changed in the way you want before TorchScript sees it.</p>
<p>You could also drop down to C++ (and potentially use PyBind to make it callable from Python) and write a custom pass that operates on a <code>Graph</code> object, see <a href="https://jott.live/markdown/Writing%20a%20Toy%20Backend%20Compiler%20for%20PyTorch" rel="nofollow noopener">this tutorial</a> for details.</p>
Implement a custom function inside the model

username: Alex_K
cooked: <p>Hello everyone!</p>
<p>I am trying to implement the function inside the model, however, no progress so far… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
I have a function which takes three arguments, one of them is an input (image transformed to PyTorch tensor) and two more numbers and returns processed image transformed to PyTorch tensor:</p>
<pre><code class="lang-auto">def process_image(image_in, variable_1, variable_2):

  img = image_in.cpu().detach().numpy()

  #do whatever to get img_out

  img_out = transforms.ToTensor()(img_out)

  return img_out
</code></pre>
<p>then I am trying to add the function in the simple NN:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self):
      super().__init__()
      self.conv1 = nn.Conv2d(3, 16, 3, 1, padding=1)
      self.conv2 = nn.Conv2d(16, 32, 3, 1, padding=1)
      self.conv3 = nn.Conv2d(32, 64, 3, 1, padding=1)
      self.fc1 = nn.Linear(1024, 300)
      self.dropout1 = nn.Dropout(0.5)
      self.fc2 = nn.Linear(300, 10)
    def forward(self, x):
      x = process_image(x, variable_1, variable_2)
      x = F.relu(self.conv1(x))
      x = F.max_pool2d(x, 2, 2)
      x = F.relu(self.conv2(x))
      x = F.max_pool2d(x, 2, 2)
      x = F.relu(self.conv3(x))
      x = F.max_pool2d(x, 2, 2)
      x = x.view(-1, 1024)
      x = F.relu(self.fc1(x))
      x = self.dropout1(x)
      x = self.fc2(x)
      return x
</code></pre>
<p>I guess I am doing it wrong, because, so far, Colab just crashes without any particular explanation… <img src="https://discuss.pytorch.org/images/emoji/apple/confounded.png?v=9" title=":confounded:" class="emoji" alt=":confounded:"><br>
Also, I’d like to make <code>variable_1</code> and <code>variable_2</code> trainable. How can I do that?</p>
<p>Thanks in advance for helping a newbie!</p>

username: ptrblck
cooked: <p>Could you run the code locally on your machine to get a proper error message?<br>
Note that all operations applied on a <code>numpy</code> array won’t be tracked by Autograd, if you don’t implement the <code>backwards</code> method manually.<br>
However, if you use <code>variable_1</code> and <code>variable_2</code> on your tensor, it should work.</p>

username: Alex_K
cooked: <p>Thanks a lot!<br>
In this case, shall I implement <code>backwards</code> method, how described here? <a href="https://discuss.pytorch.org/t/defining-backward-function-in-nn-module/5047" class="inline-onebox">Defining backward() function in nn.module?</a></p>

username: ptrblck
cooked: <p>If you cannot use PyTorch methods and would need to use numpy functions, then you would need to implement the <code>backward</code> manually.<br>
However, if all needed operations can be called in PyTorch, the backward pass will just work.</p>
torch.nn.Sigmoid vs torch.sigmoid

username: Rohan_Kumar
cooked: <p>what is the difference between these 2 sigmoid functions?</p>

username: alex.veuthey
cooked: <p>AFAIK, <code>torch.nn.Sigmoid</code> calls <code>torch.nn.functional.sigmoid</code> in the background, and according to <a href="https://discuss.pytorch.org/t/is-there-any-different-between-torch-sigmoid-and-torch-nn-functional-sigmoid/995/3">this answer</a>, the functional and <code>torch.xxx</code> calls differ with their backwards implementation (which is more efficient and GPU capable in the <code>torch.nn</code> case).</p>

username: KFrank
cooked: <p>Hi Rohan!</p>
<aside class="quote no-group" data-username="Rohan_Kumar" data-post="1" data-topic="57691" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_kumar/48/10228_2.png" class="avatar"> Rohan_Kumar:</div>
<blockquote>
<p>what is the difference between these 2 sigmoid functions?</p>
</blockquote>
</aside>
<p><code>torch.nn.Sigmoid</code> (note the capital “S”) is a <em>class</em>.  When you<br>
instantiate it, you get a <em>function object</em>, that is, an object that you<br>
can call like a function.  In contrast, <code>torch.sigmoid</code> is a function.</p>
<p>From the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#Sigmoid" rel="noopener nofollow ugc">source code</a> for <code>torch.nn.Sigmoid</code>, you can<br>
see that it calls <code>torch.sigmoid</code>, so the two are functionally<br>
same.</p>
<p>(Why even have such a class / function object?  Because,<br>
although it isn’t the case for <code>Sigmoid</code>, in many cases when<br>
you construct a pytorch function object you can pass in<br>
parameters to the constructor that control the behavior of the<br>
function.  This is useful in cases where where the caller isn’t<br>
able (or it might just be annoying) to pass in those parameters<br>
when actually calling the function.)</p>
<p>As far as Alex’s comment, he references<br>
<code>torch.nn.functional.sigmoid</code>, which is (probably) different<br>
than <code>torch.sigmoid</code>.  (Again, in any event, directly from the<br>
source code, <code>torch.nn.Sigmoid</code> calls <code>torch.sigmoid</code>, so<br>
the two are functionally the same.)</p>
<p>As for the post Alex links to, about<br>
<code>torch.nn.functional.sigmoid</code> having a different backwards<br>
implementation than <code>torch.sigmoid</code>, I suspect that this is out<br>
of date (or perhaps just  incorrect).  Its documentation shows<br>
that it’s deprecated in favor of <code>torch.sigmoid()</code> and that is<br>
calls <code>input.sigmoid()</code>.  I very much doubt that<br>
<code>torch.nn.functional.sigmoid</code> and <code>torch.sigmoid</code><br>
do anything different (but since I can’t find any code for<br>
<code>torch.sigmoid</code> or <code>tensor.sigmoid</code> it’s hard to tie this<br>
up into a neat, little package).</p>
<p>Good luck.</p>
<p>K. Frank</p>
How to Set Learning Rate and Batch Size in Data Parallel

username: Rafael_R
cooked: <p>Hi,</p>
<p>I am using data-parallel across two GPUs.</p>
<p>How to set my batch size and learning rate, as my loss in not decreasing, while in one gpu it decreased?</p>
<p>Are there any rules of thumb to do this?</p>
<p>Specifically, learning rate and batch size.</p>
<p>Also, my second GPU is not used probably:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.88       Driver Version: 418.88       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:26:00.0 Off |                  N/A |
| 27%   62C    P2    76W / 280W |   9523MiB / 11175MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  On   | 00000000:27:00.0 Off |                  N/A |
|  0%   31C    P8    11W / 280W |     10MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     24     C   python                                      9513MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>How to make sure the second gpu is also used? I am wrapping my model in data parallel.</p>
Weird performance differences with RandomSampler/torch.randperm

username: bmenn
cooked: <p>Not sure what I am doing wrong but I have had some weird issues where if I use <code>torch.utils.data.RandomSampler</code> with <code>torch.utils.data.DataLoader</code> (i.e. <code>DataLoader(shuffle=True)</code>) I get dramatically different F1 scores when I load identical data from different locations. One location is a local drive and the other is a NFS drive. Both locations contain exactly the same files when checked with md5sum over each one. I found that if I create my own version of <code>RandomSampler</code> (below) I no longer see the difference in performance. I think the issue might be with <code>randperm</code> if my code is correct (the issue exists on both CPU and GPU).</p>
<pre><code class="lang-auto">class RandomSampler(torch.utils.data.RandomSampler):
    def __iter__(self):
        n = len(self.data_source)
        if self.replacement:
            return iter(np.random.randomint(0, n, size=n).tolist())
        arr = np.arange(n)
        np.random.shuffle(arr)
        return iter(arr.tolist())
</code></pre>
<p>I would post a minimal working example here but not sure how to reduce my working code (dataset is &gt;1GB as well) to reproduce.</p>

username: ptrblck
cooked: <p>Is this effect reproducible, i.e. how many times have you trained your model from the local drive and the NFS drive?</p>

username: bmenn
cooked: <p>I’ve been able to reproduce it dozens of times now (took me a long time to figure out this was the difference). Very puzzled as to why.</p>
Cannot override torch.round() after upgrading to the latest pytorch version

username: Bohan_Zhuang
cooked: <p>I tried to override the torch.round() function and it worked well before I upgraded my pytorch version to 0.2.0.  I don’t know what’s going wrong with it and need some help!</p>
<p>##<span class="hashtag">#-------------------------------------------------------------</span></p>
<pre><code class="lang-python">import torch

class RoundNoGradient(torch.autograd.Function):
    @staticmethod
    def forward(x):
        return x.round()
    @staticmethod
    def backward(g):
        return g 


m = torch.nn.Conv2d(16, 10, 3, stride=2)
l = torch.nn.L1Loss()
input = torch.autograd.Variable(torch.randn(20, 16, 50, 10), requires_grad=True)
x = RoundNoGradient()(input)
y = m(x)
output = l(x, torch.autograd.Variable(torch.randn(20, 16, 50, 10)))
output.backward()
print(x.grad)
</code></pre>
<p>##<span class="hashtag">#-------------------------------------------------------------------------------</span><br>
And it reports:</p>
<pre><code class="lang-auto">  File "test.py", line 18, in &lt;module&gt;
    output.backward()
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 156, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 98, in backward
    variables, grad_variables, retain_graph)
TypeError: forward() takes exactly 1 argument (2 given)
</code></pre>

username: colesbury
cooked: <p>If your <code>forward()</code> call in your function is labelled with <code>@staticmethod</code> it needs to be an new-style autograd function which takes a <code>ctx</code> as the first argument:</p>
<pre><code class="lang-python">class RoundNoGradient(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x.round()
    @staticmethod
    def backward(ctx, g):
        return g 
</code></pre>
<p>then to call it:</p>
<pre><code class="lang-auto">input = torch.autograd.Variable(torch.randn(20, 16, 50, 10), requires_grad=True)
x = RoundNoGradient.apply(input)
</code></pre>
<p>However, you won’t be able to get <code>x.grad</code> because it’s an intermediate variable. You can access <code>input.grad</code> though:</p>
<pre><code class="lang-auto">print(input.grad)
</code></pre>

username: Bohan_Zhuang
cooked: <p>Thx for your instructive answer!!</p>
KL-div between 2 multivariate Gaussians without for loop

username: shivangi
cooked: <p>Hi everyone,<br>
I have 2 multivariate Gaussians and I want to compute KL-divergence between them. The shape of <code>mu1, mu2, std1, std2</code> is <code>(batch_size, 128)</code>. Currently I am computing this using for loop to cmpute this. Can this be done in a vectorized way?</p>
<pre><code class="lang-auto">def compute_kl_div(mu1, mu2, std1, std2):
    kl_sum = 0
    for i in range(batch_size):
        cov1 = torch.diag(std1[i])
        cov2 = torch.diag(std2[i])
        a = torch.logdet(cov1) - torch.logdet(cov2)
        cov2_inv = torch.matrix_power(cov2, -1)
        b = torch.trace(torch.mm(cov2_inv, cov1))
        c = torch.mm(torch.mm((mu2[i] - mu1[i]).unsqueeze(0), cov2_inv), (mu2[i] - mu1[i]).unsqueeze(1))
        kl_sum += (a - 64 + b + c)
    return 0.5*kl_sum
</code></pre>

username: tom
cooked: <p>Every of the functions seems to be batcheable or have batcheable equivalents, so you could just convert it line by line.<br>
But then, it seems that the computation by explicitly constructing the diagonal matrices is excessively inefficient – logdet or inverse are easy for diagonal matrices and hard for general ones – and I would recommend revisiting the maths here. (The GP people do a lot of these calculations, so <a href="https://github.com/cornellius-gp/gpytorch" rel="nofollow noopener">GPyTorch</a> or <a href="https://github.com/t-vi/candlegp/blob/master/candlegp/kullback_leiblers.py" rel="nofollow noopener">my ancient CandleGP</a> will have plenty of examples, even if the latter doesn’t quite have what you want.</p>
<p>It may be convenient to use <code>einsum</code> for the summations.</p>
<p>Best regards</p>
<p>Thomas</p>
Deep Contractive Autoencoder

username: enyen
cooked: <p>Hi All,</p>
<p>I am trying to implement <a href="http://www.icml-2011.org/papers/455_icmlpaper.pdf" rel="nofollow noopener">Contractive Autoencoder</a> which require gradient in loss function.<br>
When the getting jacobian is not as easy as getting the weights in multi-layer encoder, which option would you recommend?</p>
<p><strong>Option 1:</strong><br>
optim_rec = SGD([enc.parameters(), dec.parameters()])<br>
optim_con = SGD(enc.parameters())</p>
<pre><code># reconstruction loss
y = enc(x)
x_ = dec(y)
loss = smoothL1(x, x_)
optim_rec.zero_grad()
loss.backward()
optim_rec.step()

# contractive loss
x.requires_grad = True
y = enc(x)
optim_con.zero_grad()
y.backward(ones(y.size()))
x.grad.requires_grad = True
x.grad.volatile = False
loss = mean(pow(x.grad, 2))
optim_con.zero_grad()
loss.backward()
optim_con.step()
</code></pre>
<p><strong>Option 2</strong><br>
optim = SGD([enc.parameters(), dec.parameters()])</p>
<pre><code># losses
x.requires_grad = True
y = enc(x)
x_ = dec(y)
optim.zero_grad()
y.backward(ones(y.size()), retain_graph=True)
x.grad.requires_grad = True
x.grad.volatile = False
loss = [mean(pow(x.grad, 2)), smoothL1(x, x_)]
optim.zero_grad()
sum(loss).backward()
optim.step()
</code></pre>
<p>As recommeded in Autograd document, option 1 refrained from using retain_graph but requires 2 separate forward passes and optimizers. Both options work for me but I would like to know if there are more elegant methods to use Autograd.</p>
<p>Thankss.<img src="//discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: vaan
cooked: <p>One way is to use a stacked AE and train layer by layer and include the jacobian norm in each loss function</p>

username: Shisho_Sama
cooked: <p>Are you sure this method works? because I dont see any Forbineous norm calculation done here!<br>
Also I noticed using your method, the loss goes haywire! the reconstruction works regardless of the second loss, so I’d be very grateful if you could explain this and share with us the code you used for multi-layerd CAE.</p>
Dropping Python 2 support on January 1st, 2020

username: smth
cooked: <p>Almost all major open source Python packages now support both Python 3.x and Python 2.7, and many projects have been supporting these two versions of the language for several years. While we have developed tools and techniques to maintain compatibility efficiently, it is a small but constant friction in the development of a lot of code.</p>
<p>We are keen to use Python 3 to its full potential, and we currently accept the cost of writing cross-compatible code to allow a smooth transition, but we don’t intend to maintain this compatibility indefinitely. Although the transition has not been as quick as we hoped, we do see it taking place, with more and more people using, teaching and recommending Python 3.</p>
<p>The developers of the Python language extended support of Python 2.7 from 2015 to January 1, 2020, recognising that many people were still using Python 2. We believe that the extra 5 years is sufficient to transition off of Python 2, and our projects plan to stop supporting Python 2 when upstream support ends in 2020, if not before. We will then be able to simplify our code and take advantage of the many new features in the current version of the Python language and standard library.</p>
<p>In addition, significantly before 2020, many of our projects will step down Python 2.7 support to only fixing bugs, and require Python 3 for all new feature releases. Some projects have already made this transition. This too parallels support for the language itself, as Python 2.7 releases only include bugfixes and security improvements.</p>
<p>Third parties may offer paid support for our projects on old Python versions for longer than we support them ourselves. We won’t obstruct this, and it is a core principle of free and open source software that this is possible. However, if you enjoy the free, first party support for many projects including the Scientific Python stack, please start planning to move to Python 3.</p>
<p>For all of these reasons, we have pledged to  <strong>drop support for Python 2.7 no later than Jan 1st, 2020</strong> , coinciding with the Python development team’s <a href="https://www.python.org/dev/peps/pep-0373/#update" rel="nofollow noopener">timeline for dropping support for Python 2.7</a>.</p>

username: smth
action_code: pinned_globally.enabled
Zero padding prior to torch.fft

username: Carsten_Ditzel
cooked: <p>What is the most efficient way of zero padding a multi dimensional signal before using torch.fft?</p>
<p>the respective module does not consider this argument so I am wondering if placing said signal into a zero tensor at certain positions is the most suitable way of mimicking zero-padding</p>
<p>Can somebody share their thoughts on this please?</p>

username: ptrblck
cooked: <p>Placing the tensor into another tensor initialized with zeros should work.<br>
However, the more straightforward way would probably be to use <code>F.pad</code> instead.<br>
Would that work for you?</p>

username: Carsten_Ditzel
cooked: <p>thank you I will try that out</p>
The same model works differently in Python and libtorch（C++），I don‘t know why）

username: laomaup
cooked: <p>Hello, the classifier trained by Resnet50 works well in python. The same test data has poor classification accuracy in C++（libtorch）. I don’t know why and this is my main code.<br>
//read image<br>
srcImage = cv::imread(strFile, cv::ImreadModes::IMREAD_COLOR);<br>
cv::cvtColor(srcImage, traImage, cv::COLOR_BGR2RGB);<br>
cv::resize(traImage, resImage, cv::Size(224, 224),0, 0, CV_INTER_LINEAR);<br>
resImage.convertTo(detImage, CV_32F, 1.0 / 255, 0);</p>
<pre><code>   // Image conversion 
    std::vector&lt;int64_t&gt; sizes = {1, detImage.rows, detImage.cols, 3};
    at::Tensor tensorImage = torch::from_blob(detImage.data, at::IntList(sizes), at::kByte);
    tensorImage = tensorImage.to(at::kFloat);
    tensorImage = tensorImage.permute({0, 3, 1, 2});
    tensorImage = tensorImage.to(torch::kCUDA);
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.emplace_back(tensorImage);
    
    //calculate
    at::Tensor result = ptModule-&gt;forward(inputs).toTensor();
    auto max_result = result.max(1,true);</code></pre>

username: tom
cooked: <p>The Imagenet-trained models expect inputs that are standardized (to mean=0, std=1 per color over all of imagenet training).<br>
You didn’t show the Python inference code, but that would look like it is missing in the C++ code.<br>
(Generally, I’d recommend to check you’re using the same inputs when you wonder why the outputs are different.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: laomaup
cooked: <p>Thank you. I’ll check the input data is same or not.</p>