akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Partial Backprop and Weight Update

username: Novak
cooked: <p>I am trying to implement a model based on the architecture in <a href="https://arxiv.org/pdf/1906.07651.pdf" rel="nofollow noopener">Scheduled Sampling for Transformers</a>, and I’m getting lost in the details. Figure 1 in that paper is an excellent illustration of what they’re doing, in the context of a <a href="https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf" rel="nofollow noopener">Transformer model</a>:</p>
<ul>
<li>
<p>They make a single encoder taking inputs and providing an encoder output.</p>
</li>
<li>
<p>They make the first of two decoders, taking the encoder outputs and conditioning on ground truth/teacher forcing data, and provide token estimates based on that teacher forcing data.</p>
</li>
<li>
<p>They make the second of two decoders, taking identical encoder inputs but now conditioning on a token-by-token probabilistic mixture of the ground truth data and the previously generated estimates.</p>
</li>
</ul>
<p>(This is a convolutional variant of Scheduled Sampling, of course.)</p>
<p>Here is the critical trick:  The two decoders share identical weights, but in the words of the paper, they: “Only backpropagate through the decoder which makes the final predictions, based on mix between the [ground truth] target and the model predictions.”</p>
<p>I’m a little confused as to how to meet both of those criteria with PyTorch.</p>
<ol>
<li>
<p>I think I can do this by making two distinct generator objects, setting <code>requires_grad = False</code> for the first decoder, and copying the weights from the second at each step-- is this correct?</p>
</li>
<li>
<p>But I feel like I’m missing some clever way to do this all with only one copy of the decoder resident in memory.</p>
</li>
</ol>
Model(*input_tensors) vs model(input_tensors.permute(0,3,1,2))

username: Upgrade_Yourself
cooked: <p>As the title says, are these two ways of permuting the dimensions to be (batch,channel,height,width)? Or does the first syntax have nothing to do with this?<br>
Thank you</p>

username: JuanFMontesinos
cooked: <p>First syntax is python stuff.<br>
In general you can pass variable inputs in a python function. This is enabled by setting a starred variable.</p>
<pre><code class="lang-auto">def dummy(x):
</code></pre>
<p>This function will accept a single input</p>
<pre><code class="lang-auto">def dummy(*x)
</code></pre>
<p>This function will accept any amount of inputs.</p>
<p>This start is used in that case to indicate variable amount of inputs, in the rest of cases it is used to unroll an iterable.</p>
<pre><code class="lang-auto">def dummy_star(*args):
    print(args)


def dummy(args):
    print(args)

dummy(7)

dummy_star(7,8)
# dummy(7,8) # Throws error as only a single input is allowed

list_of_inputs = [7,8,9]

dummy_star(*list_of_inputs,*list_of_inputs)
print('here you are unrolling list of inputs twice,thats why args is a tuple of ints')
dummy_star(list_of_inputs,list_of_inputs)
print('here you arent unrolling list of inputs twice,thats why args is a tuple of lists')

import torch
list_tensors = torch.rand(2,1)
dummy_star(*list_tensors,*list_tensors)
dummy_star(list_tensors,list_tensors)
</code></pre>
<p>Simply if you star a tensor you will iterate over dim 0 and will send each tensor in dim 0 as an input tensor<br>
Btw: <a href="https://www.geeksforgeeks.org/args-kwargs-python/" rel="nofollow noopener">https://www.geeksforgeeks.org/args-kwargs-python/</a></p>

username: Upgrade_Yourself
cooked: <p>Thank you for your answer. I understand. So basically the asterix syntax won’t permute the dimensions but will give the model the input images one by one. What I don’t understand though is why model(*inputs) gives me an error but some coders use it. If you could tell me in what instance it works, that would help. Thanks</p>

username: JuanFMontesinos
cooked: <p>Because it depends on how the forward were coded. I still recommend you to read about *args and **kwargs but let’s illustrate it with an example.</p>
<p>if you define your forward function like:</p>
<pre><code class="lang-auto">def forward(*inputs):
   # Here inputs is a list.
   x = convolution(inputs[0])
   y = convolution(inputs[1])
   c = x+y 
   return c

output = model(a,b) # here you pass directly a and b
</code></pre>
<p>without the starring you would have to pass a list</p>
<pre><code class="lang-auto">def forward(inputs):
   # Here inputs is a list.
   x = convolution(inputs[0])
   y = convolution(inputs[1])
   c = x+y 
   return c

output = model([a,b]) # here you pass a list with a and b as forward only expects a single input
</code></pre>
<p>But still, google *args and **kwargs cos you are trying to understand a python feature a pytorch rule</p>

username: Upgrade_Yourself
cooked: <p>I googled it. Thank you for your answer.</p>
Make the gradient jacobian matrix calculation between two images faster

username: Sam3
cooked: <p>Hallo,</p>
<p>i try to make the gradient jacobian matrix calculation between two images faster, i want to apply difference approximation but i did not see any example about that.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you explain in more details what you mean by “gradient jacobian matrix calculation between two images” ?</p>

username: Sam3
cooked: <p>I mean that i want to calculate the gradient for each pixel from output image according to each pixel from input image so we have as result (256<em>256</em>256*256) value</p>

username: albanD
cooked: <p>Well, just the output matrix is going to be very expensive as it will take 4GB. So working with it will require a machine with a lot of memory.</p>
<p>But to go back to you original question, there is nothing much better than repetitive call to <code>autograd.grad</code> I’m afraid. As automatic differentiation only gives a row of the Jacobian at a time.</p>

username: Sam3
cooked: <p>autograd.grad   take alot of time  maybe 2 dayes! for this reaseon i want to make this calulation for the matrix output faster</p>

username: albanD
cooked: <p>I’m afraid this is “expected”. autograd.grad only uses automatic differentiation. And automatic differentiation is not built to compute full Jacobians. It has to construct them row by row.<br>
But since it;s so big in your case, it takes a very long time.</p>
<p>Note as well that even if you manage to compute the Jacobian matrix, since it will take 4GB just to store it, you definitely won’t be able to do any autograd with it. And you won’t be able to do much operations on it before running out of memory I’m afraid.</p>
Equivalent of model.eval()

username: EricSpeidel
cooked: <p>In trying to understand what is really going on when switching from .train() to .eval() mode, I’ve been experimenting with replacing it with the following loop within my model:</p>
<pre><code class="lang-auto">for module in self.children():
            module.training = False
</code></pre>
<p>Is this <em>exactly</em> the same as calling model.eval() from outside though? A few tests with BatchNorm and Dropout seem to look good enough; but if not, what would be required to fully replace .eval?</p>
<p>Note: This is just for understanding, I don’t think there is any benefit in doing so <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: albanD
cooked: <p>You would need to call this recursively on all the children as well if you want this to work for models with nested Modules.<br>
You can check the actual implementation <a href="https://github.com/pytorch/pytorch/blob/52b8a52e4dfcbc074be9c743bde2a8f86aa3c0e1/torch/nn/modules/module.py#L1045-L1078">here</a> if you’re curious.</p>
How to save PyTorch model in caffe2 format?

username: Marat
cooked: <p>I failed to properly load model in ONNX format possibly due to lack of support of 0.0.3 ONNX IR  in TensorRT. So I wounder is there esiest way to dump PyTorch model in caffe2 format?</p>

username: albanD
cooked: <p>Hi,</p>
<p>They are quite different libraries, so I don’t think you can do the transfer without onnx I’m afraid.</p>

username: Marat
cooked: <p>Somehow I have caffe2 library, I also “understand” how to generate pb (I am using convert-onnx-to-caffe2 tool) files but I do not know how to use these files in TensorRT parser</p>

username: albanD
cooked: <p>Sorry what I meant above was that you cannot transfer from pytorch to caffe2 without onnx.<br>
For TensorRT, have you checked the nvidia converter such as <a href="https://github.com/NVIDIA-AI-IOT/torch2trt">this</a> ?</p>

username: Marat
cooked: <p>I did not see this tool yet. Can I dump TensorRT engine and later load it in C++ code?</p>

username: albanD
cooked: <p>I’m afraid I don’t know. It might be mentioned on the repo?<br>
cc <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>If you can convert your model using this repo, you should be able to export the engine using:</p>
<pre><code class="lang-python">with open('model.engine', 'wb') as f:
    f.write(model_trt.engine.serialize())
</code></pre>

username: Marat
cooked: <p>I tried solution you offer to me and must claim that it has its own new and enchanting issues. Nevertheless <a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/ptrblck">@ptrblck</a>  thank you. I hope this tool will help me to transfer my model to TRT.</p>
Change first ResNet Layer for non Image Data

username: Filos92
cooked: <p>Hey Folks,<br>
My problem is this.<br>
Currently I have an almost arbitrarily large dataset with which I want to train a common network architecture e.g. Resnet18.<br>
My problem now is that this data set does not consist of images but only of single variables. So I have to get from a 10x1 to 3x224x224 somehow.</p>
<p>should i create a first new Layer like this:</p>
<pre><code class="lang-auto">class Resnet_Input(nn.Module):
    def __init__(self):
        super(Resnet_Input, self).__init__()
        self.fc1 = nn.Linear(10, 224*224*3)
    def forward(self, data):
        x = F.relu(self.fc1(data))
        x = x.view(-1, 3, 224, 224)
        return x
</code></pre>
<p>and stack it with nn.Sequential( Resnet_Input, ResNet18), should i modify the first layer or should i add a complete new “Residual Block”? What du you think is the best way? Do you have better ideas?</p>

username: Eta_C
cooked: <p><code>10x1</code> to <code>3x224x224</code>?<br>
That sounds crazy!</p>
<p>Referring to the structure of resnet, I think you can try to build a new network use <code>nn.Conv1d</code>, <code>nn.BatchNorm1d</code> and so on…</p>
<p>But in my understanding, if your input data have no spatial correlation, that’s no need to use convolution…(I cannot guarantee my opinion)</p>

username: Filos92
cooked: <p>First I want to check some baselines.<br>
how well do established networks work? Can I do this with more simple Networks? I also plan with going for GAN Networks (<a href="https://arxiv.org/abs/1905.12868" rel="nofollow noopener">https://arxiv.org/abs/1905.12868</a>)</p>
<p>The thing is, all my previously trained networks have been using datasets between 4000 and 8000 images. Thereby I was able to play with different parameters, because a training didn’t last very long. And after a lot of different Hyperparameters I figuered out a good Network Strukture.<br>
Now I want to test datasets larger then 300 00 to 1 Million datapoints. Training will be last a lot longer. So I want to start with a established Network, to look where am I</p>
<p>Is this the wrong way to deal with such datasets?</p>

username: ptrblck
cooked: <p>The main question would be: how would you like to “reshape” your 10 values to a tensor of <code>[3, 224, 224]</code>?<br>
I.e. you could somehow interpolate the values, just repeat them, fill with some other information?</p>

username: Filos92
cooked: <p>I will never reach 3x224x224<br>
But to fill it up with the same values could be a good idea, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Filos92
cooked: <p>Okay, maybe going for ResNet isn’t the best solution.<br>
But I did not find any good ideas how to train a Network with only two inputs and one output.<br>
Any good ideas here?</p>
Is there a way to write a custom BCE loss in pytorch?

username: alabijesujoba
cooked: <p>I am writing a custom BCE in pytorch but in some cases it returns -inf and nan most cases. Which is due to the log function.</p>
<pre><code class="lang-auto">bce_loss=y_true*torch.log2(y_pred) + (one_torch-y_pred)*torch.log2(one_torch-y_pred)
</code></pre>
<p>Is there a way to rewrite this? Note  y_pred is a sigmoid output which is between 0 and 1.</p>

username: KFrank
cooked: <p>Hello Jesujoba!</p>
<aside class="quote no-group" data-username="alabijesujoba" data-post="1" data-topic="64260" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alabijesujoba/48/18493_2.png" class="avatar"> alabijesujoba:</div>
<blockquote>
<pre><code class="lang-auto">bce_loss=y_true*torch.log2(y_pred) + (one_torch-y_pred)*torch.log2(one_torch-y_pred)
</code></pre>
<p>Is there a way to rewrite this?</p>
</blockquote>
</aside>
<p>It is certainly possible to write your own BCE loss function.</p>
<p>Two comments:</p>
<p>First, the second term in what you’ve posted has <code>y_pred</code> twice<br>
and no <code>y_true</code>.  That might just be a typo, but if it’s like that in<br>
your actual code, it won’t work.</p>
<p>Second, to reduce the likelihood of <code>nan</code>, it his better to modify<br>
your BCE loss to take a <em>logit</em> (from -infinity to infinity), rather<br>
than a probability (from 0 to 1).</p>
<p>See, for example, pytorch’s <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>.</p>
<p>This allows you to combine together, internally to your loss<br>
function, the <code>sigmoid</code> and <code>log</code>, and use the <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations" rel="noopener nofollow ugc">“log-sum-exp trick”</a><br>
to improve numerical stability.</p>
<p>By the way, what is your motivation in writing your own BCE<br>
loss function?</p>
<p>Best.</p>
<p>K. Frank</p>
MSE ouput different compared CV2 computation

username: Mrutuynjay_Wanjerkhe
cooked: <p>Hi,</p>
<p>Comparing the error value between two images using CV2 and pyTorch shows different results. Here is my code. Could you let me know what is problem here?</p>
<pre><code class="lang-auto">T = transforms.Compose([transforms.ToTensor()])
P = transforms.Compose([transforms.ToPILImage()])

def ComputFullImgError(in_img,out_img):

    inImg = T(Image.open(in_img))
    outImg = T(Image.open(out_img))

    inImg = inImg.unsqueeze(0)
    print(inImg.size())

    outImg = outImg.unsqueeze(0)
    print(outImg.size())

    l1_error = torch.nn.L1Loss()
    print('L1 Error',l1_error(inImg,outImg))

    mse_loss = torch.nn.MSELoss()
    print('MSE', mse_loss(inImg,outImg) )

def CVError(in_img,out_img):
    imageA = cv2.imread(in_img)
    imageB = cv2.imread(out_img)


    err = np.sum((imageA.astype("float") - imageB.astype("float")) ** 2)
    err /= float(imageA.shape[0] * imageA.shape[1] * imageA.shape[2])

    print('CV',err)

ComputFullImgError('input.png','target.png')
CVError('input.png','target.png')
</code></pre>
<p>Result:</p>
<pre><code class="lang-auto">torch.Size([1, 3, 1080, 1920])
torch.Size([1, 3, 1080, 1920])
L1 Error tensor(0.0103)
MSE tensor(0.0006)
CV 41.84295669367284
</code></pre>

username: JuanFMontesinos
cooked: <p>Don’t use nn.L1loss.<br>
It does average among batches and values in a batch.<br>
Besides I think that transforms normalize images between 0-1 whereas they remain 0-255 in cv2.</p>

username: Mrutuynjay_Wanjerkhe
cooked: <p>Thanks it’ the normalization error.</p>
<p>Regards,<br>
MJay</p>
Loss calculation when using manual threshold

username: Upgrade_Yourself
cooked: <p>I have a model that does multilabel classification. After calculating the outputs in training mode, I iterate over thresholds and choose the one that gives me best fbeta_score.</p>
<p>What I want to ask is do I calculate loss based on those final predictions or do I calculate loss based on the original predictions?</p>
<p>So, is the manual threshold a post-processing step to use only when the model is in eval mode?</p>
<p>Thank you for your help</p>

username: kl_divergence
cooked: <p>We generally use threshold while inferencing. I think computing loss on the original predictions (which you model spits out) is the recommended method.</p>

username: Upgrade_Yourself
cooked: <p>The problem is when using fbeta_score, I get an error:</p>
<blockquote>
<p>"ValueError: Classification metrics can’t handle a mix of multilabel-indicator and continuous-multioutput targets</p>
</blockquote>
<p>I take it to mean that I need to binarize the predictions (so going from probabilities to 0/1).<br>
To do that I need to decide on a threshold. So, if I’m going to choose one why not choose the one that gives me the best fbeta_score?</p>
Why non-consecutive indexing does not work in 1D

username: martinr
cooked: <p>I am just curious. Why multi-index selection (e.g <code>(1,2,4)</code>) works on multidimensional tensor, but does not work on 1D tensors. Does it use different data structure? Quick example:</p>
<pre><code class="lang-auto"># 1D tensor
t1 = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8])
# 2D tensor
t2 = torch.tensor([[1,2,3,4,5,6,7,8]])

# FOR 2D
# select 1st, 3rd and 5th element from 2D
t2[:, (0,2,4)]
# output: tensor([[1, 3, 5]])

# FOR 1D, try the same
t1[(0,2,4)]
# output: IndexError: too many indices for tensor of dimension 1
</code></pre>
<p>I can still select the same values by using either torch.index_select() or by doing this:<br>
<code>t1[torch.tensor([0,2,4])]</code> or <code>t1[[0,2,4]]</code> which are the same, except wrapped in a tensor or list.</p>
<p>I would just like to understand the reasoning here, why the pattern that works in 2D+ does not work in 1D? Where’s the logic? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: albanD
cooked: <p>I’m not a specialist of advanced indexing, but I think that passing a single tuple has the meaning of “these are the coordinates of the point I want”. So in your case that would be for a 3D Tensor.</p>
<p>If you want to index a single dimension, you can do: <code>t1[(0,2,4),]</code></p>
3d convolution for CIFAR

username: Daniel_Joseph
cooked: <p>Hello ,<br>
Could anyone provide me with a simple code of 3D conv on CIFAR datasets using pytorch. Thank you in advance.</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit? The CIFAR dataset yields images, so how would you like to stack these images into volumes?</p>

username: Daniel_Joseph
cooked: <p>i want to use 3d convolution on 2d input images, on any datasets , just want to learn more from codes, its easy for me.<br>
More precisely, i want to apply a 3d kernel on inputs(h,w,c),l mean, each channel need to be convolved with kernel not all with same kernel. i.e : my datasets has 100x100x5 , wanna apply kernel size of 5 channels , not the sum of these 5 kernels as the 2D conv really is.just to preserve the same depth of the image during the convolution.<br>
I hope that makes sense.</p>

username: ptrblck
cooked: <p>Based on the description, it sounds like you would like to apply a depthwise convolution, in which case you could use <code>nn.Conv2d</code> with <code>groups=in_channels</code>.</p>

username: Daniel_Joseph
cooked: <p>yeah it is. that’s what i meant. could you help me how should i do it on CIFAr , if possible.</p>

username: ptrblck
cooked: <p>Depthwise convolutions are not dependent on your dataset, so you could simply set <code>groups=in_channels</code> in each conv layer.</p>

username: ahkarami
cooked: <p><a class="mention" href="/u/daniel_joseph">@Daniel_Joseph</a><br>
There is a great reference for understanding Different kinds of Convolution Operators (3D Convolution, Spatially separable convolution, depthwise separable convolution, etc.):<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3g6anj9jkury9.cloudfront.net/assets/favicons/favicon-16x16-c1207cd2f3a20fd50de0e585b4b307a3.png" class="site-icon" width="16" height="16">
      <a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Leanpub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:95/124;"><img src="https://d2sofvawe08yqg.cloudfront.net/convolutions-for-deep-learning/medium?1575064090" class="thumbnail"></div>

<h3><a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Deep Dive into Different Types of Convolutions for Deep Learning</a></h3>

<p>This book teaches the different types of convolution operators to design Deep Neural Networks with a variety of illustrative figures and examples. The basic concepts of the convolution operator to its advanced types are described with full details in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Daniel_Joseph
cooked: <p>Thanks for sharing , I appreciate</p>
How can I use "__cuda_array_interface__" and tensor.requires_grad_() together?

username: JTiC
cooked: <p>I use package Numba to write the forward and backward cuda operation, and use it as a module. When I use it, the input tensors, with grad required, go through the modules. The example can be described as:</p>
<pre><code class="lang-auto">def forward_cudakernel():
        ...   # write by NUMBA

def backward_cudakernel():
        ...  # write by NUMBA

def forward_cuda(x)
    with torch.no_grad():
        x = numba.cuda.as_cuda_array(x)
        forward_cudakernel[blocks, threads](x)

def backward_cuda(x)
    with torch.no_grad():
        x = numba.cuda.as_cuda_array(x)
        backward_cudakernel[blocks, threads](x)

class BaseModule(Function):
    @staticmethod
    def forward():
        ...
        return forward_cuda()
    @staticmethod
    def backward():
        ...
        return grads

class Module(nn.Module):
    def forward(feature):
        fun = BaseModule.apply
        return fun(feature)

&gt;&gt;&gt; features = torch.rand(2, 256, 20, 20).cuda()
&gt;&gt;&gt; features.requires_grad_()
&gt;&gt;&gt; net = Module()
&gt;&gt;&gt; out = net(features)
&gt;&gt;&gt; out.mean.backward()
</code></pre>
<p>Where “out” and “feature” needs grads so that the module can run. However, the attr “<strong>cuda_array_interface</strong>”（cuda.as_cuda_array related） is not available when a tensor is with gradient. It seems incompatible. So, How can I use “numba” and “pytorch” together? Thanks!</p>

username: JTiC
cooked: <p>Now, I add a line <code>x = x.data</code> in forward_cuda(), discarding the gradient before using “as_cuda_array”. I find it worked, but I don’t know if there are some potential bugs remained.</p>

username: albanD
cooked: <p>As mentioned in the error message that you get if you try to use it <a href="https://github.com/pytorch/pytorch/blob/717274c001509ffbc14189edef0f7d2ee2e5b777/torch/tensor.py#L538-L542">here</a> you should use <code>.detach()</code> and not <code>.data</code>.</p>

username: JTiC
cooked: <p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Multivariate regression problem - Parameter update

username: v-prgmr
cooked: <p>Hi all,</p>
<p>I have a multivariate regression problem in hand where a multivariate function takes in the image,pointcloud as tensors and transformation matrices as tensors and outputs the projection of the pointcloud in the image.</p>
<p>The model parameters are the 3 translational parameters of the transformation matrix.</p>
<p>I have set up a nn.module class wherein the forward undergoes a series of filtering (one such filtering example is shown below), the model has three parameters with <strong>requires_grad = True</strong>, and I am not detaching or switching over to another library in the middle.</p>
<p>Example of filtering ops i do within the forward pass</p>
<pre><code class="lang-auto">    img_2D_points_tensor_with_depth = img_2D_points_tensor_with_depth[img_2D_points_tensor_with_depth[:, 0] &lt; leftImagetensor.shape[1]]

</code></pre>
<p>When i debug the forward pass, I can see that my output from the forward pass has the flag <strong>requires_grad = True</strong> too, but after i call loss.backward(), my model parameters are not updated by the optimiser. (I use SGD)</p>
<p>What do i look for in the forward pass to check if the differentiability is lost somewhere? Does a filtering op like the one i have mentioned above break differentiability anywhere?</p>
<p>My loss function is <strong>L1loss</strong></p>
<p><strong>You can find the code below</strong></p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        #parameters of the model
        self.t = nn.Parameter(torch.randn((3,1), requires_grad=True, dtype=torch.float))

    def forward(self, Left_imgtensor, Right_imgtensor, veloScan, R_0, t_0, K_00, R_cam1_cam0, t_cam1_cam0, K_01):

        Tr_cam0_velo = torch.cat([R_cam0_velo, self.t], 1)
        homo = torch.tensor([0, 0, 0, 1]).unsqueeze(0).float()
        Tr_cam0_velo = torch.cat([Tr_cam0_velo, homo], 0)

        output = multivariate_function(Left_imgtensor, veloScan, R_0, t_0, K_00, Tr_cam0_velo)

        return output 

torch.manual_seed(42)

model = Model().to(device)
print(model.state_dict())

lr = 1e-5
n_epochs = 5

optimiser = optim.sgd(model.parameters(), lr=lr)

for epoch in range(n_epochs):
    model.train()

    yhat = model(left_imgtensor, Right_imgtensor, VeloScan, R_0, t_0, K_00, R_cam1_cam0, t_cam1_cam0, K_01)
    yhat = yhat.view(1,1,512,1392)
    loss_fn = compute_loss(yhat, y_target)
    print(loss_fn)
    for p in model.parameters():
        if p.grad is not None:
            print(p.grad.data)
    loss_fn.backward()
    optimiser.step()
    optimiser.zero_grad()
</code></pre>
Addition of SubsetSequentialSampler?

username: Neofytos
cooked: <p>Hey. I believe that this does not currently exist. If there is a way to go about this please let me know otherwise, I think it would be a good addition <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using Transformer networks for images

username: Isaac_Kargar
cooked: <p>Does anyone know any useful tutorial for Transformers in vision?</p>

username: jchaykow
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/leaderj1001/Stand-Alone-Self-Attention" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/22078438?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/leaderj1001/Stand-Alone-Self-Attention" target="_blank" rel="nofollow noopener">leaderj1001/Stand-Alone-Self-Attention</a></h3>

<p>Implementing Stand-Alone Self-Attention in Vision Models using Pytorch - leaderj1001/Stand-Alone-Self-Attention</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Isaac_Kargar
cooked: <p>Thank you. I mean the usage of “attention is all you need” paper in vision. such as sequential images or image captioning</p>
Whether I can build a new network using another network's children modules

username: VictoriaYyz
cooked: <p>I wonder if I can build a new network using another network’s children module like this:</p>
<pre><code class="lang-auto">list = [] 
modules = net.children()
for m in modules:
  list.append(m)
new_net = nn.ModuleList(list)
</code></pre>
<p>Will the new network be affected by the old network?</p>

username: jchaykow
cooked: <p><code>model2 = nn.Sequential(*list(model1.children()))</code></p>

username: VictoriaYyz
cooked: <p>Will model2 and model1 share the same parameters if model2 is builded in this way?</p>

username: Eta_C
cooked: <p>Just try it.</p>
<pre><code class="lang-python">src_net = nn.Sequential(nn.Conv2d(1, 1, 1), nn.Conv2d(1, 1, 1))
dst_net = nn.Sequential(*list(src_net.children()))

print(src_net[0].weight)
print(dst_net[0].weight)
print(id(src_net[0].weight) == id(dst_net[0].weight))  # True

src_net[0].weight.data = torch.rand(*src_net[0].weight.shape)
print(src_net[0].weight)
print(dst_net[0].weight)
</code></pre>
What is the way of initialization of torch.Tensor()

username: Aniki
cooked: <p>e.g. torch.Tensor(3,4), I will get a matrix with shape of 3,4, and I notice that the mean of this matrix is 0, but the what is the specific initialization of this operation?</p>

username: ptrblck
cooked: <p>I would recommend to stick to the factory methods, e.g. <code>torch.randn</code>, <code>torch.rand</code>, <code>torch.tensor([1, 2, 3])</code> etc.<br>
<code>torch.Tensor</code> will return an uninitialized <code>FloatTensor</code>, which might be dangerous.</p>
Equivalent of THCAtomics.cuh in c++ torch

username: palver7
cooked: <p>Hi,<br>
I am looking for the equivalent of THCAtomics.cuh in the ATen/src/TH folder and I cannot find it. I want to port a program written in CUDA to C/C++ to be used in CPU and I need the functionality of THCAtomics.cuh. Can someone help?</p>
How does one perform matrix multiplication on it's transpose in a batch?

username: drewkerwin
cooked: <p>How does one perform matrix multiplication on a matrix and it’s transpose while in a batch? And I don’t wish to loop thru the batches and perform the multiplication on each of the matrices…</p>
<p>I have a batch of matrices shaped:</p>
<pre><code class="lang-auto">x.shape = [64, 16, 1000]
</code></pre>
<p>Where</p>
<pre><code class="lang-auto">batches, k_dim, other_dim = x.shape
</code></pre>
<p>It seems like the answer is to multiply the matrices by using bmm</p>
<pre><code class="lang-auto">out = torch.bmm(x, x.t())
</code></pre>
<p>or</p>
<pre><code class="lang-auto">out = torch.matmul(x, x.t())
</code></pre>
<p>But both options raise errors about shape/size.</p>
<p>What is the proper way to do this such that I get the shape</p>
<pre><code class="lang-auto">out = [64, 16, 16]
</code></pre>

username: beaupreda
cooked: <p>Hello,</p>
<p>You can use <a href="https://pytorch.org/docs/master/tensors.html?highlight=permute#torch.Tensor.permute" rel="nofollow noopener">permute</a> to change the order of the dimensions.</p>
<pre><code class="lang-auto">    x = torch.rand(size=(64, 16, 1000), dtype=torch.float32)
    out = torch.bmm(x, x.permute(0, 2, 1))
    print(out.shape)
    # torch.Size([64, 16, 16])
</code></pre>
Why Pytorch introduce Kernel(Processor) Mechanism?

username: fulltopic
cooked: <p>I am using C++ API of pytorch and I am not familiar to python API. I have noticed the kernel mechanism of pytorch source codes. That is, to register processor for certain operation in run-time (before usage) as, for example, implemented in aten/src/ATen/core/boxing|dispatch|op_registration.</p>
<p>Why this mechanism is introduced? Would it be a common use case that the user pick one kernel(processor) from one library and pick the other kernel from another library, providing running in the save device? Is that to match requirements of some complex NN? Is that related to python API usage?</p>
<p>Thank you very much!</p>

username: albanD
cooked: <p>Hi,</p>
<p>This can happen during regular NN training for example where you would do all the preprocessing on the CPU while the neural net itself would happen on a GPU (potentially some layers on TPU?).</p>

username: fulltopic
cooked: <p><a class="mention" href="/u/alband">@albanD</a>. Thank you for the information.</p>
Why the Processor (Functor) is Wrapped in a call function?

username: fulltopic
cooked: <p>The process of a certain operation is actually executed by calling <em>KernelFunction.callxxx</em>.  Take <em>callUnboxedOnly</em> for example, the sentence that invokes the operation is:</p>
<pre><code class="lang-auto">      return (*func)(getFunctor_(), std::forward&lt;Args&gt;(args)...);
</code></pre>
<p>The <em>func</em> is <em>wrap_kernel_functor_unboxed_::call</em>, <em>getFunctor_(</em>) returns the functor that executes the real operations (e.g. lstm cell).</p>
<p>I am wondering why the functor has to  be wrapped by another static function (<em>wrap_kernel_functor_unboxed_::call</em>)? How if we move the codes in <em>wrap_kernel_functor_unboxed_::call</em> into <em>KernelFunction.callUnboxedOnly</em> so we can execute the functor directly? Why we need a <em>functor_</em> then <em>unboxed_kernel_func_</em> in <em>KernelFunction</em>?</p>
<p>Thank you very much!</p>

username: fulltopic
cooked: <ol>
<li>When the KernelFunction has been stored in registry, it lost its real type. To users, it is just some KernelFunction and OperatorKernel pointers.</li>
<li>When the call() function invoked, there are just input and return arguments specialized. As it is natural for user (e.g. NN Model) to set these parameters, while it is unreasonable for user to provide template parameters, which will make the extensibility and flexibility of the framework unfeasible.</li>
<li>If we make the facade as, for example, a simple OperatorKernel.operator()(args) and make every child class of OperatorKernel to implements these operator() function, the number of the permutation of input types and the conflicts of return type for same input types make the strategy almost impossible.</li>
<li>So we wrapped the OperatorKernel again and provided a static function to make the operations in a uniform format without consideration of inheritance tree</li>
<li>Besides operator input arguments, the call function also receive a functor argument to take advantage of functor/lambda properties. (Not sure if it is best solution)</li>
<li>For boxed/unboxed functor, the specialized wrap_kernel_functor_(un)boxed struct holds the functor signature as by parameter KernelFunctor, and the OperatorKernel input argument is static_cast-ed into KernelFunctor in static call function.</li>
<li>The call function also manipulates stack for boxed kernel.</li>
</ol>
<p>Some ideas. I’ve not convinced myself completely, just put here for discussion.</p>
Adding temporal dimension to nn.Transformer, nn.Conv2d

username: vainaijr
cooked: <p>Why does nn.Transformer, nn.Conv2d not use a temporal dimension?<br>
For example, if we want to encode a string</p>
<pre><code class="lang-auto">I ate an apple
</code></pre>
<p>then, after applying nn.TransformerEncoder, we directly get an encoded representation of each word.<br>
what if we want step by step encoding, for example, after first time step, only ‘I’ is encoded, after 2nd time step, ‘I’, ‘ate’, ‘I ate’ is encoded, after third time step, ‘I’, ‘ate’, ‘I ate’, ‘an’, ‘I ate an’ is encoded, and so on, same thing for nn.Conv2d also, in the case of images, if we want to encode a part of image at one time step, another part including the first part at 2nd time step and so on?</p>
High memory usage while building PyTorch from source

username: mahmoodn
cooked: <p>How can I reduce the RAM usage of compilation from source via <code>python setup.py install</code> command? It automatically spawns many threads and I don’t want that!</p>

username: ptrblck
cooked: <p>Try to set the <code>MAX_JOBS</code> argument to your desired max. number of jobs.</p>

username: mahmoodn
cooked: <p>Is that a shell variable?<br>
export MAX_JOBS=2<br>
?</p>

username: ptrblck
cooked: <p>Yes. You could also set it in front of the install command: <code>MAX_JOBS=2 python setup.py install</code>.</p>
Apply KMeans on MNIST dataset

username: Mark_Golberg
cooked: <p>Hello,</p>
<p>I’m trying to apply KMeans clustering on MNIST data set.</p>
<p>Please see my code below:</p>
<hr>
<p>import torch<br>
from torchvision import transforms<br>
import torchvision.datasets as datasets<br>
import matplotlib.pyplot as plt<br>
from sklearn.cluster import KMeans</p>
<h2><a name="h-1-_tasks-transformscomposetransformstotensor-2-mnist_trainset-datasetsmnistrootdata-traintrue-downloadtrue-transform_tasks-3-kmeans-kmeansn_clusters10-4-kmmodel-kmeansfitmnist_trainset-1" class="anchor" href="#h-1-_tasks-transformscomposetransformstotensor-2-mnist_trainset-datasetsmnistrootdata-traintrue-downloadtrue-transform_tasks-3-kmeans-kmeansn_clusters10-4-kmmodel-kmeansfitmnist_trainset-1"></a>[1] _tasks = transforms.Compose([transforms.ToTensor()])<br>
[2] mnist_trainset = datasets.MNIST(root=‘./data’, train=True, download=True, transform=_tasks)<br>
[3] kmeans = KMeans(n_clusters=10)<br>
[4] KMmodel = kmeans.fit(mnist_trainset)</h2>
<p>I receive the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57a5e2eed07b5fe97278531947be5fd15e11afdf.png" data-download-href="https://discuss.pytorch.org/uploads/default/57a5e2eed07b5fe97278531947be5fd15e11afdf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57a5e2eed07b5fe97278531947be5fd15e11afdf.png" alt="image" data-base62-sha1="cvmWTT5pGIgayU7dZHfuvFOOORh" width="690" height="238" data-dominant-color="F5F5F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">901×312 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can someone help please???</p>
<p>THANKS</p>

username: ptrblck
cooked: <p><code>mnist_trainset</code> is a <code>Dataset</code>, while <code>sklearn</code> works on plain bumpy arrays.<br>
You could get the underlying data as a bumpy array via:</p>
<pre><code class="lang-python">data = mnist_trainset.data.numpy()
</code></pre>
<p>and normalize it manually.</p>
Unable to interpret feature maps

username: ashi
cooked: <p>I have created network to classify emotions of the people using CAER dataset.<br>
For the model I am using resnet architecture (not the pretrained one) and trying to save feature map for the last convolutional layer of last block.<br>
I am able to save the maps but when I try to view them its quite difficult to interpret not sure whether this is expected or I am doing something wrong in either capturing or saving them.<br>
Below is the code and feature maps.</p>
<p>Visualize code</p>
<pre><code class="lang-auto">writer = SummaryWriter('runs/caer')

class LayerActivations():
    features=None

    def __init__(self,model,layer_num):
        self.hook = model.layer4[layer_num].conv2.register_forward_hook(self.hook_fn)

    def hook_fn(self,module,input,output):
        self.features = output.detach()

    def remove(self):
        self.hook.remove()

def imshow_image(image):

    mean = torch.tensor([0.370, 0.225, 0.189])
    std = torch.tensor([0.172, 0.130, 0.122])
    for img,m,s in zip(image,mean,std):
        img.mul_(s).add_(m)

    writer.add_image('Image_for_activation',img)
    writer.close()

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = checkpoint['model']
    model.load_state_dict(checkpoint['state_dict'])

    model.eval()
return model

def show_activation(train_loader,dev):
    dataiter = iter(train_loader)
    images,labels = dataiter.next()

    imshow_image(images[0:2])

    model = load_checkpoint('caer_model.pth')
    features = model.module

    conv_out = LayerActivations(features,1)

    out_pred = model(images[0:2].to(dev))
    #conv_out.remove()
    act = conv_out.features

    act = torch.nn.Upsample(size=(224,224),mode='bilinear')(act)
    for i in range(512):
        filename = './features/feature_'+str(i)+'.jpg'     
        img = act[0][i] 
        torchvision.utils.save_image(img, filename, nrow=8 )
</code></pre>
<p>Feature maps:-<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43a5eec237aab54eae02e4f915e531baa9ded69c.jpeg" alt="feature_19" data-base62-sha1="9EruYparSqqRFcwUqmvsn9o4Qs4" width="224" height="224"></p>
<p>Note:- I am able to post only one feature map.</p>

username: yunusemre
cooked: <p>I think <a href="https://github.com/ramprs/grad-cam" rel="noopener nofollow ugc">Grad Cam</a> will be useful for you. It is a visualization technique for understanding which areas are dominant for CNN model decesion. Here is an example from repo:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f110891c15d63cc94dc1764153a2960e716d86cd.png" alt="2019-12-16%2014-06-48%20ekran%20g%C3%B6r%C3%BCnt%C3%BCs%C3%BC" data-base62-sha1="yoyuRb6aSrBMGnM58PvdkW2WjRr" width="623" height="220"></p>

username: ashi
cooked: <p>ok thanks.<br>
But I wanted to know whether my code is correct or not?</p>
Excessive UserWarning output for RNN training

username: Lars12345
cooked: <p>Hi All,</p>
<p>I am brand new to Pytorch, having come from Keras, and am in the process of converting an RNN model. Because the Pytorch RNN model does not currently support recurrent dropout, which Keras does, I have turned to <a href="https://github.com/keitakurita/Better_LSTM_PyTorch" rel="nofollow noopener">https://github.com/keitakurita/Better_LSTM_PyTorch</a> . The “Better LSTM” model is giving me results comparable to Keras. But what is not working very well for me is the excessive number of “User Warnings” during training.</p>
<p>Version information:</p>
<p>sys.version: 3.7.5 (default, Oct 25 2019, 15:51:11)<br>
[GCC 7.3.0]<br>
torch.<strong>version</strong>: 1.3.1<br>
device: cuda<br>
cuda version: nvcc: NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2018 NVIDIA Corporation<br>
Built on Sat_Aug_25_21:08:01_CDT_2018<br>
Cuda compilation tools, release 10.0, V10.0.130</p>
<p>Sample output:</p>
<p>Epoch 1/500/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cudnn/RNN.cpp:1268: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().<br>
Epoch 1/500 ETA 13s Step 1/609: loss: 0.942658/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cudnn/RNN.cpp:1268: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().<br>
Epoch 1/500 ETA 11s Step 2/609: loss: 0.725145/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cudnn/RNN.cpp:1268: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().<br>
Epoch 1/500 ETA 9s Step 3/609: loss: 0.644127/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cudnn/RNN.cpp:1268: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().<br>
.<br>
.<br>
.</p>
<p>I did try modifying the “Better LSTM” code to include a call to flatten_parameters at the beginning of forward(), but it made no difference to the number of UserWarning’s that were emitted.</p>
<p>Suggestions?</p>
<p>Thanks,</p>
<p>Lars</p>
Apply transforms to 4d tensor of images in my custom data loader

username: shangeth
cooked: <p>trans =  transforms.Compose([<br>
transforms.ToPILImage(),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])<br>
])</p>
<p>i want to apply this transform to 4d tensor (n, 3, 224, 224) and get a tensor in return.</p>

username: Josh_Shubert
cooked: <p>Unfortunately some of the torchvision transforms have some limitations to what dimensionality they can handle (namely transforms intended for PIL Images). However if you just wish to normalize the data, you can use transforms.Normalize() on any N dim data given that you convert the data to a Torch Tensor in your data loader.</p>
Custom forward and backward functions in conv2d layer

username: AG1991
cooked: <p>Hello everyone,</p>
<p>I am trying to build a custom convolution layer (conv2d) where the element-wise multiplication will be later replaced with an approximate multiplication.<br>
(I know it will get very slow but it is fine I will deal with this problem later.)<br>
I have managed to re-build the forward part and it works fine but the problem is with the backward and the calculation of the gradient. I have re-written everything from scratch even the Conv2DTranspose layer. You can find the code in the link below:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/AmiraGuess/MyCutomConv2d" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/37383262?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/AmiraGuess/MyCutomConv2d" target="_blank" rel="nofollow noopener">AmiraGuess/MyCutomConv2d</a></h3>

<p>Contribute to AmiraGuess/MyCutomConv2d development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>During backward pass I get the error: <strong>" function convAppxBackward returned a gradient different than None at position 2, but the corresponding forward input was not a Variable"</strong><br>
I couldn’t figure out where is the problem, I would really appreciate any help!<br>
thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is that the signature of your forward: <code>forward(ctx, x, height, width, windows, out_channels, weight, bias, n_channels):</code> does not match the order you return the gradients in the backward: <code>return grad_input, grad_weight, grad_bias, None, None, None, None, None</code>. I guess it should be <code>return grad_input, None, None, None, None, grad_weight, grad_bias, None</code>.</p>
Correct way to calculate f-1 accuracy for multi-class segmentation

username: bing
cooked: <p>I am trying to calculate f-1 accuracy for to solve multi-class segmentation problem.I am using cross-entropy loss.<br>
Below is my code -</p>
<pre><code class="lang-auto">def dice_test(loaders,model,criterion,use_cuda):
    running_loss = 0
    total_train = 0
    accuracy = 0
    for batch_idx ,(data,target) in enumerate(loaders):
        if use_cuda:
            data,target = data.cuda(),target.cuda()
        output = model(data)
        loss = criterion(output,target)
        test_loss += test_loss + ((1 / (batch_idx + 1)) * (loss.data - test_loss))
        
        _, predicted = torch.max(output.data, 1)   
        accuracy += f1_score(target,predicted&gt;0.5,average = 'micro')
        total_train += target.nelement()
    print(accuracy/total_train)
</code></pre>
<p>But I am getting <code>ValueError: unknown is not supported</code>.<br>
My dataset is highly unbalanced. Any other right way to measure accuracy is also welcomed.</p>