akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Implementing scriptable channel masking transform for multi-channel input

username: istyle0526
cooked: <p>Hi PyTorch community,</p>
<p>I’m trying to implement a channel masking transform that runs on GPU for multi-channel spectrogram input looks something like this.</p>
<pre><code class="lang-auto">class ChannelMasking(torch.nn.Module):
    def __init__(self):
        super(ChannelMasking, self).__init__()

    def forward(self, specgrams, mask_param):

        # specgrams: 4-D spectrogram (batch, channel, freq, time)
        # mask_param: Number of channels to be masked will be uniformly sampled from [0, mask_param]
        # say batch = 4, channel = 3, freq = 64, time = 16, mask_param = 2

        # trying to build (batch, channel) shaped mask. 
        mask = [
            [True, False, True], # spectro-temporal values in channel 0, 2 will be zero from batch 0
            [False, False, True], # spectro-temporal values in channel 2 will be zero from batch 1
            [False, False, False], # nothing will be changed from batch 2
            [False, False, False]  # nothing will be changed from batch 3
        ]
        mask = mask[..., None, None]

        specgrams = specgrams.masked_fill(mask, 0)
        return specgrams
</code></pre>
<p>I’m stuck with building a mask since it’s tricky to choose the numbers of channels to be removed batch-independently without using for-loop.</p>
<p>I tried to make one with <a href="https://pytorch.org/audio/stable/_modules/torchaudio/functional/functional.html#mask_along_axis_iid" rel="noopener nofollow ugc">torchaudio.functional.make_along_axis_iid</a> but felt inappropriate since this function always masks adjacent bins (I don’t want mask consecutive channels all the time) and the chance of masking bins in the corner(first and last channel) is extremely low.</p>
<p>Thank you in advance!</p>
What happend if I don't set certain parameters with `requires_grad=False` but exclude them in optimizer params?

username: whisht
cooked: <p>Hello there. I want to finetune a pretrained model on my own datasets. And I found that one usually have to do is to set some parameters to not <code>require grad</code>. Meanwhile, I should provide the parameters needed update to optimizer. So if I don’t limit the <code>requires_grad</code> but exclude these parameters I do not want to update in optimizer, what will happend? Could this way achieve the same effect as the orignal way?</p>
<pre><code class="lang-python"># original way
for np,p in model.named_parameters():
    if np in not_update:
        p.require_grad=Fasle
params = [p for p in model.parameters() if p.requires_grad]
optimizer = Adam(params,lr=0.001)

optimizer.zero_grad()
output = model(input)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()


# my way
params = [ p for np,p in model.parameters() if np not in not_update]
optimizer = Adam(params,lr=0.001)
optimizer.zero_grad()
output = model(input)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="whisht" data-post="1" data-topic="150374">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e274bd/48.png" class="avatar"> whisht:</div>
<blockquote>
<p>So if I don’t limit the <code>requires_grad</code> but exclude these parameters I do not want to update in optimizer, what will happend? Could this way achieve the same effect as the orignal way?</p>
</blockquote>
</aside>
<p>The parameters will still get their gradients calculated and accumulated in their <code>.grad</code> attribute. The optimizer will neither update these parameters nor will it set the <code>.grad</code> attribute to zero (or <code>None</code>).<br>
If you never intend to update these parameters, you could ignore this issue. However, if you want to add these parameters to an optimizer later to train them, make sure to zero out their gradients first, as their <code>.grad</code> attributes could contain large (or even overflowing) values.</p>

username: whisht
cooked: <p>Thank. In conclusion, the graident still accumalted, but won’t update these parameters. The original way that setting <code>requires_grad</code> is for saving computation cost that the parameters is removed from gard graph. Is my standpoint right?</p>

username: ptrblck
cooked: <p>Yes, you would save the memory, an accumulation kernel, and I would also prefer the explicit way to make sure you are not running into unwanted issues as previously described (e.g. when these parameters should be updated later).</p>

username: whisht
cooked: <p>Thanks again. The explicit way is more clear and definite.</p>
A memory footprint gap between the first loop and the following loop

username: JohnsonLC
cooked: <p>I have an evaluation code like this,</p>
<pre><code class="lang-auto">"""
Load models and model parameters to the GPU
"""
for items in my_dataloader:
    """
    Load items(input data) to GPU
    """
    torch.cuda.reset_peak_memory_stats()
    loss = loss_fcn()       // models forward
    print(f'Device {torch.cuda.current_device()}: ', torch.cuda.max_memory_allocated() / 1024**2, ' MB.')
</code></pre>
<p>The ouput in terminal is:</p>
<blockquote>
<p>Device 0:  9294.86962890625  MB.<br>
Device 0:  1775.291015625  MB.<br>
Device 0:  1774.791015625  MB.<br>
Device 0:  1775.2998046875  MB.<br>
…</p>
</blockquote>
<p>The question is why there is a huge memory footprint gap between the first loop and the following loops?<br>
And I also test this during training. The output is:</p>
<blockquote>
<p>Device 0:  8377.93115234375  MB.<br>
Device 0:  6737.173828125  MB.<br>
Device 0:  6686.787109375  MB.<br>
Device 0:  6740.197265625  MB.<br>
…</p>
</blockquote>
<p>There is also a gap between the first loop and the following loops. So what’s the reason? Thanks in advance.</p>

username: eqy
cooked: <p>Are you running with <code>torch.backends.cudnn.benchmark=True</code>? It could be possible that some kernels benchmarked during the first iteration are using significant memory for workspaces but not chosen to be used for later iterations.</p>

username: JohnsonLC
cooked: <p>Yes, you are right! I did run my program with <code>torch.backends.cudnn.benchmark=True</code>.</p>
<p>After I set it False, the output in terminal is:</p>
<blockquote>
<p>Device 0:  2773.99951171875  MB.<br>
Device 0:  2781.44580078125  MB.<br>
Device 0:  2782.21533203125  MB.<br>
Device 0:  2780.44189453125  MB.<br>
…</p>
</blockquote>
<p>During training, the output is:</p>
<blockquote>
<p>Device 0:  7446.09619140625  MB.<br>
Device 0:  7954.59521484375  MB.<br>
Device 0:  7948.68896484375  MB.<br>
Device 0:  7951.12646484375  MB.<br>
…</p>
</blockquote>
<p>It looks like cudnn is attempting to spend more space in the first loop to save space and time for the following loops. Thanks very much!</p>

username: eqy
cooked: <p>Sure, and to clarify on the last point, cuDNN is not exactly trying to save memory, but in the first iteration it will try many strategies for convolution, and some of these strategies will use more temporary memory (called workspaces by cuDNN). If these strategies are not the fastest ones (benchmarking only cares about speed, not space), they will not be used on the next iteration so the temporary memory usage will go down.</p>
Best interpolation for downsampling from 1200 to 288?

username: Miguel_Campos
cooked: <p>Hello community. I have been googling for long time but I didn’t find any clear answer.</p>
<p>I have big images in 1200x1200 and I need to resize them to 288x288. What do you recommend me in order to (1) best quality and (2) best quality-time balance?</p>
<p>As far as I Know, in this cases people usually uses Image.Antialias in torchvision.transform.Resize. I saw that Image.Antialias was changed by Lancoz (supported in torchvision.transforms.InterpolationMode)</p>
<p>So I think this topic is between lanczos and bicubic?</p>
<p>Thanks all !</p>
<p>EDIT1: In my application, I’m using a convolution model that usually need to be sensible to medium and small details.</p>
<p>EDIT2: Currently I’m using the default InterpolationMode.BILINEAR</p>
<pre><code class="lang-auto">T.Resize(288),
T.RandomHorizontalFlip(0.5),
T.CenterCrop(256),
T.ToTensor(),
T.Normalize([.485, .456, .406], [.229, .224, .225])
</code></pre>
Gradient of activation array

username: Roos_Kraaijveld
cooked: <p>I am trying to get the gradient of an activation vector so that it returns an array that has the same size as the activation array and not the image.</p>
<p>I found this online and this seems the most promising, but I’m not sure how to do this without the ‘cutted model’ method.</p>
<pre><code class="lang-auto">def get_gradient(self, acts, y, bottleneck_name):
        inputs = torch.autograd.Variable(torch.tensor(acts).to(device), requires_grad=True)
        targets = (y[0] * torch.ones(inputs.size(0))).long().to(device)

        cutted_model = self.get_cutted_model(bottleneck_name).to(device)
        cutted_model.eval()
        outputs = cutted_model(inputs)

        # y=[i]
        grads = -torch.autograd.grad(outputs[:, y[0]], inputs)[0]
        
        grads = grads.detach().cpu().numpy()

        cutted_model = None
        gc.collect()

        return grads

</code></pre>
<p>does anyone have any ideas?</p>
OnnxRuntime Error

username: adaptation_mod
cooked: <p>the onnx file is generated with opset 11.<br>
I’m trying to run an onnx model with onnxruntime and I got this issue.<br>
self.ort_session = ort.InferenceSession(model_path)<br>
---------log------------------------<br>
RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:40 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&amp;, onnxruntime::TensorShapeVector&amp;, bool) gsl::narrow_cast&lt;int64_t&gt;(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,16074,16074}, requested shape:{1,16074,1}<br>
----------env--------<br>
onnxruntime-1.11.1<br>
onnx-1.11.0</p>
Torch.nn.functional.conv2d implementation source code?

username: Sairam954
cooked: <p>I need to override the conv2d of pytorch, I am looking for the source which does the convolution operation to understand how it is done.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  can you please help me find the source code? I came across this <a href="https://github.com/pytorch/pytorch/blob/c780610f2d8358297cb4e4460692d496e124d64d/aten/src/ATen/native/Convolution.cpp#L481" rel="noopener nofollow ugc">Convolution.cpp</a>, but I was not successful in finding the exact line where a convolution between parameters and input is performed.</p>

username: eqy
cooked: <p>You can find some implementations for CUDA e.g., here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/ConvolutionMM2d.cu" class="inline-onebox" rel="noopener nofollow ugc">pytorch/ConvolutionMM2d.cu at master · pytorch/pytorch · GitHub</a> although the typical “fast” implementations that are used are dispatched from the cuDNN library and not open-sourced.</p>
<p>On CPU, something similar exists here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/ConvolutionMM2d.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/ConvolutionMM2d.cpp at master · pytorch/pytorch · GitHub</a> although again in practice the “fast” implementations are in MKLDNN/oneDNN.</p>
Gradient of activation arrays

username: Roos_Kraaijveld
cooked: <p>I would like to get the directional derivative of a concept using TCAV (<a href="https://github.com/rakhimovv/tcav/blob/master/tcav.py" class="inline-onebox" rel="noopener nofollow ugc">tcav/tcav.py at master · rakhimovv/tcav · GitHub</a>).</p>
<p>In order to do this, I wrote the following code:</p>
<pre><code class="lang-auto">
  model.eval()
    tcav = {}
    for ind, (img, label) in enumerate(loader):
        img.requires_grad=True
        img = img.to(device, dtype=torch.float)
        
        output = model(img)
        
        layer_activation = activation[L] #activation array of layer L
        gradients = torch.autograd.grad(outputs=layer_activation, inputs=img,
                          grad_outputs=torch.ones(layer_activation.size()).to(device),
                          create_graph=True, retain_graph=True, only_inputs=True)[0]
        
        
        grads = normalize(gradients.cpu().detach().numpy().ravel())
        
    
    return grads

</code></pre>
<p>Afterwards I need to take the dot product of the two vectors like in the code from Kim:</p>
<pre><code class="lang-auto">
 def get_direction_dir_sign(mymodel, act, cav, concept, class_id):
        """Get the sign of directional derivative.
        Args:
            mymodel: a model class instance
            act: activations of one bottleneck to get gradient with respect to.
            cav: an instance of cav
            concept: one concept
            class_id: index of the class of interest (target) in logit layer.
        Returns:
            sign of the directional derivative
        """
        # Grad points in the direction which DECREASES probability of class
        grad = np.reshape(mymodel.get_gradient(act, [class_id], cav.bottleneck), -1)
        dot_prod = np.dot(grad, cav.get_direction(concept))
        return dot_prod &lt; 0

</code></pre>
<p>However, when I take the dot product, my vectors are not the same size. One vector is the size of my image and the other is the size of my activation array for that specific layer. What am I doing wrong?</p>
Matrix Exponential FP16 Support? Fixed order approxmation?

username: okfallah
cooked: <p>Hi all,</p>
<p>I am having problems with the memory consumption in the matrix_exp implementation in PyTorch. To my understanding, the underlying algorithm depends on matrix multiplies. Is there a reason these could not be implemented in FP16?</p>
<p>Also, the matrix_exp seems to dynamically select which order approximation to apply. This makes the memory consumption unpredictable (in some cases leading to a significant memory spike, causing CUDA to crash!). I see that there is some planning for fixing the order of approximation: <a href="https://github.com/pytorch/pytorch/blob/bd7e99cbb9e7980b89c26ae3fa5596f6e4aaebc4/aten/src/ATen/native/LinearAlgebra.cpp#L2157" class="inline-onebox" rel="noopener nofollow ugc">pytorch/LinearAlgebra.cpp at bd7e99cbb9e7980b89c26ae3fa5596f6e4aaebc4 · pytorch/pytorch · GitHub</a>. I am wondering if the only way to change this would be to build from source?</p>
IoU of predicted and target masks in mask-rcnn

username: chandlerbing65nm
cooked: <p>Here’s my mask IoU calculation:</p>
<pre><code class="lang-auto">    intersection = torch.sum((mask1 + mask2) &gt; 1, -1)
    union = torch.sum((mask1 + mask2) &gt; 0, -1)
    iou_score = intersection / (union + 1e-6)
</code></pre>
<p>I wonder if the <code>&gt;1</code> and <code>&gt;0</code> are correct for <code>intersection</code> and <code>union</code> of masks. I tried to print the values of target masks in Mask-rcnn of which have a shape of (batch, 28, 28). I’m seeing values of either <code>zeros</code> and <code>ones</code> through the whole array.</p>
<p>What does <code>zeros array</code> means, the mask with the object or an empty mask?</p>
What is the reverse for sentencepiece_numericalizer

username: Fhunmie
cooked: <p>I have trained a Sentence piece model using <code>generate_sp_model()``. I used the pre-trained model for  </code>sentencepiece_numericalizer``` to change my text to numeric values before converting to tensors. I then went ahead to train my transformer model. Now I have outputs how do I get the text back from my transformer output (i.e Decode from ids to text). For example</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torchtext.data.functional import sentencepiece_numericalizer
&gt;&gt;&gt; sp_id_generator = sentencepiece_numericalizer(sp_model)
&gt;&gt;&gt; list_a = ["sentencepiece encode as pieces", "examples to   try!"]
&gt;&gt;&gt; list(sp_id_generator(list_a))
    [[9858, 9249, 1629, 1305, 1809, 53, 842],
     [2347, 13, 9, 150, 37]]
</code></pre>
<p>How do get I <code>"sentencepiece encode as pieces", "examples to   try!"</code> back from the ids</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation, but there is no inplace operation?

username: promach
cooked: <p>For <a href="https://gist.github.com/buttercutter/b6f526c56e20f029d68e6f9041c3f5c0/17ce540da7b4e66a73a812e0a93041b52e9fd9c4#file-gdas-py-L612-L679" rel="noopener nofollow ugc">this training code</a> , why the <a href="https://pastebin.com/vzRB6dUT" rel="noopener nofollow ugc">runtime error when there is no inplace operation</a> ?</p>

username: promach
cooked: <p>The above runtime error is solved with <a href="https://github.com/buttercutter/gdas/commit/a7ca6e2a5ba319ae2be1df49906b1726baaa33e6" rel="noopener nofollow ugc">this github commit</a></p>
How to fastly seperate a tensor into k subtensor with a specific range?

username: Maxwell_Albert
cooked: <p>Here is a problem about how to sort the origin tensor into k subtensor with a restricted value range.<br>
The example is shown below.<br>
There is a list from 1 to 8.</p>
<pre><code class="lang-auto">[1,5,3,2,6,7,8,4]
</code></pre>
<p>I want to split it into 2 subtensors like,</p>
<pre><code class="lang-auto">[1,4,3,2]
[6,7,8,5]
</code></pre>
<p>I only care about the order between the subtensors but not in the subtensors.<br>
I hope to find a effienct way to do it.<br>
One way is to find kthvalue of the tensor by using <code>torch.kthvalue</code> and use <code>torch.mask_select</code> to find it.<br>
But <code>torch.kthvalue</code> could be high cost when tensor size is too big.<br>
The other way is to use shell sort. But there are no efficient shell sort in python.<br>
Could you please give me some suggestions?</p>

username: eqy
cooked: <p>What is the speed difference you are seeing with this approach compared to e.g., calling <code>topk</code> multiple times?</p>

username: Maxwell_Albert
cooked: <p>Actually if I call <code>torch.topk</code>, I need to do it multiple times.<br>
And if I want to seperate the tensor into four subtensors. k should first be set to total_size/4. And then I should mask or somehow delete the elements chose in the first run.<br>
Like</p>
<pre><code class="lang-auto">input = torch.rand([16])
src1, index1 = torch.topk(input, k = 4,sorted = False)
input = input.index_fill(index1, -1*e9)
...
</code></pre>
<p>Is there a way to select the medium part of the tensor, please?</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Maxwell_Albert" data-post="3" data-topic="150246">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maxwell_albert/48/31003_2.png" class="avatar"> Maxwell_Albert:</div>
<blockquote>
<p>Actually if I call <code>torch.topk</code>, I need to do it multiple times.</p>
</blockquote>
</aside>
<p>Did you compare the timing of <code>torch.sort()</code> with this method of calling <code>torch.topk()</code> multiple times?<br>
How does <code>torch.sort()</code> perform compared to this method?</p>

username: Maxwell_Albert
cooked: <p>Actually I have tested torch.sort.<br>
It performs better than <code>torch.topk</code> when I want to separate tensor to more subtensors.<br>
But it is still so slow at cpu.<br>
Acutally I am confused that a tensor with size12845056. Sorting it needs about 2seconds.<br>
Both <code>torch.topk</code> and <code>torch.sort</code> costs a lot more than a linear or a convolutional layer.<br>
Why this happens</p>
Is there a better solution to loading custom c++ class other than torch.classes.load_library

username: sudo42
cooked: <p>I am new to C++ extensions, so far it has been working great but the only thing that annoys me is that we need to hard code <code>torch.classes.load_library(&lt;path-to-dll&gt;)</code> in our program which is not portable. Is there a way to make class extensions the same as functions, i.e. using  <code>import &lt;class-name&gt;</code>?</p>
Torch.rfft in old PyTorch and torch.fft.rfft in new PyTorch

username: NASRIN_RAHIMI
cooked: <p>Hello,</p>
<p>I have a code in old version Pytorch (1.4) which works fine. I want to use new version of Pytorch(1/9), however, I can not get the same result using torch.fft.rfft()</p>
<p>The code in Pytorch 1.4:</p>
<pre><code class="lang-auto">import torch 
X = torch.Tensor([[1,2,3,0], [4,5,6,1],[7,8,9,2] , [10,11,12,3]])
fftfull = torch.rfft(X,2)
print("fftful {}".format(fftfull))
</code></pre>
<p>The result:</p>
<pre><code class="lang-auto">fftful tensor([[[ 84.,   0.],
         [ -8., -20.],
         [ 20.,   0.]],

        [[-20.,  20.],
         [  4.,   4.],
         [ -4.,   4.]],

        [[-20.,   0.],
         [  0.,   4.],
         [ -4.,   0.]],

        [[-20., -20.],
         [ -4.,   4.],
         [ -4.,  -4.]]]
</code></pre>
<p>The code in Pytorch 1.9:</p>
<pre><code class="lang-auto">import torch 
import torch.fft
X = torch.Tensor([[1,2,3,0], [4,5,6,1],[7,8,9,2] , [10,11,12,3]])
fftfull = torch.fft.rfft(X, dim=-1)
print("fftful {}".format(fftfull))

</code></pre>
<p>The result:</p>
<pre><code class="lang-auto">fftful tensor([[ 6.+0.j, -2.-2.j,  2.+0.j],
        [16.+0.j, -2.-4.j,  4.+0.j],
        [26.+0.j, -2.-6.j,  6.+0.j],
        [36.+0.j, -2.-8.j,  8.+0.j]])

</code></pre>
<p>Why the results are not the same. How can I get the same values in Pytorch 1.9?<br>
Thank you for your help in advance.</p>

username: NASRIN_RAHIMI
cooked: <p>The problem solved by <code>fftfull = torch.fft.rfft2(X)</code> in PyTorch v1.9.</p>
<p>The 2 in <code>fftful = torch.rfft(X,2)</code> in PyTorch 1.4 shows that we want to take 2-dimensional  fft of X, therefore the equivalent in PyTorch 1.9 will be <code>fftfull = torch.fft.rfft2(X)</code> , and <code>fftfull = torch.fft.rfft(X, dim=-1)</code> is 1d fft of last dimension of input.</p>
Transformer - get contribution matrix

username: papyrus
cooked: <p>Hi everyone,</p>
<p>I am trying to experiment with <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">Transformers</a>. After training, I would like to visualize the contribution matrix from an input. If I understood correctly, the contribution matrix is a square one, indicating the contribution of each element with other elements from the sequence (diagonal values are high). This matrix is computed as in the linked picture, Q and K being respectively the query and key matrix, and d the number of features of each element.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7f944d16b98ef76492f24120bc0b38f47eb1457.jpeg" alt="Selection_005" data-base62-sha1="nXXNcp9zcdB3BFddxnMALxrNWHZ" width="405" height="186"></p>
<p>Is there a way to easily get the result of this operation in order to visualize this contribution matrix?</p>
<p>I think that these contribution matrices can be obtained from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" rel="noopener nofollow ugc">TransformerEncoderLayer</a>. So, in the function at line 350 in this file: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/transformer.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/modules/transformer.py at main · pytorch/pytorch · GitHub</a>, I call the function <code>self.self_attn</code> with parameter <code>need_weights=True, average_attn_weights=False</code>, and then save the output_weights as tensors. Is this the right way ?</p>
<p>Thanks for the help</p>

username: JuanFMontesinos
cooked: <aside class="onebox allowlistedgeneric" data-onebox-src="https://www.juanmontesinos.com/posts/2022/02/20/transformer-activations">
  <header class="source">
      <img src="https://www.juanmontesinos.com/images/favicon-32x32.png?v=M44lzPylqQ" class="site-icon" width="" height="">

      <a href="https://www.juanmontesinos.com/posts/2022/02/20/transformer-activations" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 20 February 2022">Juan Montesinos – 20 Feb 22</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.juanmontesinos.com/posts/2022/02/20/transformer-activations" target="_blank" rel="noopener nofollow ugc">Visualizing Pytorch’s transformer activations</a></h3>

  <p>Transformers are a trendy architecture nowadays. The paralellism and possibility of working with sequences of different lenghts allowed this architecture to achieve awesome results in different fields. Today we are gonna learn how to visualize the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: papyrus
cooked: <p>Ok, thanks for the answer. So it seems it is the right way to do this.</p>
<p>Nevertheless, it seems like the obtained matrices don’t have diagonal values that are high, and there aren’t symmetric.</p>
<p>I think I miss something, is it possible to have more information on how to display those matrices?</p>
How to feed mfcc features into Resnet/VGG

username: Gorgen
cooked: <p>I have trouble with how to feed mfcc features into conv2d layer(N,C,H, W). However, the shape mfcc feature is audio_feature (N, time frames, Mel). I don’t know how to do it? or just unsqueeze(1) like audio_feature.unsqueeze(1)<br>
(N, 1, time frames, Mel). Is it true?<br>
How to do it? Thanks<br>
best wishes</p>
<p>By the way, feed mfcc features into VGG/Resnet</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Gorgen" data-post="1" data-topic="150299">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f14d63/48.png" class="avatar"> Gorgen:</div>
<blockquote>
<p>just unsqueeze(1) like audio_feature.unsqueeze(1)<br>
(N, 1, time frames, Mel).</p>
</blockquote>
</aside>
<p>Technically this is possible.<br>
However, I have seen some papers using MFCC Spectrogram as a direct input to CNNs as well.</p>
<p>Disclaimer: I am not an expert in speech processing / and do not know a lot of literature.</p>

username: Gorgen
cooked: <p>Thank you for reply. Do you know how to feed mfcc features directly into Resent? I mean if the shape of mfcc feature is [batch, time_frame, mel], how to do it?</p>
<p>Thanks<br>
best wishes</p>
How to convert coordinates to tensors?

username: ugurokuduro
cooked: <p>Hi, I am trying to develop a Lane Detector using Pytorch. Basically, I’m reading the video frame by frame using cv2, then finding edges using Canny Edge Detector and then using average lane algorithm I’m getting lane’s coordinates.</p>
<p>These coordinates look like [[x1, y1, x2, y2], [x3, y3, x4, y4]] and my .csv dataset is looks like:</p>
<pre><code class="lang-auto">image-0.jpg,"[[216, 312, 1458, 615], [315, 314, 465, 165]]"
image-1.jpg,"[[165, 951, 468, 1654], [465, 654, 416, 654]]"
...
</code></pre>
<p>*(I’m saving images frame by frame from the video to specify the road lanes. That’s why my database has names of every frame. image-0.jpg, image-1.jpg, image-2, etc.)</p>
<p>*(([x1, y1, x2, y2] is left road lane, [x3, y3, x4, y4] is right road lane))</p>
<p>But in the <code>__getitem__</code> method of my dataset class, I’m getting this error:</p>
<pre><code class="lang-auto">    y_label = torch.tensor(int(self.annotations.iloc[index, 1]))
ValueError: invalid literal for int() with base 10: '[[449, 576, 353, 600], [696, 576, 722, 600]]'
</code></pre>
<p>How can I fix this code? I want these coordinates to be readable by PyTorch so I can train my model on these coordinates and road images that I save. Should I convert this list of coordinates to list of tensors?</p>
<p>My full database code:</p>
<pre><code class="lang-auto">import os

import pandas as pd

import torch

from torch.utils.data import Dataset

from torchvision import transforms

from skimage import io

class RoadLanesDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):

        self.annotations = pd.read_csv(csv_file)

        self.root_dir = root_dir

        self.transform = transform

    def __len__(self):

        return len(self.annotations)

    def __getitem__(self, index):

        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])

        image = io.imread(img_path)

        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

        if self.transform:

            image = self.transform(image)

        return (image, y_label)
</code></pre>

username: ptrblck
cooked: <p>The issue is caused as the nested list is a string and cannot be directly parsed to an <code>int</code>.<br>
Trying to unwrap the nested list also won’t be easy, since not the actual integers are stored as string, but the entire nested list.<br>
I would thus recommend to use e.g. <code>pandas</code> for the parsing:</p>
<pre><code class="lang-python">a = "[[216, 312, 1458, 615], [315, 314, 465, 165]]"
int(a)
# ValueError: invalid literal for int() with base 10: '[[216, 312, 1458, 615], [315, 314, 465, 165]]'

b = pd.read_json(a)
arr = b.values
print(arr)
# [[ 216  312 1458  615]
#  [ 315  314  465  165]]

x = torch.from_numpy(arr)
print(x)
# tensor([[ 216,  312, 1458,  615],
#         [ 315,  314,  465,  165]])
</code></pre>
Cross validation with Pytorch

username: rubijade
cooked: <p>I read about cross-validation and I implemented it recently. My question is, after the cross-validation process, we have to test the model, right? Because it is the training set that is used for the cross-validation process (the training set is split Train/test). If yes, does that mean that I have to save the model after each fold? Which one do I have to use then, as I will have 5 saved models in the case of 5 K-fold cross-validation? Or can I save the model and load it at each fold before the training?</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="rubijade" data-post="1" data-topic="150275">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ec9cab/48.png" class="avatar"> rubijade:</div>
<blockquote>
<p>I will have 5 saved models in the case of 5 K-fold cross-validation</p>
</blockquote>
</aside>
<p>In my understanding, the model should be randomly initialized at the start of the training in each fold.<br>
After training the model using the training set of a particular fold, find out the performance on the test set of that fold and save this number. (saving the model, data configuration is optional depending on the need for reproducibility).<br>
At the end of all <code>k</code> folds, (typically) take the mean of the performances.</p>
DNN performance suddenly crashes

username: Sam_Lerman
cooked: <p>It goes from 60% training acc to near 0. I suspect some kind of floating point error occurred. Is there a way to check this / prevent this?</p>

username: eqy
cooked: <p>You may want to inspect the model outputs and intermediate activations for <code>NaN</code> values e.g., via <a href="https://pytorch.org/docs/stable/generated/torch.isnan.html" class="inline-onebox" rel="noopener nofollow ugc">torch.isnan — PyTorch 1.11.0 documentation</a>.</p>

username: osama-usuf
cooked: <p>You might also want to log your loss function - in my experience just logging the epoch-wise loss helps. One simple case where this may happen is if you choose a really high network learning rate, and the loss ends up exploding and going to inf., which is NaN and can lead to drops in network accuracies.</p>

username: rubijade
cooked: <p>Try to decrease or increase the learning rate, maybe when there is no change in the loss after some epoch. <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html" class="inline-onebox" rel="noopener nofollow ugc">ReduceLROnPlateau — PyTorch 1.11.0 documentation</a><br>
You can also stop the training earlier.</p>
Parameters for torch.utils.data.TensorDataset

username: ksalkar
cooked: <p>training_set = TensorDataset(training_img_set,training_label_set)</p>
<p>I wanted to use this line of code to wrap my image data and labels into one tensor, but the variable training_img_set is in the form of a list of images which gives an error. How do I convert it to a tensor format so I can use it as a parameter for the function?</p>

username: Sairam954
cooked: <p><a class="mention" href="/u/ksalkar">@ksalkar</a> where are you getting these list of images from? Can you paste the error here?<br>
Have a look at this thread which might be of some help  <a href="https://discuss.pytorch.org/t/make-a-tensordataset-and-dataloader-with-multiple-inputs-parameters/26605">https://discuss.pytorch.org/t/make-a-tensordataset-and-dataloader-with-multiple-inputs-parameters/26605</a></p>
How to convert a method to work on any tensor dimension

username: Sairam954
cooked: <p>Hi team,</p>
<p>I have defined a method which takes a decimal value and creates a representation of it know as temporal coding.</p>
<pre><code class="lang-auto">def getDecimalToUnary(input, bitwidth, encode="TC"):
    """This method accepts an input in decimal format and generates an unary temporal encoded representation of the input.
        The length of the unary bit stream depends on the bitwidth

    Args:
        input (int): The input decimal that needs to be transistion encoded
        bitwidth (int): The bit stream length is 2**bitwidth
        encode (str, optional): _description_. Defaults to 'TC'.  TC represents temporal encoding
    """
    if encode == "TC":
        unary_code = torch.cat(
            (torch.zeros(2**bitwidth - input), torch.ones(input)), 0
        )
        return unary_code
    else:
        print("Other encoding schemes are not implemented")
</code></pre>
<p>Rules of generating temporal code of a decimal are:</p>
<ol>
<li>The generated temporal code length should be equal to 2**bitwidth</li>
<li>The total number of 1’s in the bitstream should be equal to decimal value.</li>
<li>All the 1’s should be together at the end of the bitstream<br>
For example:<br>
If input is 5 and bitwidth is 3. Then, the unary bitstream is of length 2**3 = 8bits.<br>
The temporal representation of 5 is 00011111.<br>
The above code is able to work on single decimal value, however I want to have this method work for any dimension tensor.<br>
For instance, for tensor([1,2],[2,5])  with bitwidth 3, the output of the method should be<br>
tensor([[[0,0,0,0,0,0,0,1],[0,0,0,0,0,0,1,1]],[[0,0,0,0,0,0,1,1],[0,0,0,1,1,1,1,1]]])</li>
</ol>
How to improve my model accuracy?

username: Shahzad_Akram
cooked: <p>I’m learning PyTorch and tried my concepts on my own custom data. I created two lists of random numbers and combined the corresponding items in a new list as X and another list containing the sum of corresponding items as labels y. In simple I want to train a model in such a way that to make it work as an addition calculator. But, I’m getting a very huge loss, I tried tweaking different parameters but getting almost the same results.</p>
<p><strong>Here’s my code:</strong></p>
<pre><code class="lang-auto">import torch as T
import torch.nn as nn
import torch.nn.functional as F
import random

randomList1 = random.sample(range(0, 10000), 10000)
randomList2 = random.sample(range(0, 10000), 10000)

xList = []
yList = []

for (item1, item2) in zip(randomList1, randomList2):
    xList.append([item1, item2])

for x in xList:
    yList.append(x[0] + x[1])


dataset = T.torch.utils.data.TensorDataset(
    T.tensor(xList, dtype=T.float), T.tensor(yList, dtype=T.float))

train_batch = T.torch.utils.data.DataLoader(
    dataset, batch_size=10, shuffle=True)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(2, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, 1)

    def forward(self, x):
        x = T.relu(self.fc1(x))
        x = T.tanh(self.fc2(x))
        x = T.tanh(self.fc3(x))
        x = T.tanh(self.fc4(x))
        return x


net = Net()


loss_fn = nn.MSELoss()
optimizer = T.optim.Adam(net.parameters(), lr=0.0001)


for epoch in range(3):
    for data in train_batch:
        X, y = data
        net.zero_grad()
        output = net(X.view(-1, 2))
        loss = T.sqrt(loss_fn(output, y))       
        loss.backward()
        optimizer.step()
    print(loss)
</code></pre>
<p><strong>Output:</strong></p>
<pre><code class="lang-auto">tensor(11254.4688, grad_fn=&lt;SqrtBackward0&gt;)
tensor(9933.7793, grad_fn=&lt;SqrtBackward0&gt;)
tensor(11308.9463, grad_fn=&lt;SqrtBackward0&gt;)
</code></pre>
What is the gradient of torch.unique

username: Jason_Liu1
cooked: <p>Hi,</p>
<p>I’m wondering what is the gradient of torch.unique. Can I do this or Is there an alternative way to implement something like:<br>
y = model(X)<br>
loss = len(torch.unique(y))<br>
…backprop…</p>
<p>Many thanks indeed!</p>

username: KFrank
cooked: <p>Hi Jason!</p>
<aside class="quote no-group" data-username="Jason_Liu1" data-post="1" data-topic="150138" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jason_liu1/48/49022_2.png" class="avatar"> Jason_Liu1:</div>
<blockquote>
<p>I’m wondering what is the gradient of torch.unique.</p>
</blockquote>
</aside>
<p><code>torch.unique()</code> doesn’t really have a sensible gradient*.  What would<br>
you want it to be?</p>
<p>Oddly, <code>unique()</code> does have a companion backward function, even<br>
though backpropagation through it isn’t implemented:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt; t = torch.tensor ([1.0, 2.0, 1.0, 3.0, 1.0, 4.0, 2.0], requires_grad = True)
&gt;&gt;&gt; u = torch.unique (t)
&gt;&gt;&gt; u
tensor([1., 2., 3., 4.], grad_fn=&lt;Unique2Backward0&gt;)
&gt;&gt;&gt; u.sum().backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
NotImplementedError: the derivative for '_unique2' is not implemented.
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>loss = len(torch.unique(y))</p>
</blockquote>
</aside>
<p>Regardless of whether  you can make sense of backpropagating through<br>
<code>unique()</code>, <code>len()</code> returns an integer, so it is inherently not differentiable<br>
(and it doesn’t return a pytorch tensor as would be required for autograd<br>
to operate on it).</p>
<p>*)  You could arguably define <code>unique()</code> to return a view into the original<br>
tensor, in which case you could backpropagate through it.  But this seems<br>
a bit contrived, and in any event, would be ambiguous.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Jason_Liu1
cooked: <p>Hi Frank,</p>
<p>Thank you very much for your explanation. I’m trying to calculate the entropy of the model’s output (which is a sequence) as loss, but it seems not very practical.</p>
<p>Best regards,<br>
Jason</p>
Adding Noise to Decoders in Autoencoders

username: utkarsh23
cooked: <p>Hi,<br>
I am a little confused about how I can add random noise to decoders of the autoencoders. It can be imagined that there are two inputs to the decoder, one is the output of encoders, and one is random noise.<br>
Please help.</p>
<p>Thank you</p>

username: a_d
cooked: <p>I suppose you need to add noise to the encoder output before feeding it to the decoder. If that is so, then the you could add Gaussian noise as follows -</p>
<pre><code class="lang-auto">output = Encoder(input)
output = output + torch.sqrt(desired_variance) * torch.randn(output.shape)
model_output = Decoder(output)
</code></pre>

username: utkarsh23
cooked: <p>Thank you for the response. I am trying it now.<br>
I want to ask one thing about how to choose the desired_variance value?<br>
Since I am getting error “TypeError: sqrt(): argument ‘input’ (position 1) must be Tensor, not float”.<br>
So the desired_Variance has to be a tensor with different values?</p>
<p>Just to be more clear, I have attached a picture to show what I wanted.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/352d08b1409f9aa4763c9fda36417fac04652b80.png" data-download-href="https://discuss.pytorch.org/uploads/default/352d08b1409f9aa4763c9fda36417fac04652b80" title="random_noise"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/352d08b1409f9aa4763c9fda36417fac04652b80.png" alt="random_noise" data-base62-sha1="7ApN1AGJe2eJlTxnD5EW1tTxvsQ" width="690" height="332" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">random_noise</span><span class="informations">852×410 3.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: a_d
cooked: <blockquote>
<p>I want to ask one thing about how to choose the desired_variance value?</p>
</blockquote>
<p>This would depend on how noisy data would you like to train on. Too noisy data would lead to garbage features and might do more harm. It would ultimately be set heuristically, however need more context to set it.</p>
<p>By the picture I would assume you want to add noise to the encoder output and feed it to the decoder</p>
<p>torch.sqrt would require a tensor.<br>
Do the following -</p>
<pre><code class="lang-auto">output = outuput + (desired_variance ** 0.5) * torch.randn(output.shape)
</code></pre>
<p>Remember that if you do the following in the forward calculation, it will be included in the backprop. However, if you perform this step in the training loop, I think it will still be taken in consideration, so you might want to keep that in mind</p>
Slicing image into square patches

username: Mika_S
cooked: <p>Is there a slicing function to slice an image into square patches of size <code>p</code>. Right now I am doing this in few lines of code but if there is a function that does this I would just replace.</p>

username: SimonW
cooked: <p>Look into <code>torchvision.transforms.*crop</code> methods. <a href="http://pytorch.org/docs/master/torchvision/transforms.html">http://pytorch.org/docs/master/torchvision/transforms.html</a><br>
Afaik, there is no default method for an arbitrary location crop, but it is easily implementable using <code>torchvision.transforms.Lambda</code>.</p>

username: Pranay_Dugar
cooked: <p>Quite late but for reference for others, you can use the unfold function:</p>
<p>patches = img_t.data.unfold(0, 3, 3).unfold(1, 8, 8).unfold(2, 8, 8)</p>
<p>Here is the demo code to test:</p>
<pre><code class="lang-auto">import torch
from torchvision import transforms
import matplotlib.pyplot as plt

%matplotlib inline

transt = transforms.ToTensor()
transp = transforms.ToPILImage()
img_t = transt(Image.open('cifar/train/10000_automobile.png'))

#torch.Tensor.unfold(dimension, size, step)
#slices the images into 8*8 size patches
patches = img_t.data.unfold(0, 3, 3).unfold(1, 8, 8).unfold(2, 8, 8)


print(patches[0][0][0].shape)

def visualize(patches):
    """Imshow for Tensor."""    
    fig = plt.figure(figsize=(4, 4))
    for i in range(4):
        for j in range(4):
            inp = trans1(patches[0][i][j])
            inp = np.array(inp)

            ax = fig.add_subplot(4, 4, ((i*4)+j)+1, xticks=[], yticks=[])
            plt.imshow(inp)

visualize(patches)
</code></pre>
Implicit sum in method autograd.Fonction.backward

username: Xav
cooked: <p>I have create a new autograd Fonction with both method forward and backward.<br>
one of my inputs of forward is a scalar and the linked gradient in bacward is a multli dimentional tensor. In general those two tensor has to be the same size but it works and compute the sum over the gradient tensor.<br>
I did not find where in the documentation it says that in this case it will do a sum, and i did not understand the documentation about backward method :<br>
“Computes the sum of gradients of given tensors with respect to graph leaves”.<br>
It is the sum of what here??</p>
Exclude layers that are not supported by QNNPACK in model quantization

username: wentian_zhao
cooked: <p>I want to quantize a model that contains Conv3d layers and run the model on Android devices. When I follow the tutorial (<a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.11.0 documentation</a>) to perform static quantization, the Android device throws the following exception when loading the quantized model:</p>
<pre><code class="lang-auto">com.facebook.jni.CppException: prepack/__setstate__: QNNPACK only supports Conv2d now. ()
</code></pre>
<p>Thus, I want to exclude the Conv3d layers during quantization (i.e. the Conv3d layer is not quantized), and the quantization is done as follows:</p>
<pre><code class="lang-auto">import torch, torch.nn as nn
from torch.quantization import QuantStub, DeQuantStub, get_qconfig_propagation_list
from torch.utils.mobile_optimizer import optimize_for_mobile

input_fp32 = torch.rand(size=(1, 3, 16, 224, 224))
# just an example model
model_fp32 = nn.Sequential(
    QuantStub(),
    nn.Conv3d(in_channels=3, out_channels=3, kernel_size=5, stride=1),
    nn.Linear(220, 220),
    DeQuantStub(),
)
model_fp32.eval()
model_fp32.qconfig = torch.quantization.get_default_qconfig('qnnpack')
model_fp32 = torch.quantization.prepare(model_fp32)
qconfig_propagation_list = list(filter(lambda x: x != torch.nn.modules.conv.Conv3d, get_qconfig_propagation_list()))

model_fp32_prepared = torch.quantization.prepare(model_fp32, allow_list=qconfig_propagation_list)
model_fp32_prepared(input_fp32)
model_int8 = torch.quantization.convert(model_fp32_prepared)

traced_model = torch.jit.trace(model_int8, input_fp32)
traced_script_module_optimized = optimize_for_mobile(traced_model)
traced_script_module_optimized._save_for_lite_interpreter('model_quantized_int8.pkl')
</code></pre>
<p>The same exception that says <code>QNNPACK only supports Conv2d now. ()</code> persists when loading the quantized model. Did I make mistake in the above code? Or does it mean that when running quantized models on Android devices, the model must not contain any layer that is not supported by QNNPACK, even if these layers are not quantized?</p>
Per-parameter for ModuleList

username: JosAr
cooked: <p>Hi there,<br>
I have a model with a variable amount of custom Layers, therefore I use torch.nn.ModuleList - now I would like to set the per parameter option in my optimizer, but the parameters are not grouped with a name (as in torch.nn.ModuleDict). How can I still set per-parameter options?</p>
<pre><code class="lang-auto">
class my_model(torch.nn.Module):
    def __init__(self, channel_dimensions):
        super().__init__()
        self.layers = torch.nn.ModuleList([CustomLayer(channel_dimension[i+1]), channel_dimension[i]) for i in range(len(channel_dimensions))])
    def forward(self, x)
        for layer in self.layers:
            x = layer(x)
        return x
</code></pre>
<p>Maybe something like</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': MyModel.layers[-1].parameters()}
            ], lr=1e-2, momentum=0.9)

</code></pre>
<p>?</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="JosAr" data-post="1" data-topic="150199">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/josar/48/39901_2.png" class="avatar"> JosAr:</div>
<blockquote>
<p><code>{'params': MyModel.layers[-1].parameters()}</code></p>
</blockquote>
</aside>
<p>Assuming that <code>MyModel</code> is an instance of the class <code>my_model</code>, your approach is correct.</p>