akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reduce output dimensions of batch Hessian without looping

YZ_John: <p>Hi all,</p>
<p>I used the method <a href="https://discuss.pytorch.org/t/computing-batch-jacobian-efficiently/80771/4">here</a> to compute batch Hessian of the output of a neural network f with respect to its input at given batch points x, where x is of the shape [B, N], with B being the batch size and N being the input dimension of f.</p>
<p>The code produces a tensor of shape [B, N, B, N], where Output [i, :, i, :] =Hessian of f at x[i, :]. I further reduce the Hessian tensor in a desired shape [B, N, N], by looping over range (B) and setting Final_output [i, :, :] = Output [i, :, i, :].</p>
<p>My question is whether it is possible to perform such a reduction without using loop?</p>
How to use AdaptiveLogSoftmaxWithLoss() jointly with CTC loss

galactica147: <p>Hi, I wonder if there is a way to use AdaptiveLogSoftmaxWithLoss() instead of the regular log_softmax() in a LSTM model with CTCLoss().</p>
<p>With log_softmax(), it’s boils down to essentially this:</p>
<pre><code class="lang-auto">output = self.LSTM(input_size, lstm_size, num_layers)
output = self.Linear(fc_input, fc_output)
output = F.log_softmax(output, dim=2)   # [batch, seq_length, class]

# to compute CTC loss
ctc_loss = self.ctcLoss(output, labels, original_input_lengths, original_label_lengths)
</code></pre>
<p>where all the LSTM, linear, and ctcLoss are standard pytorch layers/functions.</p>
<p>This is working all good except that we’d also like to improve model perf when <code>num_classes</code> is big.<br>
Then <code>AdaptiveLogSoftmaxWithLoss()</code> seems to be a great success for regular classification problems.<br>
So i tried something like this:</p>
<pre><code class="lang-auto">output = self.LSTM(input_size, lstm_size, num_layers)
output = output.view(batch_size, -1)
output, loss = self.AdaptiveSoftmax(output, labels)

# to compute CTC loss
ctc_loss = self.ctcLoss(output, labels, original_input_lengths, original_label_lengths)
</code></pre>
<p>This doesn’t seem to work as the output of <code>AdaptiveLogSoftmaxWithLoss()</code> is of shape: (batch_size, n_classes), while <code>ctcLoss()</code> expects the shape to be (batch_size, seq_length, num_classes).</p>
<p>So i wonder what would be the best way to use <code>AdaptiveLogSoftmaxWithLoss()</code> for sequence problems using CTC loss?</p>

galactica147: <p>i guess i could manually compute <code>AdaptiveLogSoftmaxWithLoss</code> for each time step in the sequence and then assemble the results into the original shape. But is there a better way to do so?</p>
Don't have built-in NCCL in distributed package

zeming_hou: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64d9fac4b28bf99bedf6fd12abf597781a3bb2d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/64d9fac4b28bf99bedf6fd12abf597781a3bb2d3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64d9fac4b28bf99bedf6fd12abf597781a3bb2d3.png" alt="image" data-base62-sha1="eoaLuUEMCUinEyqHf1CtHrgC2pd" width="690" height="177" data-dominant-color="491517"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1369×352 18.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

pritamdamania87: <p><a class="mention" href="/u/zeming_hou">@zeming_hou</a> Did you compile PyTorch from source or did you install it via some of the pre-built binaries? In either case, could you share the commands you used to install PyTorch?</p>
How to choose num_worker when using DDP?

Olivier-CR: <p>Hi,</p>
<p>I’m using DDP on a machine with 64 vCPUs and 8 GPUs<br>
In what range shall I tune the num_workers of the data loader? since the script will run 8 times in the machine (one process per GPU), is my understanding correct that in order to avoid CPU contention, num_workers should be &lt;= cpu_count / 8 ?</p>

pritamdamania87: <aside class="quote no-group" data-username="Olivier-CR" data-post="1" data-topic="140978">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/c5a1d2/48.png" class="avatar"> Olivier-CR:</div>
<blockquote>
<p>is my understanding correct that in order to avoid CPU contention, num_workers should be &lt;= cpu_count / 8 ?</p>
</blockquote>
</aside>
<p>Yes this is correct if your dataloader is CPU intensive, if the dataloader is IO intensive, you could have more num_workers to load data from the filesystem/network with a higher degree of parallelism (until you saturate the filesystem/nw bandwidht).</p>
Simulation of Nvidia Jetson & Co

pulsar: <p>Hey folks,<br>
As title, I was wondering if it was possible to simulate the computation capability by Nvidia Jetson Families (E.g number of cores, Ram…).<br>
I need to start some experiments in order to check the computational time over a limited hardware as Jetson Nano etc…<br>
Is there any way to simulate it? Is it possible from pytorch to do it?<br>
Thanks in advance for your possible information.</p>
MIssing CommHook symbols for PT1.10.0

YangFei: <p>I was running into this error when using PT1.10.0</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.8/site-packages/smdistributed/modelparallel/torch/smplib.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTVN4c10d17AllReduceCommHookE
</code></pre>
<p>Same issue does not exist in PT1.8. This hook is defined in <a href="https://github.com/pytorch/pytorch/blob/4829dcea0920ba89719a658238a44f7c931d536e/torch/csrc/distributed/c10d/default_comm_hooks.hpp" rel="noopener nofollow ugc">default_comm_hooks.hpp</a><br>
So I’m checking the symbols from lib torch:<br>
In PT1.8</p>
<pre><code class="lang-auto">nm -gDC /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so | grep AllreduceCommHook
0000000000a06ee0 T c10d::AllReduceCommHook::runHook(c10d::GradBucket&amp;)
000000000125d518 V typeinfo for c10d::AllReduceCommHook
0000000000dd7ca0 V typeinfo name for c10d::AllReduceCommHook
000000000125d568 V vtable for c10d::AllReduceCommHook
</code></pre>
<p>But in PT 1.10.0</p>
<pre><code class="lang-auto">nm -gDC /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so | grep AllreduceCommHook
</code></pre>
<p>returns nothing.<br>
I also observed the same behavior for <code>FP16CompressCommHook</code> which is defined in the same file. Could someone let me know where can I find those symbols in PT1.10.0?</p>
DataParallel with custom layer parameters

randophilus: <p>I am currently testing a custom layer where I am taking a regular conv layer, but I am also introducing an intermediate loss where I get the MSEloss between the image patch and the filter weights that caused the highest output per image. My current implementation calculates the full loss in the custom layer itself and uses <code>self.register_parameter('unsupervised_loss',nn.Parameter(torch.zeros(1)) )</code> to store my intermediate loss.</p>
<p>While this works in the network by itself, it only returns the original value of 0 when I wrap it with dataparallel. This problem is similar to <a href="https://discuss.pytorch.org/t/dataparallel-get-updated-module-attributes/15140">[DataParallel] Get updated module attributes</a> which does not have an answer, but I also want to specify that I need the gradients to also work. In <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html" rel="noopener nofollow ugc">The original documentation</a> there is a warning about this behavior, stating that you can use in place updates on device 0, but how would I extend this to my specific issue where the operations are input-specific and not like incrementing a counter on device 0.</p>
<p>I see two possible options that I might test: either I send the unsupervised losses to device 0 somehow, or I instead rewrite the function to return both the conv output and the unsupervised loss as a tuple. Both of these options seem messy though.</p>

mrshenli: <aside class="quote no-group" data-username="randophilus" data-post="1" data-topic="140019">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/randophilus/48/30455_2.png" class="avatar"> randophilus:</div>
<blockquote>
<p>While this works in the network by itself, it only returns the original value of 0 when I wrap it with dataparallel.</p>
</blockquote>
</aside>
<p>I believe this is because DataParallel (DP) would replicate the module to each provided device in every forward. So the module instance stored in DataParallel and the module instance used by forward on every device are different.</p>
<p>However, even if they are the same instance, it’s still problematic. Suppose we run DP on two devices, then, one of the device would override the value of <code>unsupervised_loss</code> written by another device. Is this an expected behavior?</p>
<blockquote>
<p>I instead rewrite the function to return both the conv output and the unsupervised loss as a tuple.</p>
</blockquote>
<p>This seems a reasonable workaround.</p>

randophilus: <p>I ended up having to do the tuple output to get things working. Unfortunately this messes up my original implementation of the network since my custom layer was being used in a sequential. Had to rewrite some code to fix that.</p>
Implementing LSTMCell. Problem with DataParallel on multi-GPU due to in-place operation

wilbertcaine: <p>I’m trying to implement LSTM using LSTMCell. Here is my implementation</p>
<pre><code class="lang-auto">class lstm(nn.Module):
    def __init__(self, input_size, output_size, hidden_size, n_layers, batch_size):
        super(lstm, self).__init__()
        self.input_size = input_size
        self.output_size = output_size
        self.hidden_size = hidden_size
        self.batch_size = batch_size
        self.n_layers = n_layers
        self.embed = nn.Linear(input_size, hidden_size)
        self.lstm = nn.ModuleList([nn.LSTMCell(hidden_size, hidden_size) for i in range(self.n_layers)])
        self.output = nn.Sequential(
                nn.Linear(hidden_size, output_size),
                #nn.BatchNorm1d(output_size),
                nn.Tanh())
        z = torch.zeros(batch_size, self.hidden_size)
        hc = torch.unsqueeze(torch.stack((z, z)), 0)
        hidden = torch.Tensor([])
        for i in range(self.n_layers):
            hidden = torch.cat((hidden, hc))
        self.register_buffer('hidden', hidden)

    def init_hidden(self):
        self.hidden.fill_(0)

    def forward(self, input):
        embedded = self.embed(input.view(-1, self.input_size))
        h_in = embedded
        for i in range(self.n_layers):
            hc = torch.unbind(self.hidden[i])
            lstm_out = self.lstm[i](h_in, hc)
            lstm_out = torch.stack(lstm_out)
            self.hidden[i].copy_(lstm_out)
            h_in = self.hidden[i][0]

        return self.output(h_in)
</code></pre>
<p>However, I figured that hidden has requires_grad set to True during training. Thus, the in-place copy_ creates an error as follows.</p>
<blockquote>
<p>RuntimeError: Output 0 of SelectBackward0 is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.</p>
</blockquote>
<p>On the other hand, I believe that in-place updates are the only way to keep the next hidden state and next cell state in DataParallel with multi-GPU.</p>
<p>I can’t figure out a workaround for this issue. Any help would be appreciated!</p>
Is there an elegant way to get response variables separately from predictor variables out of a torch.utils.data.Dataset object?

alexanderguyer: <p>Looking at the documentation and source code for <code>torch.utils.data.Dataset</code>, I’m not seeing a natural way to extract response variables (e.g. labels / targets, like per-example class labels) other than via <code>__getitem__()</code>, which <em>also</em> extracts the predictor variables. I think this is a problem with the interface failing to separate concerns, but perhaps there’s a solution.</p>
<p>I’m researching open set recognition, so I often need to partition datasets “by class”, which means determining which class each data point belongs to. I’ve recently been working with a subset of ImageNet 1K, and I’m often working on a server where the file system is networked, so it’s extremely expensive to load images from disk, and it’s difficult to justify doing so when I don’t actually <em>need</em> the images, but rather just the class labels.</p>
<p>My current solution has been to build individual wrapper classes for each torchvision.datasets.XXX dataset class that I’ve been using to expose a function for extracting just the labels, often having to refer to the corresponding dataset’s source code to see how it stores and loads labels. But this a) depends on implementation details and thus can break from any torchvision update, and b) requires a lot of extra work.</p>
<p>Is there an easier, more universal way to extract just response variables, annotations, etc. from a dataset without also extracting predictor variables? Or is this impossible given the current nature of the <code>torch.utils.data.Dataset</code> interface?</p>
Forward() method: "RuntimeError: No grad accumulator for a saved leaf!"

zfzhang: <p>Hi,<br>
I’m writing a Pytorch CUDA extension, which processes a tensor point-wise.<br>
As a tensor is recommended to be made contiguous before calling CUDA kernel, and I try to avoid calling tensor.contiguous() where possible.<br>
One scenario is that a 3D tensor x is not contiguous, but x.view(-1) is contiguous, so in the Python code I try the following, which triggers the above “<em>RuntimeError: No grad accumulator for a saved leaf!</em>” error message.</p>
<pre><code class="lang-auto">class myfunc(Function):
    @staticmethod
    def forward(ctx, x):
        x = x.view(-1).contiguous() # this is what causes the error !
        ctx.save_for_backward(x)
        return mycppclass.forward(x)
        
    @staticmethod
    def backward(ctx, dy):
        return mycppclass.backward(dy.contiguous(), *ctx.saved_variables)
</code></pre>
<p>Any suggestions?<br>
Thank you!</p>
Calculating meta-gradients with approximate inner-loop gradient

Harbar-Inbound: <p>I am currently working on a re-implementation of a (model agnostic) meta-learning paper (<a href="https://openaccess.thecvf.com/content/ICCV2021/html/Jamal_A_Lazy_Approach_to_Long-Horizon_Gradient-Based_Meta-Learning_ICCV_2021_paper.html" rel="noopener nofollow ugc">Lazy MAML - ICCV2021</a>), which approximates the 'gradient through the inner-loop process as a single step. As such, now the inner-learner could perform many steps (say k=10…20). Then to calculate the meta-gradient we approximate the expensive computation as though a single inner-loop step (or leap) has been taken.</p>
<p>Converting this logic into actual code gives me some headaches, as I fail to see how this can be properly implemented. My current idea boils down to the following four steps, on the last one I run into problems.</p>
<p>First, let us define that φ denotes the inner-loop model, and θ the outer-loop model (meta-learner)</p>
<ol>
<li>
<p>Do the inner-loop optimization. For this, I plan to use autograd, without retain_graph enabled. For my baseline implementation of MAML/ANIL/BOIL with learn2learn this needs to be enabled to allow for the outer-loop backprop.</p>
</li>
<li>
<p>Calculate the single step from θ to φ, i.e. <em>grad = φ - θ</em>, for which we can loop over their parameters, and calculate the single gradient that approximates the inner-loop. I will call this the ‘derived’ gradient.</p>
</li>
<li>
<p>Using φ we can calculate the loss on the query set, which we need to calculate the meta-gradient.</p>
</li>
<li>
<p>Now I need to perform the backpropagation ‘through the inner-loop’ using my ‘derived’ gradient update.</p>
</li>
</ol>
<p>For step 4, I consider the following, but I feel like I am abusing some of PyTorch’s functionality. First I could perform a normal MAML update using θ, to get ѱ, and perform a backward pass with <code>autograd.grad</code> with <strong>retain_graph</strong> enabled. (Using learn2learn’s update_module functionality to make the updates differentiable) However, using backward hooks I can set the gradients to my ‘derived’ gradients instead of the actual modules gradient, such that ѱ == ɸ, but <strong>with</strong> the computational graph needed.<br>
Now, I have a (fudged) computational graph for autograd and can perform the second-order derivative with the loss that I get from step 3. However, I would need to re-compute this loss (after first taking a step with my ‘derived’ gradient), as ѱ and the loss are otherwise unrelated.</p>
<p>Although I feel like this <strong>should</strong> work, I feel uncertain about my approach and feel like there should be a more sophisticated way. (Or a correct way, if I am mistaken in my approach). E.g. constructing the computational graph differently, or using a method such as implicit MAML (without <em>l2</em> weight regularisation). I am currently writing code to verify my idea against my MAML implementation to see if in the case of a single inner-update step my implementation.</p>
Creating batches of sequences

Timothy35964154: <p>Hello There. I’m currently working on a LSTM Autoencoder. I have a big amount of samples. Each sample contains 120 features. For now, I’m creating sequences of length 1, batch_size is equal to 1 and everything is working fine. I first convert my data array to a list and then using the following function, I convert them to sequences of length 1:</p>
<pre><code class="lang-auto">def dataset(mydatalist):
    dataset = [torch.tensor(s).unsqueeze(1) for s in mydatalist] 
 
    n_seq, seq_len, n_features = torch.stack(dataset).shape # n_seq,4,1
    return dataset, seq_len, n_features 
</code></pre>
<p>Then for training, I write the procedure “for seq_true in train_dataset:” which states the batch_size of 1. But as I have a large amount of samples, the training procedure is too slow. So I want to increase the batch_size in order to achieve better performance. Could anyone please help me with that? I know that it maybe a simple question but everything I try leads to shape-related errors in the LSTM network.<br>
Also it would be very nice if you could also point out how to create sequences with length more than 1 alongside increasing the batch_size.<br>
Many thanks in advance.</p>

vdw: <p>I’m not 100% sure what you problem is. Typically, in case of RNNs going from batch_size = 1 to batch_size &gt; 1 requires to handle sequence of different lengths. However, you say</p>
<blockquote>
<p>I’m creating sequences of length 1</p>
</blockquote>
<p>which is a bit odd in itself as it’s not really a sequence :). In you code snippet it looks like <code>seq_len=4</code>. So this is all a bit confusing.</p>
<p>Just in case the problem might be sequences of different lengths: My approach is to create batches with sequences of the same length. This is particularly for autoencoders easy since input and target sequences are the same. We talked about this extensively in <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284">this post</a>. If it helps, my code for that is available on <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/utils/data/text/dataset.py" rel="noopener nofollow ugc">Github</a>.</p>
Outstanding memory allocation during pack_sequence / pack_padded_sequence

Zador: <p>Hi,.<br>
I noticed some odd (and crippling) behaviour in the function <code>torch.nn.utils.rnn.pack_sequence</code>. See my code:</p>
<pre><code class="lang-auto">inputs= list(self.buffer.values())
print(torch.cuda.memory_allocated()/1e9)
print(torch.cuda.max_memory_allocated()/1e9)
inputs = [torch.cat(input, dim=0) for input in inputs]
print(torch.cuda.memory_allocated()/1e9)
print(torch.cuda.max_memory_allocated()/1e9)
inputs, batch_sizes, sorted_indices, unsorted_indices = pack_sequence(inputs, enforce_sorted=False)
print(torch.cuda.memory_allocated()/1e9)
print(torch.cuda.max_memory_allocated()/1e9)
</code></pre>
<p>Code is structured this way to demonstrate my problem. Below you will see 4 examples of what is printed by this sequence of codes. You will see that the memory allocated by the list of tensors I input into  <code>pack_sequence</code> is outstandingly smaller than the maximum allocated memory, which occurs during the packing (logically true as before packing, maximum is much smaller). 4 outputs are a result of different datapoints (achieved by setting different random seeds; am using shuffled data laoders).</p>
<p>Output 1:</p>
<pre><code class="lang-auto">0.05142272
0.05142272
0.101368832
0.101368832
0.10063104
3.67196672
</code></pre>
<p>Output 2:</p>
<pre><code class="lang-auto">0.055465984
0.055465984
0.105938432
0.105938432
0.104708096
0.995354112
</code></pre>
<p>Output 3:</p>
<pre><code class="lang-auto">0.061822976
0.061822976
0.111620608
0.111620608
0.111021056
1.934670848
</code></pre>
<p>Output 4:</p>
<pre><code class="lang-auto">0.052079616
0.052079616
0.101535232
0.101535232
0.101254144
0.52041728
</code></pre>
<p>Additionally, I would like to note that all tensors inside of <code>inputs</code>are already stored in my single cuda device.</p>
<p>Could anybody explain what is going on and how I can avoid this? I suspect this is because during padding there is a lot more information stored. If this is the case is there no way this function can be run by sparsifying? I would happily trade memory for speed here (<strong>without</strong> batchifying this procedure).</p>
<p>Thanks!</p>
Dataloader locked when I increase the size of training set

Ning_Albert: <p>When I increase my training set from <code>226</code> to <code>998</code>, the dataloader locked and the program just stay there. When I click stop button, the ternimal shows that the code stop here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd550d17ab7ffa7a11b56ebc833e844113e93472.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd550d17ab7ffa7a11b56ebc833e844113e93472" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd550d17ab7ffa7a11b56ebc833e844113e93472_2_690x234.png" alt="image" data-base62-sha1="tis6mGa1whho2jzR9wEbrhHqc5c" width="690" height="234" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd550d17ab7ffa7a11b56ebc833e844113e93472_2_690x234.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd550d17ab7ffa7a11b56ebc833e844113e93472_2_1035x351.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd550d17ab7ffa7a11b56ebc833e844113e93472_2_1380x468.png 2x" data-dominant-color="2F3033"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1576×535 77.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I try to change <code>num-workers = 0</code> but it doesn’t work. I can’t reduce the <code>batch_size</code> because that will impare the performance of my model.</p>
C++ Export a torch module to JIT

Titus: <p>Hello,<br>
I’m kind of confused by the documentation. I am looking for a method to save / export my <code>torch::nn::module</code> (a simple CNN network of few layers) to a jit  so I can later use <code>torch::jit::load()</code> instead of <code>torch::load()</code>.<br>
I’ve found <code>torch::serialize::OutputArchive</code> but I don’t understand how to use it and if its matches my problematic.<br>
Do you have any insights on this ?</p>
<p>Thanks</p>
Reusing variables with same name

Bomin_Kim: <p>I am trying to build a model that iterates two different layers, and below is my implementation.</p>
<pre><code class="lang-auto">class mymodel(nn.Module):
    def __init__(self, k_iters):
        super().__init__()
        self.k_iters = k_iters
        self.layer1 = ..
        self.layer2 =..

    def forward(self, x):
        x_k = x
        for k in range(self.k_iters):
            z_k = self.layer1(x_k)
            x_k = self.layer2(z_k)
        return x_k
</code></pre>
<p>Would there be any problem when I reuse the variables “z_k” and “x_k” for many times? I wonder if it interferes with the backward operation or tracking gradient…</p>

shivammehta007: <p>No, it will have no issues with the variable name. For future references, if pytorch won’t be able to do something it will throw an error and tell you to explicitly clone the tensor.</p>
When opening RGB image in grayscale, can 2 colors be collapsed in the same gray level?

Olivier-CR: <p>I am working on Audi A2D2 dataset 2D segmentation.<br>
The segmentation mask are provided as 3-channel, RGB images (despite having only 55 classes).<br>
In order to process the labels faster in my data loader, I’d like to open them as grayscale even though they are RGB. Because there are only 55 different RGB pixel values, it should be possible to convert to gray scale and keeping unique pixel value for the 55 classes, since there are 256 possible shades of grey.</p>
<pre><code class="lang-auto">from torchvision.io import read_image
from torchvision.io.image import ImageReadMode

read_image(path, mode=ImageReadMode.RGB)
</code></pre>
<p>I’m wondering: <strong>is there a risk of 2 RGB pixel being assigned the same grayscale value, when opening an RGB picture with <code>ImageReadMode.GRAY?</code></strong></p>

KFrank: <p>Hi Olivier!</p>
<aside class="quote no-group" data-username="Olivier-CR" data-post="1" data-topic="140976" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/c5a1d2/48.png" class="avatar"> Olivier-CR:</div>
<blockquote>
<p><strong>is there a risk of 2 RGB pixel being assigned the same grayscale value</strong></p>
</blockquote>
</aside>
<p>Yes, the risk is real, and such an occurrence is in fact quite likely:  See the<br>
so-called <a href="https://en.wikipedia.org/wiki/Birthday_problem" rel="noopener nofollow ugc">“birthday paradox.”</a></p>
<p>If the person who prepared the segmentation masks were kind, then he<br>
could have purposely chosen RGB values that map to distinct grayscale<br>
values.  (And if he were diabolical, he could have chosen 55 RGB values<br>
that all map to the same grayscale value.)  But if the RGB values were<br>
chosen somewhat “randomly,” then you will likely have such a collision.</p>
<p>So …   Do you feel lucky?</p>
<p>Best.</p>
<p>K. Frank</p>

Olivier-CR: <p><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=10" title=":joy:" class="emoji" alt=":joy:"> haha I understand thanks <a class="mention" href="/u/kfrank">@KFrank</a> ! I think I’ll pre-process myself once before the training (now it’s done in the <strong>getitem</strong> on the fly and taking up a lot of CPU)</p>
L2-Normalizing the weights

Ashima_Garg: <p>Hi, I used the following two implementations. With Implementation 2, I am getting better accuracy.</p>
<p>But I am not clear of how <code>nn.utils.weight_norm</code> will change the performance. The PyTorch documentation reads that <code>nn.utils.weight_norm</code> is just used to decouple the norm vector and the angle. Then why is there difference in the numerical value?</p>
<p>Implementation 1</p>
<pre><code class="lang-auto">def __init__(self):
    super(MyModel, self).__init__()
    self.linear = nn.Linear(2, 2)

def forward(self, x):
    weight = F.normalize(self.linear.weight)
    out = torch.mm(x, weight.t()) + self.linear.bias
    return out
</code></pre>
<p>Implementation 2</p>
<pre><code class="lang-auto">def __init__(self):
    super(MyModel, self).__init__()
    self.linear = nn.utils.weight_norm(nn.Linear(2, 2))

def forward(self, x):
    self.linear.weight = F.normalize(self.linear.weight)
    out = self.linear(x)
    return out
</code></pre>
<p>Please let me know what is the right way of using L2-normalized weights for classification.<br>
Thanks!</p>
What is the local batch size when using DistributedSampler?

Olivier-CR: <p>When doing <code>data_loader = DataLoader(my_dataset, sampler=DistributedSampler(dataset), batch_size=N)</code> in a DDP distributed training script, what is the number of records each GPU/worker/process/script (unsure what is the most accepted name) receives at each iteration?</p>
<p>Does each DDP GPU receive N records, or N/gpu_count?</p>

huahuanZ: <p>The batch size on local workers would be <code>N_global_batch_size//N_workers</code></p>

Olivier-CR: <p>thanks! and <code>N_global_batch_size</code> would be the value you set as batch_size in the <code>DataLoader</code> ?</p>

huahuanZ: <p>No. You need to manually set <code>batch_size=N_global_batch_size//N_workers</code> in the DataLoader. <code>N_global_batch_size</code> is the <strong>real batch size</strong>.</p>
For CUDA extension implementing element-wise function, what is the best shape for the kernel launch?

zfzhang: <p>Hi,<br>
I’m writing a Pytorch CUDA extension that implements an element-wise function.<br>
Assuming we don’t know whether the input is <strong>2D or 3D or 4D</strong>, but it is known to have the first dimension as <strong>batch</strong>. How do we choose the best kernel shape (grid size, block size) at launch?</p>
<p>Thank you!</p>

zfzhang: <p>I found a similar question with answer, may be the solution.</p><aside class="quote quote-modified" data-post="1" data-topic="79297">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/separius/48/4165_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-write-a-custom-point-wise-cuda-kernel/79297">How to write a custom point-wise cuda kernel?</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    I want to write a custom point-wise cuda kernel and I can do that easily with cpp_extension, and a custom cuda kernel luncher. The problem is that I don’t know how to set a good block_size. 
As far as I can tell CUDA_tensor_apply2 handles those parameters and simplifies my work. But I need CUDA_tensor_apply3 which has been removed from the aten. 
It seems like I should use TensorIterator with gpu_kernel but I can not include it into my code. Should I include &lt;ATen/native/CUDALoops.cuh&gt;? If so, I…
  </blockquote>
</aside>
Converting BERT models to ONNX

Nimit: <p>Hi there,</p>
<p>I am trying to convert a BERT model to ONNX. However, I think there is some discrepancy in the ONNX conversion module. I ran the sample conversion presented here on the website: <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html" class="inline-onebox" rel="noopener nofollow ugc">(optional) Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime — PyTorch Tutorials 1.10.1+cu102 documentation</a></p>
<p>I did not find the test to succeed and there is a significant numerical error between the PyTorch model and the ONNX version of the model generated by the conversion.</p>
<p>Python (3.8.10)<br>
ONNX (1.10.2)<br>
NumPy (1.21.2)<br>
ONNXRuntime (1.10.0)</p>

Nimit: <p>Update: I have the latest branch of PyTorch (built from source) and ONNX (built from source).</p>
<p>Here’s a sample output when I run the code:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "sample_convert.py", line 96, in &lt;module&gt;
    np.testing.assert_allclose(to_numpy(torch_out), ort_outs[0], rtol=1e-03, atol=1e-05)
  File "/usr/local/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1531, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.001, atol=1e-05

Mismatched elements: 451416 / 451584 (100%)
Max absolute difference: 7.256867
Max relative difference: 317452.75
 x: array([[[[ 0.178333,  0.240012,  0.491639, ...,  0.368835,  0.33173 ,
           0.327067],
         [ 0.153973,  0.266167,  0.592167, ...,  0.480333,  0.449961,...
 y: array([[[[ 0.620332,  0.832479,  1.062318, ..., -0.410807, -0.471132,
          -0.290673],
         [ 0.496837,  0.832677,  1.17443 , ..., -0.337473, -0.433076,...
</code></pre>
Partial transfer learning EfficientNet

VirginieBfd: <p>Hi, can someone ping me to an example of partial transfer learning with EfficientNet? For instance, unfreezing the last two blocks of the network?</p>
<p>Thanks in advance!</p>

ptrblck: <p>You can freeze all parameters of the model first via:</p>
<pre><code class="lang-python">for param in model.parameters():
    param.requires_grad_(False)
</code></pre>
<p>and later unfreeze the desired blocks by printing the model (via <code>print(model)</code>) and use the corresponding module names to unfreeze their parameters. E.g. assuming the last two “blocks” are named <code>block1</code> and <code>block2</code>, this should work:</p>
<pre><code class="lang-python">for param in model.block1.parameters():
    param.requires_grad_(True)
for param in model.block2.parameters():
    param.requires_grad_(True)
</code></pre>

VirginieBfd: <p>Indeed, thanks for the suggestion. A bit granular but works!</p>
<pre><code class="lang-auto">def get_model(model_name):
    model = timm.create_model(model_name, pretrained=True)
    model.reset_classifier(NUM_CLASSES)
    return model

model = get_model('efficientnet_b0')
for name, _ in model.named_children():
    print(name)

for param in model.parameters():
    param.requires_grad_(False)

blocks_to_retrain = 1
count = 0
for name, block in model.blocks.named_children():
    if count &gt;= (len(model.blocks) - blocks_to_retrain):
        print(f'Unfreeze block {name}')
        for pname, params in block.named_parameters():
            if 'bn' not in pname:
                params.requires_grad = True
    else:
        print(f'Keep block {name} frozen')
    count += 1


for name, child in model.named_children():
    if name in ['conv_head', 'act2', 'global_pool', 'classifier']:
        print(f'Unfreeze block {name}')
        for params in child.parameters():
            params.requires_grad == True
</code></pre>
<p>Here an example of partial transfer learning</p>

AI-P-K: <p>I use EfficientNet as feature extractor and only train the las layer. The training works however at each epoch i get this warning</p>
<p>UserWarning: An output with one or more elements was resized since it had shape [1204224], which does not match the required output shape [8, 3, 224, 224].This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at  /opt/conda/conda-bld/pytorch_1641456362397/work/aten/src/ATen/native/Resize.cpp:24.)<br>
return torch.stack(batch, 0, out=out)</p>
<p>Does anyone know what it is?</p>
Convert a variable into numpy then reconvert it back

JZArray: <p>Hello all! I have a question about the autograd, please look at the following the example.</p>
<blockquote>
<p>out = net(input)<br>
out = out.detach().numpy()<br>
new_output = function_1(out)  # function_1 contains some numpy operations<br>
new_output = torch.from_numpy(new_output ).float()<br>
new_output  = Variable(new_output , requires_grad=True).to(device)<br>
loss = loss_function(new_output, label)<br>
loss.backward</p>
</blockquote>
<p>When I call ‘loss.backward’, will the parameters in net also be updated?</p>
<p>Thanks</p>

InnovArul: <p>In short, No. The parameters of the model will not receive gradients as you have <code>.detach()</code>ed the computation graph from the loss function.</p>

s11: <aside class="quote no-group" data-username="JZArray" data-post="1" data-topic="138807">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jzarray/48/38667_2.png" class="avatar"> JZArray:</div>
<blockquote>
<p>es_grad</p>
</blockquote>
</aside>
<p>Hi, do you know how to solve this problem with other operations?  Can I perform Numpy operations between the output of a network and loss calculation?</p>

Andrea_Rosasco: <p>Nope, to my understanding, if you want to backpropagate through them, they have to be torch operations. You should re-implement those functions using torch.</p>

s11: <p>Ok. Thank you for your answer.</p>

JZArray: <p>No, you can’t, but in torch there are lots of similar operations as numpy</p>
Use autograd for customized mathematical function

Zeeyuu: <p>Hi, I would like to clarify is it possible to use autograd to find the derivative or to do gradient descent on a customized function? If yes a simple example is appreciated (e.g. how to find the derivative/ do gradient descent for f(x_1,x_2) = 2 * x_1^3 + 3 * x_2^2 where x_1, x_2  and f(x_1, x_2) are all scalars). Thanks!</p>

anantguptadbl: <p><a class="mention" href="/u/zeeyuu">@Zeeyuu</a> Cubing and squaring are higher order polynomical equations.But If you are able to represent the formula in a series of matrix operations, you can easily use autograd to get the Jacobian</p>

Zeeyuu: <p>Do I need to define my function as a nn.Module? If so can you provide a toy example?<br>
Thanks a lot.</p>

anantguptadbl: <p>example formula<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30aa7d1def1cf3d5e79a2273e8384c4fe0080608.png" alt="image" data-base62-sha1="6WwbIAjkKoZ4AtXDEhYgrHacZqE" width="253" height="52"></p>
<pre><code class="lang-auto">x = torch.Tensor([1,2,3])
W = torch.nn.Parameter(torch.Tensor([[0.4],[0.5],[-0.6]]), requires_grad=True)
c = torch.matmul(x, W)
d = torch.nn.functional.sigmoid(c)
loss = torch.tensor([1.]) - d
loss.backward()
W.grad
</code></pre>
<pre><code class="lang-auto"># Results
tensor([[-0.2403],
        [-0.4805],
        [-0.7208]])
</code></pre>
Gradient flow of MSE loss after multiplication with mask

cbd: <p>I want the loss function like ||Mask · (output − target )||2. Should the below code works fine for gradient flow? <strong>Note</strong>: Mask is bool(True or False).</p>
<pre><code class="lang-auto">
import torch.nn as nn
import torch.nn.functional as F
import torch
from torchsummary import summary

class TwoLayerNet(nn.Module):
    def __init__(self):
        super(TwoLayerNet, self).__init__()
        conv_layers=[]
        self.conv=[]      

        conv_layers.append(nn.Sequential(nn.Conv2d(3, 5, kernel_size=3, stride=1,padding=1, bias=True) ) )
        conv_layers.append(nn.Sequential(nn.Conv2d(5, 5, kernel_size=3, stride=1,padding=1, bias= True),nn.LeakyReLU(0.2,True) ))
        self.conv = nn.Sequential(*conv_layers)     
                     
    def forward(self, x):
        x1=self.conv(x)
        return x1

torch.manual_seed(88)
x = torch.ones(1,3,2,2)
tar=torch.ones(1,5,2,2)*0.1

model = TwoLayerNet()
y_pred = model(x)
mask=y_pred&gt;tar
print("mask",mask)
crit=torch.nn.MSELoss()
loss= mask*crit(y_pred,tar)</code></pre>

wbsst: <p>I’m facing the same error, when I add <code>requires_grad=True</code> it works but with no improvement at all the loss is the same for all steps no changes at all</p>

cbd: <p>Just check to follow optimizer.step() after backward(). It should work. My case is different. It multiply with mask to calculate loss function.</p>

wbsst: <p>check <a href="https://discuss.pytorch.org/t/custom-loss-function-no-grad-and-oom-error/140954">here</a> to get an idea about the problem I’m facing I think it’s similar to yours</p>
How to get heatmap?

Cverlpeng: <p>Hi,<br>
In many papers, I will see the heat map, how to get it, what is the API of pytorch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cd13ae879e5acf7491d4cb141e20d1abaa39983f.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd13ae879e5acf7491d4cb141e20d1abaa39983f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cd13ae879e5acf7491d4cb141e20d1abaa39983f_2_652x500.png" alt="image" data-base62-sha1="tgc33WbEkxnuoVEq0tFrq4Wojzp" width="652" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cd13ae879e5acf7491d4cb141e20d1abaa39983f_2_652x500.png, https://discuss.pytorch.org/uploads/default/original/2X/c/cd13ae879e5acf7491d4cb141e20d1abaa39983f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/cd13ae879e5acf7491d4cb141e20d1abaa39983f.png 2x" data-dominant-color="33349A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">675×517 58.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0cf3d15ae15957ba230f3034504d4f8280530766.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0cf3d15ae15957ba230f3034504d4f8280530766" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0cf3d15ae15957ba230f3034504d4f8280530766_2_363x500.jpeg" alt="image" data-base62-sha1="1QA6CLaCe7CVXhSDhxQSSPPW0ES" width="363" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0cf3d15ae15957ba230f3034504d4f8280530766_2_363x500.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/0/0cf3d15ae15957ba230f3034504d4f8280530766.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0cf3d15ae15957ba230f3034504d4f8280530766.jpeg 2x" data-dominant-color="2C3BA3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">508×699 166 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>thanks</p>

Balamurali_M: <p>You can convert the torch to numpy.<br>
Use plt.imshow(arr,map=‘jet’)</p>
<p>This is how people find heatmap and report in paper.</p>

Cverlpeng: <p>plt.imshow(arr, map=‘jet’)<br>
arr: is  output of conv2d  ?<br>
thanks</p>

good_standard: <p>Please, tell me a bit in detail about “arr”?.</p>
<p>I use deeplabv3+ → predict output, so arr is output, is it right?</p>
NoneType' object has no attribute 'shape'

Samanyu-V: <p>so this is my code</p>
<pre><code>sentence = tokenize(sentence)
X = bag_of_words(sentence,all_words)
X = X.reshape(1,X.shape[0])
X = torch.from_numpy(X).to(device)
</code></pre>
<p>and this is the error given<br>
AttributeError: ‘NoneType’ object has no attribute ‘shape’</p>

talhaanwarch: <p>what is bag_of_words</p>

Samanyu-V: <p>Its basically a neural network function…</p>
<p>def bag_of_words(tokenized_sentence,words):<br>
sentence_words = [stem(word)for word in tokenized_sentence]<br>
bag = np.zeros(len(words),dtype=np.float32)</p>
<p>for idx , w in enumerate(words):<br>
if w in sentence_words:<br>
bag[idx] = 1</p>
<pre><code>        return bag
</code></pre>
<p>the code for bag of words</p>

talhaanwarch: <p>i did not get this error</p>
<pre><code class="lang-auto">sentence='i am going to school'
vocab=['i','go','there','he','she','it','college','school']

import nltk
from nltk.stem.porter import PorterStemmer
stemmer=PorterStemmer()

def tokenize(sentence):
    return sentence.split()

def bag_of_words(tokenized_sentence,words):
    sentence_words = [stemmer.stem(word) for word in tokenized_sentence]
    bag = np.zeros(len(words),dtype=np.float32)

    for idx , w in enumerate(words):
        if w in sentence_words:
            bag[idx] = 1
    return bag

sentence = tokenize(sentence)
X = bag_of_words(sentence,vocab)
print(X)
</code></pre>
Memory leak while training model generated by torch.fx.symbolic_trace() in data parallel mode

shengchunnan: <p><strong>There is memory leak when I train a model in data parallel mode if this model is generated by  torch.fx.symbolic_trace().</strong><br>
<strong>There seems to be no memory leak if I turn off one of the follow options:</strong></p>
<ul>
<li>data parallel mode (<strong><code>torch.nn.DataParallel</code></strong>)</li>
<li>This model is <strong><code>torch.fx.GraphModule</code></strong> created via <strong><code>torch.fx.Tracer</code></strong>
</li>
</ul>
<p>The python script that can reproduce this BUG is attached here. This program would leak over 1GB<br>
memory per 10 training epochs.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.fx as fx
import torchvision
import torchvision.transforms as transforms

batch_size = 64

class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()

        self.backbone = torchvision.models.densenet121()
        self.linear = nn.Linear(1000, 10)

    def forward(self, x):
        out = self.backbone(x)
        out = self.linear(out)

        return out

train_dataset = torchvision.datasets.CIFAR10(root='data/',
                                                train=True,
                                                transform=transforms.ToTensor(),
                                                download=True)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                            batch_size=batch_size, 
                                            shuffle=True)

model = MyNet()
model = fx.symbolic_trace(model)

model = model.to('cuda')
model = nn.DataParallel(model)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(
                model.parameters(),
                lr=0.01,
                momentum=0.8,
                nesterov=True,
                weight_decay=0.0001)

for epoch in range(1000):
    print('epoch:', epoch)
    for i, (images, labels) in enumerate(train_loader):

        images = images.to('cuda')
        labels = labels.to('cuda')
        
        outputs = model(images)

        loss = criterion(outputs, labels)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        _, predicted = torch.max(outputs.reshape(-1, 10).data, 1)
        correctness = (predicted == labels).sum().item() / batch_size

        print('\ttraining loss: {:3.4f}'.format(loss.item()))
        print('\ttraining correctness: {:3.4f}'.format(correctness))
</code></pre>
<h2>
<a name="pytorch-1100-1" class="anchor" href="#pytorch-1100-1"></a>PyTorch: 1.10.0</h2>
<h2>
<a name="system-ubuntu-1804-2" class="anchor" href="#system-ubuntu-1804-2"></a>System: ubuntu 18.04</h2>
<h2>
<a name="cuda-102-3" class="anchor" href="#cuda-102-3"></a>CUDA: 10.2</h2>