akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get value from c10::Dict<c10::IValue, c10::IValue>

username: lamhoangtung
cooked: <p>Hi, I’m using a TorchScript Model on Pytorch C++ Frontend.</p>
<p>The model in Python return an <code>output</code> dict as  <code>Dict[str, List[torch.Tensor]]</code>.</p>
<p>When I use it in C++, it return a <code>c10::Dict&lt;c10::IValue, c10::IValue&gt;</code>. But what is the equivalents of this Python code:</p>
<pre><code class="lang-python">value_a = output['key_a']
value_b = output['key_b']
</code></pre>
<p>in C++ to get value from <code>c10::Dict</code></p>
<p>I have tried this but it does not work.</p>
<pre><code class="lang-cpp">torch::IValue key_a("key_a");
torch::IValue key_b("key_b");
c10::IValue value_a = output[key_a];
c10::IValue value_b = output[key_b];
std::cout &lt;&lt; value_a &lt;&lt; std::endl;
std::cout &lt;&lt; value_b &lt;&lt; std::endl;
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">error: type 'c10::Dict&lt;c10::IValue, c10::IValue&gt;' does not provide a subscript operator
</code></pre>
<p>Anyone can give me a little help here. Thanks</p>

username: lamhoangtung
cooked: <p>Also asked in here <a href="https://stackoverflow.com/questions/63917001/get-value-from-c10dictc10ivalue-c10ivalue-in-pytorch-c" rel="nofollow noopener">https://stackoverflow.com/questions/63917001/get-value-from-c10dictc10ivalue-c10ivalue-in-pytorch-c</a></p>

username: lamhoangtung
cooked: <p>As answer by <a href="https://stackoverflow.com/users/10886420/szymon-maszke" rel="nofollow noopener">https://stackoverflow.com/users/10886420/szymon-maszke</a> on the StackOverFlow question. The correct way to do this is:</p>
<pre><code class="lang-cpp">auto a = output.at("key_a")
</code></pre>
Calculated padded input size per channel: (254 x 446 x 1). Kernel size: (3 x 3 x 64). Kernel size can't be greater than actual input size

username: lifeblack
cooked: <p>Hi there, I have two problems…<br>
I have a tensor with this shape: [16, 1, 256, 448, 1] , I applied a conv layer {kernel size: 3<em>3</em>1 and channels: 64} and the output is:[16, 64, 254, 446, 1] .<br>
I wrote this code for the conv layer:<br>
<code>self.conv_1= nn.Sequential(             nn.Conv3d(1,64, kernel_size=(3,3)),             nn.ReLU()         )</code><br>
How to change it that the output be [16,64, 256, 448, 1]<br>
then, I pass this tensor over a conv layer{kernel size: 3<em>3</em>64 and channels: 64} I encountered an error:</p>
<pre><code class="lang-auto">RuntimeError: Calculated padded input size per channel: (254 x 446 x 1). Kernel size: (3 x 3 x 64). Kernel size can't be greater than actual input size
</code></pre>
<p>I wrote this code for the conv layer:</p>
<pre><code class="lang-auto">self.conv_2= nn.Sequential(
            nn.Conv3d(64,64, kernel_size=(3,3,64)),
            nn.ReLU()
        )
</code></pre>
<p>Would you mind helping me solve these problems</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lifeblack" data-post="1" data-topic="96147">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lifeblack/48/18210_2.png" class="avatar"> lifeblack:</div>
<blockquote>
<p>How to change it that the output be [16,64, 256, 448, 1]</p>
</blockquote>
</aside>
<p>You can add <code>padding</code> to the convolution to keep the spatial sizes.</p>
<aside class="quote no-group quote-modified" data-username="lifeblack" data-post="1" data-topic="96147">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lifeblack/48/18210_2.png" class="avatar"> lifeblack:</div>
<blockquote>
<p>then, I pass this tensor over a conv layer […]</p>
</blockquote>
</aside>
<p>The <code>conv_2</code> layer uses a kernel size of 64 in the last spatial dimension, while the activation input has a size of 1 in this dim. This won’t work, since the (padded) input activation needs to have at least the same size as the kernel.</p>
Too much time Dataloader to run

username: Maicol_Polvere
cooked: <p>`def training(epoch, net, device, criterion, train_data, optimizer,tb):<br>
“”"<br>
Run one training epoch<br>
:param epoch: Current epoch<br>
:param net: Network<br>
:param device: Torch device<br>
:param criterion: function to evaluate the loss<br>
:param train_data: Training Dataset<br>
:param optimizer: Optimizer<br>
:return:  Average Losses for Epoch<br>
“”"</p>
<pre><code>results = 0
batch_idx = 0
# change flag of training
net.train()

tic = time.perf_counter()

for sample in train_data:  
    
    toc = time.perf_counter()      
    
    batch_idx +=1

    # load of the normalized image
    img = sample['img'].to(device)
    
    # load of the ground bounding boxes
    ground_bb = sample['bb'].to(device)`
</code></pre>
<p>Each cycle takes too much time at loading the data. In fact trying to time the interval between before the for loop and the initial of the for loop it takes about 1 minute. How I can speed this each cycle?</p>

username: ptrblck
cooked: <p>The first iteration can be slower, since the first batch would have to be loaded. However, the following iterations should be faster, if the actual model workload is not tiny.<br>
If you want to avoid this slowdown in each epoch, you could use <a href="https://discuss.pytorch.org/t/enumerate-dataloader-slow/87778/4">this workaround</a>.</p>

username: Maicol_Polvere
cooked: <p>Thank you so much. However I have a slowdown not only at the beginning of every epoch but also in each cycle of <code>for sample in train_data</code>. I think this slowdown is caused by the data transformation (rotation and zooming) inside the__getitem__ method of my custom dataset. How I can perform in parallel data loading and training?</p>

username: ptrblck
cooked: <p>If you are using multiple workers in the <code>DataLoader</code> via <code>num_workers&gt;=1</code>, each worker will load the batch in the background while the GPU is busy with the model training.</p>

username: Maicol_Polvere
cooked: <p>I have done it setting num_workers = 8. However for 8 training cycle I have not to wait the loading of the data but after that the <strong>for loop</strong> of training stops and wait for loading of data.</p>

username: ptrblck
cooked: <p>If you are seeing the slowdown after a full epoch, please refer to my previous post for a workaround.</p>

username: Maicol_Polvere
cooked: <p>unfortunately I see a slowdown after each batch</p>

username: ptrblck
cooked: <p>This could indicate that your training loop might be increasing the computation graph and thus not the data loading but the training is slower.<br>
Could you remove the complete model and training, and just time the data loading loop?</p>

username: terro
cooked: <p>Thank you for your answer. I have found the problem. In <strong>getitem</strong>  method in my custom dataset images transformations are performed using function from skiimage that takes too much time to perform. I have substituted them with opencv functions that performs more or less the same operations very very faster. In this way loading the batches are more fast and It solves my problem</p>
CycleGan - Turn off training

username: chaslie
cooked: <p>morning all,</p>
<p>I was looking at a Keras cycleGan and they use the following parameters to deactivate training in the combined model of generator1 → discriminator 1 → generator -&gt;2</p>
<pre><code class="lang-auto">def define_composite_model(g_model_1, d_model, g_model_2, image_shape):
	# ensure the model we're updating is trainable
	g_model_1.trainable = True
	# mark discriminator as not trainable
	d_model.trainable = False
	# mark other generator model as not trainable
	g_model_2.trainable = False
	# discriminator element
</code></pre>
<p>Would the equilant be:</p>
<blockquote>
<p>g_model_2=g_model_2.train(mode=false)</p>
</blockquote>
<p>where g_model_2 is defined by a class? if so why do i get the error message???</p>
<blockquote>
<p>TypeError: train() missing 1 required positional argument: ‘self’</p>
</blockquote>
<p>thanks in advance.</p>
<p>Chaslie</p>

username: ptrblck
cooked: <p>Most likely <code>model.train()</code> is not corresponding to the <code>trainable</code> argument in Keras.<br>
<code>model.train()</code> and <code>model.eval()</code> switches the behavior of some modules, e.g. it’ll disable dropout during <code>eval</code> and use the running stats in batchnorm layers.<br>
If you want to freeze parameters, you can set <code>param.requires_grad=False</code>.<br>
However, note that optimizers with running estimates or momentum can still update these parameters even if the gradient is zero.</p>

username: chaslie
cooked: <p>Hi Ptrblck,</p>
<p>would</p>
<blockquote>
<p>with torch.no_grad():</p>
</blockquote>
<p>for that particuler training block work?</p>

username: ptrblck
cooked: <p><code>with torch.no_grad()</code> would work, if you don’t need any gradients in parameters, which are used before this block, since the <code>no_grad()</code> guard would not create the computation graph for this section.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python"># setup
lin1 = nn.Linear(1, 1)
lin2 = nn.Linear(1, 1)
lin3 = nn.Linear(1, 1)

x = torch.randn(1, 1)

# case1
with torch.no_grad():
    out = lin1(x)
    out = lin2(out)
out = lin3(out)
out.mean().backward()

print(lin1.weight.grad, lin2.weight.grad, lin3.weight.grad)
&gt; None None tensor([[-0.8122]])
lin1.zero_grad(), lin2.zero_grad(), lin3.zero_grad()

# case2
out = lin1(x)
with torch.no_grad():
    out = lin2(out)
out = lin3(out)
out.mean().backward()

print(lin1.weight.grad, lin2.weight.grad, lin3.weight.grad) # !!! lin1 doesn't have grads !!!
&gt; None None tensor([[-0.8122]])
lin1.zero_grad(), lin2.zero_grad(), lin3.zero_grad()

# case3
for param in lin2.parameters():
    param.requires_grad = False

out = lin1(x)
out = lin2(out)
out = lin3(out)
out.mean().backward()
print(lin1.weight.grad, lin2.weight.grad, lin3.weight.grad)
&gt; tensor([[-0.0768]]) None tensor([[-0.8122]])
</code></pre>
Torch.backends.cudnn.enabled = False

username: freezek
cooked: <p>So, what’s the corresponding code in C++?</p>

username: freezek
cooked: <p>I suffer from different results between cpu and gpu…</p>

username: ptrblck
cooked: <p><code>at::globalContext().setBenchmarkCuDNN(flag)</code> should work in C++.<br>
However, <code>benchmark</code> is off by default so I guess that’s not the root cause of the non-deterministic results.<br>
You could try to use <code>at::globalContext().deterministicCuDNN()</code> or <code>at::globalContext().setDeterministic()</code> in the nightly binaries and refer to <a href="https://pytorch.org/docs/stable/notes/randomness.html">this document</a> for more information.</p>
Warm up for DataLoader

username: ekosman
cooked: <p>I’m using <code>torch.utils.data.DataLoader</code> with a custom <code>data.Dataset</code> for loading video batches.<br>
Since video is quiet heavy, I’m experiencing some latency issues, especially with the first batches of every epoch. For example:</p>
<pre><code class="lang-auto">2020-09-14 00:16:42,229 Epoch 120/200      Iteration 0/322    Loss: 9.586519445292652e-05    Time: 22.14 seconds/iteration
2020-09-14 00:18:24,911 Epoch 120/200      Iteration 50/322    Loss: 0.00014619871217291802    Time: 2.447 seconds/iteration
2020-09-14 00:19:35,139 Epoch 120/200      Iteration 100/322    Loss: 9.589765249984339e-05    Time: 1.931 seconds/iteration
2020-09-14 00:21:04,289 Epoch 120/200      Iteration 150/322    Loss: 8.543662988813594e-05    Time: 1.882 seconds/iteration
2020-09-14 00:22:16,219 Epoch 120/200      Iteration 200/322    Loss: 9.707502613309771e-05    Time: 1.772 seconds/iteration
2020-09-14 00:23:53,514 Epoch 120/200      Iteration 250/322    Loss: 0.00016338478599209338    Time: 1.806 seconds/iteration
2020-09-14 00:25:03,165 Epoch 120/200      Iteration 300/322    Loss: 0.00013553885219153017    Time: 1.738 seconds/iteration
</code></pre>
<p>Notice how long are the first iterations of the training of this epoch, and it actually happens in every epoch. I assume it is because the <strong>iter</strong> method just resets the loader for each epoch, so I was wondering if there’s any solution for warming it up? I already maximized <code>num_workers</code> by setting it to 16.</p>

username: ptrblck
cooked: <p>You can try to use <a href="https://discuss.pytorch.org/t/enumerate-dataloader-slow/87778/4">this workaround</a>.</p>
Precision,recall and f1 score values in EXP

username: Khola
cooked: <p>I am using doing binary segmentation using Signet, IoU loss. the problem is Precision, recall and f1 score values become exp. kindly guide me if there is an issue with my loss function or there is any other issue.i have pasted the loss  code below<br>
Thank you</p>
<pre><code class="lang-auto">
class IoULoss(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(IoULoss, self).__init__()

    def forward(self, inputs, targets, smooth=1):

        inputs = inputs.view(-1)
        targets = targets.view(-1)

        intersection = (inputs * targets).sum()
        total = (inputs + targets).sum()
        union = total - intersection 
        
        IoU = (intersection + smooth)/(union + smooth)
                
        return 1 - IoU
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/284b6a71835ddc4a2e47296c1c5d8d261a8db362.png" alt="results" data-base62-sha1="5KsFW84yfhSekI7Ki06SYb738SS" width="482" height="238"></p>

username: ptrblck
cooked: <p>The result table indicates that the precision, recall, F1, and Ji are all zero. You can ignore the small numbers and basically treat them as zero values, so you should check your metric calculation as well as the results of your model.</p>

username: Khola
cooked: <p>If I stop and then resume the training everything works fine. I have used PyTorch anomaly detection and even tried to clip gradient between -5 to 5 but it’s also not working. but the problem is  what can be the issue then</p>
<ol>
<li>Data does not contain any nan value.</li>
<li>Changing the model does not work.</li>
<li>Gradient clipping does not work.</li>
<li>learning rate is 1e^-3 at the start of training and updating it using lr scheduler</li>
<li>Apparently there is no issue in metric and loss calculation code.</li>
</ol>
<p>then what can be the issue<br>
kindly guide me</p>
<p>Any help will be appreciated<br>
Thanks</p>

username: ptrblck
cooked: <p>I don’t see any NaNs in your output, so I’m unsure why you are checking for these invalid values.<br>
In your table the scientific notation is used, which is indicating values close to zero.</p>
What is the input of dog_response

username: Dawars
cooked: <p>The <a href="https://kornia.readthedocs.io/en/latest/feature.html#kornia.feature.dog_response" rel="nofollow noopener">documentation</a> says that the input is a “Gaussian 5d input” with dimensions [B,C,D,H,W]. It is not clear to me how to get this representation and what it means exactly.</p>
<p>I’m trying to extract patches to use with HardNet.<br>
Based on reading the SIFT paper I think I need to combine the following functions:</p>
<ol>
<li>ScalePyramid() to get the different octaves/levels</li>
<li>dog_response - which subtracts neighbouring level in each octave</li>
<li>Non maximum suppression - to get the feature points</li>
<li>extract_patches_from_pyramid() - to get the local patches to be fed to HardNet</li>
</ol>
<p>I’m not sure how to combine these functions since the inputs/outputs don’t seem the straightforwardly match.<br>
An example demonstrating this use case would be much appreciated.</p>
<p>I know about this example which shows how to extract the patches around keypoints: <a href="https://github.com/kornia/kornia/blob/master/examples/feature_detection/extract_local_patches.ipynb" rel="nofollow noopener">https://github.com/kornia/kornia/blob/master/examples/feature_detection/extract_local_patches.ipynb</a></p>

username: Dawars
cooked: <p>I found this repo which does exactly what I want: <a href="https://github.com/ducha-aiki/extract_patches" rel="nofollow noopener">https://github.com/ducha-aiki/extract_patches</a></p>
<p>Thanks <a class="mention" href="/u/ducha-aiki">@ducha-aiki</a> !</p>

username: ducha-aiki
cooked: <p>You don’t need to combine anything, it is what ScaleSpaceDetector does. <a href="https://kornia.readthedocs.io/en/latest/feature.html#kornia.feature.ScaleSpaceDetector" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/feature.html#kornia.feature.ScaleSpaceDetector</a></p>
<p>Look at this example: <a href="https://github.com/kornia/kornia-examples/blob/master/image-matching-example.ipynb" rel="nofollow noopener">https://github.com/kornia/kornia-examples/blob/master/image-matching-example.ipynb</a><br>
and instead of <code>resp = BlobHessian()</code> use <code>resp = BlobDoG()</code> and pass <code>scale_space_response=True</code> when creating <code>ScaleSpaceDetector</code></p>
Differentiable indexing for the channels

username: tremblerz
cooked: <p>I am trying to write a neural network which can learn to assign scores to each channel. The following code is not able to run backprop. Here is a toy example to illustrate the problem</p>
<pre><code class="lang-auto">class ChannelScorer(nn.Module):
    def __init__(self):
        super(ChannelScorer, self).__init__()
        self.layer1 = nn.Conv2d(in_channels=100, out_channels=200, kernel_size=3, stride=2)
        self.layer2 = nn.Linear(1800, 100)
    def forward(self, x):
        z1 = self.layer1(x)
        z1 = torch.flatten(z1, start_dim=1)
        z2 = self.layer2(z1)
        return z2

channel_scorer = ChannelScorer()

x = torch.rand(32, 100, 8, 8)
y = torch.rand(32, 100, 8, 8)
indices_score = channel_scorer(x)
indices = torch.topk(indices_score, 50)[1]
x[:, indices, :, :] = 0

loss = (x-y)**2
loss = torch.mean(loss)

loss.backward()
</code></pre>
<p>I am getting error as</p>
<blockquote>
<p>97     Variable._execution_engine.run_backward(<br>
98         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 99         allow_unreachable=True)  # allow_unreachable flag<br>
100<br>
101<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>My guess is that the indexing using the topk indices is making this whole thing non-differentiable. Is there a way to circumvent this situation?</p>

username: ptrblck
cooked: <p>The returned <code>indices</code> by <code>topk</code> do not have a <code>grad_fn</code>, since this operation is not differentiable, and will thus detach the computation graph.</p>
Is that normal to train with 2 sets and/or 2 diff loss?

username: thao
cooked: <p>I have 2 train sets: one with label and one with no label.<br>
When training, i’m simultaneously load  one batch from labelled set, calculate loss in 1 way; one batch from unlabeled set, calculate loss in other way. Finally I sum them (2 loss) and <code>loss.backward()</code> .</p>
<p>Is this way ok ? it’s quite uncommon in my mind so just ask if the engine truthfully know how to back-propagate  ?<br>
Thank you.</p>

username: tom
cooked: <p>Semi-supervised learning is a relatively common technique to deal with situations in which labeled data isn’t abundant and might look like this.<br>
PyTorch’s autograd it is much the same if the loss has 1, 2, or 10 terms - losses of minibatches typically are just sums or means of the individual items under the hood, too.<br>
If you have very separate computations, you could also forward with the first, backward, forward for the second and backward again. PyTorch will accumulate the gradients of the two (which is why you need to zero gradients), and it might be much easier on GPU memory / you can afford larger batch sizes.</p>
<p>Best regards</p>
<p>Thomas</p>
AssertionError: action shape did not match expected=(1,) actual=(1, 1)

username: datascguy
cooked: <p>Hello,</p>
<p>I am trying to replicate the following code: <a href="https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail" rel="nofollow noopener">https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail</a> and run it on a server. However, when I run it on the server side, I get the following error:</p>
<p>AssertionError: action shape did not match expected=(1,) actual=(1, 1)</p>
<p>It seems like there is a problem with the dimensions of the action space, but I don’t get the same Error when I try to train it locally on my laptop (only CPU).</p>
<p>Thank you in advance.</p>

username: ptrblck
cooked: <p>I guess you might be using different PyTorch versions, where this additional check was either added or removed.<br>
Could you post the stack trace so that we can have a look at the error?</p>
Matrix multiplication on matrix with dimension >= 4

username: qiminchen
cooked: <p>I have two tensors <code>a = torch.randn(2, 3, 10, 25, 25)  # B x C x *</code> and <code>b = a.permute(0, 2, 1, 3, 4)  # 2 x 10 x 3 x 25 x 25</code>, I want to do matrix multiplication on <code>a</code> and <code>b</code> to get the output with size of <code>2 x 3 x 3 x 25 x 25</code>,</p>
<pre><code class="lang-auto">a = torch.randn(2, 3, 10, 25, 25)
b = a.permute(0, 2, 1, 3, 4)
c = torch.matmul(a, b)
</code></pre>
<p>but I ran into this error,</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (10) must match the size of tensor b (3) at non-singleton dimension 2
</code></pre>
<p>so I check the <a href="https://pytorch.org/docs/stable/generated/torch.matmul.html#torch.matmul" rel="nofollow noopener">torch.matmul</a> and found out that I should permute <code>a = a.permute(0, 3, 4, 1, 2)  # 2 x 25 x 25 x 3 x 10</code> and <code>b</code> and then do matrix multiplication. But in training, pytorch is in the default channel first so is swapping the channel to last, doing matmul and swapping channel back to first the only solution?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="qiminchen" data-post="1" data-topic="96143">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qiminchen/48/20829_2.png" class="avatar"> qiminchen:</div>
<blockquote>
<p>But in training, pytorch is in the default channel first so is swapping the channel to last, doing matmul and swapping channel back to first the only solution?</p>
</blockquote>
</aside>
<p>The matrix multiplication doesn’t “check” for the memory format, i.e. if the tensors are in channels-first or channels-last, but just uses both tensors in the provided shapes.<br>
It’s on the user to make sure the result is expected, so I would recommend to create tensors with known values and compare the output to this reference.</p>
Update conv layer according to another gradients

username: Frida
cooked: <p>Hi<br>
Any idea how can I use an unlearned kernel and still updating the original filter using the response to that filter?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Would you like to keep the “unlearned kernel” in its original state and use the gradients to update a clone of this kernel?<br>
If so, note that the clone of this kernel would eventually get gradients for completely different parameters.<br>
I’m unsure if I understood the question correctly, so please feel free to add more information about the use case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Convert a tensor of images to PIL image

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>I want to convert a tensor of images to PIL images.</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as transforms

tran1 = transforms.ToPILImage()
x = torch.randn(64, 3, 32, 32) # 64 images here
pil_image_single = tran1(x[0]) # this works fine
pil_image_batch = tran1(x) # this does not work
</code></pre>
<p>Can somebody tell me if there is any efficient way to do the final line without going through a loop?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I don’t think there is a way to achieve this without a loop, since the underlying <code>PIL.Image.fromarray</code> operation expects a single numpy array, if I’m not mistaken.</p>
Improve existing checkpoint in Pytorch

username: gnadaf
cooked: <p>Hello,<br>
My goal is to improve the existing <strong>checkpoint mechanism</strong>. Please share your suggestions or opinions</p>

username: ptrblck
cooked: <p>What do you think is missing or not working properly at the moment? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: gnadaf
cooked: <p>I am referring the paper “On Efficient Constructions of Checkpoints” <a href="https://proceedings.icml.cc/static/paper_files/icml/2020/5071-Paper.pdf" rel="nofollow noopener">https://proceedings.icml.cc/static/paper_files/icml/2020/5071-Paper.pdf</a></p>
<p>I am not sure this is implemented or not, my implementation is based on this paper.<br>
If code for this research project is implemented, could you please point me or share the link.</p>

username: ptrblck
cooked: <p>I’m not aware on an implementation of this algorithm.</p>
How is the data of tensor iterated within `serial_for_each` func of TensorIterator.cpp？

username: Bob_Liao
cooked: <pre><code class="lang-auto">void TensorIterator::serial_for_each(loop2d_t loop, Range range) const {
  if (range.size() == 0) {
    return;
  }
  auto strides = get_strides();
  while (strides.size() &lt; 2 * ntensors()) {//why would this multiplied by 2?    
      strides.push_back(0);
  }

  auto base_ptrs = get_base_ptrs();
  if (ndim() &lt;= 1) {
    auto ptrs = get_data_ptrs(base_ptrs, { range.begin });
    loop(ptrs.data(), strides.data(), range.size(), 1);
  } else {
    auto counter = DimCounter(shape_, range);
    while (!counter.is_done()) {
      auto ptrs = get_data_ptrs(base_ptrs, counter.values);
      auto step = counter.max_2d_step();
      loop(ptrs.data(), strides.data(), step[0], step[1]);
      counter.increment(step);
    }
  }
}
</code></pre>
<p>I can’t figure the DimCounter out. How does the iteration carries out? I am really new to Pytorch Sourcecode. Are there any resources about TensorIterator details? Thanks.</p>

username: Bob_Liao
cooked: <p>No one shows interest on this. OK, I have found some articles about Pytorch Internals about <code>TensorIterator</code>:</p>
<ul>
<li>
<a href="https://labs.quansight.org/blog/2020/04/pytorch-tensoriterator-internals/" rel="nofollow noopener">https://labs.quansight.org/blog/2020/04/pytorch-tensoriterator-internals/</a><br>
If you can read contents written in Chinese, you can also refer:</li>
<li>
<a href="https://zhuanlan.zhihu.com/p/129778637" rel="nofollow noopener">https://zhuanlan.zhihu.com/p/129778637</a><br>
This is super useful to help yourself understand details about TensorIterator.<br>
I still have some confusions. If you are also interested in this, please contact with me. I am glad to have discussions with you to help us each other to understand this completely. Thanks.</li>
</ul>
How to freeze specific channels in a layer?

username: ProGamerGov
cooked: <p>Freezing layers is straightforward enough, but how do I freeze specific channels in a layer?</p>

username: pfloat
cooked: <p>I think this can be done manually only by zeroing your weights after loss.backward().<br>
Check this topic:</p><aside class="quote quote-modified" data-post="1" data-topic="9457">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e8c25b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/freezing-part-of-the-layer-weights/9457">Freezing part of the Layer Weights</a> 
  </div>
  <blockquote>
    Hello Everyone, 
How could I freeze some parts of the layer weights to zero and not the entire layer. 
I tried below code, but it doesn’t freeze the specific parts(1:10 array in 2nd dimension) of the layer weights. 
I am new to ML &amp; started with Pytorch. Appreciate any help. Thanks. 
for child in model_ft.children(): 
print(“Freezing Parameters(1-&gt;10) on the Convolution Layer”,child) 
for param in child.parameters(): 
param.data[:,1:10,:,:].zero_() 
param.data[:,1:10,:,:].requires_grad = False 
o…
  </blockquote>
</aside>
Output of a subnetwork as weights of backbone network

username: SakuraTea
cooked: <p>Hi,</p>
<p>I am trying to implement the network structure in this paper <a href="https://arxiv.org/abs/1605.09673" rel="nofollow noopener">https://arxiv.org/abs/1605.09673</a> with PyTorch and have run into some problems.</p>
<p>May I ask how can I replace the weight of the backbone network with the output of another subnetwork? More importantly, how can I pass the gradient of weight to the output of subnetwork?</p>
<p>This is what I have done now:</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self):
        # some other stuff

        self.backbone = nn.Conv2d(
            # some parameters
        )

        self.weight_generator = nn.Sequential(
            nn.Linear(4096,576),
            nn.Tanh()
        )


     def forward(self, x):
        # some other stuff

        weight = self.weight_generator(x).view(50,64,3,3)

        self.backbone.weight.data = nn.Parameter(weight, requires_grad=True)
        out = self.backbone(x)

        return out
</code></pre>
<p>Current problem is that when I <code>print(list(myNetwork.weight_generator.children())[0].weight.grad)</code>, it always output <code>None</code>. Can anyone help me with it?</p>
<p>Thanks in advance for any help!</p>
Creating custom RNNs that work with packed sequences

username: mtanti
cooked: <p>I’m curious on whether PyTorch provides convenience functions for working with packed sequences. If I implement a custom RNN (not one of those provided by PyTorch), how would I work with a packed sequence?</p>

username: vdw
cooked: <p>Well, an alternative solution might be to ignore padding and packing (for the time being). I usually <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">create batches that ensure the input and/or target sequences have the same lengths</a>. But there might be <a href="https://discuss.pytorch.org/t/is-padding-and-packing-of-sequences-really-needed/66478">caveats</a>.</p>
Distributed Data Parallel allreduce

username: Sourabh_Daptardar
cooked: <p>Is there a way to verify if allreduce operation is getting called in a multinode DDP training with nccl backend ? In my training the results of single node and distributed training appear similar.  <a class="mention" href="/u/mrshenli">@mrshenli</a> <a class="mention" href="/u/apaszke">@apaszke</a></p>

username: mrshenli
cooked: <p>One option is to use <a href="https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/" rel="noopener nofollow ugc"><code>nvprof</code></a>.</p>
<blockquote>
<p>In my training the results of single node and distributed training appear similar.</p>
</blockquote>
<p>You mean speed is similar? What is the batch size fed into each DDP instance? When using DDP, the <code>batch_size</code> should be updated to <code>original_batch_size/world_size</code>.</p>

username: Sourabh_Daptardar
cooked: <p>No. I have divided the batch size by the world size.<br>
Will check out nvprof and also create minimal working example as I can not share code.</p>
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [1, 3, 96, 128]

username: Khawar_Islam
cooked: <p>I am using UNet</p>
<p>I have images with masks (black background and highlighted portion contain different RGB colors) and 12 classes. I want to do training using uNET. but i got error. “RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [1, 3, 360, 640]”</p>
<p>Please help me</p>
<p>Kind Regards,<br>
Khawar</p>
<p>Error</p>
<pre><code class="lang-auto">  File "/home/khawar/Pytorch-UNet/train.py", line 185, in &lt;module&gt;
    val_percent=args.val / 100)
  File "/home/khawar/Pytorch-UNet/train.py", line 84, in train_net
    loss = criterion(masks_pred, true_masks)
  File "/home/khawar/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 948, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/khawar/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2422, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/khawar/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2220, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [1, 3, 96, 128]
</code></pre>

username: ptrblck
cooked: <p>The target tensor for a multi-class segmentation use case using <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code> should have the shape <code>[batch_size, height, width]</code> and contain the class indices in the range <code>[0, nb_classes-1]</code>.<br>
If your current target is an RGB image containing color codes for each class, you would have to map these colors to class indices first, e.g. by using a lookup table.</p>

username: gokulp01
cooked: <p>Hi! Even I am facing the same error, what exactly do you mean by a look up table. You mean convert RGB to monochrome?</p>

username: ptrblck
cooked: <p>No, I meant you would have to convert the colors to class indices.<br>
If your dataset doesn’t provide the mapping, you could create your own as described <a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">here</a>.</p>

username: gokulp01
cooked: <p>Thanks a lot! It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torch tensor and numpy array

username: blackbirdbarber
cooked: <p>When we do conversion from tensor to numpy array via <code>.numpy()</code> method is it true that tensor and numpy array share the same storage?</p>

username: CedricPicron
cooked: <p>Yes, see <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.numpy" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.numpy</a>.</p>

username: blackbirdbarber
cooked: <p>OK, but do we check this via code? I would like to prove this since I don’t trust documents in general.</p>

username: CedricPicron
cooked: <p>You can for example run</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch

&gt;&gt;&gt; tensor = torch.arange(6)
&gt;&gt;&gt; array = tensor.numpy()

&gt;&gt;&gt; tensor
tensor([0, 1, 2, 3, 4, 5])
&gt;&gt;&gt; array
array([0, 1, 2, 3, 4, 5])

&gt;&gt;&gt; tensor[0] = 10
&gt;&gt;&gt; tensor
tensor([10,  1,  2,  3,  4,  5])
&gt;&gt;&gt; array
array([10,  1,  2,  3,  4,  5])

&gt;&gt;&gt; array[1] = -2
&gt;&gt;&gt; tensor
tensor([10, -2,  2,  3,  4,  5])
&gt;&gt;&gt; array
array([10, -2,  2,  3,  4,  5])
</code></pre>
<p>You can clearly see that both <code>tensor</code> and <code>array</code> share storage.</p>
Mapping from pytorch to libtorch for batchnorm1d

username: Chen0729
cooked: <p>Hi, All</p>
<p>I recently try to map the following line from pytorch to libtorch</p>
<p>self.bn1 = nn.BatchNorm1d(500, affine = False, eps=1e-6, momentum = 0.5)</p>
<p>and I write in libtroch</p>
<p>bn1 = torch::nn::BatchNorm1d(500, false, 1e-6, 0.5);</p>
<p>It gives me an error no viable overloaded “=”</p>
<p>Any suggestions on fix the error please ?</p>
<p>Thanks.</p>

username: Chen0729
cooked: <p>It is also incorrect if I use<br>
bn1 = register_module(“bn1”, torch::nn::BatchNorm1d(500, false, 1e-6, 0.5));</p>

username: Chen0729
cooked: <p>torch::nn::Sequential bn1;<br>
bn1-&gt;push_back(torch::nn::BatchNorm1d(500, false, 1e-6, 0.5));</p>
<p>The above has no error, is the above expression correct ?</p>
Image Classification

username: hheavenknowss
cooked: <p>when I use pytorch, resnet50 to classify my images, which have 3 classed, there is a problem,<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3553787c02abcc6c28508175dbccd3a09d9ef78.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3553787c02abcc6c28508175dbccd3a09d9ef78" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3553787c02abcc6c28508175dbccd3a09d9ef78.png" alt="image" data-base62-sha1="u9xk0jITsPoAxm9qtVZ6lp0oCLu" width="690" height="254" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1882×695 60.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
is there anyone know what happened? please help me!</p>

username: InnovArul
cooked: <p>It seems your dataset class does not resize all the images to same size?<br>
The width (dimension 2) of images do not match for all images.</p>

username: hheavenknowss
cooked: <p>yes,I found the issue, thanks</p>
Exporting torchvision FCN model to ONNX and running in onnxjs - opset confusion

username: afg1
cooked: <p>I’m trying to export an FCN from torchvision using the following code:</p>
<pre><code class="lang-auto">model= models.segmentation.fcn_resnet101(pretrained=True, progress=True, num_classes=21, aux_loss=None)
model.eval()
x = torch.randn(1,3,512,512)
torch_out = model(x)
torch.onnx.export(model, x, "seg_rn.onnx", 
                    export_params=True, 
                    opset_version=11, 
                    do_constant_folding=True, 
                    verbose=True
                    )
</code></pre>
<p>When exporting the model, I need minimum opset 11 to support the way pytorch’s interpolation works, and this is confirmed by the output of the onnx model when running in the python onnx runtime.</p>
<p>Running in the python onnx runtime is fine in python, but when I load the model in a browser using onnxjs, like this:</p>
<pre><code class="lang-auto">var session = new InferenceSession();
const modelURL = "./models/seg_rn.onnx";
await session.loadModel(modelURL);
</code></pre>
<p>I get <code>Uncaught (in promise) TypeError: cannot resolve operator 'Shape' with opsets: ai.onnx v11</code></p>
<p>If I go and create my own copies of bits from <code>torchvision.models.segmentation</code>, I can get rid of the error about Shape (by speciying a static shape for the input and telling the interpolation what the resizing factor should be), but then I get another error: <code>Uncaught (in promise) TypeError: cannot resolve operator 'MaxPool' with opsets: ai.onnx v11</code> but this time in reference to MaxPool. Ignoring tests and outputting with opset v10 results in a loadable model, but one which will be incorrect.</p>
<p>What is going on? Is there any way forward, or am I basically stuck?</p>
<p>I think opset v13 might have all the ops I need (gleaned from onnxjs github commits recently). How big a job is it to implement the new opset for pytorch? I’m not afraid to get stuck in, I just don’t really know where to start.</p>
Dynamic Input for ONNX.js using a Pytorch trained model

username: Ashraf_Ali
cooked: <p>So I’ve got this autoencoder that I’ve trained and now I wanna deploy it to a website.<br>
I came across ONNX.js and liked it because it can run natively on the browser(Saving me my much needed Student cloud credits <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">)</p>
<p>However I noticed that onnx requires a dummy input so that it can trace the graph and this requires a fixed input size.</p>
<pre><code class="lang-auto">dummy = torch.randn(1, 3, 1920, 1080)
torch.onnx.export(transformer, dummy, save_path, opset_version=11)
</code></pre>
<p>I’m almost certain that on the ONNX.js side I need the input to be the same shape.<br>
Is there anyway around this, because I need dynamic shape inputs.<br>
Or should I just used Flask and host the model?</p>
<p>Thanks!</p>

username: afg1
cooked: <p>I’m by no means an expert, but I think you can use the <code>dynamic_axes</code> optional argument to <code>onnx.export</code></p>
<p>In the tutorial <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html#running-the-model-on-an-image-using-onnx-runtime" rel="nofollow noopener">here</a> (about a quarter of the way down) the example uses the dynamic_axes argument to have a dynamic batch size:</p>
<pre><code class="lang-auto">                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable lenght axes
                                'output' : {0 : 'batch_size'}})
</code></pre>
<p>I <em>assume</em> that can also be done for the other axes, but I haven’t tried it - my use case has images which are guaranteed to be the same size, so I don’t need it.</p>
<p>I have seen some discussion that would suggest this is not the case though, so maybe someone with more experience can weigh in?</p>
How can I use multi-hot embedding efficiently?

username: springtostring
cooked: <pre><code>Hello everyone，I encountered a problem in my code.
</code></pre>
<p>I want use embedding trick to encode my multi-hot vector ，such as  [1,1,0,1,0,0,0,0],[1,0,0,0,0,1,0].<br>
However， I find there is not  multi-hot embedding in Pytorch.I  writed a function to  achieve multi-hot  embedding，but it is used in for-loop，so it is so slow and time-consuming.<br>
Can you help me solve this problem？Thank you！</p>
Confused about the images display of tensorboard

username: bruce
cooked: <p>I use tensorboard to record my image,here is the output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0d8b6336306a93fcc8f3cb3c8a8c772dd85c105.png" alt="pic1" data-base62-sha1="rvZYzLZxwZTxaGlGF513vWGfN0p" width="354" height="339"></p>
<p>and I use the matplotlib.pyplot.imshow,The ploted image is in the reply</p>
<p>the raw Image series format was tiff，int32.<br>
After reading the picture, the data type is torch.float32, however it is not work even if i changed the type to int32<br>
here is part of my code:<br>
‘’’<br>
writer = SummaryWriter(‘runs/another_scalar_example’)</p>
<p><span class="hashtag-raw">#img</span> = torch.cat((img,img,img),dim=1)</p>
<p>imagegrid = torchvision.utils.make_grid(img,nrow=16)<br>
imagesample = img[0]<br>
imagesample = imagesample.type(torch.ShortTensor)</p>
<p>writer.add_image(‘jb’,imagesample,global_step=1,dataformats=‘CHW’)<br>
writer.add_image(‘countdown’,imagegrid,dataformats=‘CHW’)<br>
‘’’</p>

username: bruce
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7abffcbdc93ea6720e5830f7df28ddcceeaafd2.png" alt="pic2" data-base62-sha1="nVif8yogOOfdTdNNMCZkndqniiC" width="326" height="322"></p>
Forward hook is not always called

username: deJQK
cooked: <p>Hi I am using some forward hook, and apply the model on the data. It seems that during the forward, for the last batch (which has less data than the batch size), the forward hook is not always called. I do not understand the reason.</p>
<p>The code for hook is like this:</p>
<pre><code class="lang-auto">mapping_hooks = []
model_wrapper.range_dict = {}
def get_input_range(mem, name):
  def get_input_range_hook(module, input, output):
    print('HOOK')
    ...
    mem[name + f'{module.bar} bar' + str(output.device)] = cal_range
  return get_input_range_hook

for name_, module_ in model_wrapper.named_modules():
  mapping_hooks.append(module_.register_forward_hook(get_input_range(model_wrapper.range_dict, name_)))

with torch.no_grad():
  for input in data:
    for bar in bar_list:
      model_wrapper.apply(lambda m: setattr(m, 'foo', bar))
      model_wrapper(input)
      range_dict = {**model_wrapper.range_dict}
      for k, v in range_dict.items():
        print('bar: ', bar)
        print('v: ', v.shape)

for hook in mapping_hooks:
  hook.remove()
mapping_hooks = []
</code></pre>
<p>And in the final log, ‘HOOK’ is printed for every batch, except for the last batch and the second bar value (it is correctly printed for the first bar value for the last batch). The shape of v printed is also incorrect for the second bar value for the last batch, which should be [80, 3, 32, 32], but is [128, 3, 32, 32] (128 is the batch size, 80 is the number of data for the last batch, for cifar10 with 50k images and batch size of 128). I only use one GPU, and model_wrapper is an nn.DataParallel instance.</p>
<p>Thanks a lot.</p>
<p>First update: it seems that hook is called only once for every batch, only for the first bar value.</p>
<p>Second update: I update the code a little to be more like the real code (add ‘bar’ information and device information into the keys for the range_dict). Also, it seems that the original description is correct, i.e., the hook is called correctly for every bar value and every batch, except for the last batch, where the hook is only called for the first bar value.</p>

username: deJQK
cooked: <p>Sorry I found the problem.</p>
Is there a way to get the model definition out of a saved model in Pytorch?

username: Shisho_Sama
cooked: <p>hello everyone, I hope you are having a great time.<br>
Is there a way to access the model definition of a saved model? I know we have something kind of similar in jit traced models but I’m not sure if we have something like this in normal models.<br>
what I’m looking for is to get my hands on the model definition that I have previously pruned! since the model is resnet18, its cumbersome to manually create/edit a new definition file for this model so I wonder whats my best options at the  moment that I have my saved model, but without any definitions.<br>
I dont want to train again and then save the traced model, this will work, but the training takes couple of days and I dont want to waste the days trained so far unless absolutely necessary.</p>
<p>Any suggestion /help is greatly appreciated<br>
Thanks a lot in advance</p>

username: ptrblck
cooked: <p>If you are following the recommended way of storing the <code>state_dicts</code> only, you would have to store the model definition separately. I don’t think there is a clean way to store source code in Python without the danger of breaking it in several ways (but maybe you’ve seen something useful).</p>

username: Shisho_Sama
cooked: <p>Thanks I must have been vague since you got me wrong.<br>
to be more precise, I’m looking for a way to get an updated definition(i.e. latest representation of the model as is) based on the latest changes to the model structure, not the initial model definition.<br>
For example during pruning some featuremaps are pruned and thus some layers that initially had x number of feature maps, now have x’ number. when the model is saved, the state_dicts are saved and thus I still need to find a (sane) way to also extract the model definition and save it along with the weights.<br>
What are my options here?<br>
I hope this makes it clearer.<br>
Thanks a lot in advance</p>