akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Modify TorchScript model parameters using libtorch C++ API

pmandke: <p>Not a Contribution.</p>
<p>Is it possible to access and modify the parameters of a TorchScript model using the libtorch C++ API?</p>
<p>Suppose for eg, that we want to load the parameters in a std vector/map, perform some operations on them and write them back to the TorchScript model. I believe this is possible using the torch::nn::Module API, but if we do not have the model definition in C++, would something like this be doable using the TorchScript model?</p>
<p>Thanks!</p>

pmandke: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, any pointers? Or know anyone who could help? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I’m not entirely sure, as I haven’t experimented with such a use case, but based on this Python behavior:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
traced = torch.jit.script(model)

print(traced.weight)

x = torch.randn(1, 10)
out = traced(x)

with torch.no_grad():
    traced.weight.copy_(torch.ones_like(traced.weight))
out = traced(x)
</code></pre>
<p>I would guess you could also directly manipulate the parameters of the scripted models.<br>
Are you able to access these parameters or are you getting an error?</p>

pmandke: <p>Thanks for your response.</p>
<p>I tried a similar approach (something like <a href="https://discuss.pytorch.org/t/how-to-copy-network-parameters-in-libtorch-c-api/32221/2">this</a>), but I can only get const references to the parameters, so cannot manipulate their values for a scripted model. The manipulation only works with a torch::nn::Module instance.</p>
<p>Is there any API in libtorch that would get me either pointers or non-const references to a scripted models’ parameters?</p>

ptrblck: <p>Ah OK, I didn’t know that. As I haven’t used this approach before, maybe <a class="mention" href="/u/tom">@tom</a> has experimented with a similar use case or would know if this use case is supported at all?</p>

hoyden: <p><a class="mention" href="/u/pmandke">@pmandke</a> Do you solve it ? I met the same question</p>
Masked Linear Layer which only updates a subset of the Input vector

devpouya: <p>Hi all,</p>
<p>I have the following setting. Imagine a tensor A as input of shape [n,m]. I want to pass the entire tensor through a linear layer, but I only want one of the rows to be updated and the rest to remain the same. So after NN(A) I want only the i-th row to be updated and all other rows to be the same as in the input.</p>
<p>Do any of you know of a way to achieve this, without copying and cloning tensors? I don’t want to break the differentiability.</p>
<p>TLDR: How can I define a layer which applies the Identity function to every row except row i and a linear function to row i?</p>
<p>Thanks in advance</p>

InnovArul: <aside class="quote no-group" data-username="devpouya" data-post="1" data-topic="138788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/devpouya/48/44703_2.png" class="avatar"> devpouya:</div>
<blockquote>
<p>I want to pass the entire tensor through a linear layer, but I only want one of the rows to be updated and the rest to remain the same</p>
</blockquote>
</aside>
<p>Question: Why do you want to pass the entire input through the linear layer if only one row needs to be transformed? Isn’t it a waste of computation?</p>

devpouya: <p>The problem is that my input is from a pre-trained model which I want to fine-tune, but at the same time, I want to update parts of the input.</p>
<p>I understand that this is a bit ill-defined/not-nice conceptually. Would be happy to hear any thought you have.</p>

JanoschMenke: <p>Split the input. Feed the separate parts through different layers: One does the linear transform and the other performs the identity.<br>
Merge the output of the two layers.</p>
Inference fails with cpu with sync bn on while training with multi-gpus

two_Two: <p>After training with multiple gpus with sync_bn on, I would like to do inference on CPU, However, raise the error:<br>
<code>ValueError: SyncBatchNorm expected input tensor to be on GPU</code></p>
<p>Is there any way to solve this issue ?  Thanks</p>

ptrblck: <p>I don’t know how you are adding <code>SyncBatchNorm</code> to the model, but in case you are using <code>convert_sync_batchnorm</code>, remove the operation and keep the standard batchnorm layers.<br>
If you are manually adding <code>SyncBatchNorm</code> layers to your model, you might need to replace them with their non-sync equivalent and copy their internal states over.</p>

two_Two: <p>As far as I know, The states that I need to move include : weights, bias,running_mean,running_var, is there any else that I need to transfer over ?<br>
Mentation about replacing <code>SyncBatchNorm</code> layers, could you provide simple example how to do this, maybe something like :</p>
<pre><code class="lang-auto">for  name , module in model.named_modules():
    if isinstance(module, nn.SyncBatchNorm) : 
        #do something 
</code></pre>
Efficient ways for DataSet and DataLoader

kartikpodugu: <p>I created a custom dataset in 2 ways.</p>
<ol>
<li>created a single csv file of all my training/testing data and then loaded this file in <strong>init</strong>, and <strong>getitem</strong> is very simple indexing.</li>
<li>created multiple small csv files for all my training/testing data. created a very simple <strong>init</strong> function, where I load some root folder name, no. of files in the folder and so on, and created a complicated <strong>getitem</strong> function, where based on the index of the training example requested, i calcualte the file which has the required training example, and then load the required data.</li>
</ol>
<p>First way worked like a charm when data is very small. As data exploded, creating a single big file with all training examples was in itself a very time consuming one, and also memory requirements of the program also grew exponentially, causing data creation script to hang. But, once i created this big file, loading into program took time, but get item is very simple.</p>
<p>Second way, i need not create a merged file with all training examples. So, no big files. No big memory requirements in data creation and data loading. I thought, this is efficient way. But, second one increasing training times.</p>
<p>What is the best way to handle this issue?</p>

mMagmer: <p>You can use bigger <code>num_workers</code> in <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">dataloader</a> to set how many subprocesses to use for data loading.<br>
<code>pin_memory=True</code> can also help if you’re using gpu.<br>
<strong>warning:</strong> it’s not working perfectly on windows machine. and you should read dataloader doc for more info about <strong>Platform-specific behaviors</strong>.</p>

kartikpodugu: <p>already using num_workers. Haven’t tried pin_memory. Will try that.</p>
Pick one random value from tensors of different sizes

Jonas_ar: <p>Hello everyone,</p>
<p>i want to extract one random value from a number of different tensor, how to get the random number out of the tensor should not depend on its size.</p>
<p>Just to get an idea here are some examples of sizes.<br>
So either 4, 2 or 1 indicies</p>
<p>|features.0.0.weight | torch.Size([16, 3, 3, 3])|<br>
|features.0.1.weight | torch.Size([16])|<br>
|features.16.1.weight | torch.Size([960])|<br>
|classifier.0.weight | torch.Size([1280, 960])|<br>
|classifier.3.weight | torch.Size([1000, 1280])|</p>
<p>I started quite generic and easy and thought just pick 4 random values that choose the index location of the value within the tensor:</p>
<p>size_tensor = sd[location].size() <span class="hashtag">#The</span> location basically the name of the weight tensor which I state in<br>
the beginning, where the sd is the state_dict()</p>
<p>a = random.randint(0,size_tensor[0]-1)</p>
<pre><code>b = random.randint(0,size_tensor[1]-1)

c = random.randint(0,size_tensor[2]-1)

d = random.randint(0,size_tensor[3]-1)
</code></pre>
<p>random_value_from_tensor = weight_tensor[a][b][c][d]</p>
<p>So in the example of taking features.0.0.weight as the location the size is 16, 3, 3, 3 so the code choses the random value as “weight_tensor[random value between 0 and 15][0 and 2]” and so on.</p>
<p>This works as intendet. But my problem is that I have to do that for every form or size for the different tensors, if I choose one with torch.Size([16])| for example, the code obviously does not work, since there is only one index for the values in the weight tensor, instead of 4.</p>
<p>So the question is, if there is a easier way that does not depent on the torch.size to extract one random value from the tensor? Basically “rand_value(tensor) = one random value from tensor”<br>
And if not, does anyone have a better idea to pick random values and not having to code exceptions and random values for every from of torch.size between 1 to 4 indices?</p>

Azerus: <p>Hello,</p>
<p>Does this function solve your problem?</p>
<pre><code class="lang-auto">import random

def get_rand_value(tensor):
    number_of_elements = tensor.numel()
    random_element_number = random.randrange(number_of_elements)
    value_of_random_element = tensor.flatten()[random_element_number]
    return value_of_random_element
</code></pre>
<p>Regards,<br>
Thomas</p>

Jonas_ar: <p>Yes, that solves my problem, thank you very much!</p>
Why reserved_memory+allocated memory is bigger than the memory shown in nvidia-smi?

ZiQing_Li: <p>It seems like that reserved_memory+allocated memory is bigger than the memory shown in nvidia-smi. My allocated mem is 3199MB and my reserved mem is 8766 MB, which equals 11965MB, but the mem shown in my nvidia-smi is 11131MB.</p>
<p>By the way, there is another question, I met an OOM when trying to allocate about 1000MB space but there was still about 8766MB in reserved space. The reserved space is designed to be reused, why is there still an OOM error?</p>

ptrblck: <p>The reserved memory reports the allocated memory and the cached memory so you shouldn’t sum them together. Take a look at <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management">the memory management docs</a> for more information.</p>

ZiQing_Li: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply!<br>
I have checked the docs. But I still have some confusion over the memory. I met an OOM error. <code>RuntimeError: CUDA out of memory. Tried to allocate 1.53 GiB (GPU 0; 11.75 GiB total capacity; 5.97 GiB already allocated; 449.69 MiB free; 8.99 GiB reserved in total by PyTorch)</code>.<br>
It seemed that I still have about 3G (8.99GiB-5.97GiB) cached memory to allocate tensors, but the allocation failed. So I guess 1.53 GiB tensors might be moved to free memory and then removed to cache memory, or the cached memory was occupied by Pytorch internal context management but why could it occupy so large space.<br>
Did I make anything wrong? Thanks!</p>

ptrblck: <p>The error message points out that <code>449.69MiB</code> are only free and the rest might be scattered and lost due to e.g. memory fragmentation.<br>
The CUDA context is not captured by the allocated or reserved memory, as PyTorch doesn’t check its memory usage.</p>
Convert unknown labels to yolov5

elan1337: <p>Hello datascience-community,</p>
<p>i need your kind assistance.</p>
<p>I own a dataset of  images with unknown label format, wich is:</p>
<p><strong>angry_actor_104.jpg 0 28 113 226 141 22.9362 0</strong></p>
<p>It indicates an image as follows:</p>
<p><strong>image_name face_id_in_image face_box_top face_box_left face_box_right face_box_bottom face_box_cofidence expression_label</strong></p>
<p>My question is: How can this be <strong>converted into the yolov5 format</strong>?</p>
<p>I have been looking this up for a long time and hope someone can help.<br>
Thank you very much in advance.</p>
<p>Best regards<br>
Phil</p>

thecho7: <p>Hi, from rewriting your dataset,</p>
<p>image_name:  angry_actor_104.jpg<br>
face_id_in_image: 0<br>
face_box_top: 28<br>
face_box_left: 113<br>
face_box_right: 226<br>
face_box_bottom: 141<br>
face_box_cofidence: 22.9362<br>
expression_label: 0</p>
<p>YOLOv5 handles the dataset both image and the corresponding annotation have the same file name.<br>
The annotation only includes the followings,<br>
label, box_center_x, box_center_y, width, height<br>
and all coordinates are [0, 1).</p>
<p>Consequently, your annotation should be<br>
filename: angry_actor_104.txt<br>
Inside the file,<br>
expression_label bbox_center_x bbox_center_y bbox_width bbox_height</p>
<p>if there are multiple bbox inside an image, your annotation text file should be<br>
expression_label1 bbox_center_x1 bbox_center_y1 bbox_width1 bbox_height1<br>
expression_label2 bbox_center_x2 bbox_center_y2 bbox_width2 bbox_height2<br>
…</p>
Pytorch on a raberry Pi 4 (32-bit OS)

yannic310: <p>I tried to install Pytorch on a rasberry PI 4. The OS is the 32-bit from rasberry pi.<br>
I tried it with pip, wheels and building PyTorch from the source. But it didn’t work.</p>
<p>Installing it with wheels, I got always the error that the wheel was not compatible with your currunt version.</p>

ptrblck: <p><a class="mention" href="/u/tom">@tom</a> provided some PyTorch builds for Raspberry Pi <a href="https://discuss.pytorch.org/t/raspberry-arm64-binaries/94751/6">here</a>, which might be useful for you.</p>

tom: <p>The problem with 32bit ARM is that PyTorch cannot be expected to fully work.<br>
Chances are that the compute part works because it is used for android as well, but a long time ago, I noticed that <a href="https://github.com/pytorch/pytorch/issues/27040" rel="noopener nofollow ugc">JIT tracing didn’t work on 32bit ARM</a>. As 64bit Raspberry Pi OS is a thing (and will run 32bit things, too), I stopped building and using 32bit PyTorch packages.<br>
Now, this was two years ago and depending on your optimism and constraints you could see if things have improved for 32bit ARM, but personally I would be weary of subtle bugs in PyTorch on 32bit creating debugging headaches.</p>
<p>Best regards</p>
<p>Thomas</p>
Crf loss being negative during training

shayue111: <p>I implement a version of Linear Chain CRF based on Pytorch framework. After testing, I use that with NER dataset. I found the crf loss, aka NLLoss, being negative with the train process going by.</p>
<p>In the implementation of my linear chain crf, there are not <code>&lt;START&gt;</code> and <code>&lt;END&gt;</code> tag, which I guess the main reason for the negative loss.</p>
<p>Does someone have the same problem?</p>
Why the reserved_memory space generated by the forwarding of Net can't be released?

ZiQing_Li: <h2>
<a name="questions-and-help-1" class="anchor" href="#questions-and-help-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=10" title=":question:" class="emoji" alt=":question:"> Questions and Help</h2>
<p>The demo code is listed below. Why after I used the <code>torch.cuda.empty_cache()</code>, only part of the reserved space generated by the forwarding of Net could be released? For the second epoch, the reserved_space will get larger while doing the same operation as the first epoch. And how does the reserved_memory work? Why is the reserved memory much larger than allocated memory?</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = torch.nn.Conv2d(3, 28, (3, 3)) # in out kernel size
        self.maxpool1 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.maxpool2 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
    def forward(self, x):
        x = self.conv(x)
        x = self.maxpool1(x)
        x = self.maxpool2(x)
        return x

def train():
    net = Net().to('cuda')
    for i in range(5):
        with torch.no_grad():
            frames_batches = torch.randn(512 , 3, 224, 224).to('cuda')
            
            # before forward 1st | memory_reserved: 296MB | memory allocated: 294MB
            pred = net(frames_batches)
            # after forward 1st epoch | memory_reserved: 5014MB | memory allocated: 465MB
            # after forward 2nd epoch | memory_reserved: 5688MB | memory allocated: 465MB
            torch.cuda.empty_cache()
            # after empty_cache 1st epoch | memory_reserved: 1644MB | memory allocated: 465MB
if __name__ == '__main__':
    train()
</code></pre>
Is there a nice way to get all layer properties?

Karthik_Ganesan: <p>What’s a nice way to get all the properties for a given layer type, maybe in an iteratable way?</p>
<p>For example, for an <code>nn.Linear</code> layer, I am reading currently getting them as:</p>
<pre><code class="lang-auto">for name, layer in model.named_modules():
    if isinstance(layer, nn.Linear):
        in_features = model._modules[name].in_features
        out_features = model._modules[name].out_features
        weight = model._modules[name].weight
        bias = model._modules[name].bias
</code></pre>
<p>Not only am I accessing a protected member of the class but it’s also error prone, since I want to do this for other layer types as well. is there a nice way to get all the layer properties? I am trying to copy all these and set them for my own custom version of each layer!</p>

ptrblck: <p>You could check if the <code>.__dict__</code> or <code>dir()</code> approach would work for you:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
print(lin.__dict__)
print(dir(lin))
</code></pre>
<p>However, both approaches would still need some parsing, checks, etc. since a lot of internals would be printed.</p>

googlebot: <p>from the snippet, it seems that you want parameters not properties (python attributes), that’s simply done with module.named_parameters([recurse=False perhaps])</p>

Karthik_Ganesan: <p>Thank you for your amazing help as always, <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p><a class="mention" href="/u/googlebot">@googlebot</a> thank you! But I wanted to include things such as ‘in_channels’ and ‘out_channels’ which are not parameters, if I understand correctly.</p>

googlebot: <p>Right, these are not well formalized, though for built-in modules you can get relevant names from <code>module.__constants__</code> (this attribute was added to reference module’s scalars in JIT mode, but mostly reflects constructor arguments)</p>

Karthik_Ganesan: <p>Thank you. Yes, I might do a mix of <code>__constants__</code> and <code>__parameters__</code> to get all the details I want.</p>
The different results with each training time, even the same code, environment,..everything,

Vincent24: <p>Hello everybody!<br>
I did as following and had the problem:</p>
<p>Code:<br>
def seed_torch(seed=0):<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)#  if you are using multi-GPU.<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = True</p>
<p>seed_torch()</p>
<p>def main():<br>
…<br>
rawnet.train()<br>
for i, data in enumerate(train_dataloader):<br>
seed_torch()<br>
…<br>
…<br>
seed_torch()<br>
rawnet.eval()<br>
seed_torch()<br>
with torch.no_grad():<br>
for i, data in enumerate(val_dataloader):<br>
seed_torch()<br>
…</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38c337789a7d6cab95aeb85ae17f95cd0fbcfd28.png" alt="image" data-base62-sha1="868ZguBghMDiC0f6siElRWgd48g" width="344" height="333"></p>
<p>The different results with each training time, even the same code, environment,…everything. Particularly, I have almost the same results of training loss. However, I have DIFFERENT results of validation loss, even putting seed_torch() everywhere in loops. Please help me solve this problem!!!. Thank you!</p>

ptrblck: <p>Could you check, if your model is using some of the non-deterministic methods mentioned in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>?</p>

Lin_Jia: <p>Are your results very different? I have an MNIST app, the loss and accuracy varies a little bit every time I run the app. But the variation is tiny. For example, in terms of accuracy, the change is below 0.5%.</p>

Vincent24: <p>Yes!, I wrote my code as <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">reproducibility docs </a>.</p>
<p>you can see my above subprogram:</p>
<p>def seed_torch(seed=0):<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)<br>
np.random.seed(seed)<span class="hashtag">#as</span> reproducibility docs<br>
torch.manual_seed(seed)# as reproducibility docs<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)<br>
torch.backends.cudnn.benchmark = False# as reproducibility docs<br>
torch.backends.cudnn.deterministic = True# as reproducibility docs</p>
<p>do you know what the problem is?</p>

Vincent24: <p>yes. The validation loss is very different but the training loss is little different. You can see as above figure</p>

Dicko871: <p>Hi <a class="mention" href="/u/vincent24">@Vincent24</a> , did you manage to solve this issue? I am facing exactly the same problem.</p>
Multi-output regression using Hugging Face library

Shreyas_Mangalgi: <p>Hi,<br>
I have been using the Hugging Face library, specifically the ‘<em>robertaforsequenceclassification</em>’ model. I wanted to train a multi-output regression model and for that I have set the number of out classes to two. Essentially, I want to predict a geo-coordinate as output. But, the hugging face document mentions that for regression tasks, the number of classes needs to be 1, in which case an RMSE loss is automatically used. While if number of classes is greater than one, a cross entropy loss is used. Is there a way to use the ‘<em>robertaforsequenceclassification</em>’ model for multi-output regression?</p>

hivacruz: <p>Hello <a class="mention" href="/u/shreyas_mangalgi">@Shreyas_Mangalgi</a></p>
<p>Did you find any way to do multi-output regression with the Transformers library?</p>
<p>Thanks!</p>
Gradient calculation of multiple outputs different using torch.stack

ferqui: <p>Hi, I’m training a recurrent network and I want to know the intermidiate gradients of the output of the network over time, but when calculating the gradient I notice that is different if I calculate the gradient in each time independently or if I calculate it together using torch.stack, but I don’t fully understand why. Below is an example that has different value for dE_dz. Can someone help me to understand it? Thank you</p>
<pre><code class="lang-auto">for t in range(T):
    out, state = network(inputs[:,t], state)
    out.retain_grad()
    spikes.append(out)
...
loss.backward()
dE_dz = torch.stack([s.grad for s in spikes], 1)
</code></pre>
<pre><code class="lang-auto">for t in range(T):
    out, state = network(inputs[:,t], state)
    spikes.append(out)
...
spikes = torch.stack(spikes)
spikes.retain_grad()
loss.backward()
dE_dz = spikes.grad
</code></pre>
How to use torch.no_grad() if my loss function needs to compute the Jacobian matrix

Le_ZHENG: <p>Hi, I know that using with torch.no_grad() when validating the model can save lots of time and memory. However, since part of my loss function needs to compute the Jacobian matrix, which is computed using torch.autograd.grad. What should I do in this case? Do you have any suggestions?</p>

ndronen: <p>There are some options, none of which are completely satisfactory:</p>
<ul>
<li>Compute a metric and not the loss during validation.</li>
<li>If memory is limited, you may have to reduce the batch size during validation, and not use the <code>torch.no_grad()</code> context manager, so you can compute the loss.</li>
</ul>
Custom Entropy Loss function

FCHR18: <p>Hi<br>
We are making a project about quantisation in a neural network layer and need to calculate the entropy of these quantised values. An example of the data that the entropy should get as input is: x = [1/8,1/8,4/8,2/8]. Currently we have the following function for entropy</p>
<p>def Ixxhat(bitdepth,xhat):<br>
hist = torch.histc(xhat, 2**bitdepth, 0, 1)/(xhat.shape[0]*xhat.shape[1])<br>
xhatProbs = torch.tensor([hist[i] * torch.log2(hist[i]) if hist[i] &gt; 0 else 0 for i in range(len(hist))])</p>
<pre><code>return - torch.sum(xhatProbs), hist
</code></pre>
<p>We read that as long as we use the torch functions it should work, but can someone verify this for this function and/or design an entropy function we could use instead <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks in advance</p>

KFrank: <p>Hi Frederik!</p>
<aside class="quote no-group" data-username="FCHR18" data-post="1" data-topic="138659" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/958977/48.png" class="avatar"> FCHR18:</div>
<blockquote>
<p>We read that as long as we use the torch functions it should work, but can someone verify this</p>
</blockquote>
</aside>
<p>No, this won’t work.</p>
<p>Being discrete in nature, the histogramming operation is not (usefully)<br>
differentiable, so, mathematically, there is no way to backpropagate<br>
through it.</p>
<p>It is true – for reasons that I do not understand – that <code>torch.histc()</code><br>
does have a companion <code>HistcBackward</code> function.  It doesn’t, however,<br>
implement a derivative for <code>histc()</code> (because it wouldn’t make sense):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; hist = torch.histc (torch.randn (10, requires_grad = True), bins = 10)
&gt;&gt;&gt; hist.grad_fn
&lt;HistcBackward object at 0x000001F87CB2E5F8&gt;
&gt;&gt;&gt; hist.sum().backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: the derivative for 'histc' is not implemented.
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

FCHR18: <p>Hi KFrank<br>
Thank you for the answer. Do you know how we could implement the entropy in pytorch?</p>
<ul>
<li>Frederik <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
</ul>

ndronen: <p>What’s your goal? Is it to minimize KL divergence between histograms of activations at each layer of a network to find the best parameters for quantization?</p>
Convert text from print(model) back to a neural network

sven_t: <p>Hey PyTorch community,</p>
<p>Nice to meet you all here! I got an interesting task to solve…</p>
<p>After we build the network using <code>class Net(nn.Module)</code>, we could display the network by using <code>print(model)</code> as shown in <a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html" rel="noopener nofollow ugc">Tutorial of Build the Neural Network</a>.</p>
<p>I would like to know whether there is any PyTorch function which could convert back the text generated from <code>print(model)</code> to a PyTorch code of neural network? Let’s neglect the forward pass at this moment…</p>
<p>Many thanks for your help!</p>
<p>Cheers,<br>
Sven</p>
Batch size is getting changed

Sumera_Rounaq: <p>In my code I defined batch size as 50 and after few lines of code I am getting batch size as 42. I did not make any change in batch size and it is getting changed automatically.<br>
What could be the reason behind it?</p>

my3bikaht: <p>You have 50*n+42 (42,92,142 etc)  training samples, so on nth batch loop you get 42 samples per batch</p>

Sumera_Rounaq: <p>Could you please elaborate it little more? What is “n” over here? In my case I have 146 training examples.</p>

my3bikaht: <p>Batch size can’t change on it’s own. The only reasonable suggestion here is that remaining number of elements from the pool during the last batch is less than 50: if you have 146 examples your last batch’ size should be 46. Since it’s 42, you’re missing something with dataloader or your object’s shape.</p>

Sumera_Rounaq: <p>Thank you so much. I got you. By mistake I wrote 146 it was 292. Now it makes sense.</p>
Question about the correctness of AdaptiveAvgPool2d

Jie-KUN: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e97fd655b48404998da5fed41363b4fbe9143c6.png" alt="QQ图片20211207170427" data-base62-sha1="mCZ4M7EMa17S2Z6PbmmJBS3YYaq" width="367" height="207"></p>
<p>In my view, the results of the  AdaptiveAvgPool2d and AvgPool2d should be the same. But I try the above code, it turns out that they are not the same. Why can help me solve this problem?</p>

Jie-KUN: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/296797acb4387a4b17e90d8e06ea4b0affec256b.png" data-download-href="https://discuss.pytorch.org/uploads/default/296797acb4387a4b17e90d8e06ea4b0affec256b" title="res"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/296797acb4387a4b17e90d8e06ea4b0affec256b.png" alt="res" data-base62-sha1="5Uhwn9dHudromjSjVqTsaBoK1D5" width="526" height="500" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">res</span><span class="informations">748×711 5.47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Implement Ordered dropout

zak_lah: <p>Actually, I want to implement ordered dropout as tackled in the fjord paper. The behavior of ordered dropout is dropping adjacent components of the model instead of random neurons. I am working on the Resnet18 model, and as a first step, I should override the forward method. Can someone give me some hints or ideas ?</p>

ptrblck: <p>Your idea to override the <code>forward</code> method sounds right and you could either derive a custom class from the <code>ResNet</code> class provided by <code>torchvision</code> or copy-paste the code directly from <a href="https://github.com/pytorch/vision/blob/93b26da3623c9134d64f026a234d97caab0675f2/torchvision/models/resnet.py">here</a> (you might need to fix some imports, too).<br>
Once this is done, define your custom dropout layer as an <code>nn.Module</code> and use it in your model.<br>
Don’t forget to switch its behavior using the internal <code>self.training</code> flag so that you can disable it after calling <code>model.eval()</code> (assuming this dropout layer is also a no-op during evaluation).</p>

zak_lah: <p>Thank you sir for your answer</p>
Biases are all zeros for QAT

MrOCW: <p>Hi, I have trained a YOLOv5 QAT model and exported it to ONNX. I noticed in ONNX that the biases are all set to 0. Is this supposed to be the case for QAT modules? Is this due to fuse_modules that causes the biases to be 0 in ONNX?</p>

supriyar: <p>Hi <a class="mention" href="/u/mrocw">@MrOCW</a>, afaik there is no supported way to train models using pytorch quantization and then export the quantized models to ONNX. We currently do not plan to add support for ONNX export of pytorch quantized models. If this is important for your use case, feel free to submit a feature request or PR and our team can take a look.</p>

MrOCW: <p>Hi <a class="mention" href="/u/supriyar">@supriyar</a>, I am exporting the pre-quantized model with the fake quants and their q params, not the quantized model.</p>

supriyar: <p>I see, if this happens for conv layers that have fusion with batch_norm then the possible reason is here <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/intrinsic/qat/modules/conv_fused.py#L104-L107" class="inline-onebox" rel="noopener nofollow ugc">pytorch/conv_fused.py at master · pytorch/pytorch · GitHub</a><br>
where we set the biases to zero and add the original as part of the batch_norm operator.</p>
Matrix multiplication along specific dimension

Anurag_Dalal: <p>So I want to multiply 2 matrices that has dimensions:</p>
<pre><code class="lang-auto">torch.Size([10, 16, 240, 320])
torch.Size([10, 32, 240, 320])
</code></pre>
<p>now I want the output to be [10, 16, 32] (it will multiply the last 2 dimensions element-wise and sum them)</p>
<p>The code that generates the 2 metrics:</p>
<pre><code class="lang-python">import torch

b = 10
h1 = 480
w1 = 640

h2 = 240
w2 = 320

m = 16
n = 32
# task 1: interpolate F1 [h1,w1] to [h2,w2] --&gt; [h,w]
# task 2: multiply channel wise, sum them and divide by h*w, output-&gt; [b,m,n]
F1 = torch.rand([b, m, h1, w1])
F2 = torch.rand([b, n, h2, w2])

F1 = torch.nn.functional.interpolate(F1, [h2, w2], mode = 'bicubic', align_corners = True)
</code></pre>

Eta_C: <pre><code class="lang-python">out = torch.matmul(F1.view(-1, h2 * w2), F1.view(-1, h2 * w2) .t())
</code></pre>

Anurag_Dalal: <p>Thank you for your reply.<br>
So, I did</p>
<pre><code class="lang-python">out = torch.matmul(F1.view(-1, h2 * w2), F2.view(-1, h2 * w2) .t())
print(out.shape)
</code></pre>
<p>And the output shape is:</p>
<pre><code class="lang-auto">torch.Size([160, 320])
</code></pre>
<p>But it should be [10, 16, 32]</p>

Anurag_Dalal: <pre><code class="lang-python">def FSP(layer1, layer2):
    b = layer1.shape[0]

    h2 = layer2.shape[2]
    w2 = layer2.shape[3]

    m = layer1.shape[1]
    n = layer2.shape[1]

    mid = torch.nn.functional.interpolate(layer1, [h2, w2], mode='bicubic', align_corners=True)
    F3 = torch.zeros(b, m, n)
    for batch in range(b):
        for i in range(m):
            for j in range(n):
                F3[batch, i, j] = torch.mean(torch.mul(mid[batch, i, :, :], layer2[batch, j, :, :]))

    return F3
</code></pre>
<p>Actually, this is what I am trying to implement, but without using loops.</p>
How to Export Real-Time-Capable LSTM to ONNX

cwitkowitz: <p>I am having trouble getting a model with several LSTMs to export to ONNX properly. The main issue is that I intend to use the model in an online fashion, i.e. feeding in one frame of data at a time.</p>
<p>My LSTM code is similar to the following:</p>
<pre><code class="lang-auto">class MyLSTM(torch.nn.Module):
    def __init__(self, dim_in, dim_out):
        super().__init__()

        self.deployed = False

        self.hidden = torch.zeros(1, 1, dim_out)
        self.cell = torch.zeros(1, 1, dim_out)

        self.lstm = torch.nn.LSTM(input_size=dim_in,
                                  hidden_size=dim_out,
                                  batch_first=True,
                                  bidirectional=False)

    def deploy(self):
        self.deployed = True

    def forward(self, x):
        if self.deployed:
            out, (self.hidden, self.cell) = self.lstm(x, (self.hidden, self.cell))
        else:
            out, _ = self.lstm(x)

        return out
</code></pre>
<p>Obviously, before I export the model to ONNX, I call <code>deploy()</code>. The PyTorch model works as expected, and I even tried saving it as a <code>ScriptModule</code> with <code>torch.jit</code>. However, when I load and attempt to use the exported model using <code>onnxruntime</code>, it’s behavior suggests that it never updates the hidden/cell state.</p>
<p>Is it possible to do this? If so, how?</p>
Multi gpu training need advice

R_S: <p>hello,</p>
<p>I have 2 gpus (8 + 8)gb. How can i use both the gpu for my training. I am using<br>
<code>model = nn.DataParallel(model)</code><br>
But, the gpu memeory does not seem to be utilized fully. Is this the right way?</p>
<p>edit: should i buy single 16gb gpu or can i use 2 8gb gpus?</p>
<p>thank you</p>
Hardware accelerated NNAPI tests

dennism: <p>I went through the speed benchmark for Android page to test if our hardware could potentially be used with PyTorch models for accelerating something we’re working on.</p>
<p>Actual results are that the NNAPI models are much much slower:</p>
<blockquote>
<p>./speed_benchmark_torch --pthreadpool_size=1 --model=mobilenetv2-quant_core-cpu.pt --use_bundled_input=0 --warmup=5 --iter=50<br>
Starting benchmark.<br>
Running warmup runs.<br>
Main runs.<br>
Main run finished. Microseconds per iter: 97858.2. Iters per second: 10.2189<br>
./speed_benchmark_torch --pthreadpool_size=1 --model=mobilenetv2-quant_core-nnapi.pt  --use_bundled_input=0 --warmup=5<br>
Starting benchmark.<br>
Running warmup runs.<br>
Main runs.<br>
Main run finished. Microseconds per iter: 560976. Iters per second: 1.78261</p>
</blockquote>
<p>Quick sanity check… did I run this test right to get expected faster results with the NNapi model?</p>
<p>I compiled PyTorch from sources and just used the latest origin/master (relatively recent).  My platform is Android 11 and the hardware is NXP i.mx8m which claims to support hardware acceleration for NNapi (It has a built in NPU).</p>
<p>NXP however only provides demo acceleration for TensorFlow.  If I run that demo, the results are a significant speedup for NPU acceleration.</p>
<p>So my goal is to find out which module is failing.  Can someone suggest a known working phone or android device that will show PyTorch successfully hardware accelerated?  If I can get desired results there, I can then move on to find out if our hardware is not behaving properly and get support from NXP.</p>
<p>Any other debugging advice appreciated.</p>
<p>Thanks for work on a cool project!</p>

axitkhurana: <p>Hi <a class="mention" href="/u/dennism">@dennism</a> Try turning on verbose adb logs and checking if the ops are running on the accelerator?</p>
<p><a href="https://developer.android.com/ndk/guides/neuralnetworks#nnapi_logs" class="onebox" target="_blank" rel="noopener nofollow ugc">https://developer.android.com/ndk/guides/neuralnetworks#nnapi_logs</a></p>
<p>Sounds like some part of the graph is running on the CPU which goes to the NNAPI reference implementation which might be slower than regular Pytorch code</p>
Conda build compiler for cpp_extension

wconnell: <p>I am using a library that calls <code>torch.utils.cpp_extension.load()</code> and I believe I am having trouble specifying the correct compiler.  I receive a compiler warning:</p>
<blockquote>
<p>!! WARNING !!<br>
!!!<br>
Your compiler (c++) is not compatible with the compiler Pytorch was<br>
built with for this platform, which is g++ on linux. Please<br>
use g++ to to compile your extension. Alternatively, you may<br>
compile PyTorch from source using c++, and then you can also use<br>
c++ to compile your extension.<br>
See <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md</a> for help<br>
with compiling PyTorch from source.<br>
!!!<br>
!! WARNING !!</p>
</blockquote>
<p>I used conda to build pytorch so I believe the incorrect system default compiler is being called. So I have tried assiging the <code>CXX</code> environment variable per the <a href="https://pytorch.org/docs/stable/cpp_extension.html" rel="noopener nofollow ugc">documentation</a>:</p>
<blockquote>
<p>To compile the sources, the default system compiler (<code>c++</code> ) is used, which can be overridden by setting the <code>CXX</code> environment variable.</p>
</blockquote>
<p>However I do not think I am specifying the correct path to the compiler that was used to build pytorch with conda on my linux system. Is this diagnosis on the correct path? If so, how do I determine the path to the g++ compiler used to build pytorch?</p>
Why my linear layer is expecting 4-dimensional weight?

GamarChoba: <p>I am very new to Deep learning. I am working on the CIFAR10 dataset and created a CNN model which is as below.</p>
<pre><code class="lang-auto">class Net2(nn.Module):
  def __init__(self):
    super(Net2, self).__init__()
    self.conv1 = nn.Conv2d(3, 32, 5, 1)
    self.fc1 = nn.Linear(32 * 4 * 4, 512)
    self.fc2 = nn.Linear(512,10)


  def forward(self, x):
    x = x.view(x.size(0), -1)
    x = F.max_pool2d(F.relu(self.conv1(x)),(2,2))
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x
    

net2 = Net2().to(device)
</code></pre>
<p>My assignment requirements are to create a model with:</p>
<p>Convolutional layer with 32 filters, kernel size of 5x5 and stride of 1.<br>
Max Pooling layer with kernel size of 2x2 and default stride.<br>
ReLU Activation Layers.<br>
Linear layer with output of 512.<br>
ReLU Activation Layers.<br>
A linear layer with output of 10.</p>
<p>Which I guess I wrote. But I am assuming that I am going to the wrong path. Please help me to write the correct model and also the reason behind those arguments in Conv2d and Linear layers.</p>
<p>The error which I am getting from my code is as below:</p>
<pre><code class="lang-auto">RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 3, 5, 5], but got 2-dimensional input of size [1024, 3072] instead
</code></pre>

AlphaBetaGamma96: <aside class="quote no-group" data-username="GamarChoba" data-post="1" data-topic="138711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ad7895/48.png" class="avatar"> GamarChoba:</div>
<blockquote>
<pre><code class="lang-auto">    x = x.view(x.size(0), -1)
</code></pre>
</blockquote>
</aside>
<p>This reshapes your input into a 2d Tensor, <code>nn.Conv2d</code> requires the input be 4D.</p>
How to reduce processing time of TensorImageUtils.bitmapToFloat32Tensor function

user21: <p>After converting the bitmap generated by Android into a Tensor through the TensorImageUtils.bitmapToFloat32Tensor function, we have been receiving the result value through inference.</p>
<p>However, the processing time of that function is about 2 seconds, so it takes a lot of time to get the full result.</p>
<p>In bitmap, an image is drawn in real time, and each time a bitmap is obtained and inferred.<br>
The result is delayed compared to the real-time video, so matching is not possible.</p>

szewaiyuen6: <p><a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> Wondering if you may have some thoughts on this. Thank you!</p>
DCGAN Tutorial: Multiple backward calls

James_e: <p>I have a question regarding the <code>.backward()</code> calls in the DCGAN tutorial.</p>
<p>When training the discriminator, an error for the real data, <code>errD_real</code>, and an error for the fake data, <code>errD_fake</code>, is calculated. Each of these have <code>.backward()</code> called on them <strong>before</strong> the <code>.step()</code> method is called on the optimiser.</p>
<p>How does this work? Would the second <code>.backward()</code> call not overwrite the gradient values stored in <code>.grad</code> attribute of the optimiser?</p>
<p>Thanks for your help!</p>

ptrblck: <aside class="quote no-group" data-username="James_e" data-post="1" data-topic="138605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_e/48/12325_2.png" class="avatar"> James_e:</div>
<blockquote>
<p>How does this work? Would the second <code>.backward()</code> call not overwrite the gradient values stored in <code>.grad</code> attribute of the optimiser?</p>
</blockquote>
</aside>
<p>No, each <code>backward</code> call accumulates the gradients in the <code>.grad</code> attribute of the used parameters, which is also why you have to call <code>optimizer/model.zero_grad()</code> before calculating the gradients of a new iteration.</p>

James_e: <p>Would this therefore be the same as calling <code>(errD_real + errD_fake).backward()</code>?</p>

ptrblck: <p>Yes, the resulting gradient would be the same, but you would use more memory, since both computation graphs needs to be kept before the final <code>backward()</code> operation is called. If you call <code>backward()</code> separately, the intermediate tensors would be freed right afterwards. You would thus save memory but pay for it with two <code>backward</code> operations.</p>