akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataParallel splits up input parameters

Liam: <p>I ran into an issue related to the DataParallel class I’m not sure how to solve. Here’s a minimal example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from   torch.nn import functional as F
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class Model(nn.Module):
    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)
    '''
    def forward(self, input, param):
        x = self.fc(input)
        return x
    '''
    def forward(self, input, param):
        w, b = param
        print("Shape of w: ", w.shape)
        output = F.linear(input, w, b)
        return output

input = torch.randn(10, 3).cuda()
w, b = torch.randn(4, 3).cuda(), torch.randn(4).cuda()
model = Model(3, 4)
model.to(device)

# Single GPU
output = model(input, [w, b])
print("Outside: input size", input.size(), "output_size", output.size())

# DataParallel
model = nn.DataParallel(model)
output = model(input, [w, b])
print("Outside: input size", input.size(), "output_size", output.size())
</code></pre>
<p>Let’s say somehow I need to redefine the forward method with nn.functional calls, so every forward propagation comes with newly-defined input parameters. The problem is when I try to parallelize the model, the DataParallel class seems to split up not only the data but also my input parameters, causing above output sizes to be diffferent.</p>
<p>This is what I get with 2 GPUs available:</p>
<blockquote>
<p>Shape of w:  torch.Size([4, 3])<br>
Outside: input size torch.Size([10, 3]) output_size torch.Size([10, 4])<br>
Shape of w:  torch.Size([2, 3])<br>
Shape of w:  torch.Size([2, 3])<br>
Outside: input size torch.Size([10, 3]) output_size torch.Size([10, 2])</p>
</blockquote>
<p>How can I fix this?</p>
<p>I’m doing this for a MAML implementation and this seems the only way to do it, as load_state_dict() aren’t able to perserve previous computational graphs.</p>

Yanli_Zhao: <p>dataparallel will scatter/split input and args[w, b]  to different devices, but replicate the model.parameters().</p>
<p>Why do you need to put [w,b] as args? model.parameters() will include them, right?</p>

Liam: <p>Thank you for your response. In the MAML setting, we need two sets of parameters to be preserved, so I use one of them as model.parameters() and the other loaded with nn.functional operations.</p>
<p>Because eventually the loss comes from forward propagations through both parameters, the computational graphs need to be preserved as well, so it can’t be done with load_state_dict operations, or storing them in separate models.</p>
<p>This is related to the discussion here: <a href="https://discuss.pytorch.org/t/gradient-computation-in-meta-learning-algorithms/6039" class="inline-onebox">Gradient computation in meta-learning algorithms</a></p>
PyTorch: torchsummary for loaded model

gakadam: <p>I have the following code to print the model summary of a previously saved model:</p>
<pre><code class="lang-auto">import torch                                                                                                                                              

model = torch.load('results/model_best.pth.tar')
print(model) # works

from torchsummary import summary

summary(model, (3, 224, 224)) # error
</code></pre>
<p>The print method works okay, but the summary method has the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "ModelSummary.py", line XX, in &lt;module&gt;
    summary(model, (3, 224, 224))
  File "anaconda3/lib/python3.8/site-packages/torchsummary/torchsummary.py", line 68, in summary
    model.apply(register_hook)
AttributeError: 'dict' object has no attribute 'apply'
</code></pre>
<p>How to rectify this? (I tried the summary method on a model from torchvision database. It worked okay).</p>
<p>Thank you.</p>

ccoding: <p>Maybe you should instantiate it when you use <code>summary</code> function to show it.<br>
Try like this</p>
<pre><code class="lang-auto">model_dict = torch.load('results/model_best.pth.tar')
model = YourModel() #Your Model Class Here
model.load_state_dict(model_dict)
summary(model, (3, 224, 224)) 
</code></pre>
Using nn.DataParallel(model, device_ids=[0]) results in lost imports

Glerk: <p>I have a main python file in the same folder as another file I import. If I use the nn.DataParallel function I get an error saying the secondary file cannot be found. To get around this I add the folder to my global path and it seems to work BUT this shouldn’t be needed.</p>
<p>Interestingly when I run it using debug the import statement at the top of main runs fine, so it’s something related to DataParallel.</p>
PyTorch to ONNX no batch normalization layer

ruka: <p>I have some very standard CNN-BatchNorm-relu combinations in my model, after I use torch.onnx.export(), the BatchNorm layer doesn’t exist any more in onnx model, I carefully checked the model and found that BN has been fused in CNN layer.<br>
This happens after I update my pytorch to 1.7, my code used to work in 1.6. Is there a way to keep BN layer even after it is converted to onnx model?</p>

0e6c384a462aeae391b3: <p>Same problem, the BatchNorm Layers disappear after exporting to onnx  format…<br>
How did you find BN has been fused in CNN layer? In the source code of PyTorch?</p>

Eta_C: <p>See <a href="https://github.com/pytorch/pytorch/issues/49226" rel="noopener nofollow ugc">This Issue</a></p>
Propsal for "version first supported" documentation feature

KFrank: <p>Hello Forum!</p>
<p>How hard would it be (Yeah, yeah, I know …   famous last words …)<br>
to add to the documentation the version of pytorch to which a feature<br>
(function, class, etc.) was first added?</p>
<p>I recall seeing something like this in various java documentation.  I’m<br>
thinking of something like the javadoc <code>@since</code> tag.</p>
<p>Best.</p>
<p>K. Frank</p>

richard: <p>I think this is a good idea but I’m not sure how difficult it would be to implement. A python decorator that behaves like <code>@since</code> could indeed work, I’d expect it to modify a function’s docstring and add a “since vX.X” line.</p>
<p>Please open us a feature request issue on our issue tracker! <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox">Issues · pytorch/pytorch · GitHub</a></p>

KFrank: <p>Hi Richard!</p>
<aside class="quote no-group" data-username="richard" data-post="2" data-topic="108448" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p>Please open us a feature request issue on our issue tracker!</p>
</blockquote>
</aside>
<p>I apologize, but I don’t have a github account at the moment.  (I used<br>
to have one, but google munged my credentials.)</p>
<p>Best.</p>
<p>K. Frank</p>
Is there some architecture overview of glow?

111453: <p>I am new to glow. And I am watching for some materials to study glow. After surfing the git repo <a href="https://github.com/pytorch/glow" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/glow: Compiler for Neural Network hardware accelerators</a>, I found that there are some documents and wiki. But they are not organized structurally. Do you have any suggestions for me on how to ramp up?</p>
<p>What I want maybe something like the overview of glow’s architecture.</p>

jfix: <p>Hi <span class="mention">@tanyo</span>, the <a href="https://arxiv.org/abs/1805.00907" rel="noopener nofollow ugc">Glow arXiv paper</a> may be useful. Otherwise I am not sure there is a much better doc other than what exists in the docs folder on the Github page. If you have questions feel free to ask via a Github Issue or here.</p>

111453: <p>Thanks <a class="mention" href="/u/jfix">@jfix</a>. I also found the slides sharing at IEEE: <a href="https://ieee-region6.org/scv-cis/wp-content/uploads/sites/6/2019/03/IEEE_Glow.pdf" rel="noopener nofollow ugc">https://ieee-region6.org/scv-cis/wp-content/uploads/sites/6/2019/03/IEEE_Glow.pdf</a></p>
Train test image quality difference

Sai_Gayatri_Prasad: <p>Hello,</p>
<p>I am currently training my age-gender estimation model on images from various datasets ( with different image quality if it makes sense) and will be testing it on images obtained from either a webcam or CCTV.</p>
<p>I plan to add image quality enhancements like increasing sharpness and contrast for the test set. I was wondering if there are any similar experiments performed and how the results were.</p>
<p>Intuitively, I understand that the model should have no problem predicting in better quality images and would like to check more sources.</p>
<p>Thank You</p>
Stop backpropagation of sequence models for padded sequences

Thomas_Ricatte: <p>Hello,</p>
<p>I am working on NLP tagging models (i.e. I have to produce one tag per input token). In this context, I am working with:</p>
<ul>
<li>padded sequences</li>
<li>BertModel / BiLSTM stack</li>
</ul>
<p>Ideally, I would prevent any padding side-effect (aka no backpropagation on padded states). What is the correct way to handle this ? I found the PackedSequence object but it’s not clear how it behaves especially when several models are stacked.</p>
<p>Thanks for your help or any pointer on the best practice on this topic !</p>

Abhilash_Srivastava: <p>Use masking over the padded sequence.<br>
You can either use either create your own mask or used <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.masked_fill_" rel="noopener nofollow ugc">masked_fill</a>.<br>
Checkout <a href="https://charon.me/posts/pytorch/pytorch_seq2seq_4/#4-packed-padded-sequences-masking-inference-and-bleu" rel="noopener nofollow ugc">this</a> blogpost.</p>
Is it possible the edit the weight matrix after the forward pass but before the backward pass?

mk100000: <p>Hi,</p>
<p>I’m looking to see if one can edit a weight matrix right before backpropagation takes place, but after the forward method is applied, so that the forward weight matrices are not identical to the ones used during backpropagation. To set a simple example,</p>
<pre><code class="lang-auto">some_const = torch.tensor([5],requires_grad=False,dtype=torch.float)
other_const = torch.tensor([3],requires_grad=False,dtype=torch.float)

x = torch.tensor([1.000],dtype=torch.float,requires_grad=True)
y = some_const*x**5

with torch.no_grad():
   some_const.copy_(other_const)

y.backward()
print(x.grad) # wish to have 15 here
</code></pre>
<p>when I try the above, I get an error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>I’m not really familiar with the topic, but I have heard that something called feedback alignment is very close to what I’m trying to do, in case that helps.</p>
<p>Any help is greatly appreciated.</p>

albanD: <p>Hi,</p>
<p>The problem here is that the original value of <code>some_const</code> was required and you modified it.<br>
Hence the error you’re seeing.</p>
<p>If you want explicitly change that value so that autograd will compute wrong gradients, then I don’t think you should be doing that <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> In particular, it is an implementation detail of the different formulas which Tensor is saved and used or not. So you cannot predict what effect changing this Tensor inplace will have because it might not be used in the backward computation at all.</p>

mk100000: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Well, to spill the beans a little bit, I’m working on a method that produces a mask on the weights, which is applied during the <code>backward()</code> method, just for experimentation purposes. The <a href="https://www.nature.com/articles/ncomms13276" rel="noopener nofollow ugc">feedback alignment paper</a> does something similar, by using a random weight matrix during backpropagation which is asymmetric to the forward weight matrix. Maybe the small example I provided above doesn’t fully represent the complexity of the task… I’m just curious if it can be done. Could I subclass <code>torch.autograd.Function</code> and implement my own <code>backward()</code> method?</p>
<p>Thanks</p>

albanD: <p>If you just want to modify the gradients, you can either add a hook to the Tensor with <code>register_hook()</code> and whatever is returned by that hook will be used as gradient.<br>
Or if it is more involved, you can do that with a custom Function as well.</p>
<p>But modifying the weights data directly is dangerous and might break for no reason in the future.</p>
Passing arguments to forward method

Megh_Bhalerao: <p>Hi All,<br>
I am instantiating an architecture like this:</p>
<pre><code class="lang-python">class Discriminator_classwise(nn.Module):
    def __init__(self, inc=4096, num_class=126):
        super(Discriminator, self).__init__()
        self.classfier_list = []
        for _ in range(num_class):
            self.classfier_list.append(nn.Sequential(nn.Linear(inc,512),nn.Linear(512,512),nn.Linear(512,2)))

    def forward(self, x, reverse=False, eta=1.0, choose_class = 0):
        if reverse:
            x = grad_reverse(x, eta)
        which_classifier = self.classfier_list[choose_class]
        x_out = which_classifier(x)
        return x_out
</code></pre>
<p>Essentially, what I am doing is I want to choose <strong>a</strong> classifier from the list of classifier according to the class that is predicted for my input.</p>
<p>Could someone help me in writing a forward function for this, which takes an argument which is a class index and returns the forward passed tensor through the desired classifier?</p>
<p>Thanks in advance,<br>
Megh</p>

albanD: <p>Hi,</p>
<p>You simply need to make your list a <code>ModuleList</code> so that it is tracked properly:</p>
<pre><code class="lang-auto">self.classfier_list = nn.ModuleList()
</code></pre>
<p>And then the code you shared will work just fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Megh_Bhalerao: <p>Thanks a lot <a class="mention" href="/u/alband">@albanD</a>. But I am wondering, why will a normal list not work? Anyways, I will just access the appropriate network from the list that is it, right?</p>
<p>Does this make sense? Please let me know if something is unclear.</p>

albanD: <p>A regular list doesn’t work because the nn construct are not opening plain list for speed reasons.<br>
But the ModuleList will behave just like a regular python list <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Initialization error when using torch.distributed

xdwang0726: <p>I got this error when using initialization torch.distirbuted.</p>
<pre><code class="lang-auto">  File "run_classifier_gcn.py", line 393, in &lt;module&gt;
    main()
  File "run_classifier_gcn.py", line 309, in main
    rank=args.local_rank, world_size=args.world_size)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 422, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 119, in _tcp_rendezvous_handler
    raise _error("rank parameter missing")
ValueError: Error initializing torch.distributed using tcp:// rendezvous: rank parameter missing
</code></pre>
<p>Where actually I have add the rank parameter when initialization.</p>
<pre><code class="lang-auto">    dist.init_process_group(backend=args.dist_backend, init_method='tcp://{}:{}'.format(ip_address, args.port),
                            rank=args.local_rank, world_size=args.world_size)
</code></pre>

Yanli_Zhao: <aside class="quote no-group" data-username="xdwang0726" data-post="1" data-topic="108294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xdwang0726/48/17441_2.png" class="avatar"> xdwang0726:</div>
<blockquote>
<p><code>rank parameter missing</code></p>
</blockquote>
</aside>
<p>‘rank parameter missing’ ==&gt; do you want to check whether the ‘args.local_rank’ is valid?</p>

xdwang0726: <p>never mind, I got it work, yeah, I put the wrong rank at first, thanks!</p>
Handling RGBA images

satanarmy: <p>When we have alpha channel present in images, reading the data with ImageFolder converts it to RGB. I’m a Pytorch noobie. How would one write custom Datalodaer/Dataset to read images with 4 channels?</p>

Scott_Hoang: <p>what is your image file format?</p>

satanarmy: <p>Images are PNG with 32 bit format, 8 bits for each of the four channel.</p>

ptrblck: <p><code>ImageFolder</code> uses the <code>pil_loader</code> by default, if <code>accimage</code> is not the used backend and if no other loader was specified.<br>
This loader converts the images to <code>RGB</code> as seen <a href="https://github.com/pytorch/vision/blob/cf534fdaee048a7edce90a5577508985c58c3602/torchvision/datasets/folder.py#L152-L156">here</a>.<br>
To use the <code>RGBA</code> channels, you could write a custom loader (basically just copy-paste the <code>pil_loader</code> and remove the <code>convert</code> call or convert to <code>RGBA</code>) and pass it as the <code>loader</code> argument to <code>ImageFolder</code>.</p>

satanarmy: <p>This worked. Thanks!</p>

VirginieBfd: <p>Uploading my solution:</p>
<pre><code class="lang-auto">from PIL import Image
from torchvision import datasets

def custom_loader(path):
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGBA')
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),data_transforms[x], 
                  loader=custom_loader)
                  for x in ['train', 'val']}
</code></pre>
Implementation doubt on Monte Carlo Dropout for pretrained model

Paulo_R_Junior: <p>I know that I should apply Dropout on training and testing, get several results for prediction and so on. But, considering a pretrained model, how should I add Dropout layers?</p>
<ul>
<li>on the fully-connected layers;</li>
<li>after every convolutional layer.</li>
</ul>
<p>I’ve read some papers and implementations where one applies Dropout at fully-connected layers only, using a pretrained model; however, when using a custom model, usually one adds Dropout after every conv layer.</p>
<p>Using Dropout on the fully-connected layers only can be considered an actual uncertainty estimation of the model?</p>
Functions with (almost) same output but (very) different gradients

informator: <p>Hello. I have implemented in two different ways my own loss function, which involves computing a (one-dimensional) integral. The. first one is using “torch.trapz” to compute the integral and the other one is computing the integral exactly in closed-form.</p>
<p>I have done a lot of experiments to confirm that the two implementations are correct and do return the same results. Essentially, there are just tiny numerical approximation differences, for example, 15020.1720 instead of 15020.1724. In other words, we can assume that the two implementations are identical.</p>
<p>One thing I do not understand is why I get (very) different gradients for my parameters (using backwards on my loss). I was expecting to get (almost) the same gradients given that the two functions compute the same quantity.</p>
<p>Am I missing something? I know it is hard to say without seeing the code, but what would be the typical error for a non-expert in this case? What should I look at?</p>
<p>Thank you.</p>
Pytorch debugging

Sathsara_Rasantha: <p>I am getting this error while training a Bidirectional attention flow model for question answering.</p>
<h2>Starting training …</h2>
<p>Starting batch: 0</p>
<p>IndexError Traceback (most recent call last)<br>
in ()<br>
----&gt; 1 train(model, train_dataset)</p>
<p>2 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)<br>
2262 .format(input.size(0), target.size(0)))<br>
2263 if dim == 2:<br>
-&gt; 2264 ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
2265 elif dim == 4:<br>
2266 ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)</p>
<p>IndexError: Target 545 is out of bounds.</p>
<p>It seems that there is some problem with the way I set the target values in data preprocessing.<br>
What are the best ways to debug these kinds of errors in pytorch. Are there any tutorials or books I can refer to ?</p>

ptrblck: <aside class="quote no-group" data-username="Sathsara_Rasantha" data-post="1" data-topic="107583">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sathsara_rasantha/48/32726_2.png" class="avatar"> Sathsara_Rasantha:</div>
<blockquote>
<p>What are the best ways to debug these kinds of errors in pytorch. Are there any tutorials or books I can refer to ?</p>
</blockquote>
</aside>
<p>The first step would be to check the error message, which provides good information for this particular failure: the target contains indices, which are out of bounds in <code>nll_loss</code>.</p>
<p>From a general point I don’t know, if there are any good books to debug Python applications.<br>
If you are using the GPU, rerun the script with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>, as you might often get the wrong lines of code in the stack trace due to the asynchronous CUDA operations.<br>
Alternatively, you could rerun the code on the CPU and should see a better error message.<br>
If you need to debug further, you can use e.g. <code>pdb</code> or <code>gdb</code> for the backend.</p>
Plateau-ing training loss and LOW train_accuracy/test_accuracy

promach: <p>With regards to the thread topic, could anyone help to advise what is wrong with the <a href="https://gist.github.com/promach/52c5199e98647c694163abd0b3af3dae#file-net-py-L152-L167" rel="noopener nofollow ugc">loss_function() computation logic</a> (especially <code>policy_output_discrete</code> and <code>value_output_discrete</code>) in my NN ?</p>
<pre><code class="lang-auto"># Forward Pass
policy_output, value_output = net(_board_features_and_turn)

# Since both policy_output and value_output are of continuous probability nature,
# we need to change them to discrete number for loss_function() computation
policy_output_discrete = torch.zeros(len(_score), NUM_OF_POSSIBLE_MOVES, requires_grad=True)
if USE_CUDA:
    policy_output_discrete = policy_output_discrete.cuda()

for topk_index in range(len(_score)):  # functionally equivalent to softmax()
    policy_output_discrete[topk_index][policy_output.topk(1).indices[topk_index]] = 1

# substract 1 because score is one of these [-1, 0, 1] values
value_output_discrete = torch.topk(value_output, 1).indices - 1

# Loss at each iteration by comparing to target(moves)
loss1 = loss_function(policy_output_discrete, move)
# Loss at each iteration by comparing to target(score)
loss2 = loss_function(value_output_discrete, _score)

loss = loss1 + loss2

# Backpropagating gradient of loss
optimizer.zero_grad()
loss.backward()
</code></pre>
What do TensorDataset and DataLoader do?

Jude_Capachietti: <p>I am used to using numpy arrays in the form X,y and fitting a model to those.  I can’t understand what Datasets and Dataloaders do to the X and y vectors.  I have searched on the internet a fair amount and I still cannot figure out what those functions do.</p>
<p>I am hoping someone on here can give me a simple quick explanation of what these functions do and are for.  Here’s an example of where how I use these functions:</p>
<p>trainset = torch.utils.data.TensorDataset(X_train, y_train)</p>
<p>trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, drop_last=True)</p>
<p>Why do I do these to load data into a Neural Network, instead of just using X_train, y_train?</p>

ptrblck: <p>You can use the plain tensors as <code>X_train</code> and <code>y_train</code>, if you are able to load them completely (and push to the GPU without sacrificing too much memory).<br>
The <code>Dataset</code> is ab abstraction to be able to load and process each sample of your dataset lazily, while the <code>DataLoader</code> takes care of shuffling/sampling/weigthed sampling, batching, using multiprocessing to load the data, use pinned memory etc.<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> might be helpful to see the advantages of using this approach.<br>
That being said, you are of course fine to use the tensors directly, which might also be faster if you are using a tiny dataset.</p>

Jude_Capachietti: <p>Hey thank you so much ptrblck.  This is very helpful to my understanding!  I feel it is good for me to understand what these functions do, so it is not like some black box that I am too scared to touch!</p>
<p>I see the difference now, and I will use a DataLoader now for the helpful reasons you mentioned.</p>
<p>All the best!</p>
How to create 3D Gaussian window?

Anno: <p>I saw that creating a Gaussian window works for 1D tensors with <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.windows.gaussian.html" rel="noopener nofollow ugc">scipy.signal.window.gaussian</a>, but how would it work for 3D tensors? I expect for example a tensor of shape (64, 64, 32) to have a value of 1.0 at the central voxel and 0.0 at the borders.</p>
<p>Thank you in advance!</p>
Adaptive filtering

Peter_Featherstone: <p>Can anyone explain why this ONNX export fails, and whether they can propose a fix. I am doing adaptive filtering. There is a network that predicts a set of filters, one for each batch, then each filter is convolved with the input. I’ve posted an <a href="https://github.com/pytorch/pytorch/issues/50348" rel="noopener nofollow ugc">issue</a> on github but i am calling on the community to help me.</p>
<pre><code class="lang-auto">    class Localiser(nn.Module):
        def __init__(self, nfeats):
            super(Localiser, self).__init__()
            self.net = nn.Sequential(nn.Conv1d(1,128,3,padding=1),
                                     nn.AdaptiveAvgPool1d(1))
            self.fc = nn.Linear(128,nfeats)

        def forward(self, x):
            x = self.net(x)
            x = x.flatten(1)
            x = self.fc(x)
            return x

    @torch.jit.script
    def adaptive_filter(x, w, padding: int):
        B = x.shape[0]
        w = w.unsqueeze(1)
        x = x.view(1,B,-1)
        x = F.conv1d(x, w, padding=padding, groups=B)
        x = x.view(B,1,-1)
        return x

    class Equalizer(nn.Module):
        def __init__(self, ntaps):
            super(Equalizer, self).__init__()
            self.loc = Localiser(ntaps)
            self.ntaps = ntaps

        def forward(self,x):
            w = self.loc(x)
            x = adaptive_filter(x, w, self.ntaps//2)
            return x

    x1 = torch.randn(4, 1, 1024)
    x2 = torch.randn(8, 1, 512) * 10

    model = Equalizer(13)
    y2 = model(x2)

    torch.onnx.export(model,
                      (x1,),
                      "file.onnx",
                      opset_version=12,
                      input_names=['x'],
                      output_names=['y'],
                      dynamic_axes={'x': [0,2], 'y': [0,2]})
    print("exported")
</code></pre>
Loss decreasing rapidly but accuracy is constant

Bixqu: <p>I am training a model in pytorch-geometric, for graph classification. Even with a small model, I am seeing my loss go to very low values, but the accuracy is fixed. How can this be?</p>
<pre><code class="lang-auto">Epoch: 001, Loss: -170941725.82055, Train Acc: 0.82073, Test Acc: 0.84599
Epoch: 002, Loss: -765036750.91172, Train Acc: 0.82073, Test Acc: 0.84599
Epoch: 003, Loss: -1879149609.86397, Train Acc: 0.82073, Test Acc: 0.84599
Epoch: 004, Loss: -3669840125.40666, Train Acc: 0.82073, Test Acc: 0.84599
Epoch: 005, Loss: -6205604898.08394, Train Acc: 0.82073, Test Acc: 0.84599
</code></pre>

neuralpat: <p>I think we’ll need more of your code to be able to help. Would you be able to show all of it ?</p>
How to use pretrained resnet-50 to get features?

cainiaoxueai: <p>Hello!<br>
I want to get 2048 features from a picture by using pretrained resnet-50,is it ok by these code?</p>
<p>resnet50_feature_extractor = models.resnet50(pretrained = True)<br>
resnet50_feature_extractor.fc = nn.Linear(2048, 2048)<br>
nn.init.eye_(resnet50_feature_extractor.fc.weight)</p>
<p>Thanks for your help!</p>

user_123454321: <p>I think it is mostly correct, but I think you need to zero the bias of the fc layer.<br>
Another line of code using</p>
<pre><code class="lang-auto">nn.init.zeros_(resnet50_feature_extractor.fc.bias)
</code></pre>
<p>I usually use</p>
<pre><code class="lang-auto">resnet50_feature_extractor = nn.Sequential(*list(models.resnet50(pretrained = True).children())[:-1])
</code></pre>
<p>but in this way you need to reshape the output by squeezing the spatial dimensions.</p>
Linear projection in residual networks

spring: <p>Hi,</p>
<p>If the dimensions of F(X) and X are different in H(X) = F(X) + X, X must be linearly projected. Is matrix W_s used at this time a parameter to be learned? or Is it a defined(or fixed) special matrix?</p>
<p>Linear projection is like the below<br>
y = F(x, {Wi}) + W_s*x</p>
<p>*Paper: <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=7780459" rel="noopener nofollow ugc">https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=7780459</a></p>

InnovArul: <p>The parameters (if applicable) in the shortcut path are trainable.</p>
<p>Actual implementation of shortcut path in ResNet is here:</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L213-L217" target="_blank" rel="noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L213-L217" target="_blank" rel="noopener">pytorch/vision/blob/master/torchvision/models/resnet.py#L213-L217</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="213" style="counter-reset: li-counter 212 ;">
<li>if stride != 1 or self.inplanes != planes * block.expansion:</li><li>    downsample = nn.Sequential(</li><li>        conv1x1(self.inplanes, planes * block.expansion, stride),</li><li>        norm_layer(planes * block.expansion),</li><li>    )</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


spring: <p>Thanks for your kind response!</p>
Pytorch hyper parameter tuning library

lyjdwh: <p><code>kerastuner</code> provides the following feature. Is there any hyper parameter tuning library that <code>pytorch</code> can use also provides this feature?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e84e6bb10af3091613251af71067bae393cf2bd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/e84e6bb10af3091613251af71067bae393cf2bd3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e84e6bb10af3091613251af71067bae393cf2bd3_2_690x445.png" alt="image" data-base62-sha1="x94MTACyid6GHUg4IthfWHc4Hkf" width="690" height="445" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e84e6bb10af3091613251af71067bae393cf2bd3_2_690x445.png, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e84e6bb10af3091613251af71067bae393cf2bd3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e84e6bb10af3091613251af71067bae393cf2bd3.png 2x" data-dominant-color="F2F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">938×605 65.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

googlebot: <p>TPE algorithm seems to work better for that, so I’d suggest <a href="http://optuna.org/" rel="noopener nofollow ugc">optuna</a> as a more recent alternative to hyperopt, where it first appeared.</p>

lyjdwh: <p>I mean the tunable parameter is dynamic. Is there another library that has this feature?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91af64ecd891c9998abb45cf8576feddbeea36c9.png" data-download-href="https://discuss.pytorch.org/uploads/default/91af64ecd891c9998abb45cf8576feddbeea36c9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91af64ecd891c9998abb45cf8576feddbeea36c9_2_690x319.png" alt="image" data-base62-sha1="kMN0QqzdOwpecZHVDzf9ovLjdDH" width="690" height="319" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91af64ecd891c9998abb45cf8576feddbeea36c9_2_690x319.png, https://discuss.pytorch.org/uploads/default/original/3X/9/1/91af64ecd891c9998abb45cf8576feddbeea36c9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/1/91af64ecd891c9998abb45cf8576feddbeea36c9.png 2x" data-dominant-color="EFEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">733×339 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

lyjdwh: <p>After I read optuna’s documentation, I  found it indeed has this feature!</p>
<p>Thanks very much!</p>
Data loader for large dataset having multiple folders with multiple images and labels

Saurav_Gupta1: <p>I am new to PyTorch and have a small issue with creating Data Loaders for huge datasets. I have a folder “/train” with two folders “/images” and “/labels”. Now, these folders further have 1000 folders that contain 1000 images and 1000 labels in each. I am new to creating custom data loaders. After reading the PyTorch documentation I was able to create the following class but since the dataset is too big 350 GB my code will not work. Can someone please point me in the right direction?</p>
<pre><code class="lang-auto">import numpy as np
import os 
from torch.utils.data import Dataset
from PIL import Image

class CustomDataset(Dataset):
    def __init__(self, data_root):
        self.data_root = data_root
        self.images = []
        self.labels = []
        for filenames in os.listdir(self.data_root)
            data = np.load(data_root+'/'+filename)
            img = data["images"]
            lbl = data["labels"]
            for image, label in zip(img,lbl):
                img = Image.fromarray(image)
                self.images.append(img)
                self.labels.append(label)

    def __len__(self):
        return len(self.filenames)*1000

    def __getitem__(self, index):
        return self.images[index], self.labels[index]
</code></pre>

ptrblck: <p>You could lazily load the data by defining the paths etc. in the <code>__init__</code> method and move the actual loading to the <code>__getitem__</code> method using the already defined paths and the <code>index</code>.</p>

Saurav_Gupta1: <p>I could imagine then I need a for loop inside <code> __getitem__</code> method for going into all the folders that contain the images. Or I need two index values 1. for going into a folder 2. for getting an image.<br>
I thought of another approach that is to run a script that creates a .txt file that contains full paths to all the images with its labels.<br>
After this, I just need an index value to get an image at a location that is at line number:index of the .txt file.</p>

ptrblck: <p>The second approach sounds like the right one and would be the usual workflow.</p>
Data in Cifar10 dataset, torchvision.datasets shown as decimal values

Rohi: <p>I imported Cifar10 dataset from torchvision.datasets, I was expecting the pixel values should be between [0-255], but I can see values as decimal. I belive I havent done any normalization.</p>
<p>Is it the correct data which I am seeing, or some transformations are happening on it?</p>
<p>below is the code,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96dfce1ff0acd97d33929a4beaa74ae7d1bc9d2e.png" data-download-href="https://discuss.pytorch.org/uploads/default/96dfce1ff0acd97d33929a4beaa74ae7d1bc9d2e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96dfce1ff0acd97d33929a4beaa74ae7d1bc9d2e.png" alt="image" data-base62-sha1="lwH7uU7ABxs29mVsMuZtD6lAr0i" width="573" height="500" data-dominant-color="343332"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">952×830 44.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>transforms.ToTensor</code> normalizes the tensors to the range <code>[0, 1]</code> as described in <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor">the docs</a>.</p>

Rohi: <p>oh yeah, true. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Rohi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Also I was wondering how does the lables get converted to one_hot_endcoding, although I didnt do any one_hot_encoding for labels seperately but  model absolutely fine.  is there anything that does one_hot_encoding for labels? Thank You:)</p>

ptrblck: <p>If you are working on a multi-class classification, which I assume since you are using the <code>CIFAR10</code> dataset, then the target is expected in the shape <code>[batch_size]</code> containing the class indices in the range <code>[0, nb_classes-1]</code>, so it’s not one-hour encoded.<br>
The <code>torchvision</code>.datasets` CIFAR10 dataset will return the target in the expected format.</p>
TorchScript does not support assigning output of module to element of tensor

jwdink: <p>I would like to define a module in torch-script that contains other modules. In the forward pass, a matrix would be populated by assigning the output of each sub-module to different elements of the matrix.</p>
<p>Here’s a simplified example:</p>
<pre><code class="lang-auto">import torch
from torch import jit, nn, Tensor


class Module1(jit.ScriptModule):
    def __init__(self, first: torch.nn.Module, second: torch.nn.Module):
        super(Module1, self).__init__()
        self.first = first
        self.second = second

    @jit.script_method
    def forward(self, input: Tensor) -&gt; Tensor:
        out = torch.eye(2)
        out[0,0] = self.first(input)
        out[1,1] = self.second(input)
        return out


def test1():
    m1 = Module1(first=nn.Linear(1, 1), second=nn.Linear(1, 1))
    out = m1(torch.ones(1))
    print(out)


if __name__ == '__main__':
    test1()
</code></pre>
<p>This runs fine with <code>PYTORCH_JIT=0 python3 test.py</code>, but an error is thrown with jit:</p>
<pre><code class="lang-auto">RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "torch_kalman/rewrite/test.py", line 14, in forward
    def forward(self, input: torch.Tensor) -&gt; torch.Tensor:
        out = torch.eye(2)
        out[0,0] = self.first(input)
        ~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        out[1,1] = self.second(input)
        return out
RuntimeError: output with shape [] doesn't match the broadcast shape [1]
</code></pre>
<p>I am not sure why the interpreter thinks that the module outputs have zero extent. Is this a known unsupported feature? I could not find it in documentation. Or is this a bug?</p>
<p>EDIT: Maybe “output of a module” is not needed in the title – seems like nothing generated at runtime can be assigned to tensor-elements? For example, the following attempt at a workaround hits the same error:</p>
<pre><code class="lang-auto">    @jit.script_method
    def forward(self, input: Tensor) -&gt; Tensor:
        out = jit.annotate(List[Tensor], [])
        out += [self.first(input)]
        out += [self.second(input)]
        out = torch.stack(out)
        out2 = torch.eye(len(out))
        for i in range(len(out)):
            out2[i, i] = out[i]
        return out2
</code></pre>

jwdink: <p>The following seems to work:</p>
<pre><code class="lang-auto">    def forward(self, input: Tensor) -&gt; Tensor:
        out = torch.eye(2)
        out[slice(0, 1), slice(0, 1)] = self.first(input)
        out[slice(1, 2), slice(1, 2)] = self.second(input)
        return out
</code></pre>

googlebot: <p>that’s a quirk with zero-dimensional tensors, out[0,0] and out[0,0:1] are not equivalent, and broadcasting from 1-dim to 0-dim is like a “special case”, that is apparently not handled in TorchScript mode.</p>
SSIM/MS-SSIM in pytorch

alwynmathew: <p>It would be great to have SSIM/MS-SSIM in pytorch.</p>
<p>I have opened an issue <a href="https://github.com/pytorch/pytorch/issues/6934" rel="nofollow noopener">#6934</a> for SSIM/MS-SSIM feature request.</p>

francois-rozet: <p>Hello, I’ve created <a href="https://github.com/francois-rozet/piqa" rel="noopener nofollow ugc"><code>piqa</code></a>, a PyTorch IQA metrics library. SSIM and MS-SSIM are implemented, along with PSNR and others.</p>
Glow - Object detection

Alberto_Gonzalez_Iso: <p>Hello,<br>
I want to use mobilenetv2 SSD in an NXP ARM M7 board.</p>
<p>The thing is, how can I compile my object detection model using glow??</p>
<p>I have seen that in the /bin folder it is a object-detector but I have no idea how to use it.</p>

jfix: <p>Hi, we have some documentation in Glow for saving models and targeting e.g. ARM, see <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" rel="noopener nofollow ugc">here</a>. Also I think NXP has their own documentation for doing this, e.g. see <a href="https://www.youtube.com/watch?v=XTW9D9WZIUY" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with eIQ ML Software for Glow Neural Network Compiler - YouTube</a> or other related videos.</p>
Simple Autoencoder to Cortex-M4 (CMSIS-NN)

bun_cha_bowl: <p>Hey</p>
<p>My goal is to train a simple autoencoder with pytorch, somehow quantize the parameters so that they can be used with CMSIS-NN and run the model on a Cortex-M4-Microcontroller.</p>
<p>So far I’ve found a video which explains how to quantize a model for the mnist dataset: <a href="https://www.youtube.com/watch?v=XTW9D9WZIUY" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=XTW9D9WZIUY</a><br>
I haven’t found clear information though on a more general level though.</p>
<p>Is it possible to archive my goal using pytorch und glow?</p>
<p>If yes, are there any examples of something similar? Or is there documentation that describes the necessary steps?</p>
<p>Thank you</p>
<p>Bun Cha</p>

jfix: <p>Hi, you should be able to follow very similar steps just using your model and some inputs that are reasonably close to what you expect to see in the real world, and then run with the <code>-dump-profile=example.yml</code> option. This runs the model in a floating point mode on the CPU to generate a profile of values flowing through your graph. Then that profile can be used when reloading the model, with <code>--load-profile=example.yml</code>, which quantizes the model with that profile, and compiles it.</p>
Glow bundle on small embedded systems: weights.bin too big

groncarolo: <p>I’m implementing a de-noiser using deep learning.</p>
<p>Here’s the steps I followed:</p>
<ol>
<li>I have a pytorch model with 467k parameters (in float) that is 467*4 = 1.9MB big</li>
<li>I converted it into ONNX format: size still 1.9MB</li>
<li>I followed the AOT instructions and got a bundle for my arm-m33 architecture from the ONNX model.</li>
<li>I checked the results of the inference on arm and they match with those of pytorch.</li>
<li>I had a look at the model.weights.bin created in the bundle and it is 20MB, this is a problem for me</li>
</ol>
<p>Unfortunately the client design is not going to have that kind of memory.</p>
<p>Is there any way to shrink the weights down? (ideally to a size close to the ONNX model)</p>

jfix: <p>Sorry for the delayed response – one way to understand what is going on here is to look at the output graph via <code>--dump-graph-DAG=file.dot</code> and then use dot to look over the <code>Constant</code> nodes in the graph to see if any were split or duplicated. This could happen if e.g. there are multiple users and constant folding occurred. It may be necessary to add some option to prevent duplication of weights to the graph optimizer, but not sure.</p>
Torch Dataset class won't emit right index from h5 file

localh: <p>Why won’t my Torch dataset class emit the right index from an h5 file?</p>
<p>It is basically emitting the same first batch over and over and over.</p>
<pre><code class="lang-auto"># prepare torch data set
class H5DataSet(torch.utils.data.Dataset):
    '''
    This functions loads H5 files for deep learning.

    Parameters
    ----------
    transform : optionable, callable flag
        Whether or not we need to transform the data with a separate class

    Returns
    -------
    self.dataset  : torch tensor
        A tensor containing embeddings with the following
        dimensions: [layers, batch_sz, tokens, features]
    '''
    def __init__(self, path):
        self.file_path = path
        self.dataset = None
        with h5py.File(self.file_path, 'r') as file:
            self.dataset_len = len(file["embeds"])

    def __getitem__(self, index):
        if self.dataset is None:
            # embeds are shaped: (13, 8512, 64, 768)
            self.dataset = h5py.File(self.file_path, 'r')["embeds"][:, index, :, :]
        return self.dataset

    def __len__(self):
        return self.dataset_len

embeddings_ds = H5DataSet(path='D:\\h5py_embeds\\bert_embeds.h5')

# check data
for idx, batch in enumerate(embeddings_ds):
    if idx == 0:
        batch_0 = batch
    if idx == 1:
        batch_1 = batch
        break

assert np.array_equal(batch_0, batch_1), 'not a match'
</code></pre>
<p>Edit: I’ve realized that in order to fix this, I need to change the way I build my H5 file in the first place. So, since I want to emit individual batches, I need to ensure that the first dimension in my H5 shape is batches. In other words, change my structure from:</p>
<pre><code class="lang-auto">[layers, batch_sz, tokens, features]
to
[batch_sz, layers, tokens, features]
</code></pre>
Remove isolated nodes in graph using transform

Bixqu: <p>I have a list of graphs in pytorch dataset format. I want to perform graph classification, but I fear that some of these graphs contain isolated nodes. How can I remove these nodes? I see there is <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/transforms/remove_isolated_nodes.html#RemoveIsolatedNodes" rel="noopener nofollow ugc">a transform for this</a> in Pytorch geometric, but it doesn’t say anything about how to apply it.</p>