akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't load and test pytorch model on images, which converted from tensorflow to onnx to pytorch

ml_learn: <p>getting an error like RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 224, 4, 225] to have 3 channels, but got 224 channels instead</p>
<p>initially tensorflow trained model was using efficientnet_b0 architecture and converted this model from tf to onnx by tensorflow-onnx and onnx to pytorch using onnx2pytorch library and few lines of code was :<br>
onnx_model = onnx.load(‘eff_model.onnx’)<br>
pytorch_model = ConvertModel(onnx_model)<br>
torch.save(pytorch_model, ‘effnet_model_.pth’)</p>
<p>but can’t get an outputs when loading model and predicting on real images<br>
transform = transforms.Compose([<br>
transforms.Resize(224),<br>
# transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))<br>
])<br>
model = torch.load(‘effnet_model_.pth’)<br>
test = (‘test_images/’)<br>
test_data = torchvision.datasets.ImageFolder(root = test,transform = transform)<br>
test_loader = torch.utils.data.DataLoader(test_data, batch_size = 1)</p>
<p>with torch.no_grad():<br>
for data in test_loader:<br>
images, labels = data<br>
outputs = model(images)<br>
Please help out. thanks<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> your guidance helps a lot</p>

ptrblck: <p>Your current code seems to contain multiple issues:</p>
<ul>
<li>The input shape is <code>[1 ,224, 4, 225]</code> which is neither a channels-first memory format (expected) not a channels-last one. Make sure to pass the input in the shape <code>[batch_size, channels, height, width]</code>. Also, you might want to pass the <code>size</code> argument to <code>Resize</code> as a <code>tuple</code> to make sure both spatial dimensions are equal.</li>
<li>Even after permuting the input, it would still have 4 channels while 3 are expected.</li>
</ul>
CNN only learning the mean of targets

jzy95310: <p>I’m dealing with a regression task by training a CNN with 334x334 satellite images. As I mentioned in my previous posts, I use MSE loss along with Adam optimizer, and the loss fails to converge. I now realize the reason why the loss fails to converge is that <strong>it only learns the mean of the targets</strong>. For example, if I feed it with 100 data points with mean of target = 7, then all its predictions will be very close to 7 (as shown in the picture below, where x-axis is the true value, and y-axis is the predicted value):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9ad32719b6d7afb0755d7119935b21a7cc4f4479.png" data-download-href="https://discuss.pytorch.org/uploads/default/9ad32719b6d7afb0755d7119935b21a7cc4f4479" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9ad32719b6d7afb0755d7119935b21a7cc4f4479_2_518x500.png" alt="image" data-base62-sha1="m5DVeoxUaDLdrYQKyzLTavtmW9z" width="518" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9ad32719b6d7afb0755d7119935b21a7cc4f4479_2_518x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9ad32719b6d7afb0755d7119935b21a7cc4f4479.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9ad32719b6d7afb0755d7119935b21a7cc4f4479.png 2x" data-dominant-color="F3F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">608×586 54.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
One of my guesses in this case it that my CNN does not learn any visual structures from the image, so I try different architectures such as ResNet and AlexNet, but the problem still exists. How can I prevent my network from this behavior?</p>

Hassan_Imani: <p>I had a similar issue when training a CNN for regression of numerical data.<br>
I decreased my learning rate and my problem is solved.<br>
Also check that your dataset is balanced.</p>

puneet.madan9: <p>I tried creating a simple regression model using Pytorch. The weights are converging to 0 and the bias to mean in the parameters hence all predictions are close to the mean. I tried decreasing the learning rate as well but still not able to converge.</p>
Tensor Splicing

Abhronil_Paul: <p>I have a tensor of size <code>[1,3,224,224] </code> and want to divide into 3 different tensors each of size <code>[1,1,224,224]</code>.<br>
I have tried using this snippet :</p>
<pre><code class="lang-auto">for c in range(x.shape[1]):
  torch.save(x[:, c:c+1, ...], f'channel{c}.pth')
</code></pre>

ptrblck: <p>Your code snippet should work and would save 3 tensors each in the shape of <code>[1, 1, 224, 224]</code>. Are you seeing any issues?</p>
Complexfloat supported in clam pytorch?

mahesh_bhosale: <pre><code class="lang-auto">def pdist(A, B = None, squared = False, eps = 1e-4):
    B = B if B is not None else A
    prod = torch.mm(A, B.t())
    normA = (A * A).sum(1).unsqueeze(1)
    normB = (B * B).sum(1).unsqueeze(1)
    D = clamp((normA.expand_as(prod) + normB.t().expand_as(prod) - 2 * prod), mn=0)
    return D if squared else clamp(clamp(D, mn=eps).sqrt(), mn=eps)
</code></pre>
<p>Getting error,<br>
RuntimeError: “clamp_min_cpu” not implemented for ‘ComplexFloat’</p>
<p>pytorch 1.11.0</p>
<p>How do I bypass this error?</p>

ptrblck: <p>I’ve tested it with the current nightly binary (<code>1.12.0.dev20220425</code>) which also fails with the same error message, so could you create a feature request on GitHub, please?<br>
In the meantime, you might want to split the complex tensor into its <code>.real</code> and <code>.imag</code> parts, apply the <code>torch.clamp</code> method on these parts, and rewrap the result into a complex dtype again.</p>
Reshape 3D Pytorch Tensor

Soothysay: <p>Hi,<br>
So I have a Pytorch Tensor of size [4055, 4, 80]. Now I would like to get a resultant Pytorch tensor of size [4055, (4*80)] or [4055, 320]. How can I do that?</p>

ptrblck: <p>You can flatten the last two dimensions via:</p>
<pre><code class="lang-python">x = torch.randn(4055, 4, 80)
print(x.size())
# torch.Size([4055, 4, 80])

y = x.view(x.size(0), -1) # keep dim0, flatten last dimensions
print(y.size())
# torch.Size([4055, 320])

z = torch.flatten(input=x, start_dim=1, end_dim=-1)
print(z.size())
# torch.Size([4055, 320])
</code></pre>
Map method doesn't work with custom dataset

bkh6722: <p>I have created custom dataset with <code>__get__</code>  and <code>__len__</code> functions, but using <code>map</code> method on the instance of the datasets returns me an error, that dataset has no attribtute <code>map</code></p>
<p>Here is dataset:</p>
<pre><code class="lang-auto">class trainingDataset(Dataset):
  def __init__(self):
    super().__init__()
    self.trainingAudio = audioWorker()
    self.trainingAudio.set = "training"
  def __len__(self):
    return self.trainingAudio.vocabLen
    
  def __getitem__(self, idx):
    return self.trainingAudio.getAudio(idx)
</code></pre>
<p>and here is what I am trying to do:</p>
<pre><code class="lang-auto">def prepare_dataset(batch):
  audio = batch["audio"]
  # batched output is "un-batched"
  batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
  batch["input_length"] = len(batch["input_values"])
  
  with processor.as_target_processor():
      batch["labels"] = processor(batch["sentence"]).input_ids
  return batch

testDataset = testDataset.map(prepare_dataset)
</code></pre>
<p><code>AttributeError: 'testDataset' object has no attribute 'map'</code></p>
<p>It’s looking really strange, because it was working well before <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="bkh6722" data-post="1" data-topic="150066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bkh6722/48/48400_2.png" class="avatar"> bkh6722:</div>
<blockquote>
<p>It’s looking really strange, because it was working well before</p>
</blockquote>
</aside>
<p>That’s strange, as I don’t think <code>torch.utils.data.Dataset</code> ever provided a <code>map</code> method. Also the current implementation in the <code>master</code> branch doesn’t support it as seen <a href="https://github.com/pytorch/pytorch/blob/d9f0774f983007e32caed465ab83cc016deb3a9d/torch/utils/data/dataset.py#L34-L58">here</a>.<br>
Were you maybe using the new <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.map.Mapper.html"><code>torchdata.datapipes.map.Mapper</code></a> functionality?</p>
Pth to onnx model convertion

Vishak_Raj: <p>I am trying to convert the pth format to onnx, while converting, I face this issue,</p>
<blockquote>
<p>RuntimeError: Exporting the operator _convolution_mode to ONNX opset version 13 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.</p>
</blockquote>
<p>pytorch version is 1.11.0+cpu</p>
<pre><code class="lang-auto">    cnn = nn.Sequential()
    cnn.add_module('l1', nn.Conv2d(3, 32, 2, 1, "same"))
    cnn.add_module('l2', nn.ReLU())
    cnn.add_module('l3', nn.MaxPool2d(3, 3))

    ...

    cnn.add_module('fl', nn.LogSoftmax())

    torch.onnx.export(model, torch.zeros(16, 3, 224, 224).to(device), save_file, verbose=True, opset_version=13,
                export_params=True,
                training=torch.onnx.TrainingMode.EVAL,
                do_constant_folding=True,
                input_names = ['cnn.l1'], output_names=['cnn.fl'],
                          dynamic_axes={'cnn.l1': {0: 'batch'},
                                        'cnn.fl': {0: 'batch'}
                                        })
</code></pre>
<p>In this, I got this warnings,</p>
<blockquote>
<p>UserWarning: Using padding=‘same’ with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at …/aten/src/ATen/native/Convolution.cpp:744.)</p>
</blockquote>
<p>How to successfully convert the .pth to .onnx file, thanks</p>
How to display multiple images or scalars in a row on TensorboardX?

dachun: <p>For example,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c52a8391fa66db6a322f0e05ed0e9b7cd8b8cab4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c52a8391fa66db6a322f0e05ed0e9b7cd8b8cab4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c52a8391fa66db6a322f0e05ed0e9b7cd8b8cab4_2_690x211.jpeg" alt="image" data-base62-sha1="s8d98yIhsOnirIN9kG0lvce0o72" width="690" height="211" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c52a8391fa66db6a322f0e05ed0e9b7cd8b8cab4_2_690x211.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c52a8391fa66db6a322f0e05ed0e9b7cd8b8cab4_2_1035x316.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/5/c52a8391fa66db6a322f0e05ed0e9b7cd8b8cab4.jpeg 2x" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1314×403 76.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Dynamical automated architecture search

maria.solyanik: <p>I am trying to figure out how to add layers to a model dynamically. I have <a href="https://discuss.pytorch.org/t/simple-gradient-free-genetic-code-advise-needed/149879">this</a> custom GA that does optimization with respect to weights. I want to be able to optimize for topology as well. I have looked at <a href="https://discuss.pytorch.org/t/dynamically-add-or-delete-layers/10447">this</a> post. Second answer contains a way of deleting layers dynamically, but not adding.</p>
<p>I have also seen a couple of posts with <a href="https://stackoverflow.com/questions/62937388/pytorch-dynamic-amount-of-layers" rel="noopener nofollow ugc">this implementation</a>, but I do not fully understand it. I don’t see how these allows augmenting the NN architecture.</p>
<p>The closest I got is <a href="https://medium.com/analytics-vidhya/how-to-add-additional-layers-in-a-pre-trained-model-using-pytorch-5627002c75a5" rel="noopener nofollow ugc">this</a> blog post. I am working on it right now. If you have suggestions that worked well for you, please, share!</p>
<p>Thanks in advance!</p>
iOS metal backend expected true but false

sdb4832: <p>I successfully converted a model to use the metal backend and it shows in the opnames metal_prepack::conv2d run like in the example on the pytorch site, but it doesnt show metal::copy_to_host</p>
<p>I try using it an iOS app and definitely call .metal() on the tensors (it has 2 inputs) but get the following error</p>
<pre><code class="lang-auto">Expected input1.is_metal() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
  
  Debug info for handle(s): debug_handles:{-1}, was not found.
  
Exception raised from sub_Tensor at /Users/distiller/project/aten/src/ATen/native/metal/ops/MetalBinaryElementwise.mm:234 (most recent call first):

</code></pre>
<p>Could anyone point me to a reason this would happen? The cpu version of the model works with no issues</p>
Dataloader questions for torch geometric

Thibault712: <p>Hi everyone,</p>
<p>I have two questions about torch geometric:</p>
<p>When using the Dataloader (the one of torch geometric not pytorch).</p>
<p>_Is it possible to load the batches on the fly?</p>
<p>I have a big dataset of millions entries I can’t load a list of a millions of graphs at the same time, however I can manage to load them batch/batch</p>
<p>_ Is it possible that the loader automatically lowers the batch size if it can’t manage it?</p>
<p>My graphs are not homogeneous sizewise, sometimes it will easily handle a batch of 100 and then struggle with a batch of 20. That would be great if there was an option that prevent to load a batch with too much nodes/edges and instead just lower the batchsize exceptionally.</p>

ejguan: <aside class="quote no-group" data-username="Thibault712" data-post="1" data-topic="149835">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thibault712/48/36947_2.png" class="avatar"> Thibault712:</div>
<blockquote>
<p>_Is it possible to load the batches on the fly?</p>
</blockquote>
</aside>
<p>Yes, it is. If you use <code>Dataset</code> class, you can move your data-loading logic into <code>__getitem__</code> function rather than in <code>__init__</code> function.</p>
<aside class="quote no-group" data-username="Thibault712" data-post="1" data-topic="149835">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thibault712/48/36947_2.png" class="avatar"> Thibault712:</div>
<blockquote>
<p>_ Is it possible that the loader automatically lowers the batch size if it can’t manage it?</p>
</blockquote>
</aside>
<p>You should be able to achieve such logic by using custom <code>BatchSampler</code>. If you know the size of each data entry, you can use the list of sizes to aggregate a list of data indices for every single batch.</p>
Is it possible to use Pytorch without GPU support?

mmr: <p>Hi as the question suggest, is it possible to use Pytorch without GPU support. I was using Pytorch without GPU in Google Cloud, and it complained about no finding supporting CUDA library. In <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> website, there is an option to install Pytorch without CUDA support. I believe the command is :</p>
<blockquote>
<p>conda install pytorch torchvision -c soumith</p>
</blockquote>
<p>Is this a relevant command to run Pytorch solely using CPU.</p>
<p>Thanks</p>

sg314: <ul>
<li>Yes, that would be correct.</li>
<li>PyTorch can be used without GPU (solely on CPU).</li>
<li>And the above command installs a CPU-only compatible binary.</li>
</ul>

yesh: <p>Hi,</p>
<p>I tried  “conda install pytorch torchvision -c soumith”<br>
Bt am getting an error “ModuleNotFoundError: No module named ‘torchvision.ops’”<br>
Any solutions?</p>

ptrblck: <p>The <code>soumith</code> channel was used in old PyTorch releases, so you should stick to the <code>pytorch</code> or <code>pytorch-nightly</code> channels as given in the <a href="https://pytorch.org/get-started/locally/">install commands</a>.</p>

krishna511: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Sir I am trying to install pytorch in my desktop thats without GPU facility.<br>
I have installed anaconda and tried through Command prompt using</p>
<p>conda install pytorch torchvision torchaudio cpuonly -c pytorch</p>
<p>I think its installed as I am getting : All requested packages already installed.</p>
<p>but when I am using to import it through spyder its giving error:<br>
[WinError 126] The specified module could not be found. Error loading “C:\ProgramData\Anaconda3\lib\site-packages\torch\lib\asmjit.dll” or one of its dependencies.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/pytorch-compiled-from-source-for-windows-is-failing-when-importing-torch/75567">This thread</a> might be generally useful and in particular you could post there after running <a href="https://github.com/peterjc123/pytorch_dll_load_smoketests#how-to-contribute-or-report-issues">this</a>.</p>

A.K.Koolstuff: <p>!pip install pytorch torchvision for Jupyter users</p>
Non-tehnical question regarding PyTorch branding

duspic: <p>Greetings!</p>
<p>I’m part of a newly-opened AI educational center in Croatia. In our educational program we focus on PyTorch for various DL tasks, as we wish for our students to really learn the right tools for the job (I’m a PyTorch man myself…)</p>
<p>Our AI incubator is in its finishing stages, and I started to visualise the interior design and explored what I could to to make the place feel related to the theme of AI. Usually, I’d be against brand placement, but it really seems that all of us actually endorse certain brands when it comes to programming and AI.</p>
<p>Would it be considered legally troublesome if we were to print out a couple of PyTorch logos and used them in decorating? Is there a document containing graphic standards? Is there maybe some official merch to look at?</p>
<p>Or better yet - if there’s someone who deals with this kind of stuff, please contact me freely:<br>
ivan.duspara@lipik.ai<br>
I’ve tried to find a contact but to no avail</p>
<p>Thank you : )</p>

ptrblck: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a> who might know more about these licensing issues <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Load database from file

giannis_markou: <p>I have convert the SVHN dataset to MNIST so the file that I have now is one with extension “file” and one with extension “GZ” (gzip). So I try to load this data set but I cant figure out how I could do that.<br>
The only way I know to load data is with dataloader and with data that I have directly download from pytorch. You know, like dataset.MNIST(… … …, download=True) but in my case that cant help because i have already the dataset i need</p>

ptrblck: <p>Take a look at the <a href="https://github.com/pytorch/vision/blob/184b136f205df268f9bcfec8b6a89425d08fd1e8/torchvision/datasets/mnist.py#L19-L199"><code>datasets.MNIST</code></a> implementation to see how the binary data is loaded into this <code>Dataset</code> as it seems you could use it as a template assuming your data is “MNIST-like”.</p>

giannis_markou: <p>thank you for responding. I created a custom dataloader class and i loaded my dataset. now i have a runtime error when i try to test these data the error is this:<br>
Traceback (most recent call last):<br>
File “C:\Users\giann\AppData\Local\Programs\Python\Python310\python projects\occ ood ergasia pack(meta apo call)\pytorch-tutorial.py”, line 125, in<br>
images = images.reshape(-1, 28*28).to(device)<br>
RuntimeError: shape ‘[-1, 784]’ is invalid for input of size 100</p>

ptrblck: <p>It seems your data samples have 100 elements and thus cannot be reshaped to <code>[28, 28]</code> (MNIST image size) as this shape requires 784 elements.</p>
How to load custom batches from specific folder

Carson_Lansdowne: <p>Hello – first post here, so please let me know if there is anything I can provide to augment this question. I don’t think I am posting a duplicate because I have searched for an answer pretty exhaustively at this point.</p>
<p>I think what I am trying to do is pretty simple – I am trying to write a binary classifier for <strong>multiple</strong> time series data that are in the form of images (basically a bunch of “videos”). I am loading the dataset using ImageFolder but I want to batch every series of images together. These series are sorted into a specific folder but of course, ImageFolder simply loads everything as one “series”. <strong>How can I load the data such that I can specify which images are in a batch?</strong> I think the simplest way to do this is to have each series of arbitrary length in its own folder.</p>

tom: <p>Hello,</p>
<p>I would probably write a custom dataset.<br>
It’s not all that hard and it will turn out a lot cleaner than trying to ImageFolder to abide to your bidding.</p>
<p>(To my mind, this is the philosophy behind Dataset and DataLoader: Dataset is the problem-specific part and intended to be easy to implement, DataLoader is the generic part and does all sorts of weird tricks for you and intended to be used as is.)</p>
<p>If you allow yourself to feed in the categories (I usually code them into the dataset because I prefer error messages over silent failures should a class folder be missing), it should just be a matter of a few <code>sorted(glob.glob(...))</code> and opening the images with <code>torchvision.io</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

nivek: <p>We have a new library in beta phase, <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc"><code>torchdata</code></a>. That has some <a href="https://pytorch.org/data/beta/torchdata.datapipes.iter.html" rel="noopener nofollow ugc">built-in functionalities</a> to do some of the things that you are looking for (e.g. listing files in a directory, opening them, grouping them), if iterable-style dataset is acceptable for your use case. Otherwise, writing your own <code>Dataset</code> as Thomas suggested is a good idea as well.</p>
How to get the features from the bbox prediction

chandlerbing65nm: <p>In faster rcnn, RoI’s are in reshaped to <code>(batch, 256, 7, 7)</code> and then will undergo <code>2 shared convs</code> for bbox and cls prediction. This will result in shape (batch, 1024) before it outputs the bbox pred <code>(batch, 4</code>) and cls pred  <code>(batch, 2)</code>.</p>
<p>Is there a way to transform the features from <code>(batch, 256, 7, 7)</code> to something <code>(batch, w_pred, h_pred)</code>, so that I can get the feature values inside the bounding box?</p>
<p><code>w_pred</code> and <code>h_pred</code> are width and height of the predicted bounding boxes.</p>
<p>or is there an easier way to get the feature values from the box predictions?</p>
Torchscript call other function rather than forward

jiwidi: <p>Hi!</p>
<p>When I compile my torch model to torchscript I can make use of the function <code>forward</code> by just calling the torchscript model object <code>model()</code>.</p>
<p>But when I want to use another function created on the model I cant call the function. I try to do <code>model.functionName()</code> expecting to call the function <code>functionName</code> but nothing happens.</p>
<p>Any idea on how could I call subfunctions from the model object that are not forward?</p>
<p>Thanks</p>

davidberard98: <p><a href="https://pytorch.org/docs/stable/jit.html#torch.jit.export" rel="noopener nofollow ugc">torch.jit.export</a></p>
How to iterate a ParameterList in TorchScript?

avitase: <p>Hi,</p>
<p>Is it possible to iterate a <a href="https://pytorch.org/docs/stable/generated/torch.nn.ParameterList.html" rel="noopener nofollow ugc">ParameterList</a> in TorchScript? I was surprised to see that even the simple example from the PyTorch documentation does not work, i.e., this snippet</p>
<pre><code class="lang-auto"># from https://pytorch.org/docs/stable/generated/torch.nn.ParameterList.html
class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.params = nn.ParameterList(
            [nn.Parameter(torch.randn(10, 10)) for i in range(10)]
        )

    def forward(self, x):
        for i, p in enumerate(self.params):
            x = self.params[i // 2].mm(x) + p.mm(x)
        return x


if __name__ == "__main__":
    f = MyModule()
    f = torch.jit.script(f)  # BOOM
</code></pre>
<p>works fine if not JITed but fails otherwise:</p>
<pre><code class="lang-auto">[...]
Only constant Sequential, ModueList, or ModuleDict can be used as an iterable:
  File "[...]", line 14
    def forward(self, x):
        for i, p in enumerate(self.params):
                    ~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
            x = self.params[i // 2].mm(x) + p.mm(x)
        return x
</code></pre>
<p>Are there any workarounds? In particular, I have a ModuleList and a parameter list of the same length. During the evaluation I want to pass the n-th parameter to the n-th module:</p>
<pre><code class="lang-auto">    def __init__(self, n: int) -&gt; None:
        super().__init__()

        [...]
        self.my_modules = nn.ModuleList([...])  # n modules
        self.meta_parameters = nn.ParameterList([...]) # n nn.Parameters

    def forward(self, x):
        # for f, p in zip(self.my_modules, self.meta_parameters):
        for i, f in enumerate(self.my_modules):
            x = f(x, self.meta_parameters[i])

        return x
</code></pre>
<p>Thanks for your help!<br>
Nis</p>

davidberard98: <p>should be fixed in <a href="https://github.com/pytorch/pytorch/pull/76140" class="inline-onebox" rel="noopener nofollow ugc">[JIT] support parameterlist iteration by davidberard98 · Pull Request #76140 · pytorch/pytorch · GitHub</a></p>
Using jit to convert VQA model to iPhone (MMF)

PhillipHoejbjerg: <p>I’m trying to convert a state_dict Visual Question Answering “.pth”-model using jit.script.</p>
<p>The model has been made through facebooks MMF-framework, and takes as input a picture, a question as well as utilises a Knowledge Graph.</p>
<p>While scripting the model, I’ve encountered various problems, which I’ve continuously tried to fix. However, as this is a massive framework, utilising various scripts as well as native python scripts, the quest to script the model seems endless…</p>
<p>Now, I’ve arrived at a problem that I’m not sure is fixable through Jit, and I’m asking for help.</p>
<p>Jit.script has encountered a function, which takes another function as input. This function can be various ‘Forward’-functions (if anyone is familiar with MMF, if not, you don’t have to understand this part), which means I can’t just manually implement the input-function as part of the scripted function as I’ve had to do previously.</p>
<p>As Jit needs type-hinting, I’ve had to annotate the input as a “Callable[…, Tensor]” type. However, Jit doesn’t support this annotation-type. Is there any way I can work my way around this?</p>
<p>I’ve also had to resort to deleting various Args and Kwargs in native python files such as inspect.py, ruining my Python in the process, in the hopes of scripting this model. However, that seems like the wrong course of action.</p>
<p>I feel lost, is scripting this model a lost cause or do you know of any way I can bypass these problems?</p>

davidberard98: <p>One possibility is to use torch.jit.interface <a href="https://github.com/pytorch/pytorch/blob/master/test/jit/test_module_interface.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_module_interface.py at master · pytorch/pytorch · GitHub</a></p>
nn.MultiHeadAttention attn_output_weights become the same during evaluation only

Brian1: <p>I used nn.MultiHeadAttention like below and output attn_output_weights during both training and evaluation:<br>
class New_attention(nn.Module):<br>
def <strong>init</strong>(self,num_heads):<br>
super(New_attention, self).<strong>init</strong>()<br>
self.multi_attn = nn.MultiheadAttention(3000, 5,dropout = 0.3, batch_first = True)</p>
<pre><code>def forward(self, q, k, v):
    attn_output, attn_output_weights = self.multi_attn(q, k, v)
    print(attn_output_weights)
    return attn_output, attn_output_weights
</code></pre>
<p>The weights look good during training mode. However, when I perform evaluation with model.eval() and with torch.no_grad(), the value in tensor become the same:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0079fcc2eddcec8cf216f78e1ce21f164ce3fbca.png" data-download-href="https://discuss.pytorch.org/uploads/default/0079fcc2eddcec8cf216f78e1ce21f164ce3fbca" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0079fcc2eddcec8cf216f78e1ce21f164ce3fbca.png" alt="image" data-base62-sha1="4dm77nsF0PcpwfO3xaYYc1DR2y" width="690" height="412" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">724×433 54.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>size of q is (128,1,3000)<br>
size of k is (128, 4,3000)<br>
size of v is (128,4, 3000)</p>
<p>Does anybody know why?</p>

liyunhan1993: <p>I have the same problem as yours but not the same, did you solve it?</p>
<aside class="quote quote-modified" data-post="1" data-topic="122035">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vu_nguyen/48/38605_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/multiheadattention-after-lstm-returns-the-same-output-for-all-input/122035">MultiheadAttention after LSTM returns the same output for all input</a> 
  </div>
  <blockquote>
    Hi, 
I would like to use MultiheadAttention as self-attention after applying LSTM on a single sequence. 
import torch

# shape: (sequence length, batch size, embedding dimension)
inp = torch.randn(5, 3, 10)
lstm = torch.nn.LSTM(input_size=10, hidden_size=10, num_layers=2)
self_attn = torch.nn.MultiheadAttention(embed_dim=10, num_heads=2)

x, _ = lstm(input=inp)
# x is query, key, value at the same time
out, weight = self_attn(query=x, key=x, value=x)

out
tensor([[[ 0.0137,  0.0457, -0.0169, -0.…
  </blockquote>
</aside>
Installing PyTorch with Anaconda

Denis_Kazakov: <p>I already have Anaconda and now want to install PyTorch. The manual says, " To install PyTorch with Anaconda, you will need to open an Anaconda prompt via <code>Start | Anaconda3 | Anaconda Prompt</code> ."</p>
<p>" To install PyTorch via Anaconda, and do not have a <a href="https://developer.nvidia.com/cuda-zone" rel="noopener nofollow ugc">CUDA-capable</a> system or do not require CUDA, in the above selector, choose OS: Windows, Package: Conda and CUDA: None. Then, run the command that is presented to you."</p>
<p>But Anaconda Prompt is a command line interface. There is no selector there.</p>

Matias_Vasquez: <p>On the top of the page where you are reading this, there is this selector.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cc9a9a2581daf66fcecf0b42ee5c102442c469e.png" data-download-href="https://discuss.pytorch.org/uploads/default/3cc9a9a2581daf66fcecf0b42ee5c102442c469e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cc9a9a2581daf66fcecf0b42ee5c102442c469e.png" alt="image" data-base62-sha1="8FKHVbQMAWuOMfCkfnmRVKrloOq" width="690" height="259" data-dominant-color="E8D0CD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">808×304 18.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There you have to select your settings and you get the correct command that you have to input to the command line interface.</p>
<p>e.g.</p>
<pre data-code-wrap="python"><code class="lang-python">conda install pytorch torchvision torchaudio cpuonly -c pytorch
</code></pre>

Denis_Kazakov: <p>Thank you! I see now!</p>
CUDA out of memory error with to operation

SoHardToMakeAName: <p>Hi, I got an out of memory error with the code:</p>
<pre><code class="lang-auto">model.to(device).train()
</code></pre>
<p>I was running with PyTorch 1.8.0 on a Ubuntu 20.04 machine, and used 2 Nvidia GTX 2080ti GPU for training. The CUDA version is 11.0. However, this error occured before DataParallel. The full traceback message is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "mains_single.py", line 547, in &lt;module&gt;
    main()
  File "mains_single.py", line 143, in main
    train(args)
  File "mains_single.py", line 203, in train
    save_checkpoint(args, net, 1234, 1e6)
  File "mains_single.py", line 542, in save_checkpoint
    model.to(device).train()
  File "/data_c/lzq0330/miniconda3/envs/torch18/lib/python3.6/site-packages/torch/nn/modules/module.py", line 673, in to    return self._apply(convert)
  File "/data_c/lzq0330/miniconda3/envs/torch18/lib/python3.6/site-packages/torch/nn/modules/module.py", line 387, in _apply
    module._apply(fn)
  File "/data_c/lzq0330/miniconda3/envs/torch18/lib/python3.6/site-packages/torch/nn/modules/module.py", line 387, in _apply
    module._apply(fn)
  File "/data_c/lzq0330/miniconda3/envs/torch18/lib/python3.6/site-packages/torch/nn/modules/module.py", line 409, in _apply
    param_applied = fn(param)
  File "/data_c/lzq0330/miniconda3/envs/torch18/lib/python3.6/site-packages/torch/nn/modules/module.py", line 671, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: out of memory
</code></pre>
<p>Do anyone know a possible reason?  The context of the code is:</p>
<pre><code class="lang-auto">save_checkpoint(args, net, 1234, 1e6)
</code></pre>
<pre><code class="lang-auto">def save_checkpoint(args, model, epoch, eval_loss):
    device = torch.device("cuda" if args.cuda else "cpu")
    model.eval().cpu()
    checkpoint_model_dir = args.ckp_dir
    if not os.path.exists(checkpoint_model_dir):
        os.makedirs(checkpoint_model_dir)
    ckpt_model_filename = args.dataset_name + "_" + args.model_title + "_ckpt_epoch_" + str(epoch) + ".pth"
    ckpt_model_path = os.path.join(checkpoint_model_dir, ckpt_model_filename)
    state = {"epoch": epoch, "model": model, "eval_loss": eval_loss}
    torch.save(state, ckpt_model_path)
    model.to(device).train()
    print("Checkpoint saved to {}".format(ckpt_model_path))
</code></pre>
Gradient of batched vector output w.r.t batched vector input?

paganpasta: <p>For an input <code>x [32, 1, 28, 28]</code> the output of my network is <code>y [32, 10]</code></p>
<p>Is it possible to get the gradient of each output element w.r.t the input?</p>
<p>Following this <a href="https://discuss.pytorch.org/t/gradient-of-scalar-output-w-r-t-input-in-batches/129043">thread</a> I use <code>.backward(torch.ones_like(y))</code> but the <code>x.data.grad</code> shape is <code>[32, 1, 28, 28]</code>.</p>
<p>Does the eventual gradients get added up at the end? Is there a way to customize this behaviour?</p>
<p>What I am looking for is <code>x.data.grad</code> to be <code>[32, 10, 28, 28]</code></p>
<p>An alternative way is for me to repeat the input data and compute the gradient individually for each output element.</p>

KFrank: <p>Hi Pagan!</p>
<aside class="quote no-group quote-modified" data-username="paganpasta" data-post="1" data-topic="149518" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paganpasta/48/16226_2.png" class="avatar"> paganpasta:</div>
<blockquote>
<p>Is it possible to get the gradient of the input w.r.t the output?</p>
</blockquote>
</aside>
<p>Note, your title is “Gradient of vector output w.r.t batch” (in contrast<br>
to “input w.r.t. the output”).  I answer the question in your title.</p>
<p>If the output is a vector (rather than a single scalar), you are looking<br>
for the <em>Jacobian</em> of the function that maps input to output.</p>
<p><a href="https://pytorch.org/docs/1.11/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">torch.autograd.functional.jacobian()</a> should do what you want, and<br>
automates some of the work you could have chosen to do by hand.</p>
<p>Best.</p>
<p>K. Frank</p>

paganpasta: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a> for the response. I have fixed the mistake in the title.</p>

AlphaBetaGamma96: <p>To follow on from what KFrank said, <code>torch.autograd.functional.jacobian()</code> will return an output shape which is the concatenation of the output and input, so you’ll have a repeating batch dimension. You can remove the repeating batch dim via a <code>torch.einsum</code> call,</p>
<pre><code class="lang-auto">out = torch.randn(32,10,32,1,28,28)
out = torch.einsum("bobiwh-&gt;boiwh",out)
</code></pre>
<p>Also, surely the title should be gradient of output w.r.t input for all samples in batch? You can’t have a gradient of an input w.r.t the output? Unless I’m mistaken?</p>
Size of models from torch.hub

Andrei_Cristea: <p>Greetings! Is anyone aware of a resource that shows the size (number of parameters, or size on disk, or anything like that) of each of the models available in torch.hub? I know I can load them in one by one and use something like <a href="https://github.com/jacobkimmel/pytorch_modelsize" rel="noopener nofollow ugc">this tool</a> to estimate the size, but I was wondering if anyone had already done that and posted the results anywhere?</p>
<p>Thanks!<br>
Andrei</p>
DDP + Model Parallel : save checkpoint

PierreId: <p>Hello,</p>
<p>I am working with a network made of two models:</p>
<ul>
<li>Model1: Data Parallel model parallelized with DDP</li>
<li>Model2: Model Parallel model (huge weight matrix) parallelized manually with a sub-part on each DDP process/GPU</li>
</ul>
<p>Model1 can be easily saved from any process as it is identical on each GPU.<br>
But, Model2 is distributed/split across GPUs and must be synchonized somehow.</p>
<p>Question : what would be an elegant way to save Model2 ?</p>
<p>Remarks:</p>
<ul>
<li>gathering of Model2 must be done on CPU due to its size</li>
<li>distributed.gather() is not available with NCCL backend anyways</li>
<li>I could save each part on disk in each process, wait in rank0 process (distributed.barrier()), reload everything on CPU, merge and save in dict, but…</li>
</ul>
<p>Thanks !</p>

Imahn: <p>I’m not sure I understand the question. Can you maybe take a look <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints" rel="noopener nofollow ugc">here</a>? You don’t seem to need <code>distributed.gather()</code>, only <code>distributed.barrier()</code>?</p>
How to compute loss

shkiper325: <p>Hello everyone! How to compute this loss (I’m able to sample some x from P)?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7583c780c4452272ebd7f39d96d7c20911816a5.png" alt="изображение" data-base62-sha1="sru3PDyY3AsjeGNyHRttfOubfRb" width="272" height="27"></p>
Segmentation fault (core dumped) when runnning torch.load on Nivida Jetson

abcdcadb: <p>it is super simple, i want to load yolov5 model on Nvidia jetson nano:</p>
<pre><code class="lang-auto">device =torch.device("cpu")
model = torch.load('yolov5s.pt', map_location=device)
</code></pre>
<p>and got Segmentation fault (core dumped)<br>
environment: jetpack 4.61 (ubuntu 18, python 3.6) torch 1.9<br>
appreciate any opinion</p>
How to freeze a subset of weights of a layer?

Rodolphe_Lampe: <p>I’d like to remove a subset of connections from a Conv1d. I thought about setting the value of each connection to 0 and deactivating gradient’s computations.<br>
I’ve seen a lot of google/forum results about deactivating a parameter (using requires_grad=False) but I want to deactivate a subset of the parameter’s weights, not all of them.<br>
How can I do that ?</p>
<p>Quick example of the problem</p>
<pre><code class="lang-auto">c = torch.nn.Conv1d(3,3,5)
params = list(c.parameters())
param = params[0]
param
Out[16]: 
Parameter containing:
tensor([[[-0.1008,  0.1343,  0.1057, -0.0515, -0.0182],
         [-0.1379, -0.1431,  0.1962, -0.2398, -0.1028],
         [-0.2206,  0.0444, -0.1410, -0.2043, -0.1974]],
        [[ 0.2380,  0.1988, -0.2037, -0.0818,  0.0595],
         [ 0.1872, -0.1943,  0.1382,  0.1019, -0.2491],
         [-0.0744,  0.0346,  0.0224, -0.0474, -0.0807]],
        [[-0.1499, -0.2514, -0.2398,  0.1695,  0.1787],
         [-0.2560,  0.0071,  0.0072, -0.2318,  0.0216],
         [-0.1403,  0.0443, -0.0762,  0.0318, -0.1797]]], requires_grad=True)
param[0][0][0].requires_grad = False
param[0][0][0].requires_grad
Out[18]: True
</code></pre>

albanD: <p>Hi,</p>
<p>I’m afraid the require_grad is a property of the Tensor and you cannot set it for a subset of the Tensor.</p>
<p>The way I would recommend is just to save the values you want to keep and set them back after the optimizer step.<br>
Other tricks like setting the gradients to 0 can be dangerous as optimizers with weight decay for example will still update the weights when gradient is 0.</p>
<p>Another more complex version would be to have a custom layer that has only the learnable part as parameter. Then recreate the full weights based on the fixed values (in the forward!). Then call into the conv1d functional op with this weight.</p>

asura: <p>Say <code>a</code> is a tensor, and we want to partially update only certain values within the tensor.</p>
<p>Then, can it be done this way:</p>
<pre><code class="lang-auto">a = torch.rand(5)
b = torch.rand(5)
print(b)
a.requires_grad = True
c = a.detach()
c.requires_grad = False
d = torch.where(b &gt; 0.5, a, c)
y = d * 2
y.mean().backward()
print(a.grad)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">tensor([0.6207, 0.4534, 0.2759, 0.1856, 0.9388])
tensor([0.4000, 0.0000, 0.0000, 0.0000, 0.4000])
</code></pre>

jhongtao: <p>I have the same problem. How did you finally solve this problem?</p>
Build PyTorch with CUDA 11.6

deutschgraf: <p>Hi all.<br>
I tried to create PyTorch wheel with cuda 11.6, but faced OOM error.<br>
I was following instructions from here: <a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
After necessary dependencies was installed I tried to build pytorch with command:</p>
<pre><code class="lang-bash">python3 setup.py bdist_wheel
</code></pre>
<p>But soon after compilation started, I get all my PC memory filled and get system error.<br>
How can I solve this issue?</p>

ptrblck: <p>You could limit the number of processes via <code>MAX_JOBS=nb_jobs</code> to avoid running out of memory.</p>
Gradients negative though ReLU

mokemoke: <p>I’m trying to implement Gradient Activation mapping for segmentation problems. As a basic segmentation model, I use the U-Net (<a href="https://github.com/milesial/Pytorch-UNet/blob/master/unet/unet_model.py" rel="noopener nofollow ugc">Unet_github</a>).</p>
<p>To extract the activations i hooked one of the down layers (Conv-&gt;BN-&gt;ReLU)*2. In order to get the gradients of the summed outputs of the network wrt to the down layer i hooked:</p>
<p><code>torch.autograd.grad(outputs=sum(out[:,target_class, spatial_pos]), inputs=hooked_activations)[0].cpu().data.numpy()</code></p>
<p>I’m wondering why the gradients turned out to have negative values. I checked the hooked activations and they were &gt;=0.  As i understood, the gradients should be positive as well. I changed the ReLUs within the “Down” Blocks of the Unet  to inplace=False, as I red that inplace operations are dismissed by autograd.grad.</p>
<p>Im happy for any help!</p>

tom: <p>The backpropagation computes <code>grad_in = grad_out @ grad_relu</code> by the chain rule. As things are pointwise with relu and the derivative <code>grad_relu</code> is 0 for x &lt; 0 and 1 for x &gt; 0, you find that <code>grad_in</code> is a “masked” version of <code>grad_out</code>.</p>
<aside class="quote no-group" data-username="mokemoke" data-post="1" data-topic="149875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mokemoke/48/47574_2.png" class="avatar"> mokemoke:</div>
<blockquote>
<p>I changed the ReLUs within the “Down” Blocks of the Unet to inplace=False, as I red that inplace operations are dismissed by autograd.grad.</p>
</blockquote>
</aside>
<p>If you wanted to be more selective about what sources to believe on the internet, this might be a bit of information to consider dubious. There is some delicacy about autograd and inplace and I could spend hours talking about that and how avoiding inplace is a good default policy, but I do not believe that <code>autograd.grad</code> might be dismissing operations.</p>
<p>Best regards</p>
<p>Thomas</p>

mokemoke: <p>Hi Thomas,</p>
<p>thx for the clarification, was very helpful!</p>