akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
SignalInfo in new AudioMetaData

Rima_Seiilova-Olson: <p>Hi,<br>
We’re using <code>sox_io</code> backend for torchaudio. New version of the <code>AudioMetaData</code> class returned by the <code>info</code> method doesn’t have all the data I need for analysis.<br>
<code> info</code> method in the deprecated <code>sox</code> backend used to return <code>sox_signalinfo_t</code> and ‘sox_encodinginfo_t’ that had much more detailed information about encoding, bit depth etc.<br>
What should I use to get the info about audio file’s encoding, bit depth and other details?</p>

vincentqb: <p>Thanks for the feedback! Could you create an issue on <a href="https://github.com/pytorch/audio/issues/new/choose" rel="noopener nofollow ugc">torchaudio’s github</a>? More context about this BC-breaking change is also given <a href="https://github.com/pytorch/audio/issues/903" rel="noopener nofollow ugc">here</a>.</p>
Streamlit app with PyTorch/HuggingFace Transformers crashes when deployed to Heroku

Charly_Wargnier: <p>Hi everyone! <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"></p>
<p>My very first post - it’s great to be here! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’ve an issue with a Streamlit app leveraging Pytorch with the <em>distilbart-mnli-12-3 HuggingFace</em> model.</p>
<p>I can’t seem to deploy it correctly on Heroku. I’ve tried various configs but once deployed the app crashes all time. Here’s the log trace I got from Heroku: <a href="https://pastebin.com/uzEYVQy9" rel="noopener nofollow ugc">https://pastebin.com/uzEYVQy9</a></p>
<p>It looks like the app is crashing due to memory issues, the error being:</p>
<p><code>1. at=error code=H10 desc="App crashed" method=GET path="/healthz" host=still-basin-69776.herokuapp.com request_id=54a25884-8fba-4c36-ad08-441237858746 fwd="95.144.3.223" dyno= connect= service= status=503 bytes= protocol=https</code></p>
<p>If needs be, I’ve pasted the files related to the Heroku deployment including the python file and the requirements.txt in this repo:<br>
</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/CharlyWargnier/TransformersZeroShotHerokuFiles" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:400/400;"><img src="https://avatars1.githubusercontent.com/u/27242399?s=400&amp;amp;v=4" class="thumbnail" width="400" height="400"></div>

<h3><a href="https://github.com/CharlyWargnier/TransformersZeroShotHerokuFiles" target="_blank" rel="noopener nofollow ugc">CharlyWargnier/TransformersZeroShotHerokuFiles</a></h3>


  <p><span class="label1">Contribute to CharlyWargnier/TransformersZeroShotHerokuFiles development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Thanks in advance for your precious help! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>
<p>Charly</p>
Spatial transformer network's grid generator

hyuntae: <p>I implemented a code with the paper ‘Spatial transformer Network’ and tutorial code of pytorch ‘<a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Spatial Transformer Networks Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>’.</p>
<p>But the result I expected was different to the result of the paper.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae96c3fb95de32a7db05c8cca2eb7038de35929e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ae96c3fb95de32a7db05c8cca2eb7038de35929e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae96c3fb95de32a7db05c8cca2eb7038de35929e.jpeg" alt="image" data-base62-sha1="oUu5NHHibZ8hVR60rH6gSVYHnuS" width="690" height="370" data-dominant-color="7D7E76"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">699×375 79.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the result of my code.<br>
Right pictures are the result of spatial transformer network.<br>
But the paper said that the result of spatial transformer network is not the whole input picture but the part of the the input because of the grid generator.</p>
<p>Can you help me to solve this problem?<br>
The loss function is decreased well and the accuracy is increasing well. Both graphs are converged.</p>
<p>I think that the code of the tutorial ‘affine_grid’ which calculates the location for sampling is not working well…</p>
PIL-SIMD Backend in Pytorch Docker

gabrieldernbach: <p>PIL-SIMD is said to be much faster than basic PIL and a suitable default backend.</p>
<p>Maybe this is obvious, but why is PIL-SIMD not included in the official pytorch docker container?<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://hub.docker.com/r/pytorch/pytorch" target="_blank" rel="noopener nofollow ugc">hub.docker.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://hub.docker.com/r/pytorch/pytorch" target="_blank" rel="noopener nofollow ugc">Docker Hub</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Accimage vs. Pillow-SIMD

tushar: <p>Hi,</p>
<p>I see that torchvision supports 3 backends: Pillow, Pillow-SIMD and accimage. The benchmarks tell me that Pillow-SIMD is a lot faster than Pillow, but I can’t find any sources comparing accimage and Pillow-SIMD. I have both installed, which should I use?</p>
<p>Thanks!</p>

gabrieldernbach: <p>For installation of pillow-simd please follow their github installation section.<br>
<a href="https://github.com/uploadcare/pillow-simd" rel="noopener nofollow ugc">https://github.com/uploadcare/pillow-simd</a></p>
<p>At the current state it is nicely summarized by this gist<br>
</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/ciscorn/31c7e29e8a53a8b546cd69aef520450a" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/ciscorn/31c7e29e8a53a8b546cd69aef520450a" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/ciscorn/31c7e29e8a53a8b546cd69aef520450a</a></h4>
<h5>install_pillow_simd.sh</h5>
<pre><code class="Shell">#!/bin/env bash

pip3 uninstall pillow

apt install \
    libjpeg-turbo8-dev \
    zlib1g-dev \
    libtiff5-dev \
    liblcms2-dev \
    libfreetype6-dev \</code></pre>
This file has been truncated. <a href="https://gist.github.com/ciscorn/31c7e29e8a53a8b546cd69aef520450a" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Note that Pillow-SIMD is a drop in replacement and you will have to remove Pillow before.</p>
<p>After installation there should be a new suffix to<br>
<code>$ python -c "from PIL import Image; print(Image.__version__)"</code></p>
Program block when dataloader num_worker > 0

kehuantiantang: <p>I use the newest version of <a href="https://github.com/ultralytics/yolov5" rel="noopener nofollow ugc">yoloV5</a> to training the coco image, the program successful train when num_worker = 0, if the num_worker = 0, the  program will block and spend a lot of time to acquire data. Also, if i only train in one gpu, there has no problem if num_worker &gt; 0.</p>
<p>I train in one machine with four titan xp GPUS and the pytorch == 1.7.1, cuda == 10.1, python == 3.8.5, anaconda virtual environment.</p>
<p>I have the same problem not only in yoloV5 project, sometimes it also happened in other project. The block position shows in figure blow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef5659af1a32c28a27b4f9d489ca9aec728fdacb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ef5659af1a32c28a27b4f9d489ca9aec728fdacb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef5659af1a32c28a27b4f9d489ca9aec728fdacb_2_690x252.jpeg" alt="image" data-base62-sha1="y9h7rTYK4Pxya8u2Mycf1GLeOgz" width="690" height="252" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/ef5659af1a32c28a27b4f9d489ca9aec728fdacb_2_690x252.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef5659af1a32c28a27b4f9d489ca9aec728fdacb.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef5659af1a32c28a27b4f9d489ca9aec728fdacb.jpeg 2x" data-dominant-color="454445"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">909×333 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Dose anyone have the same problem ? Thanks a lot.</p>

rvarm1: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> re: DataLoader</p>
Where is the documentation for `.features` parameter of a model?

Rafay_Khan: <p>Hi, I’m looking at the tutorial <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener"> TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL</a>.</p>
<p>In the 2nd part of the tutorial, the author changes the backbone architecture of the pre-trained model by doing the following:</p>
<pre><code class="lang-auto"># load a pre-trained model for classification and return
# only the features
backbone = torchvision.models.mobilenet_v2(pretrained=True).features
...
...
</code></pre>
<p>What is <code>.features</code> actually doing? I can’t find its documentation.<br>
Does it just return the final layer before a softmax?</p>

yash1994: <p>Yes, you are right. <code>.features</code> holds output of model without last classification layer or last fully connected layer. it is not a norm.</p>

Rafay_Khan: <p>Hey Yash, thanks for replying.</p>
<p>I’ve been looking this up, according to this <a href="https://discuss.pytorch.org/t/extracting-and-using-features-from-a-pretrained-model/20723/4">post on the forum</a> one can also slice the features layer-wise. Any idea about that?<br>
Again docs would be really helpful</p>

yash1994: <p>Yes, according to the architecture of <a href="https://pytorch.org/docs/stable/_modules/torchvision/models/mobilenet.html#mobilenet_v2" rel="nofollow noopener">MobileNet V2</a> you can slice off feature layer-wise because <code>model.features</code> is defined as <code>self.features = nn.Sequential(*features)</code> and <code>nn.Sequential</code> behaves like python list of all layers (so can be sliced). But for other models, you need to check its implementation. You check the model implementation <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">here</a>, by finding the right model and clicking on <code>[SOURCE]</code> .</p>

Rafay_Khan: <p>Thanks a lot, Yash, I guess I’ll have to check the source for every model I want to use.</p>

melike: <p>I could not find the documentation of <code>features</code> parameter either. Does anyone know where it is ?</p>
Why I use shuffle = False, the order of data are changed?

William_yang: <p>I used the dataloader for the test_data. Below is the dataset:</p>
<pre><code class="lang-auto">class Test_Data1(Dataset):
    
    # Constructor
    def __init__(self):
        
        test_data = numpy.load('x_test.npy')
        test_labels = numpy.load('y_test.npy')
        
        test_data = test_data
        self.x = test_data
        self.y = test_labels
        self.len = self.x.shape[0]
        
    # Getter
    def __getitem__(self, index):    
        return self.x[index], self.y[index]
    
    # Get length
    def __len__(self):
        return self.len
</code></pre>
<p>This is my test code:</p>
<pre><code class="lang-auto">test_dataset1 = Test_Data1()
test_loader1 = DataLoader(dataset=test_dataset1, batch_size=360, shuffle=False)
</code></pre>
<pre><code class="lang-auto">correct = 0
total = 0
for data in test_loader1:
        inputs, labels = data
        print(labels)
        labels = labels.type(torch.LongTensor)
        inputs = inputs.view(-1, 1024).double()
        outputs =model1(inputs.float())
        _, predicted = torch.max(outputs.data, 1)
        print(predicted)
        total += labels.size(0)
        correct += (predicted == labels).sum()
print(100 * correct /float(total))
</code></pre>
<p>By comparison of the output, I found that the order was changed.<br>
My order<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05aa27ca7a8923731cef61dfa8fdafcda0c99388.png" alt="image" data-base62-sha1="O6WvwZMyEH8WskAdn6glcKde0o" width="524" height="351"><br>
The output order from dataloader:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/812eba339f3d7bf4256d8ec67e25d39b4e3b499e.png" alt="image" data-base62-sha1="iqNHO4mJK9xegx7CrfCswpeTkGq" width="550" height="441"></p>
<p>Looking forward to your reply.<br>
Best regards,<br>
William</p>

InnovArul: <p>I am unable to reproduce the behavior in torch 1.6.0.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/InnovArul/b3c0f7d0cc6cbe593d315beaab062fd9" target="_blank" rel="noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/b3c0f7d0cc6cbe593d315beaab062fd9" target="_blank" rel="noopener">https://gist.github.com/InnovArul/b3c0f7d0cc6cbe593d315beaab062fd9</a></h4>
<h5>dataloader_shuffle.py</h5>
<pre><code class="Python">from torch.utils.data import DataLoader, Dataset
import torch, torch.nn as nn
import numpy as np

class DS(Dataset):
    # Constructor
    def __init__(self):
        super().__init__()      
        X = list(np.arange(15000))
        self.x = X</code></pre>
This file has been truncated. <a href="https://gist.github.com/InnovArul/b3c0f7d0cc6cbe593d315beaab062fd9" target="_blank" rel="noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Pytorch LSTM in ONNX.js - Uncaught (in promise) Error: unrecognized input '' for node: LSTM_4

christian-doucette: <p>I am trying to run a Pytorch LSTM network in browser. But I am getting this error:</p>
<pre><code class="lang-auto">graph.ts:313 Uncaught (in promise) Error: unrecognized input '' for node: LSTM_4
    at t.buildGraph (graph.ts:313)
    at new t (graph.ts:139)
    at Object.from (graph.ts:77)
    at t.load (model.ts:25)
    at session.ts:85
    at t.event (instrument.ts:294)
    at e.initialize (session.ts:81)
    at e.&lt;anonymous&gt; (session.ts:63)
    at onnx.min.js:14
    at Object.next (onnx.min.js:14)
</code></pre>
<p>How can I resolve this? Here is my code for saving the model to onnx:</p>
<pre><code class="lang-auto">net = torch.load('trained_model/trained_model.pt')
net.eval()

with torch.no_grad():
    input = torch.tensor([[1,2,3,4,5,6,7,8,9]])
    h0, c0 = net.init_hidden(1)
    output, (hn, cn) = net.forward(input, (h0,c0))


    torch.onnx.export(net, (input, (h0, c0)), 'trained_model/trained_model.onnx',
                    input_names=['input', 'h0', 'c0'],
                    output_names=['output', 'hn', 'cn'],
                    dynamic_axes={'input': {0: 'sequence'}})
</code></pre>
<p>I put input as the only dynamic axis since it is the only one that can vary in size. With this code, the model saves properly as trained_model.onnx. It does give me a warning:</p>
<pre><code class="lang-auto">UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model. 
  warnings.warn("Exporting a model to ONNX with a batch_size other than 1, "
</code></pre>
<p>This warning a little confusing since I am exporting it with a batch_size of 1:</p>
<ul>
<li>input has shape torch.Size([1, 9]) - corresponding to (batch_size, num_in_sequence)</li>
<li>h0 has shape torch.Size([2, 1, 256]) - corresponding to  (num_lstm_layers, batch_size, hidden_dim)</li>
<li>c0 also has shape torch.Size([2, 1, 256]) - same as h0</li>
</ul>
<p>But since I do define h0/c0 as inputs of the model I don’t think this relates to the problem.</p>
<p>This is my javascript code for running in the browser:</p>
<pre><code class="lang-auto">&lt;script src="https://cdn.jsdelivr.net/npm/onnxjs/dist/onnx.min.js"&gt;&lt;/script&gt;
&lt;!-- Code that consume ONNX.js --&gt;
&lt;script&gt;
  // create a session
  const myOnnxSession = new onnx.InferenceSession();

  console.log('trying to load the model')

  // load the ONNX model file
  myOnnxSession.loadModel("./trained_model.onnx").then(() =&gt; {

    console.log('successfully loaded model!')

    // after this I generate input and run the model
    // since my code fails before this it isn't relevant
  });
&lt;/script&gt;
</code></pre>
<p>Based on the console.log statements, it is failing to load to the model. How should I resolve this? If relevant I’m using Python 3.8.5, Pytorch 1.6.0, ONNX 1.8.0.</p>
How can I sum up the inverse covariance matrix in pytorch?

John_Price: <p>How can I sum up the inverse covariance matrix and the mean over a Number of samples like in the following formula in pytorch:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/355bd72283e3aa823e54d0789033dfc9f3881a0d.png" alt="math" data-base62-sha1="7C24wGGV5r5HbmQznKGqWM9KyRf" width="88" height="48"></p>
<p>Thanks in advance.</p>

John_Price: <p>is <code>torch.inverse(covariance)</code> the right way for inverting the covariance matrix in this case?</p>
AttributeError: module 'torch' has no attribute 'logical_and'

Naveen: <p>I am getting above error my torch error, my torch version are  torch 1.4.0+cpu<br>
torchvision 0.5.0+cpu.</p>
<p>any help?<br>
Thanks</p>

ptrblck: <p>This method was introduced in <code>1.5.0</code> so you would need to update PyTorch to the latest stable version.</p>

Naveen: <p>Thank you that helped it.</p>

gebbissimo: <p>Quick solution for an OR operation for earlier versions should also be</p>
<p>mask_or = (mask_x + mask_y) &gt; 0</p>
Ask for a model about torch ELMO

fan_zhimin: <p>Whether torch has open source Chinese NER code, the pre training model is based on Elmo model.</p>
<p>I want to sort out the whole process of pre training model and start with emlo, but I can’t find a good GitHub project about NER. Who can help me.</p>
<p>If you have a good reference, please post it here, thank you very much</p>
Implemented Focal Loss for Semantic Segmentation (Question)

Vival: <pre><code class="lang-auto">class FocalLoss(nn.modules.loss._WeightedLoss):

    def __init__(self, weight=None, gamma=2,reduction=None):
        super(FocalLoss, self).__init__(weight,reduction=reduction)
        self.gamma = gamma
        self.axis = 1
        self.weight = weight #weight parameter will act as the alpha parameter to balance class weights
    def forward(self, input, target):
        input = input.transpose(self.axis,-1).contiguous()
        target = target.transpose(self.axis,-1).contiguous()
        input = input.view(-1,input.shape[-1])

        ce_loss = nn.CrossEntropyLoss(ignore_index=255) 
        loss=ce_loss(input, target.view(-1)
        pt = torch.exp(-loss)

        focal_loss = ((1 - pt) ** self.gamma * loss).mean()
        return focal_loss
</code></pre>
<p>Input shape : B, C(21), H, W Target shape: B, C(1), H, W<br>
I 've implemented this to try on u-net with voc-2012 semantic segmentation dataset.<br>
Yes it works, but somehow this doesn’t feels right.</p>
<p>It seems like it’s just getting the  exponential of  the negative cross entropy loss and doing the equation of focal loss afterwards. This doesn’t makes sense because  VOC segmentation images has more than one classes per image, and if so then the weighting part for loss is not going to happen outside the cross-entropy. Then  how is the equation:<br>
<code>focal_loss = ((1 - pt) ** self.gamma * loss).mean()</code>  supposed to do his part? I am so confused please help me <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Edit: I think <code>pt</code> variable supposed be the prediction probability, but implementations i came across on the internet did like this. How can i make the <code>pt</code> variable a probability distribution from the output of cross entropy function? Or should i use <code>input=LogSoftmax(input, dim = 1)</code> kinda code at the beginning of forward?</p>
How to get input shape of model?

Peter1: <p>Hello,</p>
<p>I load a model in torchvision module.</p>
<p>but how to get the model’s input shape?</p>
<p>Thank you so much for your help.</p>

a_d: <p>If you are loading a standard model lets say VGG, you can very easily find the shape of the input tensor from a google search.<br>
Most vision models have an input shape of 3x224x224(AFAIK)<br>
But do check once…</p>

Peter1: <p>I want to know ResNet3D-18.</p>
<p>torchvision.models.video.r3d_18</p>

lucianr: <p>I am having the same problem coming from TF/Keras: for pytorch conv models i can`t tell what the input size should be when i use one implementation or another.</p>
<p>Relying on google searches seems awkward since the input tensor size should be a property of the network/architecture. I am worried might have missed something about using with torch.</p>
<p>Isn`t there any built in methods to query the input size or at least to help deduce it?</p>
<p>I even had a problem with the qubvel implementation of Unet w/ effnet encoder where the effnet-b2 encoder got shape mismatches for 260x260 imgs (the official size) but worked for 256.</p>

Alexey_Demyanchuk: <p>Most of torchvision convolutional networks could work with different image sizes, except for perhaps this:</p>
<blockquote>
<p><strong>Important</strong>: In contrast to the other models the inception_v3 expects tensors with a size of N x 3 x 299 x 299, so ensure your images are sized accordingly.</p>
</blockquote>
<p>This note is from <code>torchvision.models</code> <a href="https://pytorch.org/docs/stable/torchvision/models.html#inception-v3" rel="noopener nofollow ugc">documentation</a>. So, to be save I would at least check for documentation.</p>
<p>On the other hand not all sizes a meaningful for the networks of different depths, because they could convolve images to much or to less thus getting not so optimal receptive fields.</p>
<p>Given that, I can defenetly recommend this github <a href="https://github.com/rwightman/pytorch-image-models" rel="noopener nofollow ugc">repo</a>. There is a bunch of pretrained models and you can check the image size used for training models in the results spreadsheets <a href="https://github.com/rwightman/pytorch-image-models/blob/master/results/README.md" rel="noopener nofollow ugc">here</a> .<br>
Hope it helps!</p>
Weight Sharing between Custom Convs

L-Reichardt: <p>Hello,</p>
<p>For a Project I am using the Sparsity-Invariant Convolution which is implemented in a class. The dilation rate is an input into this class, so would have to be set when initialized.<br>
Within a Model-Block, I want to use this convolution 3 times in parallel, with differing dilation rates. Is it possible to lock the weights between these 3 convolutions so they share the same weights.</p>
<p>This weight sharing would be like DetectoRS <a href="https://arxiv.org/abs/2006.02334" rel="noopener nofollow ugc">https://arxiv.org/abs/2006.02334</a></p>
<p>Thank you in advance for your help.</p>

ptrblck: <p>Yes, weight sharing is possible and a simple approach would be to use the functional API by defining the filters and bias as <code>nn.Parameters</code> and apply them via <code>F.conv2d</code> using the different dilation rates.</p>

L-Reichardt: <p>Thank you for your answer. Would this be a correct implementation of shared weights with a common gradient?</p>
<p>Convolution In init:</p>
<pre><code class="lang-auto">self.conv = nn.Conv2d(in_channels, out_channels, kernel_size,
                              stride=stride, padding=padding,
                              groups=groups, dilation=dilation, bias=False)
</code></pre>
<p>Share convolution in forward:</p>
<pre><code class="lang-auto">shared_weight = nn.Parameter((self.conv.weight))
shared_bias = self.conv.bias
x_d = F.conv2d(x_d, shared_weight, bias=shared_bias, stride = self.conv.stride, 
                          padding=self.switch_dilation, dilation=self.switch_dilation)
</code></pre>

ptrblck: <p>Don’t re-wrap the parameters in a new <code>nn.Parameter</code>, but use them directly via <code>F.conv2d(x_d, self.conv.weight, bias=self.conv.bias, ...)</code>.</p>

L-Reichardt: <p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Strange: calling model.eval() and forward introduced new parameters to the state_dict

rahmanoladi1: <p>Hi all, please I am currently facing the following problem while running main.py from the AWD LSTM repo at <a href="https://github.com/salesforce/awd-lstm-lm/blob/master/main.py" rel="noopener nofollow ugc">https://github.com/salesforce/awd-lstm-lm/blob/master/main.py</a>. The authors use the less-preferred paradigm of saving and loading the entire trained model. So, I tried to modify their code to use the better approach of saving and loading the state_dict of the model. However, I landed an error in doing so. Specifically, when the model is initialiazed, its state_dict does not contain rnns.0.linear.module.weight. But after doing model.eval() followed by output, hidden = model(data, hidden), I found that rnns.0.linear.module.weight suddenly enters into the state_dict ! I do not know why this happens. I have already done a detailed tracing of this problem, printing the state_dict at multiple critical points in the script, and I am quite confident that the aforementioned point is where the problem occurs. The real problem is that this issue is prone to create problems during model deployment, because at that time, one would want to follow a two-step approach: First step is to define/initialize the model, and this gives a model without rnns.0.linear.module.weight in the state_dict. Second step is to load the state_dict of a pre-saved model unto the new model. But, now the pre-saved model’s state_dict would contain rnns.0.linear.module.weight and so, one lands an unexpected key error. Please kindly advice me on how to resolve this issue.</p>

ptrblck: <p>I’m not completely sure, but by skimming through the repository, it seems that some attributes are manipulated in <a href="https://github.com/salesforce/awd-lstm-lm/blob/32fcb42562aeb5c7e6c9dec3f2a3baaaf68a5cb5/weight_drop.py#L32-L43">this forward call</a>, which might add these additional parameters.<br>
If that’s the case, I think the valid approach would be to use a “fake forward pass” to initialize these parameters and load the <code>state_dict</code> afterwards.</p>
<p>Also, I agree that your workflow of creating the model beforehand and load the <code>state_dict</code> afterwards is the preferred way.</p>

rahmanoladi1: <p>Thanks Ptrblck, I am glad that what I had in mind of a dummy forward pass agrees with your answer, but the problem is, during deployment, this might double inference time.</p>
How to create a is_valid_file function for ImageFolder

Nathan_Barry: <p>I created my first CNN using Microsoft’s Dogs vs Cats dataset. Both times when I’ve ran it, the training stops because of a corrupt image. I read the ImageFolder docs and see that is_valid_file is a function, but I have no idea how to create a function that would be able to do that. Can someone give me code that can do that? Thanks.</p>

ptrblck: <p>Once you’ve created the <code>Dataset</code>, you could iterate it and use the index (and thus file path) to either fix the file (re-download, recreate) or remove it from the folder.</p>
<p>Also, what error message are you getting?</p>

M_Djamaluddin: <p>I have question why in torchvision 0.2.1 when I supply is_valid_file function there is error : “<strong>init</strong>() got an unexpected keyword argument ‘is_valid_file’”? I can see from doc the argument exist.</p>

ptrblck: <p><code>torchvision 0.2.1</code> was released on April 24, 2018 based on <a href="https://github.com/pytorch/vision/releases/tag/v0.2.1">the release notes</a>, while <code>is_valid_file</code> was added on April 25 2019 in <a href="https://github.com/pytorch/vision/pull/867">this PR</a> (so a year later).</p>
<p>Where did you find the docs which mention it in <code>0.2.1</code>?</p>

M_Djamaluddin: <p>My bad. I have 0.2.1 in my system all this time, and I thought it already includes that missing argument. I just reinstall pytorch and torchvision, and there is no problem anymore.<br>
Thanks for your time.</p>

Pyroka: <p>What does that mean? Can you show a sample code?</p>
<p>If i iterate my dataset i already get the error (caused by a corrput jpg file in my image folder) before i can do something in the for loop.</p>

ptrblck: <aside class="quote no-group" data-username="Pyroka" data-post="6" data-topic="76111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pyroka/48/32391_2.png" class="avatar"> Pyroka:</div>
<blockquote>
<p>If i iterate my dataset i already get the error</p>
</blockquote>
</aside>
<p>That’s the idea to debug the issue further. Once you’ve isolated the index, where the <code>dataset</code> fails, check the corresponding file.</p>
<p>A code snippet would be:</p>
<pre data-code-wrap="python"><code class="lang-python">for idx in range(len(dataset)):
    try:
        batch = dataset[idx]
    except Exception as e:
        print(idx)
</code></pre>
Number of Bias Parameters

HD_99: <p>How many bias parameters will object model have?</p>
<p>class linear_regression(nn.Module):</p>
<pre><code>def __init__(self,input_size,output_size):

    super(linear_regression,self).__init__()

    self.linear=nn.Linear(input_size,output_size)

def forward(self,x):

    yhat=self.linear(x)

    return yhat
</code></pre>
<p>model=linear_regression(3,10)</p>
<p>whats the answer? i put the answer = 1</p>

ptrblck: <p>Also this questions sounds like it’s coming from an assignment and is similar to <a href="https://discuss.pytorch.org/t/linear-regression-help/106296">your other post</a>.<br>
I’m sure we can help with any PyTorch-related questions, but in case I’m right and you are indeed trying to get answers for some kind of quiz, I’m hesitant to help solving it for you.<br>
Having worked at a University before, I would be quite disappointed if our students would ask for direct support for these kind of questions.</p>
Linear Regression Help

HD_99: <p>which statements are correct about the cost function for linear regression?</p>
<ol>
<li>
<p>sum or average of loss</p>
</li>
<li>
<p>measure of how well your model can predict the data</p>
</li>
<li>
<p>function of the slope and bias</p>
</li>
</ol>

ptrblck: <p>Is this some kind of assignment or what is the background of this question? <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Cannot transfer data and model to gpu

josipovic: <p>I have a deep but simple model (bunch of convs, relu, bn) and data that i feed it. I’ve moved the weights with model.cuda() and the data with .to(device) but it fails and tells me that input is torch.cuda.FloatTensor and weights are torch.FloatTensor. Which is very wierd.</p>
<p>When I don’t move data with .to(device) then it says that input is torch.FloatTensor and weights are torch.cuda.FloatTensor.</p>
<p>If I place .to within dataset class getitem then I gett multiprocessing error for cuda. Does anyone know why mysteriously when I set both model.cuda() and data.to(device) it doesnt work?</p>
<pre><code class="lang-auto">    dataset = SomeDataset()
    dataloader = DataLoader(dataset, batch_size=10, num_workers=1)

    model = Model3000()
    model.cuda()
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model = model.to(device)
    print("Using device: ", device) # this displays cuda:0

    n_iters = 100
    for i in range(1, n_iters):
        for batch_ndx, sample in enumerate(tqdm(dataloader, desc=f"Epoch {i}: ")):
            data = sample[0]
            iz = model(data)
</code></pre>

ptrblck: <p>Could you post your model definition please?<br>
If you have written a custom model, you can make sure that all parameters and submodules are properly registered, by checking the <code>model.parameters()</code> and see if all parameters are returned.</p>

josipovic: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for reply!</p>
<p>Now that I have inspected the model I wonder if there is a problem how I initialize some layers Namely b1, b2, b3, b4,b5.</p>
<pre><code class="lang-auto">class JModel(nn.Module):
    def __init__(self, b=1, r=5):
        super(Jasper, self).__init__()

        self.conv1 = ConvProc(64, 256, 11, 0.2)
        self.b1 = [Block(inc, outc, 11, 0.2, r) for inc, outc in get_sizes(256, 256, b)]
        self.b2 = [Block(inc, outc, 13, 0.2, r) for inc, outc, in get_sizes(256, 384, b)]
        self.b3 = [Block(inc, outc, 17, 0.3, r) for inc, outc, in get_sizes(384, 512, b)]
        self.b4 = [Block(inc, outc, 21, 0.3, r) for inc, outc, in get_sizes(512, 640, b)]
        self.b5 = [Block(inc, outc, 25, 0.3, r) for inc, outc, in get_sizes(640, 768, b)]
        self.blocks = self.b1 + self.b2 + self.b3 + self.b4 + self.b5
        self.conv2 = ConvProc(768, 896, 29, 0.4, dilation=2)
        self.conv3 = ConvProc(896, 1024, 29, 0.4)
        self.conv4 = nn.Conv1d(1024, 28, 1)

    def forward(self, x):
        feats = self.conv1(x)
        for block in self.blocks:
            feats = block(feats)
        feats = self.conv2(feats)
        feats = self.conv3(feats)
        feats = self.conv4(feats)
        return feats
</code></pre>
<p>EDIT: now that I’ve looked it up it seems I need nn.ModuleList?<br>
EDIT2: It works with nn.ModuleList</p>
<p>Thanks for pointing in the right direction!</p>

rayryeng: <p>Thanks for the hint.  I needed to use <code>nn.ModuleList</code> as I define a list of layers in my class implementation which was not transferring to GPU (i.e. they were not listed in <code>model.parameters()</code>).</p>
How to add PyTorch as a requirement in setup.py?

ekdnam: <p>I am trying to build my python package which uses torch and torchvision. As they cannot be installed directly with say <code>pip install torch</code>, how to add them as a requirement in setup.py?</p>

Alexey_Demyanchuk: <p>Hey. Here is one of the solutions from <a href="https://github.com/PyTorchLightning/pytorch-lightning" rel="noopener nofollow ugc">GitHub</a>:</p>
<ol>
<li>They have <code>requierments.txt</code>
</li>
<li>They define helper function to read from <code>requierments.txt</code> into list of strings like this:</li>
</ol>
<pre><code class="lang-auto">def _load_requirements(path_dir: str , file_name: str = 'requirements.txt', comment_char: str = '#') -&gt; List[str]:
    """Load requirements from a file
    &gt;&gt;&gt; _load_requirements(PROJECT_ROOT)  # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
    ['numpy...', 'torch...', ...]
    """
    with open(os.path.join(path_dir, file_name), 'r') as file:
        lines = [ln.strip() for ln in file.readlines()]
    reqs = []
    for ln in lines:
        # filer all comments
        if comment_char in ln:
            ln = ln[:ln.index(comment_char)].strip()
        # skip directly installed dependencies
        if ln.startswith('http'):
            continue
        if ln:  # if requirement is not empty
            reqs.append(ln)
    return reqs
</code></pre>
<ol start="3">
<li>In the <code>setup.py</code> they use this helper function to provide argument for <code>setup</code> function:</li>
</ol>
<pre><code class="lang-auto">setup(
    ...
    install_requires=_load_requirements(PATH_ROOT),
    ....
)
</code></pre>
<p>Should work fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ekdnam: <p>Will try to implement this. Thanks!</p>
Some doubts about converting PyTorch model to Onnx 

hanyulon: <p>I am now using PyTorch to convert PyTorch model to Onnx model. When I use torch.onnx.export function, I find that this function’s second parameter needs model inputs (x    # model input (or a tuple for multiple inputs)).<br>
What value should I set for this parameter? Is this parameter a must and why this parameter is a must?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2978662db0ace328772db931823d6020c794488.png" data-download-href="https://discuss.pytorch.org/uploads/default/a2978662db0ace328772db931823d6020c794488" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2978662db0ace328772db931823d6020c794488_2_690x341.png" alt="image" data-base62-sha1="nclZr46PUDP8ZJZc7bBveasEyw0" width="690" height="341" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2978662db0ace328772db931823d6020c794488_2_690x341.png, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2978662db0ace328772db931823d6020c794488.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2978662db0ace328772db931823d6020c794488.png 2x" data-dominant-color="EEEEF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1034×512 48.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I think tracing is used under the hood so an example input is necessary to perform the trace and create the graph in order to export the ONNX model.</p>

hanyulon: <p>So this function torch.onnx.export tries to perform the trace in onnx model (super_resolution.onnx)? But I do not have to write down any input when using onnxruntime to save a model? It’s a torch to onnx necessary step?</p>

ptrblck: <p>I’m not deeply familiar with onnxruntime and don’t know how it’s working under the hood.<br>
I assumed it’s just a runtime/serving platform and provides engines to execute the models, but might be wrong.</p>
2017 and we still can't support AMD hardware, why?

yaxattax: <p>Not all of us own NV hardware.</p>
<p>Some of us want to use these tools at home without having to make a new purchase or purchase some AWS time.</p>
<p>Not all of us are research scientists with fat grants allowing us to buy the latest NV hardware.</p>
<p>When will the vendor lock-in stop?</p>
<p>Surely “free” tools should be open to all.</p>

ClementPinard: <p><a href="https://www.quora.com/What-is-the-underlying-reason-for-AMD-GPUs-being-so-bad-at-deep-learning" class="onebox" target="_blank" rel="nofollow noopener">https://www.quora.com/What-is-the-underlying-reason-for-AMD-GPUs-being-so-bad-at-deep-learning</a><br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.redditstatic.com/desktop2x/img/favicon/android-icon-192x192.png" class="site-icon" width="192" height="192">
      <a href="https://www.reddit.com/r/Amd/comments/58m1ec/amds_future_for_deep_learning/" target="_blank" rel="nofollow noopener">reddit</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://www.reddit.com/r/Amd/comments/58m1ec/amds_future_for_deep_learning/" target="_blank" rel="nofollow noopener">r/Amd - AMD's future for Deep Learning</a></h3>

<p>45 votes and 17 comments so far on Reddit</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Lots of work need to be done to reach Nvidia’s performance. Cudnn is a huge step for performance that AMD does not have yet.</p>
<p>There was OpenCL support for torch, but pragmatically everyone just turned to CUDA devices. If someone is willing to make a cudnn equivalent maybe it would change.</p>
<p>It’s also not only a software problem. But apparently AMD is planning on producing DL oriented hardware<br>
<a href="http://www.amd.com/en-us/press-releases/Pages/radeon-instinct-2016dec12.aspx" class="onebox" target="_blank" rel="nofollow noopener">http://www.amd.com/en-us/press-releases/Pages/radeon-instinct-2016dec12.aspx</a></p>

smth: <p>Please see this thread for more details. We are waiting for HIP to be released from AMD.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/488" class="onebox" target="_blank">https://github.com/pytorch/pytorch/issues/488</a></p>

Felix_Lessange: <p>Here is 2019, not much has changed. Reviving this topic anyways. Tensorflow benchmarks for amd gpus are pretty impressive (are they misleading ?). This suggests that above comments are no longer true (?). What do you guys think is the reason for which we still can’t use amd gpus in pytorch ? <a href="http://blog.gpueater.com/en/2018/04/23/00011_tech_cifar10_bench_on_tf13/" rel="nofollow noopener">http://blog.gpueater.com/en/2018/04/23/00011_tech_cifar10_bench_on_tf13/</a></p>

gvskalyan: <p><a href="https://rocm.github.io/pytorch.html" class="onebox" target="_blank" rel="nofollow noopener">https://rocm.github.io/pytorch.html</a></p>

Madegomez: <p>Did you have a good experience using ROCm?</p>
How do I create a layer in a model with separate weights?

Stanley_C: <p>How do I create a layer with separate weights? I’m trying to have code that can “interpolate” points in a 2d plane, but interpolated points of the outputs just “stack” on the same input points, and I thought separating the weights would help reduce the stacking that I observe.</p>

InnovArul: <aside class="quote no-group" data-username="Stanley_C" data-post="1" data-topic="106272">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stanley_c/48/16353_2.png" class="avatar"> Stanley_C:</div>
<blockquote>
<p>interpolated points of the outputs just “stack” on the same input points</p>
</blockquote>
</aside>
<p>Can you elaborate on what did you mean by this?<br>
It will be great if you can post some code examples on what have you tried.</p>

Stanley_C: <p>I’m currently attempting to reimplement at first, a 2d version of Pu-net, but I’ve given up on that and went straight to a “cut-down” Pointnet Version. From the PU-net paper:</p>
<blockquote>
<p>Hence, we<br>
further add another convolution (with separate weights) for<br>
each feature set. Since we train the network to learn the r<br>
different convolutions for the r feature sets, these new features can include more diverse information, thus reducing<br>
their correlations.</p>
</blockquote>
<p>In the implementation, they seemed to be not adding a relu activation functions, so I tried that, but that didn’t seem to have the desired effect.</p>

InnovArul: <p>In my understanding (from Fig.1 of the PU-Net paper), they are applying <code>r</code> different convolutional pipelines (each pipeline has 2 conv layers with its own weights) to get r different (non-linear) affine projections of point-features. i.e., <code>N x C</code> goes to <code>r * (N x C2)</code>. I do not understand their reason for adding C_2 layer. However, its adding more capacity (parameters) to the model.</p>
<p>Implementation could be:</p>
<pre><code class="lang-auto">
def __init__(...):
    self.expansion_set = nn.ModuleList()
    for i in range(r):
        self.expansion_set.append(nn.Sequential(nn.Conv1d(C, C1, kernel_size=1), 
                                           nn.ReLU(),
                                           nn.Conv1d(C1, C2, kernel_size=1)))

    ...

def forward(...):
    ...
    r_sets = []
    for i in range(r):
        r_set.append(self.expansion_set[i](features))

</code></pre>
Why is packing/unpacking sequence slower than unpacked sequence?

praateekmahajan: <p>As <a href="https://stackoverflow.com/questions/51030782/why-do-we-pack-the-sequences-in-pytorch/56211056#56211056" rel="noopener nofollow ugc">several</a> <a href="https://medium.com/@florijan.stamenkovic_99541/rnn-language-modelling-with-pytorch-packed-batching-and-tied-weights-9d8952db35a9" rel="noopener nofollow ugc">articles</a> suggest online packing a padded sequence before passing it to RNN can save compute as it doesn’t require extra computation of the timestamps that are padded.</p>
<p>On basis of that I’m assuming that this implies, that using pack_padded_sequence should be definitely faster than forward passing the whole sequence including the pads. However I’m not observing that on my actual dataset. I have created fake data to reproduce the problem.</p>
<p>Please let me know if I’m missing something here.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/praateekmahajan/bc2ace6477d04927038a7a5acaf30349" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/praateekmahajan/bc2ace6477d04927038a7a5acaf30349" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/praateekmahajan/bc2ace6477d04927038a7a5acaf30349</a></h4>
<h5>padding_slow.ipynb</h5>
<pre><code class="Jupyter Notebook">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from torch.utils.data import Dataset, DataLoader\n",</code></pre>
This file has been truncated. <a href="https://gist.github.com/praateekmahajan/bc2ace6477d04927038a7a5acaf30349" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>