akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What to do when two classes need different lr?

alx: <p>Hi guys,</p>
<p>I’ve been stuck on this problem for a few days now.</p>
<p>There’s 5 classes to predict. When my lr = 0.00001, the accuracy of class 0 is 99% and class 1 is 0%.</p>
<p>However if I remove class 0 and change the lr to 0.0001, class 1 goes up to 99% accuracy.</p>
<p>So I need a different lr for each class. I’ve never read about this problem, what should I do?</p>
<p>Is it possible to start with a high lr and once the accuracy of class 1 is above 98% stop considering it, decrease the lr to 0.000001 and only optimize for class 0?</p>
Gumbel softmax being removed?

whoab: <p>Why this?</p>
<blockquote>
<p>This function is here for legacy reasons, may be removed from nn.Functional in the future.</p>
</blockquote>
<p>What is the newest place to get it versus nn functional?</p>

zyser: <p>Same question here, why there is a note Gumbel Softmax may be removed?</p>

mruberry: <p>The PR that added the warning (<a href="https://github.com/pytorch/pytorch/pull/13339" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/13339</a>) has a little discussion about its origin. As best I can tell, that PR’s author thought the function might be implemented via a different package in the future. I don’t think you have to worry about gumbel_softmax going away.</p>

zyser: <p>Thanks, I will use it then.</p>
How to make youcompleteme work for libtorch

wholeyu: <p>I’m new to <code>libtorch</code>, and I want to use it on ubuntu16.04 platform with vim editor. I have added <code>/usr/local/libtorch/include/torch/csrc/api/include/</code> to the <code>flags</code> in <code>.ycm_extra_conf.py</code>, but the vim editor can only complete part of the content like types and cuda related. Important things like <code>torch::nn</code> can not be completed. I want to get the correct configure for youcompleteme plugin in vim editor. Wish your help!</p>
<p>The complete like follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f92357724a0c448573ec703c86584585bd41aed.png" data-download-href="https://discuss.pytorch.org/uploads/default/2f92357724a0c448573ec703c86584585bd41aed" title="Screenshot from 2020-02-16 14-48-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2f92357724a0c448573ec703c86584585bd41aed_2_688x500.png" alt="Screenshot from 2020-02-16 14-48-58" data-base62-sha1="6MPGUwYz8DQhVtTPmaSvzBlglOd" width="688" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2f92357724a0c448573ec703c86584585bd41aed_2_688x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f92357724a0c448573ec703c86584585bd41aed.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f92357724a0c448573ec703c86584585bd41aed.png 2x" data-dominant-color="0F1719"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-02-16 14-48-58</span><span class="informations">760×552 81.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
PyTorch Android Score

vortana_say: <p>I follow the android demo app from <a href="https://github.com/pytorch/android-demo-app" rel="nofollow noopener">https://github.com/pytorch/android-demo-app</a>, and I have questions below:</p>
<p>From each prediction from the pytorch android, we get back score of each result. I know that the number that close to 0 is the best predicted result.</p>
<p>My questions are</p>
<ol>
<li>How is the score calculated internally ?</li>
<li>Why the score is negative?</li>
<li>How to get percentage of the result based on the score?</li>
</ol>
<p>I also posted in github as well: <a href="https://github.com/pytorch/android-demo-app/issues/60" rel="nofollow noopener">https://github.com/pytorch/android-demo-app/issues/60</a></p>
<p>Thanks.</p>

ptrblck: <ol>
<li>
<p>Based on the demo code it looks like one demo is using a pretrained <code>torchvision.models.resnet18</code>.<br>
This model outputs logits for each class, such that the highest logit would correspond to the predicted class. You could apply a softmax, which is not necessary to get the prediction, but will give you probabilities, which might be easier to interpret.</p>
</li>
<li>
<p>If I’m right in point 1, you should see positive and negative numbers as the logit output.</p>
</li>
<li>
<p>You could add a softmax layer to your model.</p>
</li>
</ol>

vortana_say: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the explanation.</p>
<p>For 3, I already have a trained model and serialized it to be used in the mobile app, so by adding a softmax layer to my model, does it mean I have to re-train it since the original model doesn’t have a softmax layer. Are there anyway, that we can calculate percentage based on the result of the serialized model?</p>

ptrblck: <p>No, you don’t need to retrain the model if you just want to get the probabilities with a softmax layer.<br>
I think the easiest way could be to pass your current model and the <code>nn.Softmax</code> layer to a new custom model or an <code>nn.Sequential</code> container and use this new combined model in your app.</p>
<p>Alternatively, you could compute the softmax on the mobile, as the formula is quite straightforward.</p>

vortana_say: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks.</p>
<p>I added:</p>
<h1>Add softmax layer</h1>
<p>mobilev2_loaded_model = nn.Sequential(<br>
mobilev2_loaded_model,<br>
nn.Softmax(1)<br>
)</p>
<p>and serialized model again and now I can multiple 100 to the score and get the percentage of that class.</p>
In no_grad() blocks but requires_grad is True

yaneura-no-gomi: <p>Hi, I’m new to Pytorch and I’m sorry I can’t explain it because English is not native.</p>
<p>I am doing GAN implementation now and am confused by weight initialization.<br>
In the code I am referring to, initialize the weights as follows:</p>
<pre><code class="lang-auto">def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
        nn.init.constant_(m.bias.data, 0)
        print(nn.init.constant_(m.bias.data, 0).requires_grad)
        
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

G.apply(weights_init)
D.apply(weights_init)
</code></pre>
<p>results is here:</p>
<pre><code class="lang-auto">False
False
False
False
False
False
False
False
False
False
</code></pre>
<p>But according to <a href="https://discuss.pytorch.org/t/initializing-parameters-with-weight-or-weight-data/39881">this</a> using “with no_grad()” is recommended.So ,</p>
<pre><code class="lang-auto">def weights_init(m):
    classname = m.__class__.__name__
    
    with torch.no_grad():
        if classname.find('Conv') != -1:
            nn.init.normal_(m.weight, 0.0, 0.02)
            nn.init.constant_(m.bias, 0)
            print(nn.init.constant_(m.bias, 0).requires_grad)
            
        elif classname.find('BatchNorm') != -1:
            nn.init.normal_(m.weight, 1.0, 0.02)
            nn.init.constant_(m.bias, 0)
        
G.apply(weights_init)
D.apply(weights_init)
</code></pre>
<p>but this results are:</p>
<pre><code class="lang-auto">True
True
True
True
True
True
True
True
True
True
</code></pre>
<p>I am confused<br>
What are these differences? And how does this difference affect the calculation results?</p>
<p>Thanks.</p>

ptrblck: <p>Stick to the second approach, as the first one is deprecated and might yield unwanted side effects.<br>
In a <code>torch.no_grad()</code> block all new operations won’t be tracked. The <code>requires_grad</code> attribute of your parameters will not be changed:</p>
<pre><code class="lang-python">model = models.resnet50()

with torch.no_grad():
    print(model.fc.weight.requires_grad)    
    &gt; True
    out = model(torch.randn(1, 3, 224, 224))
    print(out.requires_grad)
    &gt; False

out = model(torch.randn(1, 3, 224, 224))
print(out.requires_grad)
&gt; True
</code></pre>
How does one create a neural network where it's parameters are **NOT** leafs?

pinocchio: <p>I want a neural network where the parameters are <strong>not</strong> leafs. Is that possible?</p>
<p>I’ve tried it but get some issues: <a href="https://discuss.pytorch.org/t/how-is-it-possible-that-i-cant-create-a-deep-copy-although-all-my-parameters-are-leafs/69864" class="inline-onebox">How is it possible that I can't create a deep copy although all my parameters are leafs?</a></p>
Spatiotemporal Classification by GRU or RNN

saeed_ghorbani: <p>Hi everyone,</p>
<p>I am trying to write a code for motion classification on motion capture data.<br>
The 1-D CNN works perfectly with more than 90% accuracy but when I am trying GRU it improves for 1 or 2 %  and then plateaus at around 25% for classifying among 8 classes. RNN works a little bit better but plateaus at 30%.</p>
<p>Theses are my RNN and CNN models:</p>
<pre><code class="lang-auto">class RNN_ClassifierModel(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.hidden_channels = 100
        self.num_layers = 1
        self.input_channels = 69

        self.gru = nn.GRU(
            self.input_channels,
            self.hidden_channels,
            self.num_layers,
        )
        self.linear = nn.Linear(
            self.hidden_channels,
            len(self.config.training_sel_motions),
        )

    def forward(self, xb):
        xb = xb.permute(2,0,1)
        _, hidden_cell = self.gru(xb)
        hidden_cell = hidden_cell.squeeze(0)
        out = self.linear(hidden_cell)
        return out


class CNN_ClassifierModel(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.input_channels = 69

        self.conv1 = nn.Conv1d(
            self.input_channels,
            self.input_channels * 2,
            kernel_size=4,
            stride=2,
            padding=1,
        )
        self.conv2 = nn.Conv1d(
            self.input_channels * 2,
            self.input_channels * 4,
            kernel_size=4,
            stride=2,
            padding=1,
        )
        self.conv3 = nn.Conv1d(
            self.input_channels * 4,
            self.input_channels * 8,
            kernel_size=4,
            stride=2,
            padding=1,
        )
        self.pool = nn.MaxPool1d(2, stride=2)
        self.linear = nn.Linear(
            self.input_channels * 8, len(self.config.training_sel_motions)
        )
        self.adaptive_pool = nn.AdaptiveAvgPool1d(1)

    def forward(self, xb):
        xb = self.pool(F.relu(self.conv1(xb)))
        xb = self.pool(F.relu(self.conv2(xb)))
        xb = self.pool(F.relu(self.conv3(xb)))

        xb = self.adaptive_pool(xb)
        xb = xb.squeeze(2)
        out = self.linear(xb)
        return out
</code></pre>
<p>Everything else is the same for both models. I am using nn.CrossEntropyLoss() for loss.</p>
<p>Thanks</p>
Pytorch transformer expected input

Aviv_Shamsian: <p>Hi All,<br>
Let us say I have an input with the following dimensions <code>[4,300,15,10]</code> which correspond to <code>[batch size, frames, objects, data for each object]</code>. I want to use the transformer encoder layer and perform a self-attention on the objects using a transformer. since the input should include 3 dimensions I have folded the frame’s axis to the batch axis.<br>
Digging in the transformer encoder layer I saw that the self-attention module expects to get the batch size as the second axis instead of the first, I tried to look if there is any flag that swaps the axises but couldn’t find one. In addition, when looking at the output weights of the self-attention module the shape of the weights tensor was <code>[1200,1200,15]</code> which looks like the attention was performed on the time axis (the first axis of the batch size * number of frames).</p>
<p>What should I do to perform the attention on the subjects? Do I need to explicitly swap between the first and second axises using <code>transpose</code> or it is done automatically inside the transformer encoder layer?</p>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Aviv_Shamsian: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> maybe you can help me with it?</p>
How train my dataset(shoes) in background code?

hamzaygrtc: <p>I want to remove background but I couldn’t keep shoes like foreground. I thought that is needed train own shoes dataset. I am using following code.</p>
<p><a href="https://github.com/spmallick/learnopencv/blob/master/app-seperation-semseg/Background-Removal.py" rel="nofollow noopener">Code</a></p>
<p>(I couldn’t get good result with this models)<br>
dlab = models.segmentation.deeplabv3_resnet101(pretrained=1).eval()<br>
fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()</p>
<p>How to train my own dataset that is shoes for getting foreground? Could you tell me step by step what I will do ?</p>

Shisho_Sama: <p>Do you want to do segementation and only segment shoes for example?</p>
<p>In that case, you need to have masks for your data. simply use applications such as <a href="https://github.com/wkentaro/labelme" rel="nofollow noopener">labelme</a> (There are bunch of other apps and services you can use aside from labelme, check them out <a href="https://github.com/jsbroks/awesome-dataset-tools" rel="nofollow noopener">here(Awesom dataset tools)</a>) and create yourself a dataset, then train your network on it and you are good to go!</p>

hamzaygrtc: <p>I labeled dataset with <a href="https://github.com/wkentaro/labelme" rel="nofollow noopener">labelme</a>. How could I train the labeled dataset according following code?<br>
<a href="https://github.com/spmallick/learnopencv/blob/master/app-seperation-semseg/Background-Removal.py" rel="nofollow noopener">RemoveBackgroundCode</a></p>
[solved]Error with copy.deepcopy()

zzuczy: <p>I want to return model after train, so I copy it first:</p>
<blockquote>
<p>best_model = copy.deepcopy(model)</p>
</blockquote>
<p>but I got error:</p>
<blockquote>
<p>RuntimeError: Only Variables created explicitly by the user (graph leaves) support the deepcopy protocol at the moment</p>
</blockquote>
<p>I want to know why I got this error, and how can I solve it.</p>

JunyuMao: <p>I got the same error,you can try it :  best_model = copy.deepcopy(model.state_dict())<br>
maybe help.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pinocchio: <p>Can you be more specific how the answer given solves your problem? Because it certainly does not solve any of my problems. Don’t you still need to load those parameters to an actual model later on? How did you do that without <code>deepcopy</code>?</p>
<p>Also, can you provide a self contained part of your code that is giving you this issue? it’s impossible to know what you’re doing with so little code provided.</p>
<p>Also, you don’t need to manually harcode <code>[solved]</code> in the title, there is a button for selecting the answer that solves your question man.</p>
Why do I have to load the optimizer state dict for pytorch in order to make a good prediction, which is not for training?

JonathanSum: <p>Why do I have to load the optimizer state dict for pytorch in order to make a good prediction, which is not for training?</p>

ptrblck: <p>Could you describe the issue a bit more or post a dummy code snippet, which explains the issue?<br>
Based on your sentence it seems that something like this:</p>
<pre><code class="lang-python"># 1.
model = MyModel()
model.load_state_dict(torch.load(...))
# 2.
optimizer = torch.optim.SGD(model.parameters())
optimizer.load_state_dict(torch.load(...))
# 3. 
model.eval()
output = model(input)
</code></pre>
<p>works better when <span class="hashtag">#2</span> is skipped?</p>

JonathanSum: <p>Sorry, I should reply it at here: <a href="https://discuss.pytorch.org/t/did-any-one-know-how-to-load-the-pth-pre-trained-model-from-fastai-to-pytorch/69298/11" class="inline-onebox">Did any one know how to load the pth pre-trained model from fastai to pytorch?</a></p>
<p>On that post, Jemery said the reason why learn.model(input_image) does not work is because the opt(optimizer) problem, even though I set requires_grad to false and learn.model.eval(). I was thinking we don’t use the optimizer for training, and I was just want to do prediction. Why do I need the optimizer?</p>
<p>Does skipping <span class="hashtag">#2</span> works better normally?</p>
Lazy Evaluation Behavior

Brad_Goodman: <p>Hello !</p>
<p>I have a quick question on lazy evaluation of long running operations and if they block or not (separate answer for *.device(‘cpu’) and .device(‘cuda’), if necessary"</p>
<pre><code class="lang-auto">def MyFunc(inputTensor):
    x = SomeLongRunningFunction(inputTensor);
    y = AnotherLongRunningFunction(inputTensor);
    return x,y;
</code></pre>
<p>Does the computation of x block the computation of y, even though they are independent of each other?    If yes, is there a simple way to make the computations occur simultaneously?</p>
<p>Similar questions for loops.  Is the loop unrolled so the independent indexed computations can occur simultaneously?</p>
<pre><code class="lang-auto">def MyLoopFunc(inputTensor,N,M,device):
    x = torch.empty((N,M),device=device);
    for idx in range(N):
        x[idx,:] = LongRunningOperation(inputTensor,idx);
    return x;
</code></pre>

Perry_Deng: <p>Looks like they are lazily evaluated: <a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution</a></p>
In the tutorial,how to set the number of training and test sets?

SangYC: <p>In the tutorial,CIFAR10 dataset,Why training data has 50000，testing data has 10000.We did not set the amount of training data and test data.</p>
<p>This is code:</p>
<p>transform = transforms.Compose(<br>
[transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])</p>
<p>trainset = torchvision.datasets.CIFAR10(root=‘F:\OfficialData’, train=True,<br>
download=True, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,<br>
shuffle=True, num_workers=0)</p>
<p>testset = torchvision.datasets.CIFAR10(root=‘F:\OfficialData’, train=False,<br>
download=True, transform=transform)<br>
testloader = torch.utils.data.DataLoader(testset, batch_size=4,<br>
shuffle=False, num_workers=0)</p>
<p>classes = (‘plane’, ‘car’, ‘bird’, ‘cat’,<br>
‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’)</p>

Shisho_Sama: <p>Why didnt we specify their size? because they are the defaults. if you want you can alter it in any way and shape you like.<br>
checkout <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html</a> for more information.<br>
(quick tip, you may want to use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset" rel="nofollow noopener">torch.utils.data.Subset</a> to form different sets as you like.</p>
Proposal: deterministic random_split

artemisart: <p>Hello, I’m trying to contribute to the pytorch package to make deterministic splits easier with this function:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/32467" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/32467" target="_blank" rel="nofollow noopener">feature: deterministic random_split</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-01-21" data-time="23:32:24" data-timezone="UTC">11:32PM - 21 Jan 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/artemisart" target="_blank" rel="nofollow noopener">
          <img alt="artemisart" src="https://avatars2.githubusercontent.com/u/9201969?v=4" class="onebox-avatar-inline" width="20" height="20">
          artemisart
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
Option to provide a seed (random_state) for random_split() like the sklearn API https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html.
Motivation
Useful for deterministic sampling &amp; reproducible data generation...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: random</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
What do you think about it? Do you think it would be useful? Thanks.</p>
<p>I got no update on the issue (and tried to register to slack - no update either), should I make a PR or just wait?</p>
CNN in the Fourier Domain

a_d: <p>Hello All,<br>
I have meant to extract features from a multi-channel time-series Data from the Fourier domain, and after a lot of tries and searching, I have not been able to implement anything. Could anyone help me out with this? The architecture what I am trying to build is something like</p>
<p><strong>Data_input -&gt; FFT -&gt; Convolutional Layers -&gt; Classification Layer</strong></p>
<p>TIA</p>

A_A: <p>Hi,<br>
So you would like to extract features from frequency information and not from temporal domain which your data is currently in? I do not see a reason to perform the FFT though, the convolutional layers can behave something like band-pass filters to be sensitive to particular frequencies given the temporal data (of-course only if it helps the classification layers to classify correctly the training data).<br>
Good luck,<br>
Ahmed</p>

a_d: <p>Thanks a lot <a class="mention" href="/u/A_A">@A_A</a> for replying. So  basically what you are saying is that any kind of convolutions are sufficient?.<br>
I was actually going for some kind of a multiScale CNN architechture where I would etract features from the temporal as well the fourier domain and combine them later. Is that kind of redundant ?</p>

A_A: <p>Hi,</p>
<p>Could you please explain what is your end goal and if you have training data to achieve that goal.</p>
<p>Cheers,<br>
Ahmed</p>
How to predict Images from DataLoader_Test?

Deb_Prakash_Chatterj: <p>Hey, I made an Image Classifier and now I want my model to predict and classify the images of DataLoader_Test. Here is my code -</p>
<pre><code class="lang-auto">from google.colab import drive
drive.mount('/content/drive')
data = "/content/drive/My Drive/FT-BD"

# choose the training and test datasets
train_data = datasets.ImageFolder(data+"/train", transform=transform_train)
test_data = datasets.ImageFolder(data+"/val", transform = transform_test)
#n_classes = test_data.shape[1]
n_classes = len(test_data.classes)
print(n_classes)

batch_size = 16

dataloader_train = torch.utils.data.DataLoader(train_data, batch_size, shuffle=True, num_workers=2)
dataloader_test = torch.utils.data.DataLoader(test_data, batch_size, num_workers=2)

images, labels = next(iter(dataloader_train))
imshow_numpy(images[2].numpy())
print(images.shape)

single_loaded_img = images[0]
single_loaded_img = single_loaded_img.to(device)
single_loaded_img = single_loaded_img[None, None]
single_loaded_img = single_loaded_img.type('torch.FloatTensor') # instead of DoubleTensor

out_predict = model(single_loaded_img)
</code></pre>
<p><code>Result : RuntimeError: Expected 4-dimensional input for 4-dimensional weight 16 3 3 3, but got 5-dimensional input of size [1, 1, 3, 224, 224] instead</code></p>
<p>In the middle, there is my model, which is a CNN Classifier. I have tried many solutions, but none of them worked for me? How to load those Images from the <code>dataloader_test</code>? (In the val folder in google drive, there are two folders/classes, where the images are located.)</p>

ptrblck: <p>This line of code</p>
<aside class="quote no-group" data-username="Deb_Prakash_Chatterj" data-post="1" data-topic="69710">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deb_prakash_chatterj/48/10121_2.png" class="avatar"> Deb_Prakash_Chatterj:</div>
<blockquote>
<p>single_loaded_img = single_loaded_img[None, None]</p>
</blockquote>
</aside>
<p>unsqueezes the tensor twice, which will most likely create the issue.<br>
Remove one <code>None</code> and rerun your code.</p>

Deb_Prakash_Chatterj: <p>Hi, I very much appreciate your help and it is solved.<br>
Now I want to upload a image from my PC and try to check the image is in which class?<br>
So, for that I have to upload the image and convert it to a tensor, right? then I have to put it through the model and Predict. right?</p>
Mask like sequence_mask() - Gluon

pfinardi: <p>Hi, I’m looking for a mask funcion like npx.sequence_mask() (Documentation: <a href="https://numpy.mxnet.io/api/deepnumpy/generated/mxnet.npx.sequence_mask.html" rel="nofollow noopener">https://numpy.mxnet.io/api/deepnumpy/generated/mxnet.npx.sequence_mask.html</a>)</p>
<p>Example:</p>
<pre><code class="lang-auto">x = [[[  1.,   2.,   3.],
      [  4.,   5.,   6.]],`

     [[  7.,   8.,   9.],
      [ 10.,  11.,  12.]],

     [[ 13.,  14.,   15.],
      [ 16.,  17.,   18.]]]

// Batch 1
B1 = [[  1.,   2.,   3.],
      [  7.,   8.,   9.],
      [ 13.,  14.,  15.]]

// Batch 2
B2 = [[  4.,   5.,   6.],
      [ 10.,  11.,  12.],
      [ 16.,  17.,  18.]]

// works as identity operator when sequence_length parameter is not used
SequenceMask(x) = [[[  1.,   2.,   3.],
                    [  4.,   5.,   6.]],

                   [[  7.,   8.,   9.],
                    [ 10.,  11.,  12.]],

                   [[ 13.,  14.,   15.],
                    [ 16.,  17.,   18.]]]

// sequence_length [2,3] means 2 of batch B1 and 3 of batch B2 will be kept
// and other rows are masked with value = 1
SequenceMask(x, sequence_length=[2,3], use_sequence_length=True, value=1) =
             [[[  1.,   2.,   3.],
               [  4.,   5.,   6.]],

              [[  7.,   8.,   9.],
               [  10.,  11.,  12.]],

              [[   1.,   1.,   1.],
               [  16.,  17.,  18.]]]

</code></pre>
<p>I tried some stufss with masked_fill but no success. Any ideas?</p>
Finding sources of non-determinism for reproducability

NCTX: <p>I tried using all measures for ensuring reproducibility that I’m aware of for making sure that differences between hyper parameter changes are primarily due to the parameters and not variance caused by different random initializations:</p>
<pre><code class="lang-python">    random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
</code></pre>
<p>However, even with those options set in my main module I still get substantially different<br>
losses every epoch across multiple runs. I also tried removing all invocations of <code>cuda</code><br>
and training on the CPU to eliminate GPU non-determinism as a possible cause which didn’t help.<br>
Since I couldn’t reproduce this behavior with a small toy example I assume it is due to some mistake I made somewhere in my fairly large project. Is there some technique or tool which could help me e.g. record parts of the training process for detecting potential sources of this non-determinism for debugging?</p>
Using torch::grid_sampler_2d from C++ API

AlexanderB: <p>Hi,</p>
<p>I’m using the C+±API to implement an optical flow method.</p>
<p>I successfully implemented a convolutional network for computing optical flow on very low-resolution (24x32) image, I can scale that flow and I want to warp the image so I can continue estimating optical flow on a higher-resolution level.</p>
<p>I can’t find the values for the interpolation mode and border mode, neither in the docs nor in the code and this doesn’t compile:<br>
torch::grid_sampler_2d(img, flow, torch::kBilinear, torch::kBorder, true);</p>
<p>So, where can I find the correct constants?</p>

yf225: <p>Does the equivalent Python code uses <code>torch.nn.functional.grid_sample</code>? We can look at this for how to map it to the equivalent C++ code: <a href="https://github.com/pytorch/pytorch/blob/4baadd54d7c28eac0a2fa07686d593a376e6bd5f/test/cpp/api/functional.cpp#L440-L502" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/4baadd54d7c28eac0a2fa07686d593a376e6bd5f/test/cpp/api/functional.cpp#L440-L502</a>.</p>

AlexanderB: <p>Thank you for your reply. Seems like grid_sampler_2d is more like a low-level function, not to be used by “normal” users. I use grid_sample now and it works as expected:</p>
<p>namespace F = torch::nn::functional;<br>
F::grid_sample(img1_t, map_t, F::GridSampleFuncOptions()<br>
.mode(torch::kBilinear)<br>
.padding_mode(torch::kBorder)<br>
.align_corners(true));</p>
Calculate distribution to image

eranb: <p>Hi every one,<br>
can someone help me to understand how can I calculate the probability of an image in PyTorch?<br>
If there is some function in PyTorch that calculates it immediately?</p>

ptrblck: <p>Could you explain a bit more what you mean by “probability of an image”?<br>
Are you interested in the histogram or some sampling strategy?</p>

eranb: <p>I don’t know if it’s a histogram, that is a question in itself, but the mean idea is to strengthen the non-similarity between images, so I want to do it with probability and without the traditional way like euclidean distance.</p>

ptrblck: <p>Based on the description it seems you are generating images and would like to do it in a diverse way?<br>
I guess a GAN could do the job and base on the released number of papers I assume someone might have even worked on it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Interpolate missing values in a tensor

JuanFMontesinos: <p>Hi, <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have temporal discrete information which may have missing values.<br>
I do have a mask indicating where are those values too.</p>
<p>How can I perform an efficient interpolation filling those values?</p>
<p>In practice I have a <code> TxCxJ</code> tensor (Q). Some elements are let’s say corrupted. I would like, given a corrupted element <code>Q[t,c,j]</code>, to fill that value with an interpolation between <code>Q[t-1,c,j] and Q[t+1,c,j]</code></p>
<p>Also, in the worst case I may have several consecutive corrupted elements:<br>
<code>Q[t_0:t_1,c,j]</code><br>
to be filled with the interpolation between<br>
<code>Q[t_0-1,c,j] and Q[t_1+1,c,j]</code></p>

ptrblck: <p>Are you dealing with tensors at this point or could you use e.g. <code>pandas</code> for your use case, as it provides some convenient features to fill these missing values?</p>
About torch.utils.data.Dataloader

hbg: <p>Hi could someone tell me the difference between len(dataloader) and len(dataloader.dataset) thanks</p>

ptrblck: <p>The length of the <code>DataLoader</code> gives you the number of batches, while the length of the <code>Dataset</code> is usually defined as the number of samples.<br>
If you are writing a custom <code>Dataset</code>, you are setting the length in the <code>__len__</code> method, so that’s why I said “usually”. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

hbg: <p>thanks a lot i confirm your answer by printing len(…)<br>
en(trainloader) is the number of batches while len(train_loader.dataset) is the number of all training set which is equivalent to number of batches * batch size</p>

ptrblck: <p><code>num_batches * batch_size = num_samples</code> would only be true, if each batch is full.<br>
Otherwise the last batch might be smaller. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
PS: if you want to get rid of the last potentially smaller batch, you can specify <code>drop_last=True</code> in your <code>DataLoader</code>.</p>
How to organize data with two classes

Hannes_De_Smet: <p>Hi,</p>
<p>I am currently constructing a semi adversarial auto-encoder. Thus I have two trained networks, one classifier (Male and Female) and one Authenticator (which subject -&gt; Around 450 classes). Because I have to find the two classes of one item I need to strategically organize my data. My question is how can I label the data the best such that is not that much of a hassle?</p>
<p>Is it a good idea to create my own pytorch data base with the following  structure:<br>
{‘image’ : image , [‘gender’ : string, ‘personID’:string] }</p>
<p>Thanks in advance<br>
Kind regards</p>

ptrblck: <p>Your approach is of course valid and should work when you are properly parsing the strings to classes.<br>
Alternatively, you could also store the class indices directly in your structure.</p>
<p>Also, take a look at e.g. <code>pandas</code> which provide an easy way to store structured data.</p>
Ignore padding while batch training AutoEncoder with LSTM Encoder and Decoder

Dylan_Yung: <pre><code>def denoise_train(x: DataLoader):
    loss = 0
    x_padded = list(map(lambda s: pad_string(s), x))
    x_idx_tensor = strings_to_index_tensor(x_padded)
    noisy_x = list(map(lambda s: noise_name(s), x))
    noisy_x_padded = list(map(lambda s: pad_string(s), noisy_x))
    noisy_x_idx_tensor = strings_to_index_tensor(noisy_x_padded)
    noisy_x_rnn_tensor = to_rnn_tensor(noisy_x_idx_tensor)
    batch_sz = len(x)
    encoder_hidden = encoder.init_hidden(batch_size=batch_sz)

    for i in range(noisy_x_rnn_tensor.shape[0]):
        # LSTM requires 3 dimensional inputs
        _, encoder_hidden = encoder(noisy_x_rnn_tensor[i].unsqueeze(0), encoder_hidden)

    decoder_input = strings_to_tensor([SOS] * batch_sz)
    decoder_hidden = encoder_hidden

    for i in range(x_idx_tensor.shape[0]):
        decoder_probs, decoder_hidden = decoder(decoder_input, decoder_hidden)
        nonzero_indexes = x_idx_tensor[i]
        best_indexes = torch.squeeze(torch.argmax(decoder_probs, dim=2), dim=0)
        decoder_probs = torch.squeeze(decoder_probs, dim=0)
        best_chars = list(map(lambda idx: index_to_char(int(idx)), best_indexes))
        loss += criterion(decoder_probs, nonzero_indexes.type(torch.LongTensor))
        decoder_input = strings_to_tensor(best_chars)

    loss.backward()

    return names, noisy_x, loss.item()
</code></pre>
<p>The code I have above is for a denoising autoencoder for people names. The first few lines just noise the names then pad them so they’re all equal length for batch training. Then strings of names are then converted to index tensors where the index is the the number mapped for the character, then that gets converted into a one-hot encoded tensor that’s what <strong>to_rnn_tensor</strong> is for. <strong>encoder</strong> and <strong>decoder</strong> are both LSTMs, which is why the one-hot tensors are inputted sequentially. The hidden state is then put into the decoder which is also an LSTM.</p>
<p>My main problem is I’m trying to batch train this and not backprop on the padded characters. I’m having trouble doing this, because I can’t follow the standard packed_padded_sequences stuff cause this is an autoencoder setup. I’d like to batch train so I need to pad it, but I want pad characters to be ignored by the LSTM encoder and decoder.</p>
Pytorch - One GPU uses more training - How to mitigate?

soulslicer: <p>So my pytorch model looks like this:</p>
<pre><code>gathered_output = torch.nn.parallel.data_parallel(model, model_input, range(ngpu))
optimizer.zero_grad()
loss = F.loss(gathered_output, ground_truth)
loss.backward()
optimizer.step()
</code></pre>
<p>I am doing some large image segmentation task. And GPU 0 uses almost 20% more memory. Is there a way I can mitigate this? The data gathered back isn’t that much just a [8,128,300,400] float tensor in cuda. I have 4 GPU’s, and each GPU works on 2 batches. But I guess the gradients are all gathered back too?</p>
<p>So here I used the functional version of data_parallel, is there a difference to using the wrapped version?</p>
<p>If i compute the loss on each gpu, and only gather back the losses will i save more memory? please help</p>

ptrblck: <p>You could try to adapt the tips from <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">his blog post</a>, which also explains the work flow of <code>nn.DataParallel</code> or alternatively you could use <code>nn.DistributedDataParallel</code>.</p>
How can I use fast gradient sign method in triple losses?

S_M: <p>Hi<br>
in the triple loss network, we have three inputs. Suppose that one of the inputs is used to make an adversarial example, the fast gradient sign method is applied.<br>
my question is how can I compute the gradient of my adv image using this structure. Shall I compute the gradient? am I use triple losses here?</p>
<p>thanks in advance</p>
Input data is not added 'requires_grad=True',Through a neural network operation, the operation of backward () can be performed.Why?

SangYC: <p>This is an official tutorial code：</p>
<hr>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    # 1 input image channel, 6 output channels, 3x3 square convolution
    # kernel
    self.conv1 = nn.Conv2d(1, 6, 3)
    self.conv2 = nn.Conv2d(6, 16, 3)
    # an affine operation: y = Wx + b
    self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    # Max pooling over a (2, 2) window
    x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
    # If the size is a square you can only specify a single number
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

def num_flat_features(self, x):
    size = x.size()[1:]  # all dimensions except the batch dimension
    num_features = 1
    for s in size:
        num_features *= s
    return num_features
</code></pre>
<p>input = torch.randn(1, 1, 32, 32)<br>
out = net(input)<br>
print(out)</p>
<h2>out:tensor([[ 0.0182,  0.1101, -0.0959,  0.0026, -0.0224,  0.1947,  0.1172,  0.0107,<br>
0.0149, -0.1188]], grad_fn=)</h2>
<p>I don’t use ‘requires_grad’, Why does the output appear ‘grad_fn=’？</p>

peterjc123: <p>Parameters of newly constructed modules have requires_grad=True by default. Please read this <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="nofollow noopener">doc</a>.</p>

SangYC: <p>Thanks for your help.</p>
Will the following be part of computational graph and gradients are computed correctly?

sait: <p>consider following code will this part of computation graph correctly and gradients are computed correctly.I mean can distributions be part of graph</p>
<blockquote>
<p>import torch<br>
import torch.distributions.normal as tdn<br>
dist = tdn.Normal(mean,std) <span class="hashtag-raw">#assume</span> network predicts mean,std<br>
loss = -1 * dist.log_prob(y)</p>
</blockquote>

albanD: <p>Hi,</p>
<p>I think the <code>log_prob</code> is differentiable for Normal yes. You can check that by verifying that if <code>mean</code> or <code>std</code> has <code>requires_grad=True</code>, then <code>dist.log_prob(y).requires_grad == True</code>.</p>
<p>In general, the output requires grad, that means that the function is differentiable using the autograd.</p>

sait: <p>Another question, tdn.cdf(y) will also differentiable right?<br>
I was thinking if <strong>dist = tdn.Normal(mean,std)</strong> would mess computational graph.It will not write right?</p>
Increasing size of images in torchvision.utils.make_grid

Suraj_Subramanian: <p>I have 32 images of size 3 x 128 x 128 in a single batch. I want to display them as a grid, so I used the torchvision.utils.make_grid function as in the code below. But the images in the grid are very small. Is there anyway to increase the size of the grid ?</p>
<pre><code class="lang-auto">def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1,2,0)), interpolation='nearest')

image_data.shape # torch.Size([32, 3, 128, 128])
m = make_grid(image_datas, nrow=8, scale_each = True, normalize = True, padding = 4)
show(m)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfb41a6a341848b0ce010b05a5f3624d09c595fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/cfb41a6a341848b0ce010b05a5f3624d09c595fe" title="Screenshot 2020-02-15 at 07.39.53"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb41a6a341848b0ce010b05a5f3624d09c595fe_2_345x184.png" alt="Screenshot 2020-02-15 at 07.39.53" data-base62-sha1="tDqHKBDm1VtSNeWLYyRQ3Aelcui" width="345" height="184" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb41a6a341848b0ce010b05a5f3624d09c595fe_2_345x184.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb41a6a341848b0ce010b05a5f3624d09c595fe_2_517x276.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb41a6a341848b0ce010b05a5f3624d09c595fe_2_690x368.png 2x" data-dominant-color="8B7E77"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-02-15 at 07.39.53</span><span class="informations">842×450 420 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could increase the size of the shown image directly in <code>plt.imshow</code> or alternatively save the image and use your default viewer.</p>
<p>I assume the output is small as matplotlib tries to scale it down to fit your notebook/IDE</p>
Detach and .data

agadetsky: <p>In previous versions we did something like:</p>
<pre><code class="lang-auto">for p in model.parameters():
    p.data.add_(-lr, p.grad.data)
</code></pre>
<p>Migration guide says that now using .data is unsafe, so how to rewrite this using .detach()?</p>

agadetsky: <p>Is it correct to do the following:</p>
<pre><code class="lang-auto">for p in model.parameters():
    p = (p - lr * p.grad).detach()
</code></pre>

tom: <p>Hi,</p>
<pre><code class="lang-auto">for p in model.parameters():
    p = (p - lr * p.grad).detach()
</code></pre>
<p><strong>No! Don’t do that!</strong> Apologies for the bold font. If you assign p, you will just overwrite the name and the model parameter will be unchanged.</p>
<p>Now, if you look at the source code of an optimizer, say <a href="https://github.com/pytorch/pytorch/blob/52e4d3c4a2ca48558f218fd4f09aa9c5aa06450c/torch/optim/sgd.py#L107" rel="nofollow noopener">SGD</a>, you find that the update rule is still more or less what your original post uses:</p>
<pre><code class="lang-auto">for p in model.parameters():
    p.data.add_(-lr, p.grad.data)
</code></pre>
<p>so that will work because everyone still uses it.<br>
That said, I personally think that using <code>torch.no_grad()</code></p>
<pre><code class="lang-auto">with torch.no_grad()
    for p in model.parameters():
        p.add_(-lr, p.grad)
</code></pre>
<p>is a better way to achieve the same.<br>
But take this last advice with a grain of salt: I’m biased because I’m trying to convince people that there should be an inplace hook for some caching style applications.</p>
<p>As a general rule, I try to consult the PyTorch source code as an example as much as I can, and so far, I seem to be doing OK with that strategy.</p>
<p>Best regards</p>
<p>Thomas</p>

two_Two: <blockquote>
<p>No! Don’t do that!**  Apologies for the bold font. If you assign p, you will just overwrite the name and the model parameter will be unchanged.</p>
</blockquote>
<p>Mau I ask you why is that?  Thanks!</p>

tom: <p>Fundamentally that is how Python variables work.</p>
<ul>
<li>A plain assignment <code>=</code> as in <code>p = foo</code> just means "take whatever is on the right side and give it the name on the left side (<code>p</code>). There isn’t anything (<code>__something__</code>) called when</li>
<li>This is in stark contrast to the superficially similar <code>p[something] =</code> or <code>x.p = </code>which call <code>p.__setitem__</code> and <code>x.__setattr__</code> under the hood.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

two_Two: <p>OK,that’s really clear,Thanks !</p>
Ignore padding for LSTM batch training

Dylan_Yung: <p>I realize there is packed_padded_sequence and so on for batch training LSTMs, but that takes an entire sequence and embeds it then forwards it through the LSTM. My LSTM is built so that it just takes an input character then forward just outputs the categorical at each sequence. So I built it so that I pad the sequences before hand so they’re equal length then each index is fed in sequentially. Unfortunately this also means the first characters are pad characters (Because I use prepadding). Is there a way to get the LSTM not to backprop on inputs that are just pads using this LSTM setup?</p>
RuntimeError: Unimplemented backend QuantizedCPU

Ashish_Gupta1: <p>I tried to check the type of quantized tensor for testing! I have CNN model which I convert to quantized form. When I simply print the weights it shows dtype as qint8. But when I tired this <code>tensor.type()</code> , It throws this error…</p>
<pre><code class="lang-auto">File "test.py", line 321, in test
    print((am['ConvNet.conv4_1.weight'].type()))
RuntimeError: Unimplemented backend QuantizedCPU
</code></pre>

jerryzh168: <p>yeah, we probably haven’t implement this method… feel free to submit a patch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
UserWarning: The number of elements in the out tensor of shape [1] is 1

lenyabloko: <p>I am using HuggingFace pytorch-transformers and one of my pre-trained models refuse to fine tune giving me those UserWarnings for every torch.utils.data.DataLoader call.</p>
<p>I have described the details in  <a href="https://stackoverflow.com/questions/60218634/userwarning-the-number-of-elements-in-the-out-tensor-of-shape-1-is-1" rel="nofollow noopener">https://stackoverflow.com/questions/60218634/userwarning-the-number-of-elements-in-the-out-tensor-of-shape-1-is-1</a></p>
<p>But someone downgraded my question for some  reason. What am I missing?</p>
<p><a href="https://colab.research.google.com/drive/1mq9RZ_BX1O5vgxCM0CvPzAm9YVKnq4DQ" rel="nofollow noopener">Here is my Notebook so you can run and see the results:</a></p>
<p>Thanks for your help!</p>