akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Behaviour of WeightedRandomSampler

username: aktgpt
cooked: <p>I’m trying to implement stratified sampling in my dataset and calculate the weights for each sample in the dataset with the following function:</p>
<blockquote>
<p><code>def make_weights_for_balanced_classes(labels):</code><br>
<code> unique_labels, counts = np.unique(labels, return_counts=True)</code><br>
<code> weight_per_class = np.sum(counts) / counts</code><br>
<code> weights = [0] * len(labels)</code><br>
<code>for i, val in enumerate(labels):</code><br>
<code>   weights[i] = weight_per_class[np.where(unique_labels == val)[0]]</code><br>
<code> return weights</code></p>
<p><code>sampler = WeightedRandomSampler(weights, len(weights))</code><br>
<code>dataloader = DataLoader(             dataset, batch_size=128, sampler=sampler         )</code></p>
</blockquote>
<p>But when I’m enumerating through the <code>dataloader</code> the error occurs in my custom <code>dataset</code> <code>__getitem__</code>:</p>
<blockquote>
<p><code>list indices must be integers or slices, not list</code></p>
</blockquote>
<p>I wanted to know if this is normal that you’ll get a list of indices instead of a single index when using <code>WeightedRandomSampler</code>. Should I change my <code>dataset</code> to accept the list of indices whenever I’m using <code>WeightedRandomSampler</code>?</p>

username: ptrblck
cooked: <p>Which line of code is throwing this error?<br>
The sampler should rerun a single index and it works fine in this dummy code snippet:</p>
<pre><code class="lang-python">dataset = TensorDataset(
    torch.arange(10).float())

weights = [0.1] * 10
sampler = WeightedRandomSampler(weights, len(weights))
loader = DataLoader(dataset, sampler=sampler)

for x in loader:
    print(x)
</code></pre>

username: aktgpt
cooked: <p>Thanks for the reply. I realized after seeing your code that I had one extra dimension in my weights which was why my dataset <code>__getitem__</code>was getting a list of indices instead of single index at a time.<br>
Had to change<br>
<code>weights[i] = weight_per_class[np.where(unique_labels == val)[0]]</code><br>
to<br>
<code>weights[i] = weight_per_class[np.where(unique_labels == val)[0][0]]</code><br>
in my stratified sampling function and it works now.<br>
Thank you.</p>
Torchaudio.load normalization question

username: Laurence_J
cooked: <p>Hi, I’m new to audio signal processing and to pytorch and I’m having some trouble understanding this part of the docs of the torchaudio load function:</p>
<blockquote>
<p>normalization (bool, number, or callable, optional) – If boolean True, then output is divided by 1 &lt;&lt; 31 (assumes signed 32-bit audio), and normalizes to [-1, 1]. If number, then output is divided by that number If callable, then the output is passed as a parameter to the given function, then the output is divided by the result. (Default: True)</p>
</blockquote>
<p>From what I understand, the function assumes the file to have a <em>bit depth</em> of 32 bit, however that bit depth is rather rare. Does 32-bit audio mean indeed bit depth or something else?</p>
<p>Also I don’t understand what is the meaning of <em>output is divided by 1 &lt;&lt; 31</em>. What is meant by output and what is meant by 1 &lt;&lt; 31?</p>
<p>Thanks for your help</p>

username: ptrblck
cooked: <p>I also assume 32-bit audio corresponds to the bits per sample.</p>
<p><code>1 &lt;&lt; 31</code> is a left shift by 31 positions, so it translates to <code>1 &lt;&lt; 31 == 2**31  == 2147483648</code>, which would be the max value of each sample.<br>
If I’m not mistaken, 32bit audio would have the range <code>[−2,147,483,648, 2,147,483,647]</code>, so you would get a minimal error for the max positive value.</p>

username: Laurence_J
cooked: <p>thank you very much for the reply,</p>
<p>it must be true indeed that</p>
<blockquote>
<p>I also assume 32-bit audio corresponds to the bits per sample.</p>
</blockquote>
<p>as if i normalize with True I get a tensor with max and min values [-1,1]:<br>
max: 0.0881<br>
min: -0.1289</p>
<p>while if I use normalization=16:<br>
max: 11821056<br>
min: -17301504</p>
<p>and for normalization=False:<br>
max: 1.8914e+08<br>
min: -2.7682e+08</p>
<p>indeed the std and avg of the data loaded using normalization True are between 0 and 1 so it seems like the correct normalization for the data I’m working with.</p>
<p>Thank you very much again</p>
Obtaining true zero pixels after normalising to Imagenet stats

username: sidd.suresh97
cooked: <p>How do I obtain true zero pixels after I have applied the following transform since the values of 0 pixels change? Is there a way of obtaining the zero values along each color channel separately?</p>
<pre><code class="lang-auto"> tfms = transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])
</code></pre>
<p>Here is my use case. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b10815aa31c12c66de52a68947409a9cdf861343.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b10815aa31c12c66de52a68947409a9cdf861343" title="Screenshot 2020-02-28 at 3.21.34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b10815aa31c12c66de52a68947409a9cdf861343_2_671x500.jpeg" alt="Screenshot 2020-02-28 at 3.21.34 PM" data-base62-sha1="pg5PFZHS95ecC3ZpRZEJlUsP951" width="671" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b10815aa31c12c66de52a68947409a9cdf861343_2_671x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b10815aa31c12c66de52a68947409a9cdf861343.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b10815aa31c12c66de52a68947409a9cdf861343.jpeg 2x" data-dominant-color="ADB0AA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-02-28 at 3.21.34 PM</span><span class="informations">800×596 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You could create the masks containing zeros for the black boxes and ones everywhere else, and multiply the normalized input tensor with it.</p>
Create sentence embeddings from word embeddings by 2-layer RNN

username: Mahsa_Afsharizade
cooked: <p>I have a text processing task. I have a dataset that each sample of the dataset consists of a sentence as its input and a real score as its output.</p>
<p>Now I want to predict the score for each sentence by a 2-layer RNN networks.</p>
<p>The first layer works at word level. This layer accepts a batch of sentences. This layer produces sequentially word representations (hidden state representations in each word position).</p>
<p>The second layer works at sentence level. The input of this layer is the average of the word representations from the previous layer. This layer generates sentence representations.</p>
<p>I have problem in understanding the inputs of the second layer. I have created such a network:</p>
<pre><code class="lang-auto">import torch.nn as nn
class RNNModel(nn.Module):
def __init__(self):
    #embedding_dimension=50
    self.rnn1 = nn.RNN(input_size=50, hidden_size=50)
    self.rnn2=nn.RNN(input_size=50, hidden_size=50 )

def forward(self, input, hidden1, hidden2):
    # input is a batch of sentences
    embedded=self.embedding(input)
    output1, hidden1 = self.rnn1(embedded, hidden1)
</code></pre>
<p>I know that the size of the output1 is <code>(seq_len, batch, num_directions x hidden_size)</code> . The average of the hidden representations in rnn1 is the input for rnn2. Then the size of the input of the second layer is <code>(1, batch, num_directions x hidden_size)</code> . Averaging is done on all of the hidden representations in all of the time steps (number of time steps= seq_len).</p>
<pre><code class="lang-auto">    input2=Average(output1)
    output2, hidden2= self.rnn2(input2, hidden2)
</code></pre>
<p>I think that I can not feed input2 to rnn2 as its input. Because the size of the input2 should be <code>(seq_len, batch, input_size)</code> . But now this is <code>(1, batch, num_directions x hidden_size)</code> .</p>
<p>In the first layer, I was processing sequences of words (sentences). Each time I feed a batch of sentences to rnn1. So the size of input1 was <code>(seq_len= length of longest sentence in that batch, batch=BATCH_SIZE, input_size=word_embedding_dimension)</code> . On the other hand, in rnn2, I’m processing a sequence of sentences. I think that the size of input2 should be <code>(seq_len= BATCH_SIZE, batch=1, input_size=num_directions_of_rnn1 x hidden_size1)</code> . I’m confused about this inconsistency of sizes. I have problems in these cases:</p>
<p>(1) after averaging of the hidden representations of rnn1, how can I feed it to rnn2?</p>
<p>(2)should I reshape the average representation?</p>
<p>(3) what is the size of the input2?</p>
<p>Thanks in advance.</p>
Replace resnet 50 fc with mlp

username: Giuseppe
cooked: <p>Hy guys, I must do “features sharing”. I must pass the features of resnet 50 (2048) to a MLP</p>
<pre><code class="lang-auto">from torchvsion.models import resnet50
class pipo(nn.module):
      def __init__(self,n_input, n_output, n_hidden):
        
            super(pipo, self).__init__()
            self.model = resnet50(pretrained = True)
            self.model.fc = nn.Linear(self.model.fc.in_features, n_input)
            self.fc = nn.Linear(self.model.fc, n_hidden)
            self.mu = nn.Linear(n_hidden, n_output)
     
     def forward(self, x):
            etc...        
</code></pre>
<p>is it right in this way?</p>

username: ptrblck
cooked: <p>This looks generally good.<br>
It might be a typo, but <code>self.fc</code> should most likely be defined as:</p>
<pre><code class="lang-python">self.fc = nn.Linear(self.model.fc.out_features, n_hidden)
</code></pre>

username: Giuseppe
cooked: <p>Can you clarify me the difference between …fc.in_features() and fc.out_features()?</p>

username: ptrblck
cooked: <p>The <code>in_features</code> define the input features of your tensor for this linear layer, while <code>out_features</code> define the output features.<br>
E.g. if you are working with 2 input features for each sample (let’s say height and weight for a dog classifier), you would define <code>in_features=2</code> in your first linear layer. The number of output features depends on your architecture and you could chose any valid value, which “works”.<br>
Have a look at <a href="http://cs231n.github.io/linear-classify/">CS231n</a> to get an example of the weight matrix and the mentioned names.</p>

username: Giuseppe
cooked: <p>I have a doubt for FC layer of my resnet. I don`t know if is it right.<br>
Can you help me?<br>
Maybe do i open a new topic?</p>
<pre><code class="lang-auto">    def __init__(self, n_input, n_output, n_hidden1, n_hidden2):
        super(ActorCriticModel, self).__init__()
        self.model = resnet50(pretrained=True)
        out = self.model.fc.in_features
        self.model.fc = Identity() #identity is a class where the forward return x element
        self.fcN = nn.Linear(out, n_hidden1)
        self.fcV = nn.Linear(out, n_hidden2)
        #self.model.fc = torch.cat(self.fcN + self.fcV)
        self.muX = nn.Linear(n_hidden1, 1)
        self.muZ = nn.Linear(n_hidden1, 1)
        self.sigma = nn.Linear(n_hidden1, 1)
        self.muO1 = nn.Linear(n_hidden1, 1)
        self.muO2 = nn.Linear(n_hidden1, 1)
        self.sigmaO = nn.Linear(n_hidden1, 1)
        self.value = nn.Linear(n_hidden2, 1)
        self.distributionXZ = torch.distributions.MultivariateNormal
        self.distributionO = torch.distributions.MultivariateNormal
</code></pre>

username: ptrblck
cooked: <p>What doubts do you have about the approach?</p>
Training high-dimensional features

username: vr25
cooked: <p>Hi,</p>
<p>I have a vocabulary with a size of ~60,000. I have a dataset of document vectors with such a high dimension. I am training them on a neural network and I am unsure if it can really train such high-dimensional vectors in PyTorch.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Nothing stops you from doing it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Do you have any concerns or is your model not training at the moment?<br>
You could certainly try to reduce the feature dimension, but you could also just try out different architectures and see how it goes.</p>
Difference in Prediction scores of Pytorch and Keras model

username: Asim_Jalwana
cooked: <p>Hi,</p>
<p>I was trying to see if probability scores of pre-trained models from Keras and Pytorch match for the same images. However, I found that there is a slight difference.</p>
<p>I am confused, should this be the case?</p>
<p>The simple code is given below:<br>
import torch<br>
import torchvision.models as models<br>
from PIL import Image<br>
import numpy as np<br>
import torch.nn.functional as F</p>
<p>mean = [0.485, 0.456, 0.406]<br>
std = [0.229, 0.224, 0.225]</p>
<p>pytorchModel = models.resnet50(pretrained=True)<br>
pytorchModel.eval()</p>
<p>imagePath = ‘/home/jalwana/Data/Imagenet2012/temp/n02504013/training/n02504013_1830.JPEG’<br>
image = np.asarray(Image.open(imagePath).resize((224, 224), Image.BILINEAR), dtype=np.float32)<br>
image = image / 255.0<br>
image = np.transpose(image, (2, 0, 1))</p>
<p>image[0, :, :] -= mean[0]<br>
image[1, :, :] -= mean[1]<br>
image[2, :, :] -= mean[2]<br>
image[0, :, :] /= std[0]<br>
image[1, :, :] /= std[1]<br>
image[2, :, :] /= std[2]</p>
<p>image = torch.unsqueeze(torch.from_numpy(image), 0)<br>
probs = F.softmax(pytorchModel(image), dim=1)</p>
<h1>The keras model</h1>
<p>from keras.applications.resnet50 import ResNet50<br>
from keras.applications.resnet50 import preprocess_input</p>
<p>image = np.asarray(Image.open(imagePath).resize((224, 224), Image.BILINEAR), dtype=np.float32)<br>
preprocessedImage = np.expand_dims(preprocess_input(image), axis=0)</p>
<p>model = ResNet50()<br>
predictionsKeras = model.predict(preprocessedImage[:, :, ::-1])</p>
<p>print(probs[0, probs.argmax()])<br>
print(predictionsKeras[0, predictionsKeras.argmax()])</p>
<h1>For one image</h1>
<h1>pytorch resnet50 gave 0.8059</h1>
<h1>keras resnet50 gave 0.7875</h1>
<h1>the image belongs to ImageNet training dataset</h1>
<p>Since prediction score is not a metric, should we be considerate of it?</p>

username: ptrblck
cooked: <p>Since both models were pretrained separately, I would expect some predictions to be a bit different.<br>
Have you compared the validation performance of both models?<br>
I’m not sure, how the ResNet was pretrained in Keras and what reference implementation and training was used.</p>
ceil_mode=True for unfold

username: G.M
cooked: <p>I’m using <code>nn.Unfold</code> to implementing a custom pooling layer, however, I’d like to keep the last value, like what <code>ceil_mode=True</code> for MaxPool would do. It seems that Unfold does not support this, is there a workaround?<br>
Thanks.</p>

username: ptrblck
cooked: <p>You could pad the input to create the desired output shape.</p>
Improving testing accuracy

username: ssalome
cooked: <p>Hi<br>
I’m currently working on a classification model whose input are x-ray images, i need to classify it into the respective disease categories. I have trained resnet 18 (not pretrained )on a data-set of 100 images per class.</p>
<p>I have 1050 images for training and 450 images for testing (70:30 split).</p>
<p>On running for 50 epochs the accuracy goes to 92 percent, but the testing accuracy isn’t good at all it is just about 15 %, i think the model is overfitting,</p>
<p><strong>can anyone suggest as to what can be done to improve the testing accuracy?</strong></p>
<p>I have grey scale images and i have down-scaled them to 224 x 224 so image shape is [1,224,224].</p>
<p>I have modified the first conv2d layer to accept grey scale images and last layer of resnet to output the required number of classes.</p>
<p>I’m a beginner, it would great if anyone can help with this. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: simaiden
cooked: <p>You could divide your entire dataset into training (80) , validation (10), and testing (10) , and do a traning scheme like <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> . Also don’t forget to set the model in training mode with <code>model.train()</code> and when evaluate, either in validation or training, set to evaluation mode with <code>model.eval()</code></p>

username: ssalome
cooked: <p>Okay will try that…!</p>
[solved]How to apply BatchNorm to a part of features?

username: Roy_Paik
cooked: <p>I am struggling with quantile regression with NN.<br>
The input data shape(n_data, 308) and last dim(qunatile information) should not be normalized.<br>
May I ask how to apply batch norm partially, except some of features?<br>
-------EDIT<br>
Thanks to fs5ss1, I solved it, please refer to the code below:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.build_model()
    
    def build_model(self):
        self.short_cut = nn.Identity()
        
        self.linear1 = nn.Linear(308, 1101)
        self.bn1 = nn.BatchNorm1d(1100)
        self.leakyrelu = nn.LeakyReLU()
        
        self.linear2 = nn.Linear(1101, 308)
        self.bn2 = nn.BatchNorm1d(307)

        self.linear_mean_output = nn.Linear(308, 1)

    def forward(self, x):
        x_out = self.linear1(x)
        
        x_q = x_out[:, -1].contiguous().view([-1, 1])
        x_out = torch.cat([self.bn1(x_out[:,:-1]), x_q], axis=1)
        x_out = self.leakyrelu(x_out)
        
        x_out = self.linear2(x_out)
        x_out += self.short_cut(x)
        
        x_q = x_out[:, -1].contiguous().view([-1, 1])
        x_out = torch.cat([self.bn2(x_out[:,:-1]), x_q], axis=1)
        x_out = self.leakyrelu(x_out)
        x_out = self.linear_mean_output(x_out)
        
        return x_out
</code></pre>

username: fs4ss1
cooked: <p>I think that you can do the follow in the training loop:</p>
<pre><code class="lang-python">def forward(self, x):
    x1 = self.BatchNorm(x[:-1])
    x = torch.cat((x1, x[-1]), dim = 1)
    .
    .
    .
    return x
</code></pre>

username: Roy_Paik
cooked: <p>Wow! I will try it right away</p>
Visdom- black MNIST images

username: WNie
cooked: <p>Hi, I’m having trouble with using Visdom.  The ‘data’ is a minibatch of MNIST.<br>
After calling the method ‘images’,<br>
e.g.   ‘viz.images(data.reshape(-1,1,28,28), win=‘images’,opts=dict(title=‘images’))’ ,<br>
the export  is full of black images and I cannot figure out why this is happening.</p>

username: ptrblck
cooked: <p>Are you able to visualize these images in e.g. <code>matplotlib</code>?<br>
Also, did you normalize or processed these images in some way?</p>

username: WNie
cooked: <p>Yes, I checked the data of dataset with matplotlib.pylot.imshow and it works well, the handwritten number can be shown.<br>
The processing code and export are shown in the uploaded pictures.<br>
(upload://xv6FEc0q0KLJLDECVDhIWajBW1a.png)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c687260bd37eab68038e50e079a6584d01f33e2f.png" data-download-href="https://discuss.pytorch.org/uploads/default/c687260bd37eab68038e50e079a6584d01f33e2f" title="tim2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c687260bd37eab68038e50e079a6584d01f33e2f.png" alt="tim2" data-base62-sha1="skg5OhSvOlBRatxgdUDDliN46HZ" width="690" height="498" data-dominant-color="31312A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tim2</span><span class="informations">823×595 15.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: WNie
cooked: <p>sorry, it seems like the export picture has missed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eacbf125204393ba50667f22ce295eaca1381478.png" data-download-href="https://discuss.pytorch.org/uploads/default/eacbf125204393ba50667f22ce295eaca1381478" title="tim"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eacbf125204393ba50667f22ce295eaca1381478_2_690x392.png" alt="tim" data-base62-sha1="xv6FEc0q0KLJLDECVDhIWajBW1a" width="690" height="392" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eacbf125204393ba50667f22ce295eaca1381478_2_690x392.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eacbf125204393ba50667f22ce295eaca1381478_2_1035x588.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eacbf125204393ba50667f22ce295eaca1381478_2_1380x784.png 2x" data-dominant-color="79849C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tim</span><span class="informations">1421×809 75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Gradient of intermediate representations

username: srikarym
cooked: <p>Hi,<br>
I’m using the following network architecture:</p>
<pre><code class="lang-python">class Net(nn.module):
    def forward(x):
        x = self.base(x)
        fmap = x.view(x.size(0), -1)                               
        masked_fmap = fmap.masked_fill(self.indices_active,0)    #applies mask
        masked_fmap.requires_grad = True                          #need the grad of this activation
        return self.fc(masked_fmap), masked_fmap
</code></pre>
<p>Using masked_fill, I mask out 50% of the activations before final fc. I calculate the gradients of the masked representations this way:</p>
<pre><code class="lang-auto">model = Net()
yhat,fmap = model(x)
loss = criterion(yhat,y)
loss.backward()
gradients = fmap.grad
</code></pre>
<p>Is this the right way to calculate the gradients of intermediate activations? I observed that even though 50% of fmap is zero, the gradients are all non zero. Shouldn’t the gradients also be zero for the indices that are masked?</p>
Some questions regarding TPU

username: vainaijr
cooked: <ol>
<li>Which is preferred, using TPU, or multiple GPUs in parallel?</li>
<li>Are we going to see TPUs used in parallel in the future, similar to gpus used in parallel?</li>
<li>All of the GPU that I have heard of, work in the field of gaming / video rendering also, but I do not hear any TPU being used in such areas, how can they be primarily for neural networks, and not deliver upgraded performance in other areas?</li>
<li>When it comes to GPU, then people have them in their personal computers also, but nothing like that for TPU, why do we only hear usage of cloud TPU, and nothing like personal TPUs?</li>
<li>on Wikipedia it says, “ Compared to a <a href="https://en.wikipedia.org/wiki/Graphics_processing_unit" rel="nofollow noopener">graphics processing unit</a>, it is designed for a high volume of low precision computation (e.g. as little as <a href="https://en.wikipedia.org/wiki/8-bit" rel="nofollow noopener">8-bit</a> precision)<a href="https://en.wikipedia.org/wiki/Tensor_processing_unit#cite_note-7" rel="nofollow noopener">[7]</a> with more input/output operations per <a href="https://en.wikipedia.org/wiki/Joule" rel="nofollow noopener">joule</a>, and lacks hardware for rasterisation/<a href="https://en.wikipedia.org/wiki/Texture_mapping" rel="nofollow noopener">texture mapping</a>.”, does this mean that most of the upcoming neural network training, is going to be through quantized weight parameters (example make 32-bit to 8-bit), trained on TPUs?</li>
<li>If 5 is true, then the reason we do not use TPUs for games is that, if we represent each pixel with less number of bits, then graphics processing would go faster but it would not look convincing, so it is avoided?</li>
</ol>
Does initialization give similar result despite different learning rates?

username: WeiQin_Chuah
cooked: <p>Hi I have included an initialisation part in my code which looks like this:</p>
<pre><code class="lang-auto">&gt; for m in self.modules():
             if isinstance(m, (nn.Conv2d, nn.Conv3d)):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
             elif isinstance(m, (nn.BatchNorm2d, nn.BatchNorm3d)):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)
</code></pre>
<p>And I have tested my training with two different learning rates: (1) 0.0005 and (2) 0.01 but both give quite similar results and the loss is dropping at relatively same speed. I have also fixed the random values using the same seed.</p>
<p>Could it be because of the code I included above or other bugs in my code that cause this to happen?</p>
<p>Thank you</p>

username: albanD
cooked: <p>That would wildly depend on the optimizer you’re using and the model you try to optimize.<br>
If your model is well conditioned, the optimizer will converge fast with (almost) any learning rate.</p>

username: WeiQin_Chuah
cooked: <p>How to determine if a model is well conditioned?</p>
Grid Sample for Even Number?

username: soulslicer
cooked: <p>Let say that I want to apply grid sample to a tensor. If the tensor is a odd sized number like [1x5], and we didnt want it to change, it would be [-1, 0.5, 0, 0.5, 1].</p>
<p>But what if it is an even sized number. I know the ends have to be -1 and 1, but what about the middle?</p>
<p>Is it [-1, 0.5, 0, 1] or is it [-1, 0.25, 0.25, 1] or something else?</p>
Using tensorboard in colab

username: 111250
cooked: <p>Hello!<br>
I am using colab.<br>
I was following the tutorial using colab notebook<br>
<a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html</a><br>
when I reached to the line<br>
tensorboard --logdir=runs<br>
I got the error<br>
"  File “”, line 1<br>
tensorboard --logdir= runs<br>
^<br>
SyntaxError: can’t assign to operator"</p>
<p>Changed it to “!tensorboard --logdir=runs”<br>
I got<br>
“Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all<br>
TensorBoard 2.1.0 at <a href="http://localhost:6007/" rel="nofollow noopener">http://localhost:6007/</a> (Press CTRL+C to quit)”<br>
pressing the link got " This site can’t be reached"<br>
In short how can I run this code in colab?<br>
Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>From googling, there seems to be an extension to use tensorboard on colab.<br>
You can do in a new cell:</p>
<pre><code class="lang-auto">%load_ext tensorboard
%tensorboard --logdir mylogdir
</code></pre>
<p>And that will launch tensorboard with the given logdir <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: 111250
cooked: <p>Ok thanks !<br>
I already found a solution and now my model going to finish training<br>
this could be helpful for someone in the future<br>
<a href="https://medium.com/looka-engineering/how-to-use-tensorboard-with-pytorch-in-google-colab-1f76a938bc34" rel="nofollow noopener">https://medium.com/looka-engineering/how-to-use-tensorboard-with-pytorch-in-google-colab-1f76a938bc34</a><br>
sorry for asking before trying!</p>
Copying a tensor from one device to another

username: nirandaperera
cooked: <p>Hi,<br>
Is there a way to copy the values (data and grad) of a tensor to another in a different device?</p>
<pre><code class="lang-auto">x = torch.ones(2, 2, device='cuda:0')
y = torch.zeros(2, 2, device='cuda:1')

# something like 
x.copy_to(y)
</code></pre>
<p>And what is the implication of this on this on the grad_fn?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can move a Tensor to a specific device by doing <code>x_cuda1 = x.to("cuda:1")</code>.<br>
This is a differentiable operation. So if <code>x</code> requires gradient, then any op on <code>x_cuda1</code> will contribute to the gradients toward <code>x</code>.<br>
For the .grad field, you can do <code>x_cuda1.grad = x.grad.to("cuda:1")</code>.</p>

username: nirandaperera
cooked: <p><a class="mention" href="/u/alband">@albanD</a> say I want to use y on a different compute graph. And I only want to copy the values. Can that be accommodated?</p>

username: albanD
cooked: <p>You can use <code>.detach()</code> to break the dependency between the two.<br>
So <code>x_cuda1 = x.detach().to("cuda:1")</code>.</p>
Autograd computation time error: One of the variables needed for gradient computation has been modified by an inplace operation

username: abhishek_karanath
cooked: <p>I have a torch.autograd.variable.Variable named “unw” of shape 6x64x300 and another torch.autograd.variable.Variable named “powers” of shape 6. I need to compute the power of each of the 6 matrices in “unw” with each of the 6 values in “powers”. So I wrote down the below code:</p>
<pre><code class="lang-auto">for j in range(unw.shape[0]):
     unw[j,:,:] = torch.pow(unw[j,:,:],powers[j])
</code></pre>
<p>After computing the new “unw” I compute</p>
<pre><code class="lang-auto">s = torch.prod(unw,dim=0)
</code></pre>
<p>to get a 64x300 valued tensor.</p>
<p>Later when computing torch.autograd.grad(loss, self.model.parameters()), I get an error as below</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
<p>This is caused to due the change of “unw”. How can I resolve this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the simplest would be:</p>
<pre><code class="lang-auto">unw = torch.pow(unw, powers.view(-1, 1, 1).expand_as(unw))
</code></pre>
<p>That removes both the inplace and the for-loop <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: abhishek_karanath
cooked: <p>Thank you so much! This is perfect <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> I also had to do a .contiguous()</p>
<pre><code class="lang-auto">powers.contiguous().view(-1,1,1).expand_as(unw)
</code></pre>
Using Neural Network in loss function

username: Durotan97
cooked: <p>Hey !</p>
<p>I am quite new to using pytorch and it seems like I can not figure out how to solve my problem.</p>
<p>I am implementing a custom loss function for a neural network. The network receives an input and outputs a tensor. However, the loss which I try to use to train this network incorporates the output of a second network.</p>
<p>def action_loss(actions):<br>
l = Variable(torch.zeros(1),requires_grad=True)<br>
loss = l.clone()<br>
s = s_0<br>
for i in range(t):<br>
gen_inp = torch.cat((n,s),dim=0)<br>
action = actions[i].view(-1)<br>
gen_inp = torch.cat((gen_inp,action),dim=0)<br>
with torch.no_grad():<br>
g_out = gen(gen_inp)<br>
loss += g_out[2]<br>
s = torch.Tensor([g_out[0],g_out[1]]).to(device)<br>
loss = -loss<br>
return loss´</p>
<p>So in the gen(gen_inp) line, a second neural network is called which returns the values that I need for the loss. I do not want to change the weights of this gen network and only of the one that returns the actions vector. Currently this code does not yield any gradients and the network does not train.</p>
<p>How can i fix this problem ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>when you run in torch.no_grad, then nothing will traverse that block of code.<br>
In this case, if you just don’t want to compute gradient for that net, you remove the torch.no_grad and set its parameters not to require gradients with something like:</p>
<pre><code class="lang-auto">for p in gen.parameters():
  p.requires_grad_(False)
</code></pre>

username: Durotan97
cooked: <p>Hi albanD,</p>
<p>Thanks a lot for your help, it now works!</p>
Incorporating samples into convolution operation

username: Kamil_Adamczewski
cooked: <p>Is it possible to feed a five-dimensional tensor into the convolution operation? That is, normally we input [batch_size, channels, featuremapx, featuremapy] to the torch.Conv2D operation. Is there any way to expands the allowed dimensions possible to feed a vector:  [batch_size, samples, channels, featuremapx, featuremapy] ? For example, you feed the tensor of dimensions: [100,150,10, 64,64] and output the tensor of size [100,150,20,32,32]?</p>

username: simaiden
cooked: <p>Maybe <a href="https://pytorch.org/docs/stable/nn.html#conv3d" rel="nofollow noopener">Conv3d</a> is what you are looking for</p>
How to transform images to tensors

username: suri_g
cooked: <p>Hi, I am a newbie to  PyTorch, I am doing the image classification, please help me. how to transfer the image to tensors,<br>
Here my code :</p>
<pre><code class="lang-auto">import cv2
import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import os
import torch
import torchvision
import torchvision.transforms as transforms
file_path='dataset'
train=pd.read_csv(os.path.join(file_path,'train.csv'))
test=pd.read_csv(os.path.join(file_path,'test.csv'))
temp=[]
for img_name in train.Image:
    img_path=os.path.join(file_path,'Train Images',img_name)
    img=cv2.imread(img_path)
    img=cv2.resize(img,(64,64))
    temp.append(img)
train_x=np.asarray(temp)

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainloader = torch.utils.data.DataLoader(train_x, batch_size=4,
                                          shuffle=True, num_workers=2)
</code></pre>
<p>I tried this ,<br>
but it is  showing some module error.</p>

username: simaiden
cooked: <p>Those transforms are only for PIL Images, so you have two options:</p>
<ul>
<li>Using opencv to load the images and then convert to pil image using:</li>
</ul>
<pre><code class="lang-auto">from PIL import Image
img = cv2.imread('img_path')
pil_img = Image.fromarray(img).convert('RGB')  #img as opencv
</code></pre>
<ul>
<li>Load the image directly with PIL (better than 1)</li>
</ul>
<pre><code class="lang-auto">from PIL import Image
pil_img = Image.open(img_path).convert('RGB') # convert('L') if it's a gray scale image
</code></pre>
<p>You can resize the pil image directly or also using transforms. Also you shouldn’t load all the images into a list, because Dataloader load on the fly, but you have to set your own Dataset or using something like <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder</a></p>
MaskRCNN ONNX export issue

username: jinfagang
cooked: <p>Hello community.</p>
<p>I have exported MaskRCNN to ONNX, but my problem is that, I need convert this model into tensorrt.</p>
<p>The problem is:</p>
<ol>
<li>TensorRT does not support dynamic inputs and outputs;</li>
<li>MaskRCNN have dynamic inputs and outputs.</li>
</ol>
<p>I can fix MaskRCNN inputs to a certain size, but once that done, the outputs were also fitted.</p>
<p>Does anyone knows:</p>
<p>How to get a fixed output and a fixed input?? (Every images has different objects, so the final output is not exactly same).</p>
<p>I want a fixed output dimension, such as RPN output fixed in 6000 candidates. So that we can do the resting by ourseles.</p>

username: robot2007
cooked: <p>I checked this problem recently and I found your questions and answers here and there. I seems that we met the same problem for fasterrcnn and maskrcnn. It seems a deadend so far.</p>
Appropriate use of Convolutional neural networks

username: srl123
cooked: <p>Hi all,</p>
<p>the questions may be silly but I’ve not found a simple answer.</p>
<p>I’m trying to use the CNN to geophysical data inversion; I would like my net to produce as output some velocity models with velocity trends constant over the layers. This means that my ideal output is something like the blue curve in the left picture attached (the red is the net output). My net is characterized by 3 simple convolutional1d layers with LeakyRelu activation function and Kaiming weights inizialitation. What I don’t understand is:</p>
<ol>
<li>how to create a net whose output may try to generate flat results.</li>
<li>the loss function, reported in the right picture below shows those jumps I don’t really understand since I’m using no dropout …</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8d68963736982c84454e4be559846143d43d3ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/b8d68963736982c84454e4be559846143d43d3ac" title="Schermata del 2020-02-27 13-43-57"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8d68963736982c84454e4be559846143d43d3ac_2_690x388.png" alt="Schermata del 2020-02-27 13-43-57" data-base62-sha1="qn9uPutH2BZGIlEgEhs5uebCI0Y" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8d68963736982c84454e4be559846143d43d3ac_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8d68963736982c84454e4be559846143d43d3ac_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8d68963736982c84454e4be559846143d43d3ac_2_1380x776.png 2x" data-dominant-color="9B9998"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermata del 2020-02-27 13-43-57</span><span class="informations">1920×1080 236 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>any advise would be really helpful</p>

username: eduardo4jesus
cooked: <p>Can you put the network code here? More specifically, I am looking for the loss function and any non linear operation in the network output like to sort of subsample (decrease the resolution) of your output. A vanilla way would be to use <code>round</code>, <code>floor</code>, <code>ceil</code>. Can you specify or link something on the nature of the output. I guess that would be helpful, too.</p>
<hr>
<p>Is <a href="https://library.seg.org/doi/pdf/10.1190/geo2018-0838.1" rel="nofollow noopener">this</a> related?<br>
Fig 7, 9 and 12-c</p>
LinearLayer + BatchNorm1d with a 3D input

username: FiloCara
cooked: <p>Hello,</p>
<p>I would like to apply a <code>BatchNorm1d</code> just after a <code>LinearLayer</code>. My input is a 3D multivariate time series of shape <code>[batch_size, n_variables, timesteps]</code>. The <code>LinearLayer</code> performs the linear transformation on the third dimension so that the new shape is <code>[batch_size, n_variables, LinearLayer_out_features]</code>. My problem occurs with the <code>BatchNorm1d</code>, I would like to apply it on the third dimension but, for a 3D input, <code>BatchNorm1d</code> operation is done over the second dimension (even for a 3D tensor). Do you have any suggestion on how to do that? Something like axis permuation?</p>
<p>Thank you in advance.</p>
<p>Filo</p>

username: G.M
cooked: <p>Perhaps <code>input = input.transpose(1, 2)</code> or  <code>input = input.permute(0, 2, 1)</code>?</p>
Question about the CPU memory during inference

username: lyc6749
cooked: <p>After used the interpolate ，the gpu memory can not be released during inference。I try to use torch.cuda.empty_cache() after, but it can work only in feding the images one by one。when the images is concurrently fedding ，the GPU still increase。Can you tell me how to do？</p>

username: lyc6749
cooked: <p>is there anyone can help me ? i also have use torch.no_gard() and model.eval()</p>

username: lyc6749
cooked: <p>I install pytorch1.3 with cuda10, but run with cuda9.0. Is this will lead the problem?</p>

username: ptrblck
cooked: <p>If you install the binaries with a <code>cudatoolkit</code>, your local CUDA installation won’t be used.<br>
In case your memory increases, make sure to not store any tensors, which are still attached to the computation graph.<br>
This could usually happen, if you are storing the outputs, predictions, or the loss in a list.</p>
<p>To release the device memory, you would need to delete the object (<code>del x</code>) and can clear the cache afterwards.</p>

username: lyc6749
cooked: <p>I store the predictions in list, I have tried use <code>del x</code> ,but nothing happend。</p>

username: lyc6749
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="71252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In case your memory increases, make sure to not store any tensors, which are still attached to the computation graph.<br>
This could usually happen, if you are storing the outputs, predictions, or the loss in a list.</p>
<p>To release the device memory, you would need to delete the object ( <code>del x</code> ) and can clear the cache afterwards.</p>
</blockquote>
</aside>
<p>Can you tell me how to maksure no tensors stored?</p>

username: G.M
cooked: <p>Can u try deleting both the inputs, outputs, and predictions, then use <code>torch.cuda.empty_cache()</code>?<br>
A way to make sure that no useless tensors are stored is to declare the variables in the minimum scope possible and check all the global variables.</p>

username: lyc6749
cooked: <p>I just run the code of RDSNet based on mmdetection,  almost all variables are local variables。</p>
Split dataset into two new datasets (NOT subset)

username: Sarah_W
cooked: <p>Hi all together,</p>
<p>is it possible to split a torch dataset into two new datasets (train and validate) with indices in the range of 0 to validation length and 0 to train length? Because the random_split method returns indices in the range of 0 to the length of whole dataset for both datasets.</p>
<p>Thank you for your help!</p>

username: ptrblck
cooked: <p><code>random_split</code> returns two <code>Datasets</code> with the lengths as specified in the <code>lengths</code> argument:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.randn(1000, 1))
train_data, val_data = random_split(dataset, [900, 100])
print(len(train_data))
&gt; 900
print(len(val_data))
&gt; 100

train_data[0] # works
train_data[899] # works
train_data[900] # error

val_data[0] # works
val_data[99] # works
val_data[100] # error
</code></pre>
<p>Also answered <a href="https://discuss.pytorch.org/t/issues-with-torch-utils-data-random-split/22298/21">here</a>.</p>
What is the opposite of add_module?

username: pierrecurie
cooked: <p>So there’s an add_module method that lets me add modules, but how do I remove/replace them?  For the time being, del works, but feels really dirty.</p>
<pre><code class="lang-auto">class net(nn.Module):
    def __init__(self):
        self.relu = nn.ReLU()
    def change(self):
        try:
            del self.relu #DIRTY
        except:
            pass
        def fun(x):
            return x
        self.relu = fun #crashes without the del statement...
    def forward(self,x):
        return self.relu(x)
</code></pre>

username: tom
cooked: <p>You have funny ideas of cleanliness – just like we all do.<br>
More seriously:</p>
<ul>
<li>There is nothing wrong with <code>del</code>. This is the method Python and PyTorch forsee for removing something.</li>
<li>Even just overwriting a module with another without explicit <code>del</code> is OK.</li>
<li>What <em>does</em> seem a lot less clean (to me, but I’ll admit that I fancy myself to have very good tastes in this and well aligned with PyTorch at that) is that you sometimes have <code>self.relu</code> be a module and sometimes a function. This is not good, as modules are superspecial to PyTorch. I would recommend to either use functions throughout (e.g. <code>torch.nn.functional.relu</code>) or stay with modules when changing.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: pierrecurie
cooked: <p>Thanks for the fast response</p>
<ol>
<li>As long as del is appropriate (and doesn’t introduce bugs), I can live with that.</li>
<li>Indeed, I can overwrite modules with other modules; I shot myself in the foot because I’m trying to replace a module with a function.</li>
<li>That is indeed dirty, and I’ll think more about it.</li>
</ol>
MKL-DNN @ Windows

username: nexgus
cooked: <p>Hi there,</p>
<p>On Linux, PyTorch prebuild binary is built with MKL-DNN 0.21.</p>
<p>On WIndows (10), the prebuild binary is not built with MKL-DNN.</p>
<p>I wonder if it is possible to build PyTorch with MKL-DNN on Windows?</p>
<p>Augustus</p>

username: peterjc123
cooked: <p>Yes, it is now enabled in master.</p>
Writing a Padding Function

username: 111249
cooked: <p>I received a task on one of the platforms, it is necessary to manually implement padding with 1 layer 0 around each of the layers of the image channel. I tried everything already, but I didn’t have enough knowledge (<br>
Please help me decide.</p>
<pre><code class="lang-auto">input_images = torch.tensor(
      [[[[0,  1,  2],
         [3,  4,  5],
         [6,  7,  8]],

        [[9, 10, 11],
         [12, 13, 14],
         [15, 16, 17]],

        [[18, 19, 20],
         [21, 22, 23],
         [24, 25, 26]]],


       [[[27, 28, 29],
         [30, 31, 32],
         [33, 34, 35]],

        [[36, 37, 38],
         [39, 40, 41],
         [42, 43, 44]],

        [[45, 46, 47],
         [48, 49, 50],
         [51, 52, 53]]]])
</code></pre>
<pre><code class="lang-auto">def get_padding2d(input_images):
    padded_images= #here
    return padded_images
</code></pre>

username: ptrblck
cooked: <p>I’m quite sure I understand your use case wrong, but in case you want “standard” padding in the spatial dimensions, this should work:</p>
<pre><code class="lang-python">output = F.pad(input_images, [1, 1, 1, 1])
</code></pre>
<p>If that’s not your use case, could you print the desired output, please?</p>