akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to assign different weights for cross entropy loss?

John1231983: <p>I have a dataset A that contains the high-resolution and low-resolution of flower images. They are labeled as 6 classes. During training, I will randomly select 16 high resolution and 16 low-resolution image and feed them to the network (total we will feed 32 images–batch size =32). A cross-entropy loss will be used as a loss function to train the classifier.</p>
<p>I want to assign the higher weight for training high-resolution image and lower weight for training low-resolution image.  Is it possible in pytorch? And how to do it? The dataloader will return likes</p>
<pre><code class="lang-auto">high_res_image, class_high_res, low_res_image, class_low_res = data
</code></pre>
<p>If I used the loss likes</p>
<pre><code class="lang-auto">criterion = torch.nn.CrossEntropyLoss()
high_res_image, class_high_res, low_res_image, class_low_res = data
pred_high = networ(high_res_image)
pred_low = networ(low_res_image)
loss_total = 0.7* criterion(pred_high, class_high_res) + 0.3* criterion(pred_low, class_low_res) 
</code></pre>
<p>For above code, we just use the batch size of 16 instead of 32 because we feed the data seperately. Any good suggestion to use full 32 images and assign different weight for loss? Thanks.</p>

ptrblck: <p>You could avoid the reduction in your criterion by using:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss(reduction='none')
</code></pre>
<p>This will give you a loss values for each sample in the batch.<br>
After weighting each element, you could take the average (or normalize, sum etc.).</p>

John1231983: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: So, if we use reduction=none, then is the loss a scale or a vector? If it is scale, how can I weight the loss among sample in the batch?</p>
<p>Updated: Ok it reuturned a vector. Thanks</p>

John1231983: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: It worked. However, the loss returned a maxtrix (assume I worked on semantic segmentation with 2 class). The weights are assigned for each batch as 1.0, 2.0 3.0 and 4.0. The total loss will be take the average of the loss after multiplying with the weight. I have tried the <code>loss_total = torch.mean(loss * weights)</code> but it did not work (weights size of [3]). So, I have to use the loop for. Do you have any suggestion to make it vectorization for the loop?</p>
<p>This is my code</p>
<pre><code class="lang-auto">import numpy as np
num_class =2
b,h,w =4,8,8
input = torch.randn((b, 1, h, w), requires_grad=True)
target = torch.empty((b, h, w), dtype=torch.long).random_(num_class)
pred = torch.rand((b, num_class, h, w), dtype=torch.float)
criterion = nn.CrossEntropyLoss(reduction='none')
loss = criterion(pred, target)
weights= torch.from_numpy(np.asarray([1.0, 2.0, 3.0, 4.0]))

#loss_total = torch.mean(loss * weights)
loss_total =0
for i in range (b):
  loss_total += loss [i] * weights[i]
loss_total = torch.mean(loss_total / b)
print (loss_total)
</code></pre>
Data normalization

slavavs: <p>If I want to normalize the time series from 3 to 7, can I just use the np.interp (data, [3,7], [0,1) function? Or should I do it another way? I ask because my method does not work in eval () mode. The model is trained well, but as I understand it, Batchnorm layers do not work correctly in eval () mode.</p>
Output Pixel-wise Gradient Norm

vineeth_bhaskara: <p>Hi. I am looking for an efficient way of computing \hat{x} of dimensions (b x c x h x w) defined per sample as:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/0326eba6ed3046f60f0765238def1e75c9ac33d9.png" alt="image" data-base62-sha1="rSON7USZ6XHiYc5ituEqVGbgRr" width="284" height="89"> .</p>
<p>where x is the output of the same dimensions generated by a model with parameters \theta, and</p>
<p>i,j: index the height and width of the 2D output feature map<br>
k: indexes the channel dimension<br>
l: indexes the parameters.</p>
<p>How do I accomplish this with x.backward()?</p>
<p>If I did x.backward(torch.ones_like(x)), I would compute the sum of the gradients in the above equation in the place of sum of their absolute values.</p>
<p>Is there an efficient way of doing this?</p>
<p>EDIT:<br>
Brute-force way to do this would be to loop over each pixel in the output map and compute .backward(). But, that would be incredibly expensive. Is there an efficient way which avoids the loop?</p>

vineeth_bhaskara: <p>Could you please have a look? Thank you. <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/innovarul">@InnovArul</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Please don’t tag people, as this might discourage others to answer in the topic.<br>
We’ll have a look at all topics and try to help as best as we can <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

vineeth_bhaskara: <p>Sorry about that. Thank you.</p>
Push saved tensors to CPU memory?

cannon: <p>Hi, we’re trying to train a recurrent model which takes a sequence of images (ie. a video) as input and gives an output at every frame. We would like the gradients of the hidden state to go all the way through the video but there is not nearly enough GPU memory to store all of the saved tensors. Is there a good way to put these saved tensors into CPU memory as needed? The only way I can see right now is using a custom autograd.Function, but I need something that works with built in modules as well.</p>

albanD: <p>Hi,</p>
<p>You can easily move Tensors from cpu to gpu as these operations are differentiable in pytorch.</p>
<p>The problem though is that you won’t be able to move the buffers contained in the computational graph: there is no common interface on how to store these for each Function and so it will be special cases for each <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Pytorch autograd.grad:ValueError: only one element tensors can be converted to Python scalars

tengerye: <p>In the <a href="https://pytorch.org/docs/master/autograd.html?highlight=autograd%20grad#torch.autograd.grad" rel="noopener nofollow ugc">documentation</a> of  <code>torch.autograd.grad</code> , it is stated that, for parameters,</p>
<blockquote>
<p>parameters:</p>
<p>outputs (sequence of Tensor) – outputs of the differentiated function.</p>
<p>inputs (sequence of Tensor) – Inputs w.r.t. which the gradient will be returned (and not accumulated into .grad).</p>
</blockquote>
<p>I try the following:</p>
<pre><code class="lang-auto">a = torch.rand(2, requires_grad=True)
b = torch.rand(2, requires_grad=True)
c = a+b
d = a-b

torch.autograd.grad([c, d], [a, b]) #ValueError: only one element tensors can be converted to Python scalars
torch.autograd.grad(torch.tensor([c, d]), torch.tensor([a, b])) #RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>I would like to get gradients of a list of tensors w.r.t another list of tensors. What is the correct way to feed the parameters?</p>

albanD: <p>Hi,</p>
<p>The thing is that to backpropagate, you need to provide the original signal to backpropagate.<br>
For functions that return a scalar, then the It’s natural to use 1 to get gradients and so that’s what we do.<br>
For functions that return more than one value, there is no natural signal here. So you need to provide it with the <code>grad_outputs</code> argument.</p>
Stateless RNN vs Statefull

Michael_D: <p>Does it make sense, that <strong>Stateless RNN</strong> had a better performance than a <strong>Stateful RN</strong>?</p>
<pre><code class="lang-auto">  hidden = None
    y_pred = []
    for x_i in x.tolist():

        x_i = np.array([x_i])[:, np.newaxis]

        hidden = None # Is commented in statefull case.

        x_tensor = torch.Tensor(x_i).unsqueeze(0)
        prediction, hidden = rnn(x_tensor, hidden)
        hidden = hidden.data
        prediction = prediction.detach().numpy().flatten()
        y_pred.append(prediction)
```
![34|374x500](upload://naWxCncyBFsSp6yUcCeb5NZ7JAK.png)</code></pre>

andreaskoepf: <p>Could you explain what exactly you mean by a stateless RNN and what network topology you are using? Is rnn in your case a cell or a complete RNN? Do you pass a whole sequence or only one timestep input?</p>
<p>If the initial hidden state is not passed (None) internally a zero vector is used as the first hidden state. If conditioning on the initial hidden state is not beneficial it is possible that the ‘performace’ of the model is better than using an additional context vector.</p>

Michael_D: <p>I suppose it’s a complete RNN.</p>
<p>By Stateless, I assume that in evaluation (prediction mode) I provide <code>hidden = None</code> for each iteration instead of preserving it from output.</p>
<p>Code for RNN class:</p>
<details>
<summary>
RNN Class code</summary>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(RNN, self).__init__()
        
        self.hidden_dim=hidden_dim

        # define an RNN with specified parameters
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        
        # last, fully-connected layer
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x, hidden):
        batch_size = x.size(0)

        r_out, hidden = self.rnn(x, hidden)
        r_out = r_out.view(-1, self.hidden_dim)  
        output = self.fc(r_out)
        
        return output, hidden
</code></pre>
</details>

Michael_D: <p>Somehow I did an experiments again and I didn’t succeed to reproduce it.</p>
<p>As expected, stageful had a better results.</p>
DatasetFolder pickle loader

Noa_Lubin: <p>I’m trying to use DatasetFolder in order to use a pickle data loader with my transform and batch_size.</p>
<p>( I also tried adding to the transform: transforms.ToPILImage())</p>
<p>I tried this code which results in error because the whole list is loaded as one element (the data size is 1 and should be 500)</p>
<pre><code class="lang-auto">def pickle_loader(input):
    return pickle.load(open(input))

test_data= torchvision.datasets.DatasetFolder(root='.', loader=pickle_loader, extensions='.pickle', transform=transform)
test_loader = torch.utils.data.DataLoader(
        dataset=test_data,
        batch_size=batch_size,
        shuffle=False)



test_labels = []
for x in test_loader:
    x = Variable(x)
    out = model_conv(x)
    _, pred_label = torch.max(out.data, 1)
    test_labels.append(pred_label)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/noay/PycharmProjects/ML3/ex_9_code.py", line 206, in main
    for x in test_loader:
  File "/home/noay/anaconda2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 179, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/noay/anaconda2/lib/python2.7/site-packages/torchvision/datasets/folder.py", line 103, in __getitem__
    sample = self.transform(sample)
  File "/home/noay/anaconda2/lib/python2.7/site-packages/torchvision/transforms/transforms.py", line 49, in __call__
    img = t(img)
  File "/home/noay/anaconda2/lib/python2.7/site-packages/torchvision/transforms/transforms.py", line 76, in __call__
    return F.to_tensor(pic)
  File "/home/noay/anaconda2/lib/python2.7/site-packages/torchvision/transforms/functional.py", line 44, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got &lt;type 'list'&gt;
</code></pre>

sdv4: <p>If you are ok with your datum being a numpy array, you can change your loader to:</p>
<pre><code class="lang-auto">def pickle_loader(input):
    item = pickle.load(open(input, 'rb'))
    return item.values
</code></pre>

dejanbatanjac: <p>You can use the <code>pil_loader</code> directly or if possible the default_loader that will try to use high performance image loader <a href="https://github.com/pytorch/accimage" rel="nofollow noopener">accimage_loader</a></p>
<pre><code class="lang-auto">def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')


def accimage_loader(path):
    import accimage
    try:
        return accimage.Image(path)
    except IOError:
        # Potentially a decoding problem, fall back to PIL.Image
        return pil_loader(path)


def default_loader(path):
    from torchvision import get_image_backend
    if get_image_backend() == 'accimage':
        return accimage_loader(path)
    else:
        return pil_loader(path)
</code></pre>
How to set values at tensor at dynamic dim?

abze6213: <p>How to set values at tensor at dynamic dim? The sizes of the tensor parameter can be dynamic.<br>
For example, for tensor = torch.rand(size=(4, 5, 6)), how to create a function like below? :</p>
<pre><code class="lang-auto">def ff(tensor, dim, pos, value):
    # this function can also be applied to tensors 
    # with shapes like [7,5,6,8], [11, 5, 7,5,6,8], etc.
    if dim == 0:
        tensor[ pos, :, :] = value
    elif dim == 1:
        tensor[:, pos, :] = value
    elif dim == 2:
        tensor[:, :, pos] = value
</code></pre>

ptrblck: <p>Depending on the operation, you could e.g. use <code>tensor.index_copy_</code>, which accepts a <code>dim</code> argument.</p>
How to decide to send which output of lstm to next layer?

pahome: <p>I read some lstm examples and I found something confused me.</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
	def __init__(self,input_feature_dim,hidden_feature_dim,hidden_layer_num,batch_size,output_dim=10, use_cuda=False, dropout=0):
		super(LSTM, self).__init__()
		self.input_feature_dim=input_feature_dim
		self.hidden_feature_dim=hidden_feature_dim
		self.hidden_layer_num=hidden_layer_num
		self.batch_size=batch_size
		self.use_cuda=use_cuda

        self.lstm=nn.LSTM(input_feature_dim,hidden_feature_dim,
hidden_layer_num, dropout=dropout).double()
		self.linear1=nn.Linear(hidden_layer_num*hidden_feature_dim, output_dim).double()

	def init_hidden(self):
		return (t.zeros(self.hidden_layer_num, self.batch_size, self.hidden_feature_dim), 
				t.zeros(self.hidden_layer_num, self.batch_size, self.hidden_feature_dim))

	def forward_v1(self,input):
		batchSize = input.size(1)
		h0 = t.zeros(self.hidden_layer_num, batchSize, self.hidden_feature_dim).double()
		c0 = t.zeros(self.hidden_layer_num, batchSize, self.hidden_feature_dim).double()
		_, hn = self.lstm(input, (h0, c0))
		hn = hn[0].view(batchSize, self.hidden_layer_num*self.hidden_feature_dim)
		output = self.linear1(hn)

		return output
    def forward_v2(self,input):
		batchSize = input.size(1)
		h0 = t.zeros(self.hidden_layer_num, batchSize, self.hidden_feature_dim).double()
		c0 = t.zeros(self.hidden_layer_num, batchSize, self.hidden_feature_dim).double()
		lstm_output, hn = self.lstm(input, (h0, c0))
		output = self.linear1(lstm_out[-1].view(self.batch_size, -1))

		return output
</code></pre>
<p>What I confuse is I saw two version of forward like above:<br>
<code>forward_v1</code> and <code>forward_v2</code>.</p>
<p>They pass different output of lstm into next full-layer<br>
I don’t know which one is appropriate. Is there any difference?</p>

phan_phan: <p>When calling <code>lstm_output, hn = self.lstm(input, (h0, c0))</code>, you get :</p>
<ul>
<li>
<p><code>lstm_output</code> from <strong>the last layer</strong>, for <strong>all the time steps</strong>.<br>
(shape:<code>[seq_len, batch, num_directions*hidden_size]</code>)</p>
</li>
<li>
<p><code>hn</code> from <strong>all the layers</strong>, for <strong>the last time step</strong><br>
( <img src="https://discuss.pytorch.org/images/emoji/apple/warning.png?v=9" title=":warning:" class="emoji" alt=":warning:"> tuple of two tensors, <em>hidden</em> and <em>cell</em>, both of shape:<code>[num_layers*num_directions, batch, hidden_size]</code>)</p>
</li>
</ul>
<p>(<a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="nofollow noopener">It’s all in the docs</a>)</p>
<p>So in your code, <code>forward_v1</code> will give an output computed from the last lstm layer, for all the time steps (and I think the <code>.view()</code> line may cause problems, it does not correspond to the shape of <code>hn[0]</code>), whereas <code>forward_v2</code> will give an output computed also from the last lstm layer (since you called <code>lstm_out[-1]</code>, but only from the last time step.</p>
Matrix Prediction

lam: <p>Hi,</p>
<p>i recently got into neural networks and machine learning and have a question.<br>
I built a small network which is supposed to take an input matrix (24x30) and an output matrix (24x30) and learn how to predict the output (basically 2d grayscale images). The idea is, that the output matrix is a subsequent development of the input matrix (ie a temporal follow up).</p>
<p>My approach is to use a 2d convolution layer, relu, pooling, and a linear activation. Now my problem is that the prediction is a 1x30 vector, which makes sense due to the linear layer and the .view part of the network.<br>
Now i clearly dont understand well enough the network but it would be nice for someone to help me out here.</p>
<p>The code is as follows:</p>
<pre><code class="lang-auto">class ConvNet(torch.nn.Module):
      def __init__(self):
        super(ConvNet, self).__init__()
        self.conv1=torch.nn.Conv2d(in_channels=args.batchsize,out_channels=16, kernel_size=5)
        self.fc1=torch.nn.Linear(in_features=2080, out_features=384)
        self.fc2=torch.nn.Linear(in_features=384,out_features= 90)
        self.fc3=torch.nn.Linear(in_features=90,out_features=30)

      def forward(self, x):
        x=x.unsqueeze(0)
        x=torch.nn.functional.max_pool2d(torch.nn.functional.relu(self.conv1(x)), (2,2))
        x=x.view(-1, self.num_flat_features(x))
        x=torch.nn.functional.relu(self.fc1(x))
        x=torch.nn.functional.relu(self.fc2(x))
        x=self.fc3(x)
        return x

      def num_flat_features(self, x):
        size=x.size()[1:]
        num_features=1
        for s in size:
          num_features *= s
        return num_features 
</code></pre>
Fusion tensors into one multidimensional

RicLaGra: <p>Hey torch-expert :),<br>
I need to create my trainloader using Dataloader class from pytorch library.<br>
The problem is that I’ve 1 tensor of feature extracted, 1 tensor for labels and 1 tensor 28x28 extracted by fashion_mnist dataset. How can I combine different tensor into one, such that I can send into my model? (resnet18). These 3 tensors are separated at this moment due by necessity. In the specific, I have (128,2), (128) and (28x28) tensors.<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>How would you like to combine these tensors?<br>
Resnet expects an image tensor of <code>[batch_size, 3, 224, 224]</code> as its input.<br>
I’m currently unsure, how to construct such a tensor using your input values.</p>

RicLaGra: <p>Exactly. I have to construct a tensor with that shape. How can I do? Thanks</p>

RicLaGra: <p>i need to replicate the train loader using these information. The main reason that i’m working only on features. After that, i need to compact these tensor into one, in order to be accepted by resnet18. These information are saved into<br>
<strong>class_list[y].append([instance, single_instance[-1], labels])</strong> where,<br>
each object is a tensor (features, 28x28 tensor, 28 tensor label).<br>
When i try to convert this into Tensor, i obtain an error of shape caused by 28x28 tensor…</p>

ptrblck: <p>I’m really not sure how you would like to create a tensor of <code>[3, 224, 224]</code> using your data.<br>
Your raw data has <code>128*2+128+28*28 = 1168</code> values, while the expected image tensor has <code>150528</code> values.<br>
Would you like to upsample the features somehow?<br>
While this could make sense for the MNIST data, I’m not sure, how you would like to treat your features.</p>
Multiple torch.nn.Parameter won't show up in parameter list

algogator: <pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super().__init__()
        ...       

        self.s = [torch.nn.Parameter(torch.arange(1, embedding_dim+1, dtype=torch.float),  requires_grad=True)]

</code></pre>
<p>I can’t see self.s in</p>
<pre><code class="lang-auto">for p in model.parameters():
    print(p)
</code></pre>
<p>but it does show up when it’s</p>
<pre><code class="lang-auto">self.s = torch.nn.Parameter(torch.arange(1, embedding_dim+1, dtype=torch.float),  requires_grad=True)
</code></pre>
<p>Is there another way to initialize multiple learn able parameters?</p>

Prashant_Kalikotay: <p>Looks like you are putting it as a list in the case where its not working.</p>
<pre><code class="lang-auto">self.s = [torch.nn.Parameter(torch.arange(1, embedding_dim+1, dtype=torch.float),  requires_grad=True)]    #The box bracket is for list initialisation.
That could be the problem
</code></pre>
<p>list is a variable and I think only tensors can be stored as parameters</p>

vainaijr: <p>if you do</p>
<pre><code class="lang-auto">self.s = nn.ParameterList([torch.nn.Parameter(torch.arange(1, embedding_dim+1, dtype=torch.float),  requires_grad=True)])
</code></pre>
<p>then it would show up</p>
Help me to make the code this simple loss

wonchulSon: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5754087e84fdea3a57aaeda07a5dce9655de7784.png" alt="image" data-base62-sha1="csxzYf3d4aqx2U9Q9t1RLnnrlSA" width="638" height="236"></p>
<p>I tried to realize this loss using pytorch but as beginner it is difficult for me…<br>
Can you make this operation to the pytorch code?<br>
It could be very helpful for me.<br>
Thank you.</p>
How to visualize the output tensor image from an autoencoder of dtype = torch.float32

n0obcoder: <p>Once my autoencoder is trained, its output is a tensor of shape [bs, h, w], where h and w are the height and width of the input image</p>
<p>i want to visualize an output image, so i choose the 1st one<br>
img = output[1, :, :]<br>
now the shape og img is [h, w]<br>
and its dtype is torch.float32</p>
<p>Now i have 2 options here,</p>
<ul>
<li>
<p>i can convert the output pixel values to uint8 [0 to 255]<br>
cv2.imshow(  (img.numpy()*255).astype(np.uin8)  )</p>
</li>
<li>
<p>or i can simply use the imshow of matplotlib which does not convert the pixel values to 0 to 255 uint8<br>
plt.imshow(img, cmap = ‘gray’)</p>
</li>
</ul>
<p>Which one is more appropriate and shows the image without losing much inforamtion ?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

LemonPy: <p>Hey,</p>
<p>Much of what you are asking is dependent on the data you have (not just it’s type, but it’s sparseness and length) and on the needs of your visualization.</p>
<p>Given that you have reduced it to these 2 options (why? maybe you want to use an external tool, so another option would be - <code>torchvision.utils.</code> <code>save_image</code> and export it to that handler) I can only say that the fact you are using cmap=‘gray’ in the second option, means that you don’t have an RGB(A) format of the data and that you are doing some transformation (or conversion) to the data on the second option as well(from the <a href="https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.imshow.html" rel="nofollow noopener">documentation</a>: “The Colormap instance or registered colormap name <strong>used to map scalar data to colors</strong>”).</p>
Question on image classification

hlsafin: <p>When doing image classification, what can someone do to increase accuracy? I have 20,000 images<br>
I am using a pretrained model (senet154), but I’m only getting 87 percent accuracy. what else can be suggested to increase accuracy other than more images and better quality of images.<br>
Thanks!</p>

andreaskoepf: <p>For some tips or inspiration check out this thread: <a href="https://forums.fast.ai/t/how-we-beat-the-5-epoch-imagewoof-leaderboard-score-some-new-techniques-to-consider/53453" rel="nofollow noopener">https://forums.fast.ai/t/how-we-beat-the-5-epoch-imagewoof-leaderboard-score-some-new-techniques-to-consider/53453</a></p>

Prashant_Kalikotay: <p>You can try different kind of weight initialisations.<br>
Eg: Xavier , He etc.</p>
<p>You can also augment the data.</p>
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 3.94 GiB total capacity; 3.36 GiB already allocated; 13.06 MiB free; 78.58 MiB cached)

NarasimmanSaravana19: <p>My PyTorch version</p>
<p>print(torch. <strong>version</strong> )<br>
1.1.0</p>
<p>Need to downgrade the PyTorch version? And how?<br>
Please guide me…Thanks</p>

DerekGloudemans: <p>I’ve run into this error when I try to load a batch onto the GPU that is too large to fit. The solution in this case is to select a smaller batch size.</p>
<p>I’ve also had this issue when I stop code during execution, I think because the GPU memory is not properly cleared in this case. The solution for me here has been to restart the kernel.</p>
<p>I’d start with checking these first. If neither of these solutions works for you, you’ll have to get some more detailed help.</p>
<p>Lastly, the following command line command provides a display of the GPU resources and their utilization, and can be useful for diagnosing issues:</p>
<blockquote>
<p>watch nvidia-smi -l 1</p>
</blockquote>

NarasimmanSaravana19: <p><a class="mention" href="/u/derekgloudemans">@DerekGloudemans</a> Thanks for the support, let me check thanks…<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

rustagiadi95: <p><strong>This error usually occurs when your are running the model without clearing the previous gradients using <code>model.zero_grad()</code>, as the gradients take up memory and get piled up to eat your GPU if not cleared.</strong></p>
<p>While the usual practice is using <code>model.zero_grad()</code> while training, hence this should not occur while training until the very first iteration of the data is costly in terms of memory on your gpu - in that case, try to use <code>torch.cuda.empty_cache()</code> to clear the cache or try to detach any useless tensors using tensor.detach() to not evaluate and store their gradients in your GPU.</p>
<p>While on testing or validation side, we don’t want to create and store gradients for any forward pass so try to run your testing or validation code under the torch.no_grad() context manager.</p>
<p>Hope this helps,<br>
Thanks.<img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"></p>
Compute loss for each batch?

John1231983: <p>I have an input size of BxCxHxW and a label size of BxHxW, where B is the batch size. We often compute the loss likes</p>
<pre><code class="lang-auto">criterion= nn.CrossEntropyWithLoss()
pred = model(input)
loss = criterion(pred, label)
</code></pre>
<p>If I want to compute the loss for each batch, then I will use</p>
<pre><code class="lang-auto">criterion= nn.CrossEntropyWithLoss()
pred = model(input)
loss = 0
for i in range (B):
    loss += criterion(pred[i:i+1,...], label[i:i+1,...])
</code></pre>
<p>Does the second approach provide same result as the first approach? Thanks</p>

ptrblck: <p>It won’t produce the same loss, as the default reduction in <code>nn.CrossEntropyLoss</code> calculates the mean loss value for the batch.<br>
If you set <code>reduction='sum'</code>, you should get the same loss.<br>
However, if you need the loss for each batch, just disable the reduction via <code>reduction='none'</code> (<a href="https://discuss.pytorch.org/t/how-to-assign-different-weights-for-cross-entropy-loss/56672/2">related topic</a>).</p>
Windows install broken

Mika_S: <p><code>pip3 install torch===1.2.0 torchvision===0.4.0 -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>does not work anymore since the links page does not have 1.2.0 anymore:<br>
<a href="https://download.pytorch.org/whl/torch_stable.html" class="onebox" target="_blank" rel="nofollow noopener">https://download.pytorch.org/whl/torch_stable.html</a></p>

ptrblck: <p>Could you try to install the wheels again, as they might have been accidentally taken down as described <a href="https://github.com/pytorch/pytorch/issues/26664" rel="nofollow noopener">here</a>.</p>
Is the rendering of the pytorch docs destroys your CPU on Google Chrome?

pinocchio: <p>I was trying to just look things up on the pytorch documentation but my Google Chrome Helper (renderer) starts consuming at least 75ish% of my CPU and sometimes going up to over 104% and sometimes even the GPU appears at the top of my Activity Monitor (something I had never ever seen).</p>
<p>What is going on? Why is the pytorch documentation consuming my computer so violently?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0db5c816637276b97e012a1fedb51ed081c125a.png" alt="24%20PM" data-base62-sha1="w5aVQ5dFjiWqSxhohaIoBRGelho" width="644" height="90"></p>
Convolution with filters of different sizes

aleingrosso: <p>Hello, I would like to build a layer which convolves an input image with many filters of different size (5x5, 10x10, 15x15…). As far as I understand, this is not possible using nn.functional.Conv2d. Using a for loop over filters turns out to be slower than simply setting the filter size to be the one of the bigger filter in Conv2d and padding the smaller filters. This seem to involve a lot of unnecessary operations, though. Is there any way to parallelize such convolutions on the GPU?</p>
Index tensor by tensor of multidimensional indices

phkb: <p>I have a 4D tensor x, and a 2D index tensor i of shape (N, 3), where i[n] is an index over the first 3 dimensions of x. I would like to extract the x values at these indices. After some trial and error, I found that the following does what I want:</p>
<p>result = x[i[:, 0], i[:, 1], i[:, 2]]</p>
<p>I was wondering if there was a better way to do so. I looked at torch.gather and torch.index_select, but they seem to be for 1D indices. Any ideas?</p>

ptrblck: <p>Would splitting the index work?</p>
<pre><code class="lang-python">x[idx.split(1, 0)]
</code></pre>

phkb: <p>Yes, thanks! I didn’t know about the split function, it’s very useful.</p>
I want to dividing pre-processing code of CNN

bemorept: <p>Hi, All.<br>
This is a pre-processing code of CNN.</p>
<p>data_transforms = {</p>
<p>‘train’: transforms.Compose([</p>
<p>transforms.RandomResizedCrop(224),</p>
<p>transforms.RandomHorizontalFlip(),</p>
<p>transforms.ToTensor(),</p>
<p>transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])</p>
<p>]),</p>
<p>‘val’: transforms.Compose([</p>
<p>transforms.Resize(256),</p>
<p>transforms.CenterCrop(224),</p>
<p>transforms.ToTensor(),</p>
<p>transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])</p>
<p>]),</p>
<p>}</p>
<p>I want to change this code to divie code that convert all input images(512x256) to 256x256 images.(x2 augmentation effect)</p>
<p>How can I code?</p>
<p>Thanks.<br>
Best,<br>
<span class="mention">@bemoregt</span>.</p>

Rishav_Sapahia: <p><code>transforms.Resize((256,256))</code> should work, if you want to resize them.</p>

bemorept: <p>I want to not resize but divide.</p>
<p>Thanks at any rate …</p>

Rishav_Sapahia: <p>Could you please tell what you mean by divide.<br>
If you are referring to dividing (512) into two parts of 256 and 256, then I guess random crop could work in this case but I am not sure if all the crops are exclusive of each other or not. If they are not, then you can increase the augmentation by more than 2* factor and of they are, then I guess it would be 2* augmentation.</p>
RuntimeError: the derivative for ‘running_mean’ is not implemented

sfancc: <p>when I use x = nf.batch_norm(x, running_mean=self.bn1_mean, running_var=self.bn1_var, weight=None, bias=None, training=False, momentum=0.1, eps=1e-05) in my net, I defined the self.bn1_mean and self.bn1_var, but when running train, there com out RuntimeError: the derivative for ‘running_mean’ is not implemented,  why ??</p>

spanev: <p>Hi <a class="mention" href="/u/sfancc">@sfancc</a>,</p>
<p>This is related to this <a href="https://github.com/pytorch/pytorch/issues/8930#issuecomment-487819546" rel="nofollow noopener">GitHub issue</a>.</p>
<p><code>running_mean</code> and <code>running_var</code> are only initialized by those value (<code>bn1_mean</code> and <code>bn1_var</code>) but are not trainable parameters.</p>

sfancc: <p>import torch.nn.functional as nf</p>
How to save memory in this case with torch.max

Zhang_Fuyang: <p>I have a tensor <code>x</code> in shape of [100, 64, 256, 256]. <code>x = [x0, x1, ..., x99]</code>, so each <code>x_i</code> is [64, 256, 256].</p>
<p>What I want is to get maximum tensor of the total 100 x_i for each i except itself.</p>
<p>E.g. for <code>x0</code>, I would like to get <code>torch.max((x1, x2, x3, ..., x99))</code> and for <code>x1</code>, I would like to get ‘torch.max((x0, x2, x3, …, x99))’</p>
<p>How to write this in memory efficiency way?</p>
<p>The way I do is fist get an <code>index</code> tensor [100, 99].</p>
<pre><code class="lang-auto">index = torch.tensor(
[[1, 2, 3, 4, 5, ..., 99],
 [0, 2, 3, 4, 5, ..., 99],
 [0, 1, 3, 4, 5, ..., 99],
 ...
 [0, 1, 2, 3, 4, ..., 98]]
)
</code></pre>
<p>then, <code>out = torch.max(x[index], 1)</code></p>
<p>BUT, this may cost so much memory, I think the problem is <code>x[index]</code> this thing is too big. Especially I need gradient to do back-propagation.</p>
<p>Is there any other better way to do this?</p>

andreaskoepf: <p>You could run a loop over the 100 entries in the first dimension, e.g.:</p>
<pre><code class="lang-auto">import torch

#x = torch.rand(100, 64, 256, 256)
x = torch.rand(100, 64, 3, 3) # reduced for testing

def max_exclusive(x, i):
    a,b = x[:i], x[i+1:]
    if a.nelement() == 0:
        return torch.max(b, 0)[0]
    if b.nelement() == 0:
        return torch.max(a, 0)[0]
    return torch.max(torch.max(a, 0)[0], torch.max(b, 0)[0])

#y = torch.cat([max_exclusive(x, i).unsqueeze(0) for i in range(x.size(0))])

# or manually if you do not want to store intermediate results:
y = x.new_empty(x.size())
for i in range(x.size(0)):
    y[i] = max_exclusive(x, i)

print(y.size())
</code></pre>
Torchvision transform for segmentation masks

n0obcoder: <p>I am trying to train a segmentation model, so I have a pairs of grayscale image and its mask. To convert these into tensors, I am using torchvision  transforms, i.e.</p>
<pre><code class="lang-auto">from torchvision import transforms
</code></pre>
<p>For the grayscale image</p>
<pre><code class="lang-auto">img_transform = transforms.Compose([transforms.ToPILImage(), transforms.ToTensor()])    
img = img_transform(img)
</code></pre>
<p>which converts my img to a tensor of dtype torch.float32<br>
and for the masks</p>
<p>But I cant use the same transform on the mask as the mask cant have float values. It needs to have int values representing the classes…</p>
<p>Pls help me with the transform that I could use to convert my masks to tensors without converting the mask values to float.</p>
<p>Thanks in advance… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>You could load the mask inside your <code>Dataset</code>'s <code>__getitem__</code> via <code>PIL.Image.open</code> and convert it to a tensor manually using <code>mask = torch.from_numpy(mask)</code>. This would make sure to keep the dtype.</p>

n0obcoder: <p>yes thats what i tried. i used cv2.imread to read the  images and converted the images to tensor using</p>
<pre><code class="lang-auto">torch.tensor(numpy_array) 
</code></pre>
<p>it seems like the correct approach.<br>
Thanks for replying <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Mixed data type convertion

z160896: <p>I have a numpy array of tensor values, [tensor(1), …, tensor(n)].<br>
how to convert it to numpy array: [1, …, n].</p>
<p>Thanks<br>
Robert</p>

ptrblck: <p><code>torch.cat(list_of_tensors).numpy()</code> should work.<br>
Let me know, if you get any errors.</p>

z160896: <p>Thanks, no, I got this error: TypeError: cat(): argument ‘tensors’ (position 1) must be tuple of Tensors, not numpy.ndarray</p>

ptrblck: <p>It seems you are already dealing with numpy arrays, not PyTorch tensors.<br>
In that case you could try to use <code>np.concatenate</code>.</p>

z160896: <p>Yes,  the array is numpy,  but contains tensor elements. I need to convert these tensor elements to none tensor values.</p>

z160896: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="56573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>concatenate</code> .</p>
</blockquote>
</aside>
<p>I fixed like this:<br>
tensor_arr = [torch.tensor(1), …, torch.tensor(n)]<br>
var = [e.item() for e in tesnsor_arr]</p>
How to restrict the output of neural network such that some dimension greater than others

ElleryL: <p>For example; My neural network <code>model</code> takes <code>x</code> as input,  and output two dimensional vector <code>y = model(x)</code> where <code>y = (y1,y2)</code> and I want to restrict the output <code>0&lt; y1 &lt; y2 &lt; 1</code>. where restricting to domain <code>(0,1)</code> is easy as I can just apply <code>sigmoid()</code> as activation function at last but how to nicely satisfy the sign constraint and the allow backprogation?</p>
<p>I’m not sure if I shuffle the <code>y</code> as  <code>y = sorted(y)</code> as last layer allows backprogration at all</p>

agadetsky: <p>Sorting is not differentiable. You can do the following trick:</p>
<pre><code class="lang-auto">y1, y2 = model(x)
y2 = torch.logsumexp([y1, y2])
</code></pre>
<p>Since LogSumExp is smooth maximum, <code>y2</code> will also be greater than <code>y1</code> (see <a href="https://en.wikipedia.org/wiki/Smooth_maximum" rel="nofollow noopener">https://en.wikipedia.org/wiki/Smooth_maximum</a>).</p>
<p>Then apply <code>sigmoid</code> or any other non-decreasing function that transforms input to <code>(0, 1)</code> range.</p>
Are tensors stored in nn.Module.register_buffer trainable?

ihexx: <p>I’m trying to implement a trainable hyperparameter as a member of an nn.Module inheritor</p>
<p>I want it to be moved to the same device as the rest of the module with <code>child_module.to(my_device)</code></p>
<p>I have done this with<br>
<code>self.log_alpha= torch.tensor(0., requires_grad=True)</code><br>
then overriden <code>child_module.to()</code> to move it manually then call the parent method as so:</p>
<pre><code class="lang-auto">    def to(self, *args, **kwargs):
        super(Trainer, self).to(*args, **kwargs)
        device, _, _ = torch._C._nn._parse_to(*args, **kwargs)
        self.log_alpha.to(device)
        self.conf.trainer_device = device
</code></pre>
<p>this little hack works just fine, but I read that the correct way to do this was with <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_buffer" rel="nofollow noopener">register_buffer</a><br>
as so:<br>
<code>self.register_buffer('log_alpha', torch.tensor(0., requires_grad=True))</code></p>
<p>However, doing it this way, <code>log_alpha</code> doesn’t get updated by its optimizer, but it does with my old hack.<br>
Am i using <code>register_buffer()</code> incorrectly, or is it not supposed to be used with trainable params?</p>

ptrblck: <p>If <code>self.log_alpha</code> is trainable (requires gradients), you should wrap it in an <code>nn.Parameter</code>, which will also make sure to move the tensor to the device.</p>
Target format for MultiLabelMarginLoss

nihalnayak: <p>I am using <code>MultiLabelMarginLoss</code> loss function. The target values to the loss function are the indices followed by -1.<br>
For example, if the correct label indices are <code>3 and 4</code>, the target tensor looks something like this -</p>
<pre><code class="lang-auto">target = torch.tensor([3, 4, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]).to(device)
</code></pre>
<p>Is this the correct format for the target?</p>

nihalnayak: <p>This is the correct tensor format for <code>MultiLabelMarginLoss</code>.</p>
Question - Define the loss function with given output and labels

judepark: <p>Hi there, I am newbie student studying natural language processing.</p>
<p>What i am writing is named entity recognition for personal study.</p>
<pre><code class="lang-auto">import torch
from torch import nn
import numpy as np


class LSTM(nn.Module):
    def __init__(self, word_vocab, embed_dim=50, hidden_size=50, num_class=18) -&gt; None:
        super().__init__()
        self.word_vocab = word_vocab
        self.word_embedding = nn.Embedding(len(word_vocab), embedding_dim=embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_size, num_layers=2, dropout=0.5, bidirectional=True)
        self.fc = nn.Linear(embed_dim * 2, num_class)

    def forward(self, x_train):
        word_embed = self.word_embedding(x_train)

        output_word, (hidden, state_word) = self.lstm(word_embed)
        outputs = self.fc(output_word)

        return outputs.float()
</code></pre>
<p>Above is the simple lstm network code that all i’ve wrote.</p>
<p>Below is the training code.</p>
<pre><code class="lang-auto">from model.loader import get_data_loader
from model.lstm import LSTM
from torch import nn
from torch.optim import Adam


if __name__ == '__main__':
    train_data_loader, test_data_loader, word2idx = get_data_loader(path='./rsc')
    lstm_model = LSTM(word2idx)
    parameters = filter(lambda p: p.requires_grad, lstm_model.parameters())

    c = nn.CrossEntropyLoss(ignore_index=0)
    optimizer = Adam(parameters, lr=1e-4)

    for step, (x_train, y_train) in enumerate(train_data_loader):
        output = lstm_model(x_train)
        loss = c(output, y_train)

        optimizer.zero_grad()
        loss.backward()

        optimizer.step()

    pass
</code></pre>
<p>Shape of output and label is torch.Size([32, 99, 18]) torch.Size([32, 99]) which means first is [batch_size * max_seq_length * num_tag], second is [batch_size * max_seq_length].</p>
<p>My question is as follows:</p>
<ul>
<li>How to use cross entropy loss function?</li>
<li>There is a  tokens in training data, So I have to exclude  tokens when calculate loss. It can be solved through <strong>nn.CrossEntropyLoss(ignore_index=0)</strong> ?</li>
</ul>
<p>Thank you for reading this questions.</p>

judepark: <p>Self-Answering.</p>
<p>To use <strong>nn.CrossEntropyLoss</strong>, manipulated shape of output and label.</p>
<pre><code class="lang-auto">outputs.view(-1, outputs.shape[2]) # dim =&gt; (batch_size * max_seq_length) * num_tag
y_train.view(y_train.shape[0] * y_train.shape[1]) (batch_size * max_seq_length)
</code></pre>
<p>For example, let batch size is 32, max_seq_length = 80, num _tag = 18, then output shape will be (2560 * 18), label shape will be (2560, ).</p>
The .grad is zero, the value change

Huaxiu_Yao: <p>I see a very strange scenario, for one parameter. The .grad is zero, while the value is change. What’s wrong? Thanks.</p>

albanD: <p>Hi,</p>
<p>There are few threads in this forum about this.<br>
Things like l2 regularization, or momentum terms in Adam will change the parameters even when the gradients are 0.</p>

Huaxiu_Yao: <p>Thanks. I found that.</p>

chen_dongdong: <p>Hi, albanD, is there any way to avoid changing the parameters and relevant statistics in such cases when the gradient is zeros.</p>

albanD: <p>That depends on your optimizer really. Chaging these parameters and statistics is the “right thing to do” if you really want to apply the optimizer.<br>
If you want to freeze some weights manually, the right way to do it is not to give it to the optimizer or you can try and set <code>weight.grad = None</code>.</p>
Cuda is available only for a short time after reboot on PyTorch

scarecrow21: <p>OS: Ubuntu 18.04<br>
GPU: Nvidia - RTX 2060, NVIDIA-SMI 430.26, Driver Version: 430.26, CUDA Version: 10.2<br>
Laptop: Lenovo Legion Y540</p>
<p>I setup my GPU for PyTorch based on this post: <a href="https://forums.fast.ai/t/successful-ubuntu-18-04-with-igpu-for-xserver-and-nvidia-gpu-for-cuda-work-setup/20128/9" rel="nofollow noopener">https://forums.fast.ai/t/successful-ubuntu-18-04-with-igpu-for-xserver-and-nvidia-gpu-for-cuda-work-setup/20128/9</a></p>
<p>I have a hybrid setup. Intel Integrated GPU is used for display and NVIDIA for deep learning. Since CUDA comes bundled with PyTorch, I haven’t installed it separately. I have Installed pytorch using: <code>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</code>.</p>
<pre><code class="lang-auto">$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation Device 3e9b
01:00.0 VGA compatible controller: NVIDIA Corporation Device 1f11 (rev a1)
</code></pre>
<p><code>torch.cuda.is_available()</code> returns True only for a short while after I boot my machine. After that, it returns False.</p>
<p>I don’t understand what the issue is. Cuda seems to available for short window of time after I boot my machine. If I miss that window, I need to reboot to be able to use the GPU in PyTorch. Please help!</p>

ptrblck: <p>Could you check some power settings and make sure the GPU isn’t deactivated to save battery power?</p>

scarecrow21: <p>I did, I did not find any such setting. In the NVIDIA X Server application, the PRIME profile is set to ‘Intel (power save)’ but that is because I’m using the Intel card for display.<br>
I will experiment some more and see in what cases does this happen. Will update this thread at a later time.</p>
Tensor Inverse in parallel over mutliple GPUs

ozdi: <p>I want to run over multiple GPUs in parallel torch.inverse(). I saw this post <a href="https://discuss.pytorch.org/t/matmul-on-multiple-gpus/33122" class="inline-onebox">Matmul on multiple GPUs</a>. Which shows that if you have multiple tensors allocated to each GPU matmul will be run in parallel. I was able to replicate this behavior for matmul but when I try to do the same thing for torch.inverse() it seems to run sequentially when I watch nvidia-smi. Any ideas?</p>

ptrblck: <p>Could you post your code snippet so that we could have a look?</p>

ozdi: <p>Thank you for the quick reply. As you can see it greatly mirrors the other post.</p>
<pre><code class="lang-python">import torch

ngpu = torch.cuda.device_count()
# This is the allocation to each GPU.
lis = []
for i in range(ngpu):
    lis.append(torch.rand(5000,5000,device = 'cuda:'+ str(i)))

# per the matmul on multiple GPUs post this should already be in parallel to my understanding
# but doesnt seem to be based on watch nvidia-smi
C_ = []
for i in range(ngpu):
    C_.append(torch.inverse(lis[i]))

</code></pre>