akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Parameter dependent on another parameter

username: algogator
cooked: <p>I’m trying to change the weights for a Conv2d after randomly initializing it</p>
<p><code>self.convs.weight = torch.nn.Parameter(data=w, requires_grad=False)</code></p>
<p>But this only trains weights for convs and not w (a tensor)</p>
<p>And convs.weight can only be assigned a Parameter if I remember correctly</p>
<p>How do I train w and not convs.weight?</p>

username: JuanFMontesinos
cooked: <p>use a functional if you don’t want to keep tracking of the weights. What are you trying to do?</p>
<p>you are making weights to be w, what does it mean training w but not weights</p>

username: algogator
cooked: <pre><code class="lang-auto">a = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
b = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
c = a + 1
d = torch.nn.Parameter(c, requires_grad=True,)
for epoch in range(n_epochs):
    yhat = d + b * x_train_tensor
    error = y_train_tensor - yhat
    loss = (error ** 2).mean()
    loss.backward()
    print(a.grad)
    print(b.grad)
    print(c.grad)
    print(d.grad)
</code></pre>
<p>Printing the gradients out I get<br>
None<br>
tensor([-0.8707])<br>
None<br>
tensor([-1.1125])</p>
<p>What I’m trying to do is learn a and not just d<br>
If that makes sense?</p>
out.backward(torch.Tensor([2.0])) doesn't work in Pytorch 1.0.3 but works in Pytorch 1.0.2

username: rwamit
cooked: <p>Hi Guys,</p>
<p>I am trying to run <code>out.backward(torch.Tensor([2.0]))</code> and I get a error shape mismatch, (which makes sense) but I am try to run the same in Pytorch 1.0.2 it is working if I print grad post this operation elements in the matrix get multiplied by 2.0.</p>
<p><code>out.backward(torch.tensor([2.0]))</code> throws error<br>
<code>RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([1]) and output[0] has a shape of torch.Size([]).</code></p>
<p>Infact, <code>out.backward(torch.tensor([1.0]))</code> doesn’t work as well in the latest version.</p>

username: albanD
cooked: <p>0 dimensional tensors have been introduced to represent tensors with no dimension and a single input value.<br>
You can use <code>torch.tensor(2.0)</code> to get a 0 dimensional Tensor that contains the value 2.</p>

username: rwamit
cooked: <p>Thanks <a class="mention" href="/u/alband">@albanD</a>, it works now but I get different output for x.grad if I use<br>
Output 1: <code>(out.backward(torch.tensor([2.0]))</code> in pytorch version <code>1.2</code>)<br>
A 2x2 square matrix where each values is 6</p>
<p>Output 2: (out.backward(torch.tensor(2.0)) in pytorch version <code>1.3</code>)<br>
A 2x2 square matrix where each values is 9</p>
<p>May you please explain the difference in results?</p>
<p>Values of x,y and z are as follows:<br>
<code>x = torch.ones(2, 2)</code><br>
<code>x.requires_grad_(True)</code><br>
<code>y = x+2</code><br>
<code>z = y * y * 3</code><br>
<code>out = z.mean()</code><br>
<code>out.backward(torch.tensor(2.0))</code><br>
<code>print(x.grad)</code></p>
<p>OUTPUT - tensor([[9., 9.],<br>
[9., 9.]])<br>
or</p>
<p>tensor([[6., 6.],<br>
[6., 6.]])</p>

username: albanD
cooked: <p>Your full function here is:</p>
<p>f(x) = ( sum_i (xi+2)^2 * 3 ) / size(x) = (sum_i (xi+2)^2 * 3 ) / 4<br>
df(x)/dxj = 2 * (xj + 2) *3 / 4<br>
given that you start with xi = 1 for all i.<br>
the gradients you expect with no scaling is: 2 * (1 + 2) *3 / 4 = 4.5<br>
So if you multiply the result by 2, you get 9 as expected.</p>
<p>I am not sure why you get 6 for your pytorch 1.2 code, but I would say this is most likely a small bug in your test code? do your start with torch.zeros() for example instead of torch.ones()?</p>

username: rwamit
cooked: <p>Sorry <a class="mention" href="/u/alband">@albanD</a> it was mistake from my end. That should return 9.0 as expected, I am using different function for whose derivative is 3.0 and hence when multiplied by 2.0 = 6.0</p>
<p>Thanks again for your help!</p>
BERT for multi-class text classification

username: Er_Hall
cooked: <p>Hi all,</p>
<p>Is it possible to use BERT from transformers library for multi-class text classification problem? Is there any tutorial.</p>
<p>Thanks</p>

username: tom
cooked: <p>The huggingface transformers library has a class for it and they also show a GLUE example:</p>
<p><a href="https://huggingface.co/transformers/model_doc/bert.html#transformers.BertForSequenceClassification" class="onebox" target="_blank" rel="nofollow noopener">https://huggingface.co/transformers/model_doc/bert.html#transformers.BertForSequenceClassification</a></p>
<p>and the examples/run_glue.py from their github repository.</p>
<p>Best regards</p>
<p>Thomas4</p>
Serialize jit optimized graphs and visualize them

username: vdantu
cooked: <p>I was curious to know if anyone know how I could serialize a jit graph into something that we can visualize. I am currently using graph-&gt;dump() and node-&gt;dump()'s to debug. I wanted to know how we could visualize the graph.</p>

username: driazati
cooked: <p>You can print out the graph as valid Python code with <code>my_module.code</code>. We used to have a <a href="https://github.com/pytorch/pytorch/blob/5949bb27b59f4f209313c42e6d28a07bb0af8e7e/torch/contrib/_graph_vis.py" rel="nofollow noopener">helper</a> to turn a <code>Graph</code> into a visualization via GraphViz, but it wasn’t maintained and was deleted.</p>

username: vdantu
cooked: <p>Thanks. I used .graph and .code to look at the graph. But wanted to get something more visual :).  I hadn’t seen the helper. I also have a interesting case. I want to use this from the C++ side <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> . Is there a similar dump function that can be invoked from C++ side or do we need to write one.</p>

username: driazati
cooked: <p>We don’t have anything like that for C++ and probably won’t (it’s outside of our scope), but you should have all the tools / APIs for <code>Graph</code> and <code>Node</code> to make something for your own purposes.</p>

username: vdantu
cooked: <p>A little late, but thanks for the response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is it possible for me to pinpoint an undetached tensor by plotting the computation graph?

username: entslscheia
cooked: <p>I am facing the following error:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>I am 90% percent sure this is because that I have some variables are used across different batches without being detached. Can I pinpoint those variables by looking at the computation graph using tensorboard? If so, what parameters should I pass to <code>add_graph()</code>?</p>

username: entslscheia
cooked: <p>Based on my understanding, this error occurs mostly when some input tensor with grad_fn that was created in previous computation is fed to the model. Normally, I can pinpoint the problem by checking the grad_fn of each input tensor, however, this time this error occurs when I am using Allennlp command line, in which a lot of data feeding operations are hided from us. I guess I can still to the same thing as previous, but with more effort to dissect the source code of Allennlp.</p>

username: tom
cooked: <p>You are still bound to have written some glue code that deals with tensors?<br>
Generally, those graphs get unwieldy much faster than just the inputs.</p>
<p>One of the things that easily happens is to not keep the leaf tensors:<br>
<code>x = torch.randn(1, 2, 3, requires_grad=True).cuda()</code> is not a leaf tensor, so computing something with <code>x</code>, doing backward, computing something with <code>x</code> and then doing backward again, it will try to backward twice through the <code>.cuda()</code> step.</p>
<p>Best regards</p>
<p>Thomas</p>

username: entslscheia
cooked: <p>Sorry, I didn’t understand you. Are you just trying to give me an example that can lead to the error I run into?<br>
Are there any general principles to locate the undetached tensors when running into this kind of error?</p>

username: tom
cooked: <p>Yeah, sorry.<br>
Most people start by looking at what they feed into the models, but you said it didn’t apply to you.</p>
<p>Best regards</p>
<p>Thomas</p>
Multi-label text calssfication

username: lzp
cooked: <p>Hello, I would like to ask, what is the format of multi-label text classification pytorch for output data processing? I am a newbie and feel the lack of information.</p>

username: ptrblck
cooked: <p>A multi-label classification (in my understanding) means that each output can belong to more than a single class.<br>
Is that correct or are you dealing with a multi-class classification, where each output belongs to exactly one particular target class?</p>
<p>For text classification I guess your output would most likely have the shape <code>[batch_size, nb_classes, seq_length]</code>.<br>
Depending on your actual use case (multi-label vs. multi-class):</p>
<ul>
<li>multi-label: your target should have the same shape as your output and you would use e.g. <code>nn.BCEWithLogitsLoss</code> as the criterion</li>
<li>multi-class: your target have the shape <code>[batch_size, seq_length]</code> and contain the class indices in the range <code>[0, nb_classes-1]</code>.</li>
</ul>
Cuda multiprocessing

username: aman_bharat
cooked: <p>Hi,<br>
I am trying to run multiprocessing in my python program. I created two processes and passed a neural network in the one process and some heavy computational function in the other. I wanted the neural net to run on GPU and the other function on CPU and thereby I defined neural net using cuda() method. But when I run the program the got the following error:<br>
<strong>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</strong></p>
<p>So I tried with  <strong>spawn</strong>  as well as  <strong>forkserver</strong>  start method, but then I got the other error:<br>
<strong>RuntimeError: cuda runtime error (71) : operation not supported at …/torch/csrc/generic/StorageSharing.cpp:245</strong></p>
<p>I have tried python3 multiprocessing and torch.multiprocessing both but nothing worked for me.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Do you have a bit more context about the second error? What is the stack trace that you get?</p>

username: aman_bharat
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a95132f9bbafda5aaeb2992d2bc441e06b6e9fd2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a95132f9bbafda5aaeb2992d2bc441e06b6e9fd2" title="1_LI"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a95132f9bbafda5aaeb2992d2bc441e06b6e9fd2_2_690x316.jpeg" alt="1_LI" data-base62-sha1="o9QEYd4438nxf2oD8TJiX49AFz4" width="690" height="316" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a95132f9bbafda5aaeb2992d2bc441e06b6e9fd2_2_690x316.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a95132f9bbafda5aaeb2992d2bc441e06b6e9fd2_2_1035x474.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a95132f9bbafda5aaeb2992d2bc441e06b6e9fd2.jpeg 2x" data-dominant-color="495052"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1_LI</span><span class="informations">1068×490 166 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> what is the reason for this error on cuda? Do you know where this is coming from?</p>

username: ptrblck
cooked: <p>Usually we see these kind of errors for IPC using CUDATensors on Windows (e.g. as explained <a href="https://pytorch.org/docs/stable/notes/windows.html#cuda-ipc-operations" rel="nofollow noopener">here</a>).<br>
However, based on the output it looks like a Linux system is being used.</p>
<p><a class="mention" href="/u/aman_bharat">@aman_bharat</a> could you post a reproducible code snippet, so that we can have a look?</p>
Finetunning object detection

username: amin_sabet
cooked: <p>I’m trying to finetune fastRcnn for object detection on a custom dataset. During each epoch after when I use the optimizer.step() function to update parameters, the model is not working in the evaluation part of the code. However, when I comment the optimizer.step() function I can get the output of the model during the evaluation.  I appreciate any comments.</p>
<pre><code class="lang-auto">model = faster_rcnn.fasterrcnn_resnet50_fpn(pretrained=True, min_size=800)
num_classes = 31
in_features = model.roi_heads.box_predictor.cls_score.in_features

#replace the pre trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
epochs = 2
model.to(device)

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)


for e in range(epochs):
  model.train()
  loss_history = list()
  for frame, label in training_loader:
      label = [{k: v.to(device) for k, v in t.items()} for t in label]
      frame = frame.to(device)
      losses_dict = model(frame,label)
      losses = sum(loss for loss in losses_dict.values())
      loss_history.append(losses.cpu().detach().numpy().item())
      optimizer.zero_grad()
      losses.backward()
      optimizer.step()
  model.eval()
  for data, label in validation_loader:
      data = data.to(device)
      output = model(data)
      print(output)

</code></pre>
<p><strong>When I don’t use the optimizer.step() the print function is:</strong></p>
<p>[{‘boxes’: tensor([[ 814.4761,    4.1513, 1232.4717,  750.0000],<br>
[ 553.6875,  124.6697, 1333.0000,  573.3456],<br>
[ 633.8101,  279.4264, 1333.0000,  661.9217],<br>
[ 619.6268,    0.0000, 1055.0597,  739.1709],<br>
[ 440.1299,  210.7000, 1203.8578,  750.0000],<br>
[ 421.0741,  663.9869, 1100.3203,  750.0000],<br>
[ 604.5875,    1.8037, 1333.0000,  325.6623],<br>
[ 349.5059,  146.8048,  807.5912,  750.0000],<br>
[ 937.7691,  210.4664, 1333.0000,  750.0000],<br>
[ 745.6071,  387.2783, 1173.9806,  750.0000]]), ‘labels’: tensor([12, 12, 12, 12, 12, 12, 12, 12, 12, 12]), ‘scores’: tensor([0.0570, 0.0562, 0.0555, 0.0540, 0.0523, 0.0520, 0.0512, 0.0502, 0.0501,<br>
0.0501])}]</p>
<p><strong>When I use the optimizer.step() the print function is:</strong></p>
<p>[{‘boxes’: tensor([], size=(0, 4), grad_fn=), ‘labels’: tensor([], dtype=torch.int64), ‘scores’: tensor([], grad_fn=)}]</p>
Best/ good pre-trained models for extracting feature from image

username: zimmer550
cooked: <p>Hello guys. As the title suggests, I am interested in extracting features from an egomotion image (an image taken from a car’s front dashcam). At the moment, I am using Resnet-50 but I don’t know whether this is the best one for this purpose. Honestly, I don’t understand how you would choose the appropriate neural network for this purpose.</p>
<p><strong>Some more context:</strong><br>
I am building a machine learning model for driving-related purposes. I want to predict the trajectory of objects that my car sees. For now, my plan is to extract features from the image at every time step and input those to an RNN encoder. With that in mind, what neural network is most likely to extract appropriate features. After extracting the features, I push them through a single fully-connected layer before going to the RNN encoder stage.</p>

username: tdchaitanya
cooked: <p>By Resnet-50 do you mean an imagenet pre-trained one? If you have enough data, one option to get some good features would be to train a network (like Resnet or VGG) from scratch using self-supervised approaches like rotation prediction. This will help learn some features specific to your data.</p>

username: donJuan
cooked: <p>Always better to init with pretrained weights, even with own data</p>
Shared embedding in PyTorch

username: Aiman_Mutasem-bellh
cooked: <p>Dear friends,</p>
<p>How I can share the same embeddings (words or characters) between encoder and decoder?</p>

username: vdw
cooked: <p>Just define the embedding first and give it to the encoder and decoder is initial parameter like I did <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/autoencoder/textrnnae.py" rel="nofollow noopener">here for example</a>. the important lines are:</p>
<pre><code>self.embedding = nn.Embedding(self.params.vocab_size, self.params.embed_dim)
...
self.encoder = Encoder(device, params, self.embedding)
self.decoder = Decoder(device, params, self.embedding, self.criterion)</code></pre>

username: Aiman_Mutasem-bellh
cooked: <p>Thank you Mr <a class="mention" href="/u/vdw">@vdw</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: vdw
cooked: <p>Mr vdw is my dad. I’m just Chris :). Happy coding!</p>

username: Aiman_Mutasem-bellh
cooked: <p>Oh, I’m sorry Mr.Chris, and thank you again for your support <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to mannually initialize weight

username: XW324
cooked: <p>I’m trying to initialize weight of a conv layer like below,</p>
<p>class some_model(nn.Module):</p>
<pre><code>def __init__(self):

    super(some_model, self).__init__()

    self.blur=nn.Conv2d(1,1,kernel_size=3,padding=1,bias=False)

    self.blur.weight[0][0][0][0]=0.0751
    self.blur.weight[0][0][0][1]=0.1238
    ...
def forward(self, x):
    xblur=self.blur(x)
    return xblur
</code></pre>
<p>But I got the following error message:<br>
ValueError: can’t optimize a non-leaf Tensor</p>
<p>What is the correct way to initialize weight mannually?</p>

username: XW324
cooked: <p>I should instead do:<br>
self.blur.weight.data[0][0][0][0]=0.0751</p>
<p>weird flex but okay…</p>

username: albanD
cooked: <p>Hi,</p>
<p>Using <code>.data</code> i discouraged, you should do:</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.blur.weight[0][0][0][0]=0.0751
    self.blur.weight[0][0][0][1]=0.1238
</code></pre>

username: XW324
cooked: <p>what does torch.no_grad() do here?</p>
<p>I know normally we would put evaluation calls inside it, since gradients are not needed. But why do we need it when initializing weights?</p>

username: albanD
cooked: <p>Since everything that happens inside this block is not recorded. The inplace operation that change the value of your weights is not recorded either. And so it is a good way to initialize the value.</p>

username: XW324
cooked: <p>I guess my question then is what does ‘being recorded’ or ‘not being recorded’ do?</p>

username: albanD
cooked: <p>That is not such an easy question.<br>
In some sense, ignore these operations when computing gradients.</p>
Efficiently build decoder output tensor

username: tomnomnom
cooked: <p>I’m working on a sequence-to-sequence model with attention. I have a loop in the decoder where the output tensor is iteratively built, one timestep at a time. I’ve tried several methods for making this loop more efficient but I’m having some trouble and I wanted to ask the community to see if there was a better way to do this.</p>
<p>Method 1 often causes an OOM error - I think due to the fact that the entire computation graph is repeatedly copied in the list.<br>
Method 2 is by far the fastest, but the graph is lost and therefore the model obviously fails to train.<br>
Method 3 works but is very slow.</p>
<p>Is there some way to “temporarily” detach from the graph so that the outputs can be stored in the list outside the loop but then reattach without losing the autograd history? Or perhaps there is some other more efficient method that I’m missing?</p>
<p><strong>Method 1 (store predictions in a list then stack)</strong></p>
<pre><code class="lang-auto">preds_list = []
probs_list = []
for timestep in range(max_len):
    decoder_out, decoder_h, attn_weights = self.decoder(dec_x, h, src)
    top_value, top_index = torch.max(decoder_out, dim=1, keepdim=True)
    preds_list.append(top_index.detach())
    probs_list.append(decoder_out)
preds = torch.stack(preds_list, dim=1)
probs = torch.stack(probs_list, dim=1)

return preds, probs
</code></pre>
<p><strong>Method 2 (retain detached predictions in a list then stack)</strong></p>
<pre><code class="lang-auto">preds_list = []
probs_list = []
for timestep in range(max_len):
    decoder_out, decoder_h, attn_weights = self.decoder(dec_x, h, src)
    top_value, top_index = torch.max(decoder_out, dim=1, keepdim=True)
    preds_list.append(top_index.detach())
    probs_list.append(decoder_out.detach())
preds = torch.stack(preds_list, dim=1)
probs = torch.stack(probs_list, dim=1).requires_grad_()

return preds, probs
</code></pre>
<p><strong>Method 3 (pre-allocate tensor and set timestep indices)</strong></p>
<pre><code class="lang-auto">probs = torch.zeros(batch_size, max_len, self.output_size)
for timestep in range(max_len):
    decoder_out, decoder_h, attn_weights = self.decoder(dec_x, h, src)
    top_value, top_index = torch.max(decoder_out, dim=1, keepdim=True)
    preds_list.append(top_index.detach())
    probs[:, timestep] = decoder_out
preds = torch.stack(preds_list, dim=1)

return preds, probs
</code></pre>
Input with different dimensions

username: donJuan
cooked: <p>I’m working with multiple matrix like inputs which has different height and width. For example assume that we have two input with shapes x_1: (400, 300) and x_2: (500, 250). My ambition is to be ably to concatenate the variables as x_c = concatenate((x_1, x_2)) for further processing in the network.</p>
<p>So my problem lies in making these two variable in the same shape. To make x_1 in the same shape as x_2 we need to upsample the first axis and downsample the second axis. I have found the following solution by using different kernel sizes. One can have multiple of these operation and not just down sample directly e.g. for adjusting the width one can have two stages, one kernel with size (1, 20) and another with size (1, 31). This may give a smoother down sampling.</p>
<p>My question is then if this is a valid approch? Will I lose too much information? Do you have any experience with this?</p>
<pre><code class="lang-python">import torch.nn as nn
import torch

class convDimAdjust(nn.Module):
	def __init__(self):
		super(convDimAdjust, self).__init__()

		self.input_channel = 1
		
		self.convWidth = nn.Conv2d(in_channels = self.input_channel,
											out_channels = self.input_channel,
											stride = 1,
											kernel_size = (1, 51))

		self.convHeight = nn.ConvTranspose2d(in_channels = self.input_channel,
														out_channels = self.input_channel,
														kernel_size = (101, 1))

	def forward(self, x):

		x = self.convWidth(x)
		x = self.convHeight(x)

		return x


def main():
	
	# Map from (400, 300) to (500, 250) using CNN 

	m = convDimAdjust()

	# Batch, channel, h, w
	x = torch.ones((1, 1, 400, 300))	

	_ = m(x)

	print(_.shape) 

	# Prints: torch.Size([1, 1, 500, 250])

</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>If you want to learn this upsampling/downsampling, I think it makes sense to do it this way.<br>
If you want classic upsampling/downsampling (with no learnable parameters), there are functions like <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.upsample" rel="nofollow noopener">upsample</a> available that use things like linear of bilinear sampling.</p>
SGD: Batch-size - Convergence tradeoff on single GPU?

username: Pkpklong
cooked: <p>Hi everyone, pytorch and MLnewbie here,</p>
<p>I’m trying to do matrix factorization via SGD, where each row in my dataset describes one entry in the target matrix. I’m trying to make use of my GPU to maximize parallel computation. In order to achieve this, I know that a larger batch must be used. I’m running into an issue where I need to consider a batch size and accuracy tradeoff. The larger the batch size, the more the GPU can be utilized, but in exchange my model converges to a local minima due to a lack of noise in the batch.</p>
<p>So my question is, is there some way where I can achieve the same performance gains of using a larger batch sizes while retaining the convergence  results of smaller batch sizes?</p>
<p>Any input would be appreciated. Thanks!</p>

username: KFrank
cooked: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="Pkpklong" data-post="1" data-topic="58113" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c68b51/48.png" class="avatar"> Pkpklong:</div>
<blockquote>
<p>The larger the batch size, the more the GPU can be utilized, but in exchange my model converges to a local minima due to a lack of noise in the batch.</p>
<p>So my question is, is there some way where I can achieve the same performance gains of using a larger batch sizes while retaining the convergence  results of smaller batch sizes?</p>
</blockquote>
</aside>
<p>You could try just adding some noise to the gradients computed<br>
using the larger batch sizes.</p>
<p>(For additive noise this is the same as subtracting learning-rate<br>
times noise from your parameters after the SGD optimizer step.)</p>
<p>You could consider applying multiplicative noise to you gradients<br>
so that the size of your noise scales, in effect, with the size of<br>
your gradients.</p>
<p>You would now have three interacting parameters to play with:<br>
batch size, size (and type) of noise, and learning rate.</p>
<p>(I don’t know how well this is likely to work, but it is certainly<br>
something people experiment with.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Pkpklong
cooked: <p>Hi Frank!</p>
<p>Originally, I thought L2 regularization already acts as the additive noise, but that’s to the loss rather than to the gradient itself. I’ll definitely try to play around adding noise to runs with larger batch sizes and see how it goes. Thanks a lot for the suggestions!</p>
<p>I also had another idea of using multiprocessing during the training phase, where I have a Pool of processes and I can just send multiple small mini batches to it to send to the GPU, so that gradients can be calculated in parallel. It is contrasting HogWild, where there are multiple processes training the same model on the same mini-batch. Have you seen a pattern like this?</p>
Shape Annotation Best Practices

username: hdkgr
cooked: <p>Hi everyone,</p>
<p>I want to improve documentation of my code to make dimensionality of tensors easily readable. Currently I’m doing this with a mix of of python type hints for dtypes and ‘manual’ code comments for shapes.<br>
Now I came across this package that enables explicit shape-hints: <a href="https://github.com/ofnote/tsalib" rel="nofollow noopener">https://github.com/ofnote/tsalib</a></p>
<p>My question: Is there any built-in functionality in pytorch that provides something similar? (Or is this planned in the future?)</p>
<p>What are best practices for annotating shapes?</p>

username: ekshaks
cooked: <p>Thanks for mentioning tsalib - I’m the tool’s author.</p>
<p>Unfortunately, there is hardly any convention right now for shape annotation - in tsalib, we’ve introduced a shorthand string notation for naming shapes (and their arithmetic derivatives) and  piggybacked on to Python’s type annotations feature to make tensor shapes explicit.  This allows shape specs to be inline with code and IMO, this is far better than adhoc comments.</p>
<p>The annotations can be extended to include dtypes too but we haven’t added that yet. Something like ‘b,t,d;f32’. Do you think that will be helpful?</p>
<p>Would love to learn more how other developers work with named shapes and how tsalib can help them.</p>

username: hdkgr
cooked: <p>Hi Nishant,<br>
thank you for the reply and the work behind tsalib. Tbh, I was reluctant to add another dependency such as your package to the project.</p>
<p>Now that pytorch 1.3 has been released, named tensors provide everything I was looking for built-in. <a href="https://pytorch.org/blog/pytorch-1-dot-3-adds-mobile-privacy-quantization-and-named-tensors/" rel="nofollow noopener">https://pytorch.org/blog/pytorch-1-dot-3-adds-mobile-privacy-quantization-and-named-tensors/</a></p>
How to use sampler on distributed system by pytorch

username: Cooper_Lee
cooked: <p>I define a kind of sampler by myself, and I want to training the model on distributed system. I do not know how to modify the code to adopt to distributed system. The following is the code of sampler.</p>
<pre><code class="lang-auto">import torch
import numpy as np


class CategoriesSampler():

    def __init__(self, label, n_batch, n_cls, n_per):
        self.n_batch = n_batch
        self.n_cls = n_cls
        self.n_per = n_per

        label = np.array(label)
        self.m_ind = []
        for i in range(max(label) + 1):
            ind = np.argwhere(label == i).reshape(-1)
            ind = torch.from_numpy(ind)

            self.m_ind.append(ind)

    def __len__(self):
        return self.n_batch
    
    def __iter__(self):
        for i_batch in range(self.n_batch):
            batch = []
            # print(len(self.m_ind))
            classes = torch.randperm(len(self.m_ind))[:self.n_cls]
            for c in classes:

                l = self.m_ind[c]

                pos = torch.randperm(len(l))[:self.n_per]
                batch.append(l[pos])
                # else:
                #     print(c)

            batch = torch.stack(batch).t().reshape(-1)
            yield batch

</code></pre>

username: ptrblck
cooked: <p>You could adapt or derive from <a href="https://github.com/pytorch/pytorch/blob/b7b73e43c004a46a501073cfb7af3a8bb9acb977/torch/utils/data/distributed.py#L7" rel="nofollow noopener"><code>DistributedRandomSampler</code></a>.<br>
There is nothing really special going on besides the splitting of the indices.</p>
Change vector value while holding gradient

username: Pcoenen
cooked: <p>Hi,</p>
<p>I’m trying to create my own layer. The layer does the following: it multiplies certain values if the input tensor and if the new value is bigger than the value at a given index, the current value gets replaced with this new value.</p>
<p>The forward function of the layer looks like this:</p>
<pre><code class="lang-auto">for vect in x:
         for index, child_indexes in self.multi_order:
                val = torch.ones(1, requires_grad=True)
                for i in child_indexes:
                    val = val * (1 - vect[i])
                val = 1 - val
                vect[index] = torch.max(vect[index], val)
        return vect
</code></pre>
<p>When I try to train a model on this, I get the flowing error</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor <span class="chcklst-box fa fa-square-o fa-fw"></span>], which is output 0 of SelectBackward, is at version 106432; expected version 106431 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>The location where the error occurs is obviously on this line:</p>
<pre><code class="lang-auto">vect[index] = torch.max(vect[index], val)
</code></pre>
<p>How can I fix this error ? And am I correctly setting the <code>requires_grad</code> parameters ?</p>
<p>Thanks in forward</p>

username: albanD
cooked: <p>You should set the requires_grad parameter to True only if you want to get the gradients for a given Tensor by reading it’s .grad field (or passing it to autograd.grad). So here I don’t think you want to set it.</p>
<p>I am not sure to understand your function. You have a return at the end of the first iteration of the loop? Also what does <code>self.multi_order</code> contains?</p>
Gradient checkpointing, transfert beetween gpu and cpu

username: Chocaponk
cooked: <p>I’m trying to transfer memory between GPU and CPU with gradient checkpointing, i use this code <a href="https://pytorch.org/docs/stable/_modules/torch/utils/checkpoint.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/checkpoint.html</a></p>
<p>In my forward function i tried to use:</p>
<pre><code class="lang-auto">tensor = args[0]
tensor = tensor.cpu()
tensor = tensor.cuda()
</code></pre>
<p>When i transfer the memory to the cpu, _grad_fn of tensor is set to None.<br>
Does anyone know how to avoid this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Within the first forward call during the checkpoint, the gradients are not tracked, so any operation will set the grad_fn to None. You can check whether or not the gradients are tracked with <code>torch.is_grad_enabled()</code>.</p>
Broadcasting - Simple

username: snip3r77
cooked: <pre><code class="lang-auto">x = torch.tensor([[1, 2, 3], [4, 5, 6]])    # x has shape (2, 3)  to convert to (1 ,2, 3,)
c = torch.tensor([1, 10, 11, 100])        # c has shape (4,)   to convert to  (4, 1, 1)

# x * y will have shape of (4, 2, 3)
y = c.view(-1, 1, 1)
</code></pre>
<p>so when we have (2, 3 ) for x and ( 4, ) for <strong>c</strong>onstant<br>
first, we extend (4) to (4,1)</p>
<p>but</p>
<p>x shape (2,3)<br>
y shape (4,1)</p>
<p>will not work because of broadcasting semantics<br>
<a href="https://pytorch.org/docs/stable/notes/broadcasting.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/broadcasting.html</a></p>
<p>hence, we need to</p>
<p>(1, 2, 3)<br>
(4, 1 ,1)</p>
<p>it can’t be this<br>
( 1 at the rear instead at the front for x is because it will fail the broadcast semantics? )</p>
<p>( 2 , 3 , 1 ) and<br>
( 4, 1 , 1 )</p>
<p>Am I correct up to here?<br>
Is there an easier way to work on broadcasting?</p>
<p>Thanks.</p>

username: spanev
cooked: <p>Hi,</p>
<p>If I understand correctly you want to do this:</p>
<pre><code class="lang-auto">x_expanded = x.expand(1, -1, -1)
x_expanded * y
</code></pre>
<p>and it will give you a the broadcasted <code>(4,2,3)</code> Tensor.</p>

username: snip3r77
cooked: <p>But I thought we need to convert them to</p>
<p>x has shape of (2, 3)  to convert to (1 ,2, 3,)<br>
c which is a constant has shape of (4,)  to convert to  (4, 1, 1)</p>
<p>and we get by</p>
<pre><code class="lang-auto">y = c.view(-1, 1, 1)

tensor([[[  1,   2,   3],
         [  4,   5,   6]],

        [[ 10,  20,  30],
         [ 40,  50,  60]],

        [[ 11,  22,  33],
         [ 44,  55,  66]],

        [[100, 200, 300],
         [400, 500, 600]]])
</code></pre>
Pytorch1.1.0 with cuda9.0 Installation Error?

username: junhocho
cooked: <p>I wanted to install Pytorch1.1.0 along with cuda9.0<br>
Referring <a href="https://pytorch.org/get-started/previous-versions/" rel="nofollow noopener">https://pytorch.org/get-started/previous-versions/</a>, Used following command.<br>
Following command fails, attempting to install pytorch1.1.0 of cuda10</p>
<p><code>conda install pytorch=1.1.0 cuda90 -c pytorch</code></p>
<pre><code class="lang-bash">Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/junho/anaconda3

  added / updated specs:
    - cuda90
    - pytorch=1.1.0


The following NEW packages will be INSTALLED:

  cuda90             pytorch/linux-64::cuda90-1.0-h6433d27_0
  cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.0.130-0
  ninja              pkgs/main/linux-64::ninja-1.9.0-py37hfd86e86_0
  pytorch            pytorch/linux-64::pytorch-1.1.0-py3.7_cuda10.0.130_cudnn7.5.1_0

</code></pre>
<p>causing</p>
<pre><code class="lang-auto">AssertionError:
The NVIDIA driver on your system is too old (found version 9010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
terminate called recursively
terminate called without an active exception
Aborted (core dumped)
</code></pre>
<p>The following command works.</p>
<p><code>conda install pytorch torchvision cudatoolkit=9.0 -c pytorch</code></p>
<pre><code class="lang-auto">Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/junho/anaconda3

  added / updated specs:
    - cudatoolkit=9.0
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-9.0            |       h13b8566_0       237.0 MB
    pytorch-1.1.0              |py3.7_cuda9.0.176_cudnn7.5.1_0       376.8 MB  pytorch
    torchvision-0.3.0          | py37_cu9.0.176_1         3.7 MB  pytorch
    ------------------------------------------------------------
                                           Total:       617.5 MB

The following NEW packages will be INSTALLED:

  cudatoolkit        pkgs/main/linux-64::cudatoolkit-9.0-h13b8566_0
  ninja              pkgs/main/linux-64::ninja-1.9.0-py37hfd86e86_0
  pytorch            pytorch/linux-64::pytorch-1.1.0-py3.7_cuda9.0.176_cudnn7.5.1_0
  torchvision        pytorch/linux-64::torchvision-0.3.0-py37_cu9.0.176_1
</code></pre>

username: smth
cooked: <p>for 1.1.0 it was <code>conda install pytorch=1.1.0 cudatoolkit=9.0 -c pytorch</code> as far as I can remember.</p>
Space2Channel Operation (Inverse of PixelShuffle)

username: Saeed_Izadi1
cooked: <p>Hello all,<br>
Does anyone know if there is any module or function for doing the inverse of nn.PixelShuffle()? That is reduce spatial dimension and add to the channel.</p>
<p>Thanks,<br>
Saeed</p>

username: IcarusWizard
cooked: <p>This code should work. For some reasons I don’t know, the community don’t like to add this function into the core implementation.</p>
<pre><code class="lang-auto">def squeeze(x, r):
    [B, C, H, W] = list(x.size())
    x = x.reshape(B, C, H//r, r, W//r, r)
    x = x.permute(0, 1, 3, 5, 2, 4)
    x = x.reshape(B, C*(r**2), H//r, W//r)
    return x
</code></pre>
How to obtain results from certain layer from a pretrained model which isn't written using Sequential()?

username: Yiru_Shen
cooked: <p>Hi, I tried to return the output from an intermediate layer of a <strong>pretrained model.</strong> Have searched many times but most of the examples are for model that’s built with Sequential() method or from model Zoo. My model is a customized model (<strong>no Sequential is used here</strong>) and when I called model._modules, it only returns an OrderDict, but <strong>the order of operation is not the same as during forward()</strong>.  So directly calling module(x) doesn’t work! My question is how to do this for model that’s not neither Sequential() nor VGG (models that are already in model zoo)?</p>
<p>Here’s my code but it doesn’t work:</p>
<pre><code class="lang-auto">class SelectiveLayer(nn.Module):
	def __init__(self, to_select, model):
		super(SelectiveLayer, self).__init__()
		for key, module in model._modules.items():
			self.add_module(key, module)
		self._to_select = to_select

	def forward(self, x):
		list = []
		for name, module in self._modules.items():
			print(name, module, x.shape)
			x = module(x)
			if name in self._to_select:
				list.append(x)
		return list

class PerceptualLoss(nn.Module):
 	def __init__(self, pretrained_model, params):
 		super(PerceptualLoss, self).__init__()
 		self.pretrained_model = pretrained_model
 		self.features = SelectiveLayer("conv5", pretrained_model)

 	def forward(self, x):
 		x = self.features(x)
 		return x
</code></pre>

username: JuanFMontesinos
cooked: <p>Hi, you can use hooks. <a href="https://www.kaggle.com/sironghuang/understanding-pytorch-hooks" rel="nofollow noopener">Small tutorial</a><br>
<a href="https://pytorch.org/docs/stable/nn.html?highlight=hook#torch.nn.Module.register_forward_hook" rel="nofollow noopener">Docs</a></p>

username: Yiru_Shen
cooked: <p>Updates here. I used hooks given the information from <a href="https://discuss.pytorch.org/t/extract-features-from-layer-of-submodule-of-a-model/20181">https://discuss.pytorch.org/t/extract-features-from-layer-of-submodule-of-a-model/20181</a>. Also thanks <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> for the information.</p>
Comparison of nn.CrossEntropyLoss with custom written cross-entropy loss?

username: Rohit_Kumar_Singh
cooked: <p>Hi,</p>
<p>I was just experimenting with pytorch. I implemented a cross-entropy loss function and softmax function as below</p>
<pre><code class="lang-auto">def xent(z,y):
    y = torch.Tensor(to_one_hot(y,3))    #to_one_hot converts a numpy 1D array to one hot encoded 2D array
    y_hat = pt_softmax(z)
    loss = -y*torch.log(y_hat)
    loss = loss.mean()
    return loss

def pt_softmax(x):
    exps = torch.exp(x - torch.max(x,dim=1)[0].unsqueeze(1))
    return exps / torch.sum(exps,dim=1).unsqueeze(1)
</code></pre>
<p>And I was comparing this loss with <code>nn.CrossEntropyLoss</code> and found that <code>nn.CrossEntropyLoss</code> converges faster on <a href="https://archive.ics.uci.edu/ml/machine-learning-databases/wine/" rel="nofollow noopener">wine dataset</a> on UCI repo. Also, the weights and gradients obtained after each epoch was different for both the losses. I am using batch gradient descent and not getting any nan values.</p>
<p>Can anyone please let me know why this is happening? Is it because of unstable implementation of <code>xent</code> or due to some other reason?</p>

username: albanD
cooked: <p>If you test with a given z and y, you should get very similar values between your version and the official one.<br>
You should check that this is the case first.</p>
<p>Then if this the true and the training behavior are still different, you can try with different random seeds. Sometimes, the random initialization can have a significant impact on the speed of convergence of the network.</p>

username: Rohit_Kumar_Singh
cooked: <p>Thanks <a class="mention" href="/u/alband">@albanD</a> for replying. You are correct the values obtained are exactly similar for same <code>z</code> and <code>y</code>. My implementation of <code>xent</code> was wrong. Here is correct implementation.</p>
<pre><code class="lang-auto">def xent(z,y):
    y = torch.Tensor(to_one_hot(y,3))
    y_hat = pt_softmax(z)
    loss = -y*torch.log(y_hat)
    loss = loss.sum()/y.shape[0]
    return loss
</code></pre>
<p>Previously I was taking mean over all the elements but I have to sum the elements first then take the average w.r.t batch size.</p>
Training on Distributed GPU System by Pytorch With Node Asynchronization Problem

username: Cooper_Lee
cooked: <p>I trained a model with pytorch 1.2 on a distributed GPU system. And the training step on each code is out-sync. For example,  the node 1 of the distributed system has already finished the training task on the epcho N, while node 2 just start the training step N-1. I wonder wether that the out-sync phenomenon would make negative effectiveness on model training. I will appreciate if anyone could give me some suggestion.</p>

username: ptrblck
cooked: <p>How did you create the distributed training procedure?<br>
Is it a manual approach or did you use <code>nn.DistributedDataParallel</code>?<br>
In the latter case, synchronization points are automatically in the training process.<br>
From the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">docs</a>:</p>
<blockquote>
<p>Across processes, DDP inserts necessary parameter synchronizations in forward passes and gradient synchronizations in backward passes.</p>
</blockquote>

username: Cooper_Lee
cooked: <p>Thanks for your reply. I use the latter case DDP. According to your answer, I need time to solve the out-sync problem  of our GPU distributed system.</p>
Confused about autograd in Conv2d

username: tomguluson92
cooked: <p>Hello，it really made me confused when reading the <a class="mention" href="/u/simonw">@SimonW</a> refered source code in <a href="https://discuss.pytorch.org/t/autograd-about-the-conv2d/12130">AutoGrad about the Conv2d</a>.</p>
<p>When i read the detail of  function <a href="https://github.com/pytorch/pytorch/blob/bc7a41af7d541e64f8b8f7318a7a2248c0119632/aten/src/THCUNN/generic/SpatialConvolutionMM.cu#L211-L486" rel="nofollow noopener">THNN_(SpatialConvolutionMM_updateGradInput)</a>, Could any warm-kinded people tell me the meaning of <code>gradInput</code>, <code>gradOutput</code> and <code>gradColums</code> and the correct order of reading Autograd mechanism used in <code>Conv2d</code>?</p>
<p>Thanks a lot, Really need your help!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The argument <code>grad_output</code> is the gradient with respect to the output given by the autograd engine.<br>
<code>grad_input</code> is the tensor where the result should be written.<br>
<code>grad_columns</code> is a temporary buffer given here for efficiency.</p>
<p>The backward functions always take <code>grad_output</code> and return <code>grad_input</code>.</p>

username: tomguluson92
cooked: <p>Thx a lot! Could i regard the <code>grad_input</code> as <code>δi,l</code> defined below? <img src="https://discuss.pytorch.org/uploads/default/original/2X/7/749062307e197c26ab3e9a769ed6251e65bf3f44.png" alt="image" data-base62-sha1="gDaJ4eb4sX7NAzIyWwY4lq9WHAw" width="374" height="53"></p>
<p>the superscript <em>l</em> reprensents the layer , <strong>W</strong> is conv2d’s weight kernel and <strong>σ(z)</strong> has a universal format like<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d424e015cfcf25939a4137ee1940fd142174c231.png" alt="image" data-base62-sha1="ugIeboD5ilT4eRXsfTbWKbp6nKN" width="203" height="59"></p>

username: albanD
cooked: <p>Yes, if you’re in the backward of the layer l,<br>
grad_input = di,l<br>
grad_output = di,l+1</p>

username: Haimin_Hunter_Zhang
cooked: <p>I am confused that why     gradColumns = weights’ * gradOutput_n ?<br>
Could any help be provided?</p>
Flattened vs unflattened loss for images?

username: xian_kgx
cooked: <p>For label with more than 1 dimension such as images, how should the loss be computed? Calculate it per image and then average over batch? Or calculate the loss on batch level?</p>

username: JuanFMontesinos
cooked: <p>They are computed element-wise (in the batch dim) and then averaged. If your sensor has N elementa, loss is computed pixel-wise for each element in the batch, averaged and then averaged in the batch dim</p>

username: xian_kgx
cooked: <p>Thanks for your swift reply, but I’m waiting for more responses to see which is a better/more common practice.</p>

username: JuanFMontesinos
cooked: <p>The standard is to average over the batch, that is why it is coded that way by default. Anyway both ways are usually the same  as if you compute L1 over each image and then average over batch is equivalent to compute L1 over all the images at once (always that each image has the same number of elements which is the case).<br>
Besides, plenty of losses cannot be computed to be element-wise (eg. Classification losses such as cross entropy)</p>
<p>Anyway, no matter how do you compute it, gradients will just be rescaled by a factor but they will have same orientation in the hyper space. You will just have to adjust learning rate.<br>
Hope it helps.</p>