akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch equivalent of numpy.take()

Jacky_Wang: <p>The mnist data loaded from dataloader is like: (batch_size,1,28,28), and I want to extract those samples with specific labels: for ex. label=1.<br>
In numpy, it can be done like:</p>
<pre><code class="lang-auto">new_label_idx=np.flatnonzero(label==1)
new_data=np.take(data,new_label_idx,axis=0)
</code></pre>
<p>However, in pytorch, torch.take does not take axis as its parameters. Is there any elegant way to do this?</p>
<p>Thanks!</p>

Yuanchao_Du: <p>you can try <a href="https://pytorch.org/docs/stable/generated/torch.index_select.html#torch.index_select" rel="noopener nofollow ugc">index_select</a></p>
<pre><code class="lang-auto">x = torch.randn(3, 4)
indices = torch.tensor([0, 2])
torch.index_select(x, 0, indices)
``</code></pre>

Jacky_Wang: <p>Thank you, yuanchao. It works!</p>
Pytorch same input but different output

woong97: <p>I’m using model.eval(). Thus it is not the problem of dropout<br>
However, I have same input but different output.</p>
<p>Thus I debug the process.<br>
The interesting thing was found in middle.<br>
Sometimes , tensor looks like 4.498e-01 and sometimes look like 4.488  (without ‘e-01’)<br>
As you see, two values are really similar but they are different. Also I don’t know why some tensor has ‘e-01’ or ‘e+00’ etc.<br>
(And I set dtype=torch.float32 to input value. Thus input has always same value and same dtype)<br>
As a result final output shows different result eventhough the input was exactly same.<br>
Can you explain reason?</p>

CedricLy: <aside class="quote no-group quote-modified" data-username="woong97" data-post="1" data-topic="108830">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/woong97/48/53120_2.png" class="avatar"> woong97:</div>
<blockquote>
<p>4.498e-01 and  4.488</p>
</blockquote>
</aside>
<p>are not the same. 4.498e-01 is the same as 0.4498.<br>
If you use model.eval() the output should be the same for the same input.</p>
<p>Can you provide an example?</p>
Curiously I don't get any gradients in one layer

foldie: <p>Hi all,<br>
I am not quite sure about the intricacies of autograd, and I ran into this issue. Wondering if anyone would know what I messed up from a quick look.<br>
I am running python 3.8.5 and torch 1.5.1.</p>
<p>I am embedding a loss inside another loss for regularization purposes. Here’s how my custom forward for my loss looks like:</p>
<pre><code class="lang-auto">    def forward(self, model, sample, reduce = True):
        regterm = 0
        for layer in model.encoder.layers:
            if isinstance(layer, Mycustomlayer):
                regterm += layer.compute_reg()
        loss, sample_size, logging_output = self.original_loss(model, sample, reduce = reduce)
        loss = loss + regterm
        return loss, sample_size, logging_output
</code></pre>
<p>where the original_loss eventually returns a construct similar to F.l1_loss, and I am adding a regularization term. Layer compute_reg() looks like this:</p>
<pre><code class="lang-auto">    def compute_reg(self, *args):
        probs = self.cdf_qz()
        reg = torch.sum((1 - probs)*self.penalty)
        total_reg = torch.mean(reg)
        return total_reg

    def cdf_qz(self):
        low, high = stretch_limits
        assert low &lt; 0
        xn = (0 - low) / (high - low)
        assert xn != 1
        logits = math.log(xn) - math.log(1 - xn)
        return torch.sigmoid(logits * lambda_multiplier - self.a).clamp(min=self.eps, max=1 - self.eps)

</code></pre>
<p>I get zero gradients on my backward pass on self.a - which should be a parameter of the network. Am I making any obvious blunders?<br>
Thanks!</p>

albanD: <p>Hi,</p>
<p>The one thing I would check is the <code>clamp</code> op that will generate 0 gradients for all the clamped values. So you want to make sure that the initial value of <code>a</code> is properly within the linear region of the clamp.</p>
<p>Otherwise, you also want to make sure that both <code>a.is_leaf()</code> and <code>a.requires_grad</code> are <code>True</code>.</p>

foldie: <p>Thanks loads for the tip! I will check the clamp, the a.is_lead() and a.requires_grad are both in order, I have double checked before.<br>
Cheers!</p>
How does PythonOp been introduced?

pytorch_lzwhard: <p>my network contains:</p>
<pre><code class="lang-auto"># A memory-efficient implementation of Swish function
 class SwishImplementation(torch.autograd.Function):
     @staticmethod
     def forward(ctx, i):
         result = i * torch.sigmoid(i)
         ctx.save_for_backward(i)
         return result
 
     @staticmethod
     def backward(ctx, grad_output):
         i = ctx.saved_tensors[0]
         sigmoid_i = torch.sigmoid(i)
         return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))
 
 class MemoryEfficientSwish(nn.Module):
     def forward(self, x):
         return SwishImplementation.apply(x)
</code></pre>
<p>after jit.trace, there are <strong>PythonOp</strong> nodes.<br>
I know that grad introduces <strong>PythonOp</strong>, but why and how, please?</p>
Training loss is not changing at all while training model

O_H: <p>I’m trying to solve a VQA classification problem. my training loss is not changing at all while training the model.</p>
<p>I put in comment the CNN model and try to run it with the text only, but still, no change in the loss function.</p>
<p>I pass through those models:</p>
<pre><code class="lang-auto">class question_lstm(nn.Module):

    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout, output_dim, que_size):
        super(question_lstm, self).__init__()
        self.hid_dim = hid_dim
        self.n_layers = n_layers
        self.embedding = nn.Embedding(input_dim, emb_dim)
        self.tanh = nn.Tanh()
        self.lstm = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout)
        self.dropout = nn.Dropout(dropout)
        #self.fc1=nn.Linear(n_layers*hid_dim,que_size)
        self.fc1=nn.Linear(n_layers*output_dim,que_size)

    def forward(self, question):
      emb_question=self.embedding(question) #(batchsize, input_dim, emb_dim=256)
      emb_question=self.dropout(emb_question)
      emb_question=self.tanh(emb_question)
      emb_question = emb_question.transpose(0, 1)  #(input_dim, batchsize, emb_dim)
      output, (hidden, cell) = self.lstm(emb_question) 
      qu_feature = torch.cat((hidden, cell), dim=2)               
      qu_feature = qu_feature.transpose(0, 1) #(batchsize=100, num_layer=2, hid_dim=2048)
      question_output =self.fc1(qu_feature)    
      return question_output
</code></pre>
<pre><code class="lang-auto">class vqamodel(nn.Module):

  def __init__(self, output_dim,input_dim, emb_dim, hid_dim, n_layers, dropout, answer_len, que_size,):

    super(vqamodel,self).__init__()
    #self.image=img_CNN(img_size,image_feature)
    self.question=question_lstm(input_dim, emb_dim, hid_dim, n_layers, dropout,output_dim,que_size)
    self.tanh=nn.Tanh()
    self.relu=nn.ReLU()
    self.dropout=nn.Dropout(dropout)
    self.fc1=nn.Linear(que_size,output_dim) 
    self.fc2=nn.Linear(output_dim,answer_len)

  def forward(self, image, question):
    question_emb=self.question(question)
    combine =question_emb   #*img_emb
    out_feature=self.fc1(combine) #(batchsize=100, output_dim=2048)
    out_feature=self.relu(out_feature) 
    out_feature=self.dropout(out_feature)
    out_feature=self.fc2(out_feature) #(batchsize=100, answer_len=1000)
    return (out_feature)
</code></pre>
<p>I’m using cross entropy loss and Adam:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(vqa_model.parameters(),lr=0.001)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
</code></pre>
<p>any idea what can cause this constant loss value?</p>
Use different final activations for different loss function for same model

Anno: <p>I would like to use different final activations for different loss functions that train the same model. In code that would mean:</p>
<pre><code class="lang-auto">model = MyModel()
criterion1 = nn.BCELoss()
criterion2 = nn.L1Loss()

out1 = model(input)
out2 = torch.sigmoid(out1)

loss = criterion1(out1, target1) + criterion2(out2, target2)

loss.backward()
optimizer.step()
</code></pre>
<p>Is this possible and if yes, how? Thank you!</p>
<p>In this minimal example is might seem to make no sense, but in my code it is a GAN that uses multiple loss functions to train the generator if that helps…</p>
Reset the parameter in GRU.weight_hh_l0 and the GRU.weight_hh_l0.grad becomes None

TimmyHuang: <p>Hi everyone, below is part of my code</p>
<pre><code class="lang-auto">def forward(self, x)
     weight = getattr(self.GRU,'weight_hh_l0')
     setattr(self.GRU,'weight_hh_l0', nn.Parameter(weight.data))
</code></pre>
<p>before I do this, the grad is non None. But it will be None after “setattr”. I’m sure weight_hh_l0 is leaf variable and the requires_grad = True. Maybe the “setattr” removes “weight_hh_l0” from the computational graph?</p>
<p>Anyone help ? Thanks</p>

pascal_notsawo: <p>I hope that <a href="https://discuss.pytorch.org/t/nn-module-add-new-parameter-setattr-vs-register-parameter/108238/2">my solution</a> to <a href="https://discuss.pytorch.org/t/nn-module-add-new-parameter-setattr-vs-register-parameter/108238">this topic</a> regarding the conditions that a model’s parameters must satisfy can help you.</p>
Jetson Nano: Functions of Pytorch do not work properly

NPhamDinh: <p>By following this <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-7-0-now-available/72048" rel="noopener nofollow ugc">guide</a>, I installed Pytorch 1.7.0 on a Nvidia Jetson Nano, which had JetPack 4.4 and Python 3.6 installed. But it seems like the installation was not proper.<br>
Normally Pytorch outputs the following:</p>
<pre><code class="lang-auto">&gt;&gt; torch.tanh(torch.tensor(4.2))
&gt;&gt; torch.tensor(1.)
</code></pre>
<p>But in the current Jetson Nano installation:</p>
<pre><code class="lang-auto">&gt;&gt; torch.tanh(torch.tensor(4.2))
&gt;&gt; torch.tensor(4.)
</code></pre>
<p>In other test scripts, the models do not seem to update their weights,<br>
which usually works in Pytorch.</p>
<p>How should I install Pytorch on the Jetson Nano? I already tried the corresponding wheel.</p>

ptrblck: <p>This was a <a href="https://github.com/pytorch/pytorch/issues/49452">known issue</a> and was fixed by fixing NEON and rebuilding the <code>1.7.1</code> Jetson wheels.</p>
BatchNorm statistics aren't updating with training

l_s: <p>I’ve made a simple model and it seems to train properly (training loss decays) but when I look at the running mean and variances they don’t seem to have been updated i.e., they’re set to the 0 and 1 vectors respectively. Track_running_stats is set to True for the bn layers.</p>
<p>relevant code snippet below</p>
<pre><code class="lang-auto">def train(train_loader, model, criterion, optimizer, device, binary_classification=False):
	
	batch_time = AverageMeter('Time', ':6.3f')
	data_time = AverageMeter('Data', ':6.3f')
	losses = AverageMeter('Loss', ':.4e')
	top1 = AverageMeter('Acc@1', ':6.2f')
	
	progress = ProgressMeter(
		len(train_loader),
		[batch_time, data_time, losses, top1],
		prefix="Train: ")

	# switch to train mode
	model.train()

	num_batches = len(train_loader)

	end = time.time()

	for batch_index, (images, target) in enumerate(train_loader):
		# print(batch_index)
		# print(images.shape,target.shape,type(images),type(target))

		# measure data loading time
		data_time.update(time.time() - end)
		images = images.to(device)

		# reformat target
		if isinstance(criterion, nn.BCEWithLogitsLoss): target = target.float()
		
		target = target.squeeze().to(device)

		# compute logits
		logits = model(images).squeeze()

		loss = criterion(logits, target)

		# measure accuracy and record loss; reshaping needed for accuracy calculation
		probabilities=get_complete_probabilities(torch.sigmoid(logits)) if binary_classification \
						else torch.softmax(logits, dim=1)


		acc1 = accuracy(probabilities, target, topk=(1,))[0]
		
		losses.update(loss.item(), images.size(0))
		top1.update(acc1.item(), images.size(0))

		# compute gradient and do SGD step
		optimizer.zero_grad()
		loss.backward()
		optimizer.step()

		# measure elapsed time
		batch_time.update(time.time() - end)
		end = time.time()

		progress.display(batch_index)


	return losses.avg, top1.avg



model=torchvision.models.resnet18()
model.fc=nn.Linear(model.fc.in_features,2)

for epoch in range(start_epoch, start_epoch + args.num_epochs):
	avg_loss, avg_acc1 = train(train_loader, model, criterion, 
										optimizer, device)
</code></pre>

ptrblck: <p>Could you post your model definition, so that we could have a look at this issue?</p>
Unknown error when sending data to GPU?

vymao: <p>When I do <code>data = data.to(device)</code>, where <code>data</code> is initially from a DataLoader, I am getting the following error:</p>
<pre><code class="lang-auto">0%|          | 0/5081 [00:00&lt;?, ?it/s]Traceback (most recent call last):
  File "/n/app/python/3.7.4/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/n/app/python/3.7.4/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/home/vym1/nn2/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 134, in reduce_tensor
    raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "
RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).
</code></pre>
<p>I’ve never see this before, and when running on a different dataset with a different model, it works fine. Does anyone know why this is occuring?</p>

ptrblck: <p>Is this particular custom <code>Dataset</code> using any multiprocessing routines or CUDA methods internally?</p>
TorchScript: how can I access functions that were exported using @torch.jit.export?

rfejgin: <p>When converting my model to TorchScript, I am using the decorator <span class="mention">@torch.jit.export</span> to mark some functions besides forward() to be exported by torch.jit.script().</p>
<p>When loading the TorchScript model in Python, I can indeed access these functions.</p>
<p>My question is regarding C++: since these functions are not included in the standard module interface, I need them to appear in a header file somewhere. Is such a header generated, and if not, how does one call the exported functions from C++?</p>
<p>Thank you!</p>

SplitInfinity: <p>You can see in <code>torch/csrc/jit/api/module.h</code> that <code>torch::jit::Module::forward</code> is implemented like so:</p>
<pre><code class="lang-auto">  IValue forward(std::vector&lt;IValue&gt; inputs) {
    return get_method("forward")(std::move(inputs));
  }
</code></pre>
<p>You should be able to call exported functions like so:</p>
<pre><code class="lang-auto">auto mod = torch::jit::load(...);
auto exported_method = mod.get_method(&lt;exported_method_name&gt;)
auto result = exported_method(...);
</code></pre>

rfejgin: <p>Thanks. Yes, I just found an earlier question that was asked on this (it seems to be missing from docs at the moment). My function does not take any arguments, but it looks like you still need to pass in an empty vector of IValues. For future reference, here’s what I ended up doing:</p>
<pre><code class="lang-auto">    std::vector&lt;torch::jit::IValue&gt; stack;
    module.get_method("my_function_name")(stack);
</code></pre>

Prashant: <p>Hey I’m trying to check my torchscript model from the python REPL. When I load a jit model with torch.jit.load(), that model doesn’t seem to have a get_method function.</p>
<p>If I’m using python, what is the alternative to get_method</p>

Prashant: <p>I seem to have answered my own question.</p>
<p>I’m supposed to use <code>model.__getattr__(func_name)(args)</code></p>

SplitInfinity: <p>I think you can also access it like any other attribute of a Python object.</p>
Batch-wise slice tensors

Zhaoyi-Yan: <p>Is it possible to remove these two <code>for</code> ?</p>
<pre><code class="lang-python">import torch

a = torch.rand(10, 10)
bz = 4
x_start_index = torch.LongTensor([2, 4, 1, 3])
x_end_index = torch.LongTensor([4, 6, 2, 6])

y_start_index = torch.LongTensor([1, 6, 5, 3])
y_end_index = torch.LongTensor([3, 9, 7, 4])

v_total = []
for i in range(bz):
    # also calculate the sum
    v_tmp = torch.sum(a[x_start_index[i]: x_end_index[i], y_start_index[i]: y_end_index[i]])
    v_total.append(v_tmp)

v_total = torch.stack(v_total)

for i in range(bz):
    # zero out the boxes
    a[x_start_index[i]: x_end_index[i], y_start_index[i]: y_end_index[i]] = 0
</code></pre>

ab-10: <p>Considering that there are <a href="https://discuss.pytorch.org/t/how-to-slice-multiple-blocks-efficiently-from-a-torch-tensor/19680">multiple</a> similar unanswered <a href="https://discuss.pytorch.org/t/slice-a-batch-with-different-coordinates-with-no-loop/55744">questions</a>, I don’t think there’s a way to do the slicing without loops.</p>

Zhaoyi-Yan: <p>Suddenly, I realized that such operation could not be implemented in an easy way.<br>
Similar questions you listed can be regarded as simplified version of my question, as they only need slicing the same size tensor with left-up coordinates given. The sliced tensors share the same size, however, in my case it is not true. I think the Pytorch team would rather perfer the case of slicing the same size instead of my case in the future.</p>

Mingfeng_Wang: <p>Only this batch slice operation takes me more than 50% time of my model, have you found the solution?</p>
Does freezzing part of the network saves GPU memory?

alwynmathew: <pre><code class="lang-auto">feat = Net1(input)
output = Net2(feat)
</code></pre>
<p>If I freeze <code>Net1</code> by <code>eval(Net1)</code>, does this save memory in my GPU?</p>
<p>The gradient need not to be computed in case of frozen layers, thus lesser memory. But I don’t see much change in the memory allocated between frozen layers and trainable layers? Is the memory difference very small?</p>

dazzle-me: <p>Try<br>
<code> with torch.no_grad():</code><br>
<code> ____feat = Net1(input)</code><br>
<code> output = Net2(feat)</code></p>
<p>This should work.</p>
<p>See <a href="https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615/2">this</a> topic for explanation.</p>

alwynmathew: <p>Thanks <a class="mention" href="/u/dazzle-me">@dazzle-me</a> but my question was does frozen layers saves GPU memory?</p>

ptrblck: <p>If you freeze modules by setting the <code>requires_grad</code> attribute of their parameters to <code>False</code>, then you would save the memory the gradients would otherwise occupy (assuming the <code>.grad</code> attributes are set to <code>None</code>).</p>
PyTorch weight initialization for Alexnet and VGG model

suwei: <p>I am new to pytorch, and I am confused when I go through the vision models of Alexnet and VGG, for Alexnet, there is no weight initialization function, and other models have., such as for VGG, there is a function  def _initialize_weights(self). So my question is, why?</p>

User4: <p>Hmm, maybe because it’s too old? But you can generate them by yourself by apply the init rule.</p>
<pre><code class="lang-auto">def init_weights(m):
    if isinstance(m, nn.Linear):
        torch.nn.init.xavier_uniform(m.weight)
        m.bias.data.fill_(0.)
    elif isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform(m.weight)

model = torchvisions.models.alexnet()
model.apply(init_weights)
</code></pre>
Performance with variable argument forward functions

jlperla: <p>I don’t have a good mental model of performance in pytorch (or how to profile with GPUs/TPUs/etc).  But basically:  Does the following pattern have any hidden performance “gotchas” due to the use of the vararg forward?</p>
<pre><code class="lang-python">class MyNet(torch.nn.Module):
    def __init__(self, N):
        super(MyNet, self).__init__()
        self.N = N
        self.net1 = nn.Linear(self.N, 1, bias=True)
        self.net2 = nn.Linear(self.N+1, 1, bias=True)
        
    def forward(self, *args):
        if len(args) == 1:
            return self.net1(args[0])
        elif len(args) == 2:
            return self.net2(torch.cat([args[0], args[1]]))
        else:
            error("Unsupported number of arguments")

# Usage
ob = MyNet(3)
X = torch.randn(3)
x = torch.randn(1)
ob(X)
ob(x,X)
</code></pre>

googlebot: <p>Nope, majority of time will be spent in inner functions. The use of cat() is much more significant than varargs.</p>

jlperla: <p>Thanks so much. Btw, is there a better way to do the <code>cat</code> in this sort of scenario… Basically to merge several branches of the computational graph in a downstream linear layer?</p>

googlebot: <p>Well, you can’t avoid merging memory blocks. Pre gradient flow you can sometimes pre-allocate a joined buffer and write to it part by part, e.g. randn(3, out=x[:3]), x[:3].normal_(), x[:3].fill_(0.1) etc.<br>
If you do “multiple branches”, it is usually slower than straightforward memory consolidation with cat().</p>
Batchnorm train() vs eval() for Multi-Instance Learning

yashsharma0906: <p>Hi Team,</p>
<p>I am using a Multi-instance learning approach for histopathology modeling. To fit all the patches coming from a single whole slide image (100kx100k pixel), I am using a batch size of 1. But in each batch, passing 64-128 patches coming from a whole slide image so batch norm should not mess up. As I am using pre-trained Resnet and the mean/std distribution of medical data will be very different from the imagenet, I have bumped up the momentum to 0.9 and re-initialized batch norm parameters (mean to zero and std to one). The model training is fine, but validation is highly unstable. Based on my experiments and evaluation done using both train and eval mode. I suspect it to be because of track running stats set to True. As I am using the train running mean/std in eval mode and in each new batch picking up a new whole slide image (64-128 patches), leading to a change in distribution at a fast speed resulting in poor validation performance. I can’t go with group norm/ instance norm as the community has not started sharing pre-trained models with these normalization layers. So:</p>
<ol>
<li>Is it wrong to set track running mean/std to False in the eval stage for cases like these? Are there any paper/example which set it to False?</li>
<li>Is running validation in train mode technically wrong? I will be just updating the running mean and std with a momentum of 0.9.</li>
<li>I have around 300 whole slide images for training, and keeping momentum around 0.1 degrades the performance. One option is to further experiment with this set-up by re-initializing batch norm parameters and to keep momentum=0.1. But this may work and start giving me stable performance even in the evaluation mode as momentum is low, and the model gets trained with a slow changing mean/std (sort of global mean/std instead of a single whole slide mean/std).</li>
</ol>
<p>Any suggestions are highly appreciated? If the day is in my favor, I get pretty good training and validation accuracy (with track running stats False) but I am hunting for a more stable/sustainable solution.</p>
<p>Thanks!</p>
Operations on pytorch distributions?

kirk86: <p>I was wondering if it would be interesting to have operations like <code>+, *, etc.</code> on distribution from <code>pytorch.distributions</code>?</p>
<p>e.g.</p>
<pre><code class="lang-auto">X  = torch.distributions.Normal(loc=1, scale=1)
Y  = torch.distributions.Normal(loc=0, scale=2)
(X+Y).mean()
(X*Y).std()
</code></pre>
<p>of course one could sample to simulate the X, Y and then perform the operations on the equivalent samples, but  I was wondering 1) if that is a good approximation and 2) would the above example be feasible?</p>

googlebot: <p>Depending on what methods you need from the resulting distribution, and whether you operate on the same family, different methods are needed (analytical formulas, monte carlo approximations, fourier transforms). So it would be hard to create a generic RV algebra mechanism, and it is a bit outside of ML realm; for example R package ‘distr’ implements this algebra, but there it is used interactively or for one-off computations, not for deep learning.<br>
You can do deep learning with sampling of course, and it is the usual way, but then the need for this arithmetic is unusual.</p>
How to Compute Recurrence Equation in PyTorch?

tannguyen1989: <p>Hi all,</p>
<p>In my current project, I need to compute the following recurrence relation.</p>
<p>Given a sequence {b} = [b_1, b_2, …, b_n], I would like to compute the sequence {a} as follows:</p>
<p>a_1 = 1<br>
a_2 = a_1 x b_1 + 1<br>
…<br>
a_n = a_n-1 x b_n-1 + 1</p>
<p>In my problem, b_i, i=1, 2, …, n, are 2-D matrices.</p>
<p>I can use FOR loop to compute the sequence {a}, but it is slow when dealing with very long sequences. I wonder if there is an efficient way to compute the sequence {a} in Pytorch or not.</p>
<p>I note that without “+1”, the sequence {a} is just the cumulative product of the sequence {b}, which can be computed fast with torch.cumprod.</p>
<p>I would like to thank you so much in advance!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Tan Nguyen</p>
PYTORCH1.6 build hava error

liu: <p>BUILD PYTORCH1.6  VS2017 error :</p>
<p>Q: \ \ pytorch1.6 \ \ pytorchaa \ \ aten \ \ SRC \ \ aten / core / boxing / impl / boxing. H (101): error c2066: it is illegal to cast to a function type (compilation source file Q: \ \ pytorch1.6 \ \ pytorchaa\build\aten\src\ATen\BackendSelectRegister.cpp ) [Q:\pytorch1.6\pytorchaa\build\caffe2\torch_ cpu.vcxproj ]</p>
<p>CODE: boxing. h<br>
template &lt;class FuncType, class Enable = void&gt;<br>
struct BoxedKernelWrapper {<br>
static_assert(<br>
sizeof(FuncType) == -1,<br>
"Function signature contains one or more unsupported parameter and/or return types. "<br>
“Look for a nearby error like "<br>
“”‘call’ is not a member of ‘c10::impl::BoxedKernelWrapper&lt;(your function type), void&gt;’” "<br>
“- (your function type) is the unsupported signature.”);<br>
};<br>
sizeof(FuncType)  is error</p>

frankdong: <p>I had the same issue as you and resolve this by using Ninja instead of Visual Studio. Looks like somehow my C++ compiler for VS is corrupted.</p>
Duplicate elements based on a kernel

davoi: <p>Hello,</p>
<p>Is there a known function able to duplicate values following this behavior?</p>
<pre><code class="lang-auto">x = torch.tensor([[1., 2.], 
                  [0., 3.]])

some_function(x, kernel=(2, 2))
&gt;&gt; [[1., 1., 2., 2.], 
    [1., 1., 2., 2.],
    [0., 0., 3., 3.],
    [0., 0., 3., 3.]]

some_function(x, kernel=(3, 2))
&gt;&gt; [[1., 1., 2., 2.], 
    [1., 1., 2., 2.],
    [1., 1., 2., 2.],
    [0., 0., 3., 3.],
    [0., 0., 3., 3.],
    [0., 0., 3., 3.]]
</code></pre>
<p>Thank you</p>

richard: <p>I don’t think there is a single operator that does what you want, but it’s possible to achieve the behavior via some broadcasting and view logic:</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([[1., 2.], [3., 4.]])
kernel = (3, 2)
x.view(2, 1, 2, 1).expand(2, kernel[0], 2, kernel[1]).contiguous().view(2 * kernel[0], 2 * kernel[1])
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor([[1., 1., 2., 2.],
        [1., 1., 2., 2.],
        [1., 1., 2., 2.],
        [3., 3., 4., 4.],
        [3., 3., 4., 4.],
        [3., 3., 4., 4.]])
</code></pre>

davoi: <p>Thank you<br>
Can be solved using Kronecker product but not added to the main branch yet (<a href="https://pytorch.org/docs/master/generated/torch.kron.html" class="inline-onebox" rel="noopener nofollow ugc">torch.kron — PyTorch master documentation</a>).</p>
Is it possible to know what GPU kernels are used and what size are they?

lmatz: <p>For example, after I built a model in python, is it possible for me to print all the specific Cuda kernel functions that will be used? Is it true that every time the kernel is launched, the size of the kernel is calculated on the fly instead of being determined statically before training starts? Thanks</p>

ptrblck: <p>It’s not possible to predict which kernels will be called in your script before running it, as this would need a sort of static code analysis, which would also need to know your current setup, since kernels are also depending on the GPU. Also, if you are using cudnn benchmarking, the actual kernel choice depends on the profiling, which is also non-deterministic and could depend on the workload your machine has at the moment.</p>
<p>Could you explain a bit what “size” of the kernel you are referring to?</p>

lmatz: <p>Thanks for the reply! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>The size means the “&lt;&lt;&lt;X,Y,Z&gt;&gt;&gt;” before the cuda kernel function call. As I went through some of the code, I now think the size for the same function is the same across different runs, but always computed again and again at runtime for the same cuda kernel function call each time. Is this correct(after benchmarking and picking the actual kernels for a static model)?</p>
<p>And for cudnn kernel, PyTorch has no way to decide the size right? “&lt;&lt;&lt;X,Y,Z&gt;&gt;&gt;” is determined inside cudnn like a blackbox.</p>

ptrblck: <p>Yes, cudnn kernel launches are not exposed.<br>
The kernel launch parameters are determined by the workload and would thus also need some kind of static analysis, which I’m not sure how it would look like.</p>
If terms cancel out, is the gradient calculated for those terms?

beisner: <p>Say I have tensors <code>a</code>, <code>b</code>, which are the result of functions that have gradients, and compute tensor <code>c</code>:</p>
<pre><code class="lang-auto">c = a + b - b
</code></pre>
<p>Then, I call</p>
<pre><code class="lang-auto">c.backward()
</code></pre>
<p>Will the gradient for <code>b</code> still be calculated? Or is Pytorch doing something symbolically smart under the hood during Autograd?</p>
<p>Thanks!</p>
BERT Embedding Vector

John_Grabner: <p>In applications like BERT, does the embedding capture the semantic meaning of the word , or does the embedding essentially learn a pseudo orthogonal friendly to the transformer it feeds?</p>
<p>Essentially the same question, in BERT like applications, is embedding equivalent to a reduced dimension orthogonal vector projected into a vector of dimension <strong>embedding_dim</strong>  where the projection is learned?</p>
Transfer learning using github repo

tash: <p>When we use pre-trained models from trochvision, we are using<br>
model = models.vgg16(pretrained = True).<br>
I am using a github repo of a network, which is not included in the torchvision. How can I assign the model?<br>
model = XXXXXX(pretrained = True).<br>
what should be this XXXXXX?</p>
<p>Any help would be appreciated. Sorry to ask a simple question,but I have been struggling with this.<br>
Thanks</p>

user_123454321: <p>Hi, you need to simply instantiate the model and and load the state dict. Something like</p>
<pre><code class="lang-auto">model = Model() # create your model
model.load_state_dict(torch.load('model.pth')) #assuming you have the model weights saved in model.pth
</code></pre>

tash: <p>Thanks a lot. I appreciate your help</p>
Pretrained Alexnet: Training Parameters

gakadam: <p>I want to do Quantization Aware Training of Alexnet on the Imagenet dataset, going from f32 to int8, to leverage GPU support.</p>
<p>I am using the same model architecture as the pre-trained model in the Torch database.</p>
<p>However, to train the model, where can I find the training parameter information, if possible, used for the pre-trained model?</p>
<p>Thank you.</p>
DistributedDataParallel loss computation

Seo: <p>Hi,<br>
I am working with DDP and I have a doubt about the loss computation per process. Currently, using DDP, I have the possibility to distribute the batch among different processes, in this way I can increase the size of each batch. For instance, if I have a batch of 128 and I use 2 processes, I will end up having an effective batch size of 128*2=256. For simplicity, I will refer to local batch (i.e., the batch seen by a single process, which is 128) and global batch (i.e., the batch seen by the entire DDP, which is 256)<br>
The doubt I have is the following. When I compute the loss for each process, this loss is averaged on the local batch and not on the global batch, thus resulting in gradient computation that depends on the local batch. When I compute the <code>loss.backward()</code>, DDP will raise hooks each time all gradients for a bucket are ready and average them among all processes. Anyway it is not clear whether DDP re-adjust the loss (divide the total loss for the global batch) or it is something I need to take care of.</p>
<p>Thank you</p>

Seo: <p>I arrived at the conclusion that there is no need to re-adjust the loss value since averaging gradients will produce an effect that is the same as using the global batch size.</p>
<p>This can be proven computing the loss for each local batch. This loss is <code>sum(l)/batch_size</code>, where <code>l</code> is the list containing the losses for each sample in the local batch. The gradient for the i-th parameter <code>wi</code> will be the gradient of <code>sum(l)</code> w.r.t. to <code>wi</code> scaled by the local batch size (e.g., 128). When we average all the gradients of the various processes we will divide by the <code>n_proc</code> which results in the scaling factor being multiplied by <code>n_gpus</code> (e.g., 2), thus resulting in a scaling factor equal to the global batch size (e.g., 256).</p>
BatchNorm2D folding/fusion

rafale77: <p>I have been trying to improve performance of a resnet50 based model for deployment and after testing scripting the pretrained model for JIT compilation and not finding any improvement from it, I am now also testing fusing of the BatchNorm layers. After successfully fusing the model, I am also not seeing any improvement from fusing 50 of the layers (verified from printing out the models before and after). This is on pytorch 1.7.1/cuda11.2 It is making me wonder if pytorch is already optimizing things so well that these tricks are no longer effective or necessary. Anyone experienced this?</p>
Inpedentent sampling along tensor dimension

gabrieldernbach: <p>How can the following be vectorized in pytorch?</p>
<p>For each <code>n</code> I want to select one of the <code>s</code> options that have <code>d</code> dimensions each.<br>
The naive for-loop looks like this</p>
<pre><code class="lang-auto">    def independent_sample(data):
        n, d, s = data.shape
        data = torch.stack([p[:, torch.randint(s, (1,))] for p in data])
        data = data.squeeze(-1)
        return data
</code></pre>

gabrieldernbach: <p>answering myself</p>
<pre><code class="lang-auto">data_ = data[range(n), :, torch.radnint(s, (n,))]
</code></pre>
Unequivalent exported ONNX format network

jianshitansuantong23: <p>Hi,<br>
I have trained a binary-weight network and would like to export it into ONNX format. However, some convolution layers are changed in the exported ONNX model. They are different than the original in a sense that they combine the next batchnorm layer with this one and have non-binary weight and bias, which doesn’t exist in the trained network.<br>
The export command I use is<br>
x=Variable(torch.randn(1,3,32,32,requires_grad=True,device=‘cuda’))     torch.onnx.export(net,x,“CNN_Medium.onnx”,export_params=True,verbose=True,input_names = [‘input’], output_names = [‘output’])<br>
Does anybody have similar issues? Are there any solutions?</p>
How to implement torch.autograd.Function with arbitary length of input

waterhorse1: <p>Hi, I want to implement torch.autograd.Function with arbitary length of input, so here is the example</p>
<pre><code class="lang-auto">class my_func(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x_1,x_2....,x_n):#the size of n can vary
        result = f(x_1,.....x_n)
        ctx.save_for_backward(result)
        return result
    @staticmethod
    def backward(ctx, grad_output):
        result, = ctx.saved_tensors
        return gradient_x_1, .....gradient_x_n# n corresponds to the length of input
</code></pre>
<p>I have tried something like input a list [x_1,x_2…x_n] into the function but it seems not to work.  Another way is to stack x_1,x_2…x_n to one single tensor but in my implementation each x represents a neural network’s parameters so it will be quite troublesome and they may not have the same dimension.</p>
<p>Really appreciate it if you can help me with that.</p>

albanD: <p>Hi,</p>
<p>You can use the vararg construct in python that looks like:</p>
<pre><code class="lang-python">def forward(ctx, special_arg, *all_others):
</code></pre>
<p>And here, <code>special_arg</code> will contain the first arg and <code>all_others</code> is a tuple with all the other positional arguments that were given.<br>
Note that <code>special_arg</code> is just an example here and you can also have only the <code>*all_other</code> is you’re happy to have all the positional arguments in a single tuple.</p>
How to load a pretrained model and train again?

Hacking_Pirate: <p>I want to train a model on a different dataset then save those weights and load them to another dataset and finetune them for the other dataset. How can I do this?<br>
(Both the datasets are similar).<br>
Thanks.</p>

Yuanchao_Du: <p>This action is called resuming, you can refer <a href="https://discuss.pytorch.org/t/how-to-resume-training/8583/2">how-to-resume-training</a></p>

Hacking_Pirate: <p>Thanks for your reply. I don’t actually want to resume training, I want to load those weights and retrain so that those weights could the model a head-start.</p>

Yuanchao_Du: <p>Do you mean</p>
<ol>
<li>train the model on datasetA</li>
<li>save the trained weights to modelA</li>
<li>fine-tune the trained modelA on datasetB</li>
</ol>
<p>if in this way, what you are looking for is <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="noopener nofollow ugc">fine-tune</a>?</p>

Hacking_Pirate: <p>If I am using a pretrained model from <code>timm</code> do I have to set  <code>requires_grad</code> to False?</p>

Yuanchao_Du: <p>No, you can refer, <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#set-model-parameters-requires-grad-attribute" rel="noopener nofollow ugc">set-model-parameters-requires-grad-attribute</a></p>
<p>basically,<br>
<code>requires_grad=True</code> ,  means you want to train or fine-tune a model<br>
<code>requires_grad=False</code>,  means you want to use the model to test or infer</p>

Hacking_Pirate: <p>Thanks a lot man, I’ll try it.</p>
Unable to load_state_dict when ModuleList is used

ManosMagnus: <p>Hello,</p>
<p>I am using ModuleList to construct a model consisting of CustomSubModule.</p>
<pre><code class="lang-python">
class CustomSubModule(T.nn.Module):
    def __init__(self, fan_in, fan_out, weight, bias, alpha, beta):
        super().__init__()

        self.no_bias_layer = T.nn.Linear(fan_in, fan_out, bias=False)

        self.no_bias_layer.weight = T.nn.Parameter(weight)

        self.bias = T.nn.Parameter(bias)

        self.register_buffer('alpha', alpha)
        self.register_buffer('beta', beta)


class CustomModule(T.nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = T.nn.ModuleList()

    def add_layer(self, layer: CustomSubModule):
        self.layers.append(layer)
</code></pre>
<p>However, I can’t load  the state_dict using the load_state_dict function</p>
<pre><code class="lang-python">custom_sub_module = CustomSubModule(10, 10, T.rand(10, 10), T.rand(10, ),
                                    T.tensor([5]), T.tensor([4]))

costum_module = CustomModule()
costum_module.add_layer(custom_sub_module)
T.save(costum_module.state_dict(), '/tmp/test_model.pt')

loaded_module = CustomModule()
loaded_module.load_state_dict(T.load('/tmp/test_model.pt'))
</code></pre>
<pre><code class="lang-python">Traceback (most recent call last):
  File "tests/save_dict_example.py", line 36, in &lt;module&gt;
    loaded_module.load_state_dict(T.load('/tmp/test_model.pt'))
  File ".venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for CustomModule:
	Unexpected key(s) in state_dict: "layers.0.bias", "layers.0.alpha", "layers.0.beta", "layers.0.no_bias_layer.weight". 
</code></pre>
<p>Is there any way to overcome this?</p>

user_123454321: <p>Should you not call add_layer on the loaded model as well. I think the layers is empty. Can you try maybe something like</p>
<pre><code class="lang-auto">loaded_module = CustomModule()
loaded_module.add_layer(CustomSubModule(10, 10, T.rand(10, 10), T.rand(10, ),
                                    T.tensor([5]), T.tensor([4])))
loaded_module.load_state_dict(T.load('/tmp/test_model.pt'))
</code></pre>
MCMC Package: Efficient&Fast Storing of StateDicts for accepting/rejecting model samples

ludwigwinkler: <p>Hi there everybody,</p>
<p>I’ve been implementing a MCMC sampling package for PyTorch models for fun.</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/ludwigwinkler/pytorch_MCMC" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/29676773?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/ludwigwinkler/pytorch_MCMC" target="_blank" rel="noopener nofollow ugc">ludwigwinkler/pytorch_MCMC</a></h3>


  <p><span class="label1">Lightweight MCMC sampling for PyTorch Models aka My Corona Project</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
So far I’ve been doing multi-processing for parallel chains on my cpu for little toy problems.</p>
<p>The next step was to modify the package such that it runs on GPU and/or multiple GPUs.<br>
So far the idea is to call the convenient state_dict function every time a parameter sample is accepted and loading an old one every time it is rejected.<br>
This includes moving a lot of data between CPU and GPU which isn’t a huge problem given the bottleneck of the forward pass (the speed up from using a GPU and the slower storing/loading should be larger than running it on a CPU).</p>
<p>Since I want to make it as fast as possible I was wondering whether there are any caveats/gotchas/things-to-pay-attention-to considering the repeated loading storing of state_dicts/parameters in PyTorch.</p>
<p>So far my “smartest” idea was to keep a parallel model with the last accepted model parameters directly on the GPU (which admittedly reduces the GPU memory available for the actual forward/backward pass) but circumvents loading the parameters from the CPU every time we reject a sampling move.</p>
<p>Does anybody have any recommendations concerning how to speed up loading/storing of state_dicts/parameters?</p>
<p>Thanks in advance! =)</p>
Debugging in Object Detection in Detectron2

draaken0: <p>I have trained a model in detectron2. But, it does lot of false prediction if i use it on supermarket as the model was trained on the images taken inside home and arranging products. So, is there any way where i can visualize output of layers of the model, i mean the anchor boxes being generated by the model? Also model predicts properly when i test it on products which are placed on the selves inside my home (similar images) but predict poorly when i take it to any super market. So, is there any proper way to reduce the false predictions.</p>

ptrblck: <p>You could capture more training data from the “supermarket” setup and finetune the model on this data domain, which would most likely reduce the prediction error.</p>

draaken0: <p>Ok, but is there any way to reduce false prediction on <strong>Similar</strong> objects, i mean like Pepsi diet can and Coke diet can, they both are in silver colour.  We cannot label everything so is there anyway se can do this ? I am reading lot of articles but couldn’t find much of it.</p>
Error in nnd get grad: an illegal memory access was encountered

Mendel123: <p>When I tries to train a model, it will crash after several iterations and an error occurs. That is</p>
<pre><code class="lang-auto">loss.backward()
  File "/opt/conda/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: an illegal memory access was encountered (launch_kernel at /opt/conda/conda-bld/pytorch_1591914880026/work/aten/src/ATen/native/cuda/CUDALoops.cuh:217)
</code></pre>
<p>After I set <code>CUDA_LAUNCH_BLOCKING</code> to True, it is</p>
<pre><code class="lang-auto">error in nnd get grad: an illegal memory access was encountered.
...
</code></pre>
<p>What does <code>nnd</code> mean? Is it <code>NaN</code>?</p>

ptrblck: <p>Are you using the latest PyTorch release (<code>1.7.1</code>) and if not could update to it and rerun your script?<br>
If you are already on the latest version, could you post a minimal code snippet to reproduce this issue and post your current setup, i.e. used GPU, CUDA, cudnn version etc.?</p>

Mendel123: <p>Sorry for the late reply… I way using Pytorch1.5 before and after I change it to 1.6.0, the error disappears. Thanks very much.</p>
What is the effect of Torch.nn.Upsample VS F.resize

Ripley: <p>Hi,</p>
<p>Just curious to know if there is a difference in terms of the pixel value or any if I use torch.nn.upsample vs F.resize? assuming both functions resize/upsample to the same pixel size and the resampling method is Nearest.</p>
<p>Thank you!</p>

ccoding: <p>They are the same inside.<br>
When you look inside the source code of <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/upsampling.html#Upsample" rel="noopener nofollow ugc">torch.nn.upsample</a>, you will find that the <code>forward</code> function of <code>upsample</code> is relized by F.resize( I think you maybe point <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=interpolate#torch.nn.functional.interpolate" rel="noopener nofollow ugc">interpolate</a> function)</p>
<pre><code class="lang-python">def forward(self, input: Tensor) -&gt; Tensor:
    return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners)
</code></pre>