akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pickup a loss function for maximizing pos_x while minimizing neg_x

username: themoonboy
cooked: <p>Hi all, I want to ask about the experience to pick up a suitable loss function for unsupervised problems. Here is a description of my current problem.</p>
<p>Assume I divide the samples into positive and negative, then get two groups of scores when passing them into my model, the dummy codes like:</p>
<pre><code class="lang-auto">pos_scores = model(pos_samples)    # (sample_num, 1)
neg_socres = model(neg_samples)    # (sample_num, 1)
pos_x = pos_scores.mean()
neg_x = neg_scores.mean()
loss = LossFunction(pos_x, neg_x)
</code></pre>
<p>Where I want to <strong>maximize the pos_scores while minimizing the neg_scores at the same time</strong>, there is no limitation of their values. Note that it is an unsupervised problem, thus no ground-truth label as a reference, only a calculated score for each sample.</p>
<p>One loss function I have tried is:</p>
<p><code> loss = neg_x - pos_x</code></p>
<p>by minimizing this loss function, the code tends to minimize neg_x while maximizing pos_x. But the resulted scores are not very brilliant. Thus, are there any other useful functions that could help to handle with this problem? Thanks all!</p>
Joint learning log linear models

username: Em_Ros
cooked: <p>Hello!</p>
<p>I want to implement a custom loss function that is a sum of log linear loss (for binary classification).</p>
<p>I tried digging in the forum and in general for a solution but I can’t seem to find how to do it. Could you maybe point me towards a source for this?</p>
<p>Thanks!</p>
Static Quantization of nn.ConstantPad2d

username: deepak_mangla
cooked: <p>I have a module like this  -<br>
<code>self.conv = nn.Sequence(nn.ConstantPad2d((1,2,1,2)), nn.Conv2d(...))</code></p>
<p>Model converts successfully into quantized form but when I try to evaluate it, I get this error -</p>
<blockquote>
<p>RuntimeError: Could not run ‘aten::empty.memory_format’ with arguments from the ‘QuantizedCPU’ backend. ‘aten::empty.memory_format’ is only available<br>
for these backends: [CPU, CUDA, MkldnnCPU, SparseCPU, SparseCUDA, BackendSelect, Autograd, Profiler, Tracer].</p>
</blockquote>
<p><strong>I think this is because quantization of <code>nn.ConstantPad2d</code> is not supported. So, any solution around it?</strong></p>
<p><em>I cannot merge ConstantPad2d and Conv2d because Conv2d don’t support  odd paddings (equivalent of nn.ConstantPad2d((1,2,1,2))) .</em></p>

username: jerryzh168
cooked: <p>I think <a class="mention" href="/u/zafar">@Zafar</a> is working on supporting constant pad right now: <a href="https://github.com/pytorch/pytorch/pull/43304" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/43304</a></p>
Which model to choose during training?

username: fred
cooked: <p>What’s the correct way to choose the epoch with the best model? Do i have to choose the epoch when the model yielded the lower validation loss?</p>
<p>Thank you.</p>

username: ayalaa2
cooked: <p>Typically you would select the epoch at which either the best validation loss occurred or when the model has converged. There’s no “has to”, but we do generally use the validation set to determine which model is best.</p>
how do I handle many Cnn outputs and loss functions?

username: shivam_bhatt
cooked: <p>I am trying it make a sudoku solver using CNNs. So I need to output 81 values (between 1 and 9, 0 is empty) and calculate 81 losses. How would I handle that? I know you can easily do this for two or three outputs but is there a more convenient way than the way outlined here <a href="https://discuss.pytorch.org/t/a-model-with-multiple-outputs/10440" class="inline-onebox">A model with multiple outputs</a> . Also, what would be the best cost function to use? I am using a resnet based architecture.</p>

username: ayalaa2
cooked: <p>So let’s first begin with representation. Our board is 9x9 so it at least makes sense that we’ll have that as our height and width. But then we need to consider how to represent the value of each tile. We could just use their value (1.0, 2.0, …, 9.0), but this is sort of problematic. This suggests that values close together are related (8.0 and 9.0 are close to each other numerically, but conceptually they’re no closer than 1.0 and 9.0 in the game of sudoku ).  So we want to avoid that sort of representation as it’ll make the problem harder for the network.</p>
<p>So instead, we probably want to explore the idea of a one-hot encoding. That is, for each tile we’ll have a 9x1 vector where there’s a single 1.0 somewhere. This sort of representation is easier for the model to understand. We can also use this for our output and treat this problem similar to how we might go about classifying an image. We can have our model take in a 9x9x9 (values x height x width) input and output a 9x9x9 output. Where in the context of a convolution, our values (the one-hots) are our channels. I would suggest we treat the outputs of our model as probabilistic logits. That is, we’ll follow up with a softmax operation to determine which value a tile should hold.</p>
<p>Now for the topic of training, this is sort of a hard problem for the game of sodoku. My first guess would be to make it so that all values in each column, row, and within a 3x3 square sum up to 45. But that’s sort of a heavily constrained problem so I don’t really know what the model will do under this case. This  problem is sort of more suited to be treated as a <em>reinforcement learning</em> problem instead. Where the model is tasked with just winning the game. I would suggest reading more into those problems.</p>
Calculate_qparams should not be called for NoopObserver

username: FruitVinegar
cooked: <p>Hi.<br>
I’m trying to use Pytorch’s quantization scheme.<br>
I’d like to quantize only weight with fake-quantization(QAT), not activation.</p>
<p>I tried this:</p>
<pre><code class="lang-auto">import torch.quantization as Q

model = load_model(my_config) # currently I'm using resnet architecture
qat_model = Q.fuse_modules(model, my_modules_to_fuse)
qat_model = Q.Qconfig(activation=Q.NoopObserver, weight=Q.FakeQuantize)
</code></pre>
<p>and this process from <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">pytorch quantization tutorial</a></p>
<pre><code class="lang-auto">for nepoch in range(8):
    train_one_epoch(qat_model, criterion, optimizer, data_loader, torch.device('cpu'), num_train_batches)
    if nepoch &gt; 3:
        # Freeze quantizer parameters
        qat_model.apply(torch.quantization.disable_observer)
    if nepoch &gt; 2:
        # Freeze batch norm mean and variance estimates
        qat_model.apply(torch.nn.intrinsic.qat.freeze_bn_stats)

    # Check the accuracy after each epoch
    quantized_model = torch.quantization.convert(qat_model.eval(), inplace=False)
    quantized_model.eval()
    top1, top5 = evaluate(quantized_model,criterion, data_loader_test, neval_batches=num_eval_batches)
    print('Epoch %d :Evaluation accuracy on %d images, %2.2f'%(nepoch, num_eval_batches * eval_batch_size, top1.avg))
</code></pre>
<p>But the program gives this error:</p>
<blockquote>
<p>calculate_qparams should not be called for NoopObserver</p>
</blockquote>
<p>the reason I why used NoopObserver is avoiding calculate_qparams for activation… but It’s confused result.<br>
How to solve this problem? any suggestion will be appreciated.<br>
Thanks.</p>

username: jerryzh168
cooked: <p>You can’t skip quantizing just by setting the observer to NoopObserver. I don’t think weight only quantization is support in convert stage. You can evaluate the accuracy of the qat module directly without convert.</p>
Dispatching of advanced indexing

username: almeetb
cooked: <p>Hello,</p>
<p>I have a backend and have registered some operations using <code>PrivateUse1</code> dispatch key. I am trying to dispatch the following:</p>
<pre><code class="lang-auto">x[0:1, :, :].copy_(y)
return x + z
</code></pre>
<p>I get three slices dispatched to handle the indexing and each return a new tensor and the <code>copy_</code> copies <code>y</code> into the tensor corresponding with the last dispatched slice but the add will take the sum of the original <code>x</code> and <code>z</code>.</p>
<p>What I expect is for the slices to be done in-place and the <code>copy_</code> to be updating the original <code>x</code> value. I have tried registering <code>index</code>, <code>index_put_</code> and <code>masked_fill_</code> but neither of these ops will be dispatched.</p>
<p>Is there some other operation I should be registering for dispatch?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/almeetb">@almeetb</a><br>
‘<a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="nofollow noopener">https://pytorch.org/cppdocs/notes/tensor_indexing.html</a>’<br>
This might help</p>

username: almeetb
cooked: <p>Thanks! based on this it seems the <code>getter</code> operations of slices are correct but I never get any <code>setter</code> ops dispatched.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/almeetb">@almeetb</a><br>
You have your own dispatchKey. It seems to me that registering setter and getter are the same process.</p>
<p>Please create an issue in pytorch github. You need to provide the follow info:</p>
<ol>
<li>pytorch versions you are working on.</li>
<li>Sample code on how you add dispatchKey</li>
<li>Sample code on how you register ops</li>
<li>Errors or logs you have</li>
</ol>
<p>assign label ‘dispatch’ to the issue you created, we will have people answer your question there. thx</p>

username: almeetb
cooked: <p>I think the problem I am having is why I see certain operations being dispatched and not others. I don’t think I have an issue with how the registration is being done. If possible would you be able to run the sample code from above and verify which ops get dispatched. I am using pytorch 1.5</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/almeetb">@almeetb</a><br>
For your example above.<br>
If you translate that into c++ apis, it should call index().<br>
If you just do it from python, it won’t dispatch any of the indexing apis you’ve tried including index(), what it actually called is<br>
‘<a href="https://github.com/pytorch/pytorch/blob/3d46e02ea1f8951bcef167c33bad0217ec9da980/torch/csrc/autograd/python_variable_indexing.cpp#L268" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/3d46e02ea1f8951bcef167c33bad0217ec9da980/torch/csrc/autograd/python_variable_indexing.cpp#L268</a>’</p>

username: almeetb
cooked: <p>Thanks for your help! tracing through that showed me what my problem was</p>
C++ access to GPU tensor element

username: Lee_Newberg
cooked: <p>I have a multi-dimensional tensor that is torch::kCUDA and I want to change (or, respectively, see) just one element in my non-CUDA, C++ CPU code.  I’d like to do the equivalent of <code>myTensor[3][1][4] = 1.1f;</code>  How would I do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You will need to use a cuda kernel if you want to change one value like that based on a pointer.<br>
Another way around this would be to use <code>.index()</code> to get a Tensor containing only the one entry that you want and the use <code>.fill_(1.1f)</code> (that has a kernel already implemented) to fill that Tensor with the value you want.<br>
Would that work for you?</p>

username: Lee_Newberg
cooked: <p>Perfect!  For getting and setting respectively I am using</p>
<pre><code class="lang-auto">float myValue = myTensor.index( myIndex ).item&lt; float &gt;();
myTensor.index( myIndex ).fill_( myValue );
</code></pre>
<p>I realize that one-by-one access to elements is a crazy thing to do with tensors and it should only be used rarely, but now I know how to do it when I need to do it rarely.</p>
Torch.from_numpy: The returned tensor is not resizable

username: devstorm
cooked: <p>Greetings from India!<br>
I was referring the book: Deep Learning with PyTorch when I came across this function:<br>
<code>from_numpy</code><br>
It says in the docstring that “the returned tensor is not resizable” but when I did something like this:<br>
<code>img = torch.from_numpy(img_arr)</code><br>
<code>img.resize_(3,720,1280)</code><br>
It returned me a resized tensor. Can you please help?<br>
I have just started with PyTorch and I am absolutely loving it though.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The short answer I would give here is: "If you just started pytorch, you should not touch <code>.resize_()</code> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> "<br>
It is a fairly tricky function that actually resize the data storage for a Tensor and can contain un-initialized memory.</p>
<p>If you want to change the shape of a Tensor (without changing the number of elements) you should use <code>.view()</code> or <code>.reshape()</code> !</p>

username: devstorm
cooked: <p>Thanks for the suggestion! My original ask was as per docstring the operation:<br>
<code>img.resize_(3,720,1280)</code> should have resulted into error but it didnt. Why?<br>
Am I missing something?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/11b67debace172253e4598dd5ecea66b762997c7.png" data-download-href="https://discuss.pytorch.org/uploads/default/11b67debace172253e4598dd5ecea66b762997c7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/11b67debace172253e4598dd5ecea66b762997c7.png" alt="image" data-base62-sha1="2wH6rW2UuuAfB6hwhYlTkr9Kv8r" width="690" height="158" data-dominant-color="F5F5F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">777×179 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>“resizable” here does not refer to the <code>.resize_()</code> function. But means you can’t change the size of the underlying data container.<br>
What I guess happens here is that the size you give to <code>.resize_()</code> does not actually need to resize the underlying data container and so it is not an issue</p>

username: devstorm
cooked: <p>Thanks for the explanation!</p>
Sample code to stop and resume training using C++ API

username: Lin_Jia
cooked: <p>Sometimes we face this situation: we have a large network, so we want to train it for a while, save the network, and resume training another day. It took me a while to implement this in C++, thus sharing the code here.</p>
<p>// your customized net:<br>
class MyNetImpl: public torch::nn::Module {<br>
…</p>
<p>}</p>
<p>// wrap this into a Torch module so that it can be restored later via torch::load<br>
TORCH_MODULE(MyNet);</p>
<p>// In the training codes:</p>
<p>if (epoch %100 == 0) {<br>
torch::save(myNet, “myNet.pt”)<br>
// If you use a stateful optimizer here such as ADAM, also, please save the optimizer.<br>
}</p>
<p>// Next time when the program restart, you wanna resume training:</p>
<p>auto myNet = MyNet(make_shared())<br>
if (“myNet.pt” file exists) {<br>
torch::load(myNet, “myNet.pt”)<br>
}</p>
<p>That is it.</p>
Difference between copy_ and _copy_from

username: almeetb
cooked: <p>Hello,</p>
<p>I have added a new backend via <code>PrivateUse1</code> dispatch key and I had registered <code>copy_</code> to be registered but I notice that there is also a <code>_copy_from</code> I was wondering what the difference between these two ops are.</p>
<p>When I call <code>.to(device)</code> on without <code>copy_</code> being registered I get an error even if <code>_copy_from</code> is registered.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/almeetb">@almeetb</a><br>
‘<a href="https://github.com/pytorch/pytorch/blob/3c66a3795a58a3866daf6dd9b26d67eded4d5982/aten/src/ATen/native/Copy.cpp#L106" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/3c66a3795a58a3866daf6dd9b26d67eded4d5982/aten/src/ATen/native/Copy.cpp#L106</a>’<br>
See comments</p>
Optimizer doesn't work but manually updating weights does

username: Diidee
cooked: <p>I implemented a custom layer and I’m having trouble getting the optimizer to work. I create my optimizer</p>
<pre><code>optimizer = optim.SGD(net.parameters(), lr=0.01)
</code></pre>
<p>and my training loop which consists of this</p>
<pre><code>optimizer.zero_grad()  
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()  
</code></pre>
<p>doesn’t update the weights. I print the weights after initializing the model, then train, and print the weights again and they are the same. I also printed the gradients during training and they are nonzero.</p>
<p>However when I run</p>
<pre><code>learning_rate = 0.01
for f in net.parameters():
    f.data.sub_(f.grad.data * learning_rate)
</code></pre>
<p>it does update the weights. Does anyone know why this is happening?</p>

username: albanD
cooked: <p>Hi,</p>
<p>How and when do you create the optimizer?<br>
Do you change the weights after creating the optimizer by any chance?</p>

username: Diidee
cooked: <p>I just realized I wasn’t updating the optimizer with the new model in between runs. Thanks!</p>
Tensorflow's tf.bias_add() . Which method of torch can be used to replace it?

username: SeanSJ
cooked: <p>tf.nn.bias_add(value_tensor ,  b )</p>

username: albanD
cooked: <p>Hi,</p>
<p>Does a <code>+</code> works? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
What are the size of value_tensor and b?</p>
How to do deterministic the random crop?

username: seyeeet
cooked: <p>I want to do random crop on frames of videos, for each video I want to have similar random crop, but different for different videos.<br>
Is there a way to do it?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I think this post might help you:</p><aside class="quote quote-modified" data-post="2" data-topic="88928">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/data-rect-transforms-randomcrop-h-w-data-i-get-an-error-somebody-help-me/88928/2">data,rect=transforms.RandomCrop((h,w))(data) I get an error,somebody help me?</a> 
  </div>
  <blockquote>
    Hi, 

RandomCrop does not return a tuple. It only crops the image and returns a Image object containing the cropped image. If you want the exact crop indices, you need to first create a RandomCrop object, then use its static method .get_params(image, output_size) which returns starting indices i and j and and the length of height and width th and tw respectively.  After this, you can call torch.nn.functional.crop(img, i, j, th, tw) to get cropped image. 
I think my explanation was a little bit c…
  </blockquote>
</aside>

<p>Bests</p>
Load saved network

username: oasjd7
cooked: <pre><code class="lang-auto">[BEFORE]
net = nn.DataParallel(net).cuda()

torch.save(net, save_folder + '/' + 'net.pt')
net = nn.DataParallel(torch.load(save_folder + '/net.pt'), device_ids=[0]).to(device)

[AFTER]
torch.save(net.module.state_dict(), 'net.pt')
net.module.load_state_dict(torch.load('/net.pt'), strict=False)
</code></pre>
<p>I saved the saved model with [BEFORE] codes.<br>
Now I’ve changed codes from [BEFORE] to [AFTER]. Because I figured out there are some problems with [BEFORE] codes when I loaded the model with DataParallel.</p>
<p>But I have to use the trained model that saved with [BEFORE] codes. How can I load ?</p>
<p>If I trained the model with gpu_id:{2,3}, then should I load this model with {2,3} ?</p>

username: ptrblck
cooked: <p>If you need to load the <code>[BEFORE]</code> model, you could directly try to use:</p>
<pre><code class="lang-python">net = torch.load(PATH)
</code></pre>
<p>and make sure the source code is in the same location (and “important” parts are unchanged).<br>
If possible, use the same machine, which should load the model to the same devices.<br>
Once this is done, save the <code>state_dict</code> properly using the <code>[AFTER]</code> approach.</p>
Slow training using LMDB dataset and nn.DataParallel

username: samra-irshad
cooked: <p>Hi everyone, I am training my segmentation model using a image set of 8146 training images. Training was taking a long time so I created LMDB data first and then trained my model but I did not see any decrease in training speed. I am also using nn.DataParallel. Any idea how can I improve the training speed? Each epoch takes more than 20 minutes <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> … I have experience with training models on tensorflow using tf.records and it was quite fast.</p>

username: user_123454321
cooked: <p>Can you see how much your GPUs are utilized using nvidia-smi. If they are not utilized fully it means the pre-processing (could be reading time from hard-drive) is taking much more time to utilize the GPU properly.</p>

username: samra-irshad
cooked: <p>I dont think nvidia-smi gives a precise picture of GPU utilization. It just tells the GPU utilization at the very moment the model is.</p>

username: user_123454321
cooked: <p>Hmmm…you can use watch command</p>
<pre><code class="lang-auto">watch -n 0.1 nvidia-smi
</code></pre>
<p>or use dmon option</p>
<pre><code class="lang-auto">nvidia-smi dmon
</code></pre>
<p>will refresh the utilization every fraction of second or so</p>
Should i use multiple instances of activation functions?

username: Topsoil
cooked: <p>Should i use multiple instances of activation functions and things like torch.nn.upsample, or just one, so basically should i do this:</p>
<pre><code class="lang-auto">class model(nn.Module):
  def __init__(self):
    super(model,self).__init__()
    self.layer1 = nn.Linear(x,y)
    self.layer2 = nn.Linear(z,w)
    self.activation1 = nn.ReLU()
    self.activation2 = nn.ReLU()
 def forward(self, input):
    out = self.layer1(input)
    out = self.activation1(out)
    out = self.layer2(out)
    out = self.activation2(out)

    return out
</code></pre>
<p>or should i do this:</p>
<pre><code class="lang-auto">class model(nn.Module):
  def __init__(self):
    super(model,self).__init__()
    self.layer1 = nn.Linear(x,y)
    self.layer2 = nn.Linear(z,w)
    self.activation = nn.ReLU()
 def forward(self, input):
    out = self.layer1(input)
    out = self.activation(out)
    out = self.layer2(out)
    out = self.activation(out)

    return out
</code></pre>

username: ptrblck
cooked: <p>If the activation function (or generally, the module) does not contain any parameter of buffers, you could just reuse them.<br>
Some activation functions, such as <code>PReLU</code> contain trainable parameters and should thus not be reused (only if this explicitly fits your use case).</p>
Network Implementation

username: Maicol_Polvere
cooked: <p>Good Morning,<br>
I would need to recreate the model showed below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c07b8b4b78a2eefea505d140031b1a9010651bf0.png" data-download-href="https://discuss.pytorch.org/uploads/default/c07b8b4b78a2eefea505d140031b1a9010651bf0" title="immagine"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c07b8b4b78a2eefea505d140031b1a9010651bf0_2_690x308.png" alt="immagine" data-base62-sha1="rsMmInp39mrURCTECZvB6HXH9sc" width="690" height="308" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c07b8b4b78a2eefea505d140031b1a9010651bf0_2_690x308.png, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c07b8b4b78a2eefea505d140031b1a9010651bf0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c07b8b4b78a2eefea505d140031b1a9010651bf0.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">immagine</span><span class="informations">925×414 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>.</p>
<p>In the paper [here] (<a href="https://arxiv.org/pdf/1412.3128.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1412.3128.pdf </a>) it is written that "the convolutional layers are interspersed with normalization and maxpooling layers at various stages and it pretrained using ImageNet.</p>
<p>I would like to implement this model or alternatively I would like to know how to adapt the pretrained version of Alexnet (since the model is derived from Alexnet). The model is used to perform regression on bounding boxes. Do you have suggestions?</p>

username: ptrblck
cooked: <p>To create a custom model, you could have a look at the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">CIFAR10 tutorial</a>, which defines a custom model architecture.</p>
<p>On the other hand, you could also use the pretrained <code>torchvision.models.alexnet</code> and change the last output layer for your regression task. Do replace the last linear layer, you could use this code snippet:</p>
<pre><code class="lang-python">model = models.alexnet()
model.classifier[6] = nn.Linear(4096, nb_output)
</code></pre>
<p>where <code>nb_output</code> defines the number of regression outputs you are dealing with.<br>
The <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#finetuning-the-convnet">Finetuning tutorial</a> gives you more information about how to further fine tune the model.<br>
Note that the tutorial is dealing with a classification task, so you would have to at least change the criterion to  e.g. <code>nn.MSELoss</code>.</p>
Discrepancy in CPU and GPU inference times

username: raindrop
cooked: <p>Hi, I have model A (a Flownet based model) and model B (a PWC-Net based model). The inference time of model B is lower than model A on GPU, but on CPU the relation is reversed. I would expect that a model which is faster on GPU would also be faster on CPU, but this doesn’t seem to be the case. Can someone please let me know if this is normal and if there are any workarounds to mitigate the issue?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>How did you measure the performance for your CPU and GPU runs?<br>
Did you properly synchronize the code for the GPU run before starting and stopping the timer?</p>
<p>If so, your scripts might either face other unrelated bottlenecks, which might not give you a proper timing, and should thus be removed (e.g. remove the data loading and use random input tensors).<br>
Also, <code>torch.backends.cudnn.benchmark = True</code> will profile the kernels in the first iteration for each new input shape and will select the fastest one, which might yield a speedup.</p>
<p>If all this was already done, then your model might suffer from a lot of kernels with tiny workloads, so that your overall GPU runtime might see the kernel launch overheads.</p>

username: raindrop
cooked: <p>Thanks for the answer. I am not sure what you meant my synchronizing the code before the GPU run. Can you please elaborate? I just timed the inference time on a single sample as follows:</p>
<pre><code class="lang-auto">net=Model().to(device).eval()
t1=time.time()
flow=net(frames)
t2=time.time()
print("Inference time =", t2-t1)
</code></pre>
<p>There is no data loading in the timed part of the code as <code>frames</code> is a preloaded tensor. I also tried adding <code>torch.backends.cudnn.benchmark = True</code> as you suggested, but it does not result in any significant change in the inference time in either case.</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/mixed-precision-training-implementation/91646/2">Here</a> is a code snippet showing how to synchronize the code before starting and stopping the timer as well as using some warmup iterations to get more stable results.</p>
<p>If you are using <code>cudnn.benchmark = True</code>, the first iteration for a new input shape will be slower, as cudnn will profile different kernels and select the fastest one, so warmup iterations would be necessary.</p>
Abort message: 'terminating with uncaught exception of type c10::Error: _ivalue_ INTERNAL ASSERT FAILED at ../torch/csrc/jit/api/object.cpp:19, please report a bug to PyTorch. (_ivalue at ../torch/csrc/jit/api/object.cpp:19)

username: KevinZ1992
cooked: <p>When i’m loading a pt file in c++:</p>
<pre><code class="lang-auto">Abort message: 'terminating with uncaught exception of type c10::Error: _ivalue_ INTERNAL ASSERT FAILED at ../torch/csrc/jit/api/object.cpp:19, please report a bug to PyTorch.  (_ivalue at ../torch/csrc/jit/api/object.cpp:19)
</code></pre>
<p>NDK version：20.0.5594570</p>
<p>Someone can help，thank you.</p>

username: ptrblck
cooked: <p>Could you explain your use case and, if possible, create a code snippet to reproduce this issue?</p>

username: KevinZ1992
cooked: <p>I just write a new jni to load my serialized model on android,  a code snippet just like</p>
<pre><code class="lang-auto">            std::istringstream pt(s);
            module_ = torch::jit::load(pt);
            module_.eval();
</code></pre>
<p>This error doesn’t appear every time，and it just occurs on Pixel 2 so far.</p>

username: ptrblck
cooked: <p>Could you please create a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a>, so that it’ll be tracked?</p>

username: KevinZ1992
cooked: <p>A new  Github issue <a href="https://github.com/pytorch/pytorch/issues/43388" rel="nofollow noopener">#43388</a> just be created.</p>
Tensor condition slice in libtorch

username: zhijian_li
cooked: <p>Hi, I have tensor slicing problem here:<br>
In python:<br>
I can use <code>output[ output &lt; 0.5 ] = 0</code> to set somevalues to <code>0</code><br>
In C++:<br>
I don’t know how to do it.</p>

username: albanD
cooked: <p>I think the <code>.index()</code> method is used to mimick the <code>[]</code> behavior.<br>
You might want to check the doc in details though as I’m not sure what this function can do exactly <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: zhijian_li
cooked: <p>Thanks for your hint, I’ve checked the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="nofollow noopener">docs</a>, there is no condition indexing method found.</p>

username: albanD
cooked: <p>Interesting.<br>
Then you’ll have to use masked_fill directly then I guess <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a5118bdabdb9cc642a7a12ceaacfd0531.html?highlight=masked_fill">https://pytorch.org/cppdocs/api/function_namespaceat_1a5118bdabdb9cc642a7a12ceaacfd0531.html?highlight=masked_fill</a></p>

username: zhijian_li
cooked: <p>any example to create a mask?</p>

username: albanD
cooked: <aside class="quote no-group" data-username="zhijian_li" data-post="1" data-topic="93564">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhijian_li/48/6415_2.png" class="avatar"> zhijian_li:</div>
<blockquote>
<p>output &lt; 0.5</p>
</blockquote>
</aside>
<p>This is actually creating a boolean Tensor with the mask you want <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

username: zhijian_li
cooked: <p>Thank you, I’ve found the solution, and tested it.<br>
<code>output.index_put_({output &lt; 0.5}, 0)</code></p>
How to solve the Riccati equation in Pytorch

username: mmutic
cooked: <p>Hi, I am trying to solve the discrete Riccati equation using pytorch tensors. I have been attempting to use the control package</p>
<pre><code class="lang-auto">import control
import torch

n=2
d=1

Q = torch.eye(n)
R = torch.eye(d)

A = torch.rand(n,n)
B = torch.rand(n,d)

(P,L,K) = control.dare(A_opt, B_opt, Q, R)
</code></pre>
<p>However, when I run this I get the following error,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec7c452fbb60f875a79efc0fe250d73cc82096c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec7c452fbb60f875a79efc0fe250d73cc82096c6" title="Screen Shot 2020-08-17 at 7.21.44 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec7c452fbb60f875a79efc0fe250d73cc82096c6_2_690x205.png" alt="Screen Shot 2020-08-17 at 7.21.44 PM" data-base62-sha1="xK2VI6F91xaCI3ieKRwlT8M6tWS" width="690" height="205" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec7c452fbb60f875a79efc0fe250d73cc82096c6_2_690x205.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec7c452fbb60f875a79efc0fe250d73cc82096c6_2_1035x307.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec7c452fbb60f875a79efc0fe250d73cc82096c6.png 2x" data-dominant-color="F5F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-08-17 at 7.21.44 PM</span><span class="informations">1124×334 53.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The control package generally works with numpy arrays. Does anyone know how to solve the Riccati equation using pytorch tensors?</p>

username: ptrblck
cooked: <p>Since the <code>control</code> library doesn’t accept PyTorch tensors, you would have to reimplement the method manually in PyTorch.<br>
I don’t know which functions <code>control.dare</code> is calling and if all functions are available in PyTorch directly.</p>

username: mmutic
cooked: <p>Thank you! I will look into it</p>
Autocompletion does not work in torch.nn module

username: khirotaka
cooked: <p>I’m using PyCharm Pro 2020.2 and in PyTorch v1.5.1, I could type torch.nn and the IDE would suggest Conv1d etc., but after updating to v1.6.0, I don’t see that anymore.</p>
<p>How can I make it like before?</p>
<p>P.S.<br>
Is this a question I should be asking on the JetBrains forum?</p>

username: albanD
cooked: <p>Hi,</p>
<p>We are working a lot on trying to add type annotations and completion so this should be getting better, not worst <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Could you open a github issue about this please? So that we can track it there!</p>

username: khirotaka
cooked: <p>Thank you for your reply.<br>
I opened the issue !</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/43379" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/43379" target="_blank" rel="nofollow noopener">Autocompletion does not work in torch.nn module</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-08-21" data-time="00:54:19" data-timezone="UTC">12:54AM - 21 Aug 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/khirotaka" target="_blank" rel="nofollow noopener">
          <img alt="khirotaka" src="https://avatars1.githubusercontent.com/u/24205015?v=4" class="onebox-avatar-inline" width="20" height="20">
          khirotaka
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
I’m using PyCharm Pro 2020.2 and in PyTorch v1.5.1, I could type torch.nn and the IDE would suggest Conv1d etc.,...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to accelerate the process of tensor index_select and index_copy?

username: Tony_Lee
cooked: <p>I have defined a customed conv2d function and the forward function is as follows : which consists a tensor slice and tensor copy process,</p>
<pre><code class="lang-auto">def forward(self, input):
	self.output.index_copy_(1, self.k_out_mask, F.conv2d(torch.index_select(input, 1, self.k_in_mask), self.weight)
	
	return self.output
</code></pre>
<p>I tested the runing time of this forward process,  it cost about 40ms,<br>
but when I tested another forward function as follows,</p>
<pre><code class="lang-auto">def forward(self, input):

	return F.conv2d(input, self.weight)
</code></pre>
<p>This function does.t have the tensor slice and copy operation, but the conv2d input and weight size are same as the above function, it cost about 14ms.<br>
Why does the tensor slice and tensor copy operation brings so much time cost ?<br>
Can anyone help me to accelerate the tensor slice and tensor copy operation ?<br>
Thanks so much !</p>

username: ptrblck
cooked: <p>Are you running your code on the CPU or GPU?<br>
In the latter case, note that CUDA calls are asynchronous, so that you would need to synchronize the code before starting and stopping the timer:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.perf_counter()
# your code
torch.cuda.synchronize()
t1 = time.perf_counter()
</code></pre>

username: Tony_Lee
cooked: <p>Thanks for your reply, I know the tensor slice and copy process should’t be added to the running time, but can I have a more solid method to speed up the tensor slice and copy process, because the tensor slice and tensor copy process which brings much time delay for the conv2d function.</p>

username: ptrblck
cooked: <p>I mean maybe you are timing the code wrong and the majority of the time is indeed spent in the convolution.<br>
Could you time your codes again using the synchronization and post the timings?</p>

username: Tony_Lee
cooked: <p>Oh, I actually run the code on cpu to test the elapse time</p>

username: Shizhe_Zang
cooked: <p>Hi Tony,</p>
<p>I have the same problem. Have you find any solution on it (accelerate tensor index_select)?</p>
<p>Best regards,</p>
<p>Shizhe</p>
PyTorch Geometric: Replicating complex message-passing model from paper

username: Coprosmo
cooked: <p>Hi there,</p>
<p>I’m relatively new to PyTorch Geometric (I’ve coded up one GNN so far, though have some experience working with PyTorch), and for some research I’m doing, I want to implement the message-passing scheme described on page 4 of <a href="https://arxiv.org/pdf/1905.10006.pdf" rel="nofollow noopener">this paper</a>. It appears to be more complex than the examples I’ve seen.</p>
<p>The data is in a tree format, and for each neighbour of a node, the message-passing algorithm operates differently (see step 2 in the referenced paper) depending on whether the neighbour is the child or parent of the node (it passes the neighbour’s embedding through one of two MLPs). I’m not sure how I could code my MPNN to recognize when a neighbor is a parent or a child - perhaps it could be encoded as a feature in the edge between them?</p>
<p>The second issue I had was that the parent and child embeddings of a node are aggregated separately - the mean of each is taken, and concatenated with the original node’s embedding (step 3 of the paper). As far as I can see, PyG only allows a simpler aggregation method where all of the neighbors are considered at once.</p>
<p>If anyone has any thoughts on how these might be approached, I’d be very interested to hear them <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch dataloader restart connection with remote data lake possibility

username: rui_zhang_331
cooked: <p>Hi Pytorch Team</p>
<p>We are using dataloader to load some data from Azure data lake, we will put the Azure data lake client in data loader init function, however, there is risk that connection may be closed or timeout, etc. so I wonder is there a way for dataloader to auto restart , and more importantly,  resume from last loaded index and item? what is best practice here?</p>
Using the same BatchNorm for several layers?

username: urkatz
cooked: <p>Hi everyone,</p>
<p>I have a several layers of CNNs operating on the same size tensor. Can I use a single BatchNorm for all of them, or do I need to set a different BatchNorm per layer?</p>
<p>In other words, in my <strong>init</strong>() can I do:</p>
<pre><code class="lang-auto">self.batchNorm = nn.BatchNorm1d(128, affine=True)
self.conv1 = nn.Conv1d(in_channels = 128, out_channels = 128, kernel_size = 3, padding = 1)
self.conv2 = nn.Conv1d(in_channels = 128, out_channels = 128, kernel_size = 3, padding = 1)
self.conv3 = nn.Conv1d(in_channels = 128, out_channels = 128, kernel_size = 3, padding = 1)
</code></pre>
<p>and in my forward pass can I do:</p>
<pre><code class="lang-auto">x = self.conv1(x)
x = self.batchNorm(x)        
x = F.relu(x)

x = self.conv2(x)
x = self.batchNorm(x)        
x = F.relu(x)

x = self.conv3(x)
x = self.batchNorm(x)        
x = F.relu(x)
</code></pre>
<p>Or do I need to declare batchNorm1, 2, and 3 for each one of these conv layers?  I don’t know how it is implemented, so I don’t know if by reusing the same BatchNorm for different layers it will screw up the statistics it uses for computing the batch norm.</p>
<p>Thanks for your help!</p>

username: urkatz
cooked: <p>Also, some follow up questions:</p>
<p>The batchNorm usually goes BEFORE the activation function (ReLU), correct?</p>
<p>Also, is the argument in the BatchNorm the size of the input, or the number channels, or the batch size? Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Batchnorm actually has some learnable parameters. So if you re-use it, it will share these learnt parameters across the different use.<br>
Also it is saving the running stats to be used in evaluation mode. And so if you re-use it, these stats will be shared.<br>
It might or might not be what you want, depending on your use case.</p>
<p>You can check the doc for the detail on how to use each argument: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d">https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d</a></p>

username: urkatz
cooked: <p>Thanks for the response!  This answers my question.</p>
<p>So I guess I need to have different BatchNorm() statements for each of the CNNs for two reasons: 1) there are learnable parameters \alpha and \beta that might be different from layer to layer, 2) it seems that the BatchNorm stores the batch mean and the variance somewhere so that it can be used at run-time.  Where exactly is that stored though?  When I save out the model using torch.save(net.state_dict(), filename) is it dumped out along with the network weights?</p>
<p>Also, I can’t understand why I only need to input the number of channels and not the length of the signal (in the 1d case) or W,H (in the 2D case). Any ideas?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes it is saved along as they Buffers.</p>
<p>You don’t need to provide the signal length as the model does not need to know it. This dimension is reduced when computing the stats.</p>
Computing gradient of discontinuous operations

username: Dipayan_Das
cooked: <p>Say we have a simple tensor operation pipeline as:</p>
<p><span class="hashtag">#w</span>, x are 1D tensors and w.requires_grad=True<br>
act = torch.matmul(w,x)<br>
output = torch.max(0,act).bool().int()  <span class="hashtag">#output</span> = 1 if act &gt; 0 else output = 0, which is simply a step function</p>
<p>Now if i want to apply autograd to such a computational operation using output.backward(), would it be possible to obtain gradients for w, and eventually optimize it ?</p>

username: KFrank
cooked: <p>Hello Dipayan!</p>
<aside class="quote no-group quote-modified" data-username="Dipayan_Das" data-post="1" data-topic="93537" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dipayan_das/48/27375_2.png" class="avatar"> Dipayan_Das:</div>
<blockquote>
<p>which is simply a step function</p>
<p>would it be possible to obtain gradients for w, and eventually optimize it ?</p>
</blockquote>
</aside>
<p>The short answer is no.</p>
<p>You have a step function, so its derivative is 0 almost everywhere<br>
and undefined at the “interesting” point where the step takes place.<br>
What would you want the gradients to be?  How would you use such<br>
gradients in a gradient-descent optimization?</p>
<p>For backpropagation / gradient descent to work, your functions need<br>
to be usefully differentiable.  The typical approach in cases where you<br>
“want” a step function is to use a differentiable “soft” approximation to<br>
the step function such as <code>sigmoid()</code> or <code>tanh()</code>.</p>
<p>Good luck.</p>
<p>K. Frank</p>
CrossEntropyLoss weight source

username: apytorch
cooked: <p>For some reason, unknown to me now, at some point I switched the source of the CrossEntropyLoss weight vector.</p>
<ul>
<li>originally: the 1/(class samples) for the entire training set</li>
<li>currently: the 1/(class samples) for the current minibatch</li>
</ul>
<p>The current method has been working fine. And I can’t find any post online mentioning using the mini batch distribution. My back-justification thinking is that since the mini batches are randomly sampled, sometimes there are significant skews in the class distributions (this is objectively true for my dataset), and therefore I should “help” the criterion.</p>
<p>However, I wonder if it’s just overthinking the issue, or worse, leaving performance on the table.</p>

username: KFrank
cooked: <p>Hi apytorch!</p>
<aside class="quote no-group" data-username="apytorch" data-post="1" data-topic="93528" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/58956e/48.png" class="avatar"> apytorch:</div>
<blockquote>
<p>However, I wonder if it’s just overthinking the issue,</p>
</blockquote>
</aside>
<p>I think that you are just overthinking it.</p>
<p>Having said that, my intuition on the matter – without any evidence<br>
to back it up – is:</p>
<ol>
<li>
<p>Class weights in the loss function are rough values to help<br>
things along a little bit.  Your training process and results<br>
should be robust with respect to the exact values of the class<br>
weights you use.</p>
</li>
<li>
<p>My feeling is that estimating your class weights from the entire<br>
training set, and then using the same weights for all batches,<br>
should be sufficient, and probably even preferable.  It is true that<br>
(unless you construct your batches so the the number of samples<br>
from each class is the same from batch to batch) your class<br>
distributions will vary from batch to batch.  But because your<br>
learning rate is “small,” your training process will, in effect,<br>
average over several batches, smoothing out the varying class<br>
distributions.  This will be even more true if you use momentum.</p>
</li>
</ol>
<p>Best.</p>
<p>K. Frank</p>