akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does my libtorch program terminate with what(): Could not find schema for aten::ones

Shane_Dev: <p>Hello,</p>
<p>I followed the <a href="https://github.com/pytorch/pytorch/blob/v1.7.0/CONTRIBUTING.md#c-development-tips" rel="noopener nofollow ugc">instructions for building libtorch</a> as a shared library, then successfully compiled and ran a simple program on Linux -</p>
<pre><code class="lang-auto">#include &lt;ATen/ATen.h&gt;

at::Tensor a = at::ones({2, 2}, at::kInt);
at::Tensor b = at::randn({2, 2});
auto c = a + b.to(at::kInt);
</code></pre>
<p>Next I compiled libtorch as a <strong>static</strong> library and successfully compiled it with my simple program. However it crashes when run -</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  Could not find schema for aten::ones.
Exception raised from findSchemaOrThrow at ../aten/src/ATen/core/dispatch/Dispatcher.cpp:81

(gdb) bt
#0  c10::Dispatcher::findSchemaOrThrow (
    this=0x555555e4b860 &lt;c10::Dispatcher::singleton()::_singleton&gt;, 
    name=0x555555a67762 "aten::ones", overload_name=0x555555a655b8 "")
    at ../aten/src/ATen/core/dispatch/Dispatcher.cpp:81
#1  0x000055555559fc75 in at::ones (size=..., options=...) at aten/src/ATen/Functions.cpp:4390
#2  0x000055555557a181 in main () at /home/shane/dev/aten_libtorch/aten_min.cpp:4
</code></pre>
<p>Here is my <a href="https://github.com/shanemcandrewai/aten_libtorch" rel="noopener nofollow ugc">CMake script</a></p>
<p>Does anyone know why?</p>
Accessing parameter grads as a list

cpsanders: <p>If I want a network’s parameters, I can easily throw them into a list using</p>
<pre><code class="lang-auto">params = list(network.parameters())
</code></pre>
<p>Is there an easy way to do this for the gradients of the network parameters? I know each gradient can be accessed using <code>grad = param.grad</code>, but storing all these gradients in an array requires me to iterate over each parameter in the network and access its <code>.grad</code>. Any help is appreciated here!</p>

albanD: <p>Hi,</p>
<p>You can use <code>grads = autograd.grad(loss, network.parameters())</code> and it will return to you a list of all the gradients.<br>
If you already did a <code>.backward()</code> that populated the <code>.grad</code> fields, you can do <code>grads = [p.grad for p in network.parameters()]</code> as well.</p>
UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples

talhak: <p>Getting the warning <code>UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples</code> when I calculate the confusion matrix of my classification model through the <code>scikit-learn</code>'s <code>confusion_matrix</code> function. I have re-confirmed that the class <code>1</code> contains images for training, validation, and test purposes just like the other classes. How can I fix this issue?</p>
<p>Here is the output:</p>
<pre><code class="lang-auto">[[115   0  17  66  96 183  18]
 [ 11   0   2  10   8  21   2]
 [ 42   0  35  68 102 172  93]
 [ 15   0   9 690  63 102  19]
 [ 12   0   7  86 315 186  13]
 [ 27   0  13  84 151 317  15]
 [  8   0  18  44  46  30 254]]
</code></pre>
<p>And here is my classification report:</p>
<pre><code class="lang-auto">              precision    recall  f1-score   support

           0      0.500     0.232     0.317       495
           1      0.000     0.000     0.000        54
           2      0.347     0.068     0.114       512
           3      0.658     0.768     0.709       898
           4      0.403     0.509     0.450       619
           5      0.314     0.522     0.392       607
           6      0.614     0.635     0.624       400

    accuracy                          0.481      3585
   macro avg      0.405     0.391     0.372      3585
weighted avg      0.475     0.481     0.451      3585
</code></pre>

huluager_wale: <p>how to fix recall,precision and f-score ill deffiend problems</p>
How is gradient calculated?

Flock1: <p>Hi,</p>
<p>I was wondering how does PyTorch calculate the gradient since I am interested in using my own loss function. I saw that <code>torch.autograd.functional.jacobian</code> calculates the derivative of the values that it gets using a formula.</p>
<p>I tried to go through the code in GitHub but I’m not able to exactly see the code which interprets the formula used (eg if we use <code>np.exp</code> to calculate the value, then the derivative is the same value, Or if we use <code>x**2</code>, then the <code>jacobian</code> will print <code>2x</code>). How does it interpret which mathematical operation is being performed and how to calculate the derivative of that.</p>
<p>Also, does it calculate the derivative of non-differentiable functions?</p>
<p>Please let me know.</p>

albanD: <p>Hi,</p>
<p>PyTorch uses automatic differentiation to compute all the gradients. See <a href="https://en.wikipedia.org/wiki/Automatic_differentiation">here</a> for more info about AD.</p>
<blockquote>
<p>Also, does it calculate the derivative of non-differentiable functions?</p>
</blockquote>
<p>For functions that are non-differentiable only at a single point, we extend the gradient with either a subgradient or by continuity.<br>
For fonctions that are more generally non-differentiable, it will raise an error if you try to give a Tensor that requires gradients to them.</p>
How to modify layers at the end of a Segmentation network in Pytorch

RohitDulam: <p>Hi everyone. As the title suggests, I want to modify the last few layers of the Semantic Segmentation networks provided by Pytorch. I want to use the output coming from the last ReLu layer to my own layer. I’m really having a tough time getting it working. So I’m trying to get the network output at the last ReLu layer which is just before the output layer.</p>
<pre><code class="lang-auto">t = torchvision.models.segmentation.deeplabv3_resnet50()
wow = nn.Sequential(*list(t.modules())[:-1])
wow(torch.ones((2, 3, 224, 224)).cuda())
</code></pre>
<p>The above code gives me an error saying</p>
<pre><code class="lang-auto">TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not collections.OrderedDict
</code></pre>
<p>I’m really not able to understand where I am going wrong. So what I want is the output at the last but one layer. But I do not know how to access it.</p>

ptrblck: <p><code>nn.Sequential</code> containers can be used for simple feed forward models, where the layers are just connected sequentially and pass a single output to the next layer.<br>
Often models are more complicated than this approach and cannot be easily mapped to <code>nn.Sequential</code>, which also seems to be the case here.<br>
Based on the error message I assume that a particular layer outputs an <code>OrderedDict</code>, which is then processed in the original <code>forward</code> method and creates the error in your approach.</p>
<p>To remove some layers I would thus recommend to replace them directly with your new layers using the layer name:</p>
<pre><code class="lang-python">model.layer = MyLayers()
</code></pre>
<p>or if you want to disable them assign an <code>nn.Identity()</code> module to the corresponding attribute.</p>

RohitDulam: <p>Thank you for your reply. Yeah, when I wanted to change the individual layers, I get a different output than when I’m running the whole network together. The segmentation.py file has a list of boolean variables which are used to change strided convolutions with dilated ones, which I don’t know how to do it while accessing individual modules of the segmentation network. Also, I just want to add a layer to the end. I guess I have to write the whole network again.</p>
Freeze BatchNorm layer lead to NaN

Chen-Wei_Xie: <p>Hi, everyone</p>
<p>I want to freeze BatchNorm while fine-tuning my resnet (I mean, use global mean/std and freeze weight and bias in BN), but the loss is so large and become nan at last:</p>
<pre><code>iter =  0 of 20000 completed, loss =  [ 15156.56640625]
iter =  1 of 20000 completed, loss =  [ nan]
iter =  2 of 20000 completed, loss =  [ nan]
</code></pre>
<p>the code I used to freeze BatchNorm is:</p>
<pre><code>def freeze_bn(model):
    for name, module in model.named_children():
        if isinstance(module, nn.BatchNorm2d):
            module.eval()
            print 'freeze: ' + name
        else:
            freeze_bn(module)
model.train()
freeze_bn(model)
</code></pre>
<p>if I delete ‘freeze_bn(model)’, the loss converge：</p>
<pre><code>iter =  0 of 20000 completed, loss =  [ 27.71678734]
iter =  1 of 20000 completed, loss =  [ 15.12455177]
iter =  2 of 20000 completed, loss =  [ 16.49391365]
iter =  3 of 20000 completed, loss =  [ 16.47186661]
iter =  4 of 20000 completed, loss =  [ 6.9540534]
iter =  5 of 20000 completed, loss =  [ 7.13955498]
iter =  6 of 20000 completed, loss =  [ 4.7441926]
iter =  7 of 20000 completed, loss =  [ 15.24151039]
iter =  8 of 20000 completed, loss =  [ 12.98035049]
iter =  9 of 20000 completed, loss =  [ 3.7848444]
iter =  10 of 20000 completed, loss =  [ 4.14818573]
iter =  11 of 20000 completed, loss =  [ 4.04237747]
iter =  12 of 20000 completed, loss =  [ 4.52667046]
iter =  13 of 20000 completed, loss =  [ 4.85921001]
iter =  14 of 20000 completed, loss =  [ 3.59978628]
</code></pre>
<p>Why the global mean and std make the loss nan?<br>
Hope for help, Thank you!</p>

ruotianluo: <pre><code>def set_bn_eval(m):
    classname = m.__class__.__name__
    if classname.find('BatchNorm') != -1:
      m.eval()

model.apply(set_bn_eval)
</code></pre>
<p>you should use apply instead of searching its children.<br>
named_children() doesn’t iteratively search submodules.</p>

highoneming: <p>What is your learning rate ??<br>
your loss is 15156 ,don’t you think it is too big?</p>

wsdgh: <p>But<br>
<code>else: freeze_bn(module)</code><br>
freeze layer recursively.</p>

Flicic_Suo: <p>I encountered with similar problem after freezing BN. Yet my loss doesn’t raise to a very high magnitude, but just shifts to NaN abruptly at some epoch. Anyway, the problem seems gone after I tuning down the learning rate, but I still don’t know the cause of it.</p>
Custom convolution layer to handle nans in image

Pablo_Jeken: <p>Hi everybody,</p>
<p>I am working on creating a custom cnn layer to process the data from a map. The data values are masked on landmass, hence I was thinking of creating a convolution that either ignores nans or forces their weight to be zero (previously converting nans in for instance -1).</p>
<p>Is this a feasible approach? Can you maybe give me some hints on how to start/which class to extend?</p>
<p>Thanks!</p>
Torch.jit.trace() fails if model forward function takes more than one argument

Himanshu_Sundriyal1: <p>My PyTorch model takes more than one argument in forward() function. That is apart from data input, it takes additional arguments for some computation. When I execute</p>
<p><code>torch.jit.trace(loadedmodel, example_inputs = data)</code></p>
<p>I get an error as forward() missing 4 required positional arguments: ‘l1’, ‘l2’, ‘criterion1’, and ‘criterion2’. Since I have already done training with this model, I loaded this model in variable loadedmodel and I am not able to figure out how can I send loadedmodel to torch.jit.trace function which expects model to take only data input in forward function. Please Help</p>

Fei_Wu: <p>You just need to specify example_inputs as a tuple:</p>
<p><code>torch.jit.trace(loadedmodel, example_inputs = (x, l1, l2, criterion1, criterion2))</code></p>
Tensor Average based on the output shape

Aron: <p>I am new to deep learning. I passed the image through my network and I have received the tensor of shape [1, 1, 48, 48] and I want to take an average and convert it to [1, 48]. How can I do that?</p>

ptrblck: <p>Depending which dimension should be used to calculate the average or mean, you could use:</p>
<pre><code class="lang-python">x = torch.randn(1, 1, 48, 48)
y = x.mean(2) # or .mean(3)
print(y.shape)
y = y.squeeze(1) # or .squeeze(0)
print(y.shape)
</code></pre>
Model giving same prediction

Sheethal00: <p>Hi, I have nueral network that is used to classify a patch as eye or not, give eye landmark and bounding box predictions.</p>
<p>All the three output layers are fully connected linear layers.</p>
<p>The problem with the model is, it is giving same output for the first output layer which has two nodes.</p>
<p>Here is my model architecture.</p>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 32, 3)<br>
torch.nn.init.xavier_normal_(self.conv1.weight)<br>
self.pool1 = nn.MaxPool2d(3,stride=2,ceil_mode=True)<br>
self.conv2 = nn.Conv2d(32, 64, 3)<br>
torch.nn.init.xavier_normal_(self.conv2.weight)<br>
self.pool2 = nn.MaxPool2d(3,stride=2,ceil_mode=True)<br>
self.conv3 = nn.Conv2d(64, 64, 3)<br>
torch.nn.init.xavier_normal_(self.conv3.weight)<br>
self.pool3 = nn.MaxPool2d(2,stride=2,ceil_mode=True)<br>
self.conv4 = nn.Conv2d(64, 128, 2)<br>
torch.nn.init.xavier_normal_(self.conv4.weight)<br>
self.fc1 = nn.Linear(3<em>1</em>128, 256)<br>
self.cls_prob_fc = nn.Linear(256,2)<br>
self.bbox_pred_fc = nn.Linear(256,4)<br>
self.lm_pred_fc = nn.Linear(256,12)</p>
<pre><code>def forward(self, x):
    x = self.pool1(F.relu(self.conv1(x)))
    x = self.pool2(F.relu(self.conv2(x)))
    x = self.pool3(F.relu(self.conv3(x)))
    x = F.relu(self.conv4(x))
    x = x.view(x.size(0), 1*3*128)
    x = F.relu(self.fc1(x))
    x_cls_prob = self.cls_prob_fc(x)
    x_bbox_pred = self.bbox_pred_fc(x)
    x_lm_pred = self.lm_pred_fc(x)    
    return x_cls_prob,x_bbox_pred,x_lm_pred
</code></pre>
<p>Could anyone please help me with this</p>

ptrblck: <p>It your model is not training properly I would generally recommend to try to overfit a small dataset, e.g. just 10 samples, and make sure your model is able to do so by playing around with some hyperparameters.<br>
If this is not possible, you might have some bugs in the code, e.g. forgetting to zero out the gradients.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Finetune and WARNING:root:NaN or Inf found in input tensor

mitr: <p>Hello everyone!<br>
I am trying to finetune one model (U-net architecture) for segmentation, which is trained on CityScapes, by using data from Mapillary Vistas. However, vistas have more classes then cityscapes. So,  I translated classes from vistas to cityscapes classes by myself.<br>
After that I started training and saw this warning: <code>WARNING:root:NaN or Inf found in input tensor</code>. I don’t understand how this appears. I hope, someone can help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Code of model I use: <a href="https://github.com/PingoLH/FCHarDNet" rel="noopener nofollow ugc">https://github.com/PingoLH/FCHarDNet</a></p>

ptrblck: <p>Which layer is raising this issue?<br>
Could you add a check for the input data and target and make sure that no invalid values are found via:</p>
<pre><code class="lang-python">print(torch.isfinite(tensor).all())
</code></pre>
DataLoader very slow when dataset is big and shuffle is True

Hui_Li: <p>I have a big dataset with lots of images and I found that the speed of dataloader is very slow. I did many tests and found when the number of images is big:</p>
<ol>
<li>Direct read the dataset is fast</li>
<li>Set shuffle = False with num_workers=0, also fast (1.1 times slower than 1st one)</li>
<li>Set shuffle = False with num_workers=8, becomes slow (2.8 times slower than 1st one)</li>
<li>Set shuffle = True with num_workers=0, slower (7.2 times slower than 1st one)</li>
<li>Set shuffle = True with num_workers=8,(9.2 times slower than 1st one)</li>
</ol>
<p>Here is the sample code:</p>
<pre><code class="lang-auto">dataset = MyTestDataset()

since = time.time()
imgs = []
for idx in range(128):
    img,label = dataset[10000+idx*1000]
    imgs.append(img)
base = time.time() - since
print("next dataset",base)

loader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=False, num_workers=0)
since = time.time()
imgs = next(iter(loader))
cost = time.time() - since
print("loader no shuffle num_workers-0",cost,cost/base)

loader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=False, num_workers=8)
since = time.time()
imgs = next(iter(loader))
cost = time.time() - since
print("loader no shuffle num_workers-8",cost,cost/base)

loader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=True, num_workers=0)
since = time.time()
imgs = next(iter(loader))
cost = time.time() - since
print("loader shuffle num_workers-0",cost,cost/base)

dataset = MyTestDataset()
loader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=True, num_workers=8)
since = time.time()
imgs = next(iter(loader))
cost = time.time() - since
print("loader shuffle num_workers-8",cost,cost/base)
</code></pre>
<p>and below is the output</p>
<pre><code class="lang-auto">next dataset 0.2839939594268799
loader no shuffle num_workers-0 0.3133578300476074 1.103396109832709
loader no shuffle num_workers-8 0.811976432800293 2.859132759157693
loader shuffle num_workers-0 2.041795253753662 7.189572827091643
loader shuffle num_workers-8 2.617912769317627 9.218198776483705
</code></pre>
<p>Seems shuffle make it 9 times slow?</p>
How to implement a split batchnorm in PyTorch?

tau: <p>The picture explains what I want to do.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9075ba21b2e1d8205a981b99e6282860c4cc5248.png" data-download-href="https://discuss.pytorch.org/uploads/default/9075ba21b2e1d8205a981b99e6282860c4cc5248" title="屏幕快照 2020-11-05 17.36.12"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9075ba21b2e1d8205a981b99e6282860c4cc5248.png" alt="屏幕快照 2020-11-05 17.36.12" data-base62-sha1="kBWZ3JItqQtqZI1trFIPhYXXgow" width="690" height="279" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕快照 2020-11-05 17.36.12</span><span class="informations">1040×421 46.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tau: <p>I read the following discussion. <a href="https://discuss.pytorch.org/t/how-does-one-make-sure-that-the-parameters-are-update-manually-in-pytorch-using-modules/6076">https://discuss.pytorch.org/t/how-does-one-make-sure-that-the-parameters-are-update-manually-in-pytorch-using-modules/6076</a></p>
<p>Can we implement such a feature without replacing the nn.Sequential()?</p>
TTS Tacotron Experienced Consultant

johnpisg: <p>Hello guys, currently we are going to start a project using Tacotron, We want to generate Text To Speech audios from a lot of Datasets we have based on human voice talents. So We are looking for a experienced consultant that can lead this project, share the knowledge with our team and work together to build awesome features.<br>
Please you can contact me here or sending me an email to <a href="mailto:devteam@snaprecordings.com">devteam@snaprecordings.com</a><br>
I appreciate your help guys.</p>

ptrblck: <p>I’ll move this post into the <code>jobs</code> category for better visibility. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Clone a variable on multiple GPUs

amin_sabet: <p>Hi,</p>
<p>In my code, I want to keep the latest input feature map to the layer and subtract it with the input feature map and then update the latest value.<br>
The code is working on a single GPU, but I get the following error when running on multiple GPUs. Any thoughts?</p>
<blockquote>
<p>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!</p>
</blockquote>
<pre><code class="lang-auto">class Subtraction(nn.Module):
    def __init__(self, channel,width,height):
        super(Subtraction, self).__init__()
        
        self.latest_input = Variable(torch.zeros(channel,width,height), requires_grad=False).cuda()

    def forward(self, input):
        #difference of feature maps
        output = input - self.latest_input
        self.latest_input = input.clone()
        return output</code></pre>

ptrblck: <p>If you are using <code>DataParallel</code> or <code>DistributedDataParallel</code>, remove the <code>cuda()</code> call in <code>self.latest_input</code>.<br>
These wrappers will push the model replica to the corresponding device for you, while you are explicitly creating this tensor on <code>cuda:0</code>, which could yield the device mismatch error.</p>
<p>PS: <code>Variable</code>s are deprecated since PyTorch <code>0.4</code> so you can use tensors directly in newer versions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

amin_sabet: <p>Thanks for your help. I’m using DataParallel. I removed “.cuda()” in “self.latest_input” statement, but It seems that “input.clone()”  copy the tensor on CPU as I get the following error.</p>
<blockquote>
<p>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!</p>
</blockquote>

ptrblck: <p>Sorry for forgetting the second part of the right approach. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>The new error is not raised by <code>input.clone()</code>, but should still be raised by <code>self.latest_input</code>, which is now on the CPU as it wasn’t properly registered.</p>
<p>To make sure <code>model.to()</code> pushes all internal parameters and buffers to the desired device you have to register <code>self.latest_input</code> as an <code>nn.Parameter</code>, if it should be trained, or as a buffer if it shouldn’t:</p>
<pre><code class="lang-python">self.latest_input = nn.Parameter(torch.zeros(channel, width, height)
# or
self.register_buffer('latest_input', torch.zeros(channel, width, height))
</code></pre>

amin_sabet: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:">.</p>
Make a binary classification RNN/LSTM  to only focus on "malicious" words to make the model more robust? (Non negative model)

OneAndOnly: <p>There are some works that make a model robust against attacks, by making the model “focus” on only malicious features, therefore addition of benign features will not affect the outcome of model and only addition of malicious content can change its prediction.</p>
<p>more on this :</p>
<aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1806.06108.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1806.06108.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1806.06108.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1806.06108.pdf" target="_blank" rel="noopener nofollow ugc">1806.06108.pdf</a></h3>

  <p class="filesize">375.14 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><a href="https://towardsdatascience.com/evading-machine-learning-malware-classifiers-ce52dabdb713" class="inline-onebox" rel="noopener nofollow ugc">Evading Machine Learning Malware Classifiers | by William Fleshman | Towards Data Science</a>  (read Non-Negative MalConv)</p>
<blockquote>
<p>Non-Negative MalConv was constrained during training to have<br>
non-negative weight matrices. The point of doing this is to prevent<br>
trivial attacks like those created against MalConv. When done<br>
properly, the non-negative weights make binary classifiers monotonic;<br>
meaning that the addition of new content can only increase the<br>
malicious score. This would make evading the model very difficult,<br>
because most evasion attacks do require adding content to the file.<br>
Fortunately for me, this implementation of Non-Negative MalConv has a<br>
subtle but critical flaw.</p>
</blockquote>
<p>my question is, how can i implement this in RNN/LSTM using pytorch?</p>
<p>my model currently takes a sequence of words and predicts whether the sentence is malicious or not, and right now addition of a lot of benign words (by beaning i mean words that appear in a lot of benign sentences) will evade the model.</p>
<p>basically i want my model to learn to predict based on only words that are malicious, meaning mostly appear in malicious samples, and addition of benign words that appear in many benign sentences will not change the prediction of the sentence (so adding many benign words will not affect the prediction)</p>
<p>how can i implement this? is this possible?</p>
How to register buffer without polluting state_dict?

pierrecurie: <p>I have a fixed tensor that needs to be moved to GPU according to DataParallel’s whims, but I don’t want to bloat the state_dict with this tensor (since it never changes).  I can’t seem to figure out a good solution.  [for a little more context, this tensor is the weight in F.conv_transpose2d]  The things I’ve considered:</p>
<ol>
<li>override _apply - for single GPU, this seems to work fine, but for multi GPU, DataParallel seems to rely on replicate… which doesn’t use _apply and instead iterates over all buffers and parameters [the same things that goes into the state_dict]</li>
<li>register_buffer (currently using) - code runs, but state_dict is bloated with unnecesary crap; also if this tensor is ever modified (accidentally or otherwise), entire network is silently corrupted</li>
<li>custom code for saving checkpoints by manually removing this tensor from state_dict every single time - ugly</li>
<li>move tensor to appropriate GPU during every forward pass - probably slow</li>
<li>construct tensor during every forward pass - probably even slower</li>
<li>any other ideas?</li>
</ol>

JuanFMontesinos: <p>Just create a flag to check whether the tensor has been allocated or not. Allocate it once.<br>
Another option is just to delete it when you stop training.</p>
<p>Anyway I think it’s not bad it stays as a buffer. It’s somehow a necessary parameter to run inference. What would you do for exporting the model, keep telling everyone those values? Seems a bit annoying to me.</p>

pierrecurie: <p>The whole point of allocating every time is that it would allocate on the correct GPU.  If it’s only allocated once, then that would be option 4).  From my tests, it seems option 4) is about as fast as 2), so maybe I can just do that (assuming it works in multi GPU context…)</p>
<p>A checkpoint is saved every epoch, so deleting it before saving is a variation of 3).  I guess that’s an option.</p>
<p>Well, ideally this buffer is instantiated with the rest of the net in *.py, so “keep telling everyone those values” isn’t much of an issue.</p>

drevicko: <p>With recent pytorch versions, you can use <code>register_buffer(persistent=False)</code> to prevent the buffer from appearing in the state_dct. This is present in 1.7 and not present in 1.4, not 100% sure when it appeared.</p>
Timing isn't working correctly

Dzhange: <p>Hi, my code runs extremely slow so I was checking each step for the reason, however, there seems to be something wrong with the timing in PyTorch</p>
<pre><code class="lang-auto">       t1 = torch.cuda.Event(enable_timing=True)
        t2 = torch.cuda.Event(enable_timing=True)        
        t3 = torch.cuda.Event(enable_timing=True)        

        torch.cuda.synchronize()
        t1.record()
        
        mask_loss_2 = nn.functional.binary_cross_entropy(out_mask, target_mask)
                
        torch.cuda.synchronize()
        t2.record()
       # self.mask_loss = nn.BCELoss(reduction='mean')
        mask_loss = self.mask_loss(out_mask, target_mask)
        
        torch.cuda.synchronize()
        t3.record()

        if show_time:
            torch.cuda.synchronize()
            print(" Loss time ", t1.elapsed_time(t2), t2.elapsed_time(t3))
</code></pre>
<p>and the results are</p>
<pre><code class="lang-auto">Loss time  594.671630859375 4.7564802169799805
</code></pre>
<p>But then I switched the order of 2 step:</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
        t1.record()
        mask_loss = self.mask_loss(out_mask, target_mask)
                
        torch.cuda.synchronize()
        t2.record()

        mask_loss_2 = nn.functional.binary_cross_entropy(out_mask, target_mask)
        
        torch.cuda.synchronize()
        t3.record()
</code></pre>
<p>and the results are</p>
<pre><code class="lang-auto">Loss time  580.7462158203125 4.7769598960876465
</code></pre>
<p>So it seems the first BCEloss in each step took much longer time than the second<br>
I also feel that this timing isn’t accurate because I’m pretty sure the major time consumption happened in some complex operations after this step. Do you know why is this happening?</p>
<p>Thanks!</p>

ptrblck: <p>I would recommend to add some warmup workloads to avoid accidentally profiling the CUDA context creation and general startup time as seen here:</p>
<pre><code class="lang-python"># warmup
x = torch.randn(1024, 1024, device='cuda')
out = torch.matmul(x, x)

# create events
t1 = torch.cuda.Event(enable_timing=True)
t2 = torch.cuda.Event(enable_timing=True)        
t3 = torch.cuda.Event(enable_timing=True)        

# profile1
t1.record()
out = torch.matmul(x, x)
         
# profile2
t2.record()
for _ in range(10):
    out = torch.matmul(x, x)
 
# end timer
t3.record()

# print results
torch.cuda.synchronize()
print(" Loss time ", t1.elapsed_time(t2), t2.elapsed_time(t3))
</code></pre>
<p>Changing the workload (single <code>matmul</code> and the <code>matmul</code> in the loop) results in the expected swapped timings.</p>
Multiple forward passes followed by one backward pass -- DistributedDataParallel fails while DataParallel works

ramprasaath: <p>I am doing 2 forward passes on a resnet and trying to compute the gradients using the outputs from the first forward pass. When using multiple GPUs this works when the model is wrapped through nn.DataParallel but not when wrapped through nn.DistributedDataParallel. Below you can find my code.</p>
<pre><code class="lang-auto">import torch
import torchvision
import torch.backends.cudnn as cudnn
import torchvision.models as models

import utils.distributed as dist

def main():
    # Get the current device as set for current distributed process.
    # Check `launch` function in `utils.distributed` module.
    device = torch.cuda.current_device()

    # create model
    model = models.resnet50().cuda(device)
    batch_size = 32

    # define loss function (criterion) and optimizer
    criterion = torch.nn.CrossEntropyLoss().to(device)

    cudnn.benchmark = True

    # Wrap model in DDP if using more than one processes.
    if dist.get_world_size() &gt; 1:
        dist.synchronize()
        model = torch.nn.parallel.DistributedDataParallel(
            model, device_ids=[device], find_unused_parameters=True
        )
        # Using DataParallel works fine
        # model = torch.nn.DataParallel(
        #     model, device_ids=[device]
        # )

    ip_1 = torch.rand(batch_size,3,224,224).cuda(device)
    op_1 = model(ip_1)
    target_1 = torch.zeros(batch_size, dtype=torch.long).cuda(device)

    ip_2 = torch.rand(batch_size,3,224,224).cuda(device)
    op_2 = model(ip_1)
    target_2 = torch.zeros(batch_size, dtype=torch.long).cuda(device)

    # loss for the first example
    loss = criterion(op_1,target_1)
    loss.backward()    #----------&gt; Fails here when DDP is used

if __name__ == "__main__":
    dist.launch(
        main,
        num_machines=1,
        num_gpus_per_machine=2,
        machine_rank=0,
        dist_url='tcp://localhost:10001',
        dist_backend='nccl'
    )
</code></pre>
<p>The error I get is</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048]] is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</code></p>
<p>In case you need to look at <code>distributed.py</code>, <a href="https://gist.github.com/ramprs/b2fdad64d23c9281253f72d16a7f6dde" rel="noopener nofollow ugc">here</a> it is. My torch version is <code>'1.7.0a0+8deb4fe'</code></p>
<p>Thanks in advance for your help</p>

pritamdamania87: <p><a class="mention" href="/u/ramprasaath">@ramprasaath</a> This looks like a bug, could you please file an issue at <a href="https://github.com/pytorch/pytorch" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch</a>?</p>
Is it something wrong about the LeNet-5 sturcutre in PyTorch Blitz?

HaFred: <p>I was looking at <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py</a>. For the network it defines, the FC3_weight connects F6_layer and output_layer without ReLU, and then put into crossentropy loss.<br>
But according to the illustration of the network, also the LeCun paper (<a href="http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf" rel="noopener nofollow ugc">http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf</a>), the output layer has been Euclidean Radial Basis Function unit, rather than further pushes into the crossentropy loss. So this illustration here does not match with the network defined below right?</p>
<p>It is kind of confusing. Thank you very much.</p>
Masked_select per sample in batch

Kevinkevin189: <p>The input tensor is a NCHW dim tensor, and I got a mask tensor with size N1HW  indicates  every pixel’s label.(0 or 1).I’d like to group within each sample by this mask and do  some matrix manipulation.such as<br>
0 group and 1 group’s inner product. Is there any good idea about how to group per sample in a batch?</p>
Get first negative element of row, else minimum

sidm: <p>Suppose I have a 2D tensor. I want to get the index of the first negative element of the tensor if there are any negative elements, else get the index of the smallest element.  Basically, I want to implement the following for loop over each row of a 2D tensor efficiently in pytorch. Is there a way to do this with pytorch operations in a single iteration over each row?</p>
<pre><code class="lang-auto">x = 10.0 # all numbers are guaranteed less than 10

for i in range(len(row)):
  if row[i] &lt; 0:
     x = i
     break

  if row[i] &lt; x:
     x = i
</code></pre>

sidm: <p>Figured it out:</p>
<pre><code class="lang-auto">            clamped = torch.clamp(rays, 5e-11, 10.0)
            minIdx = torch.argmin(clamped, dim=1)
</code></pre>
<p>-5e-11 is an upper bound on the negative numbers I’m interested in.</p>
Set cudnn.benchmark locally (embedder but not classifier)

gabrieldernbach: <p>I am using a ResNet50 as feature extractor and I would like it to run it with cudnn.benchmark=True, as I know the input shape does not change. As the number of created embeddings can differ, the following classifier will have a variable-length input, so I would prefer to set cudnn.benchmark=False for it.</p>
<p>The documentation is rather succinct and I did not find corresponding code in the repo.<br>
Where can I find more information about the benchmarks inner working?</p>

ptrblck: <p>You can use the context manager to enable/disable <code>benchmark</code> for a region:</p>
<pre><code class="lang-python">with torch.backends.cudnn.flags(enabled, benchmark, deterministic, allow_tf32):
    some_layer
</code></pre>
Way to determine the epoch except from Early Stopping

sshen82: <p>Hi,<br>
I am working in VAE, and I do know that people use Early stopping to determine a better epoch, however, is there a better way to do so? After all, using Early stopping requires validation dataset, but my model is used to cluster the data from latent variables, and also, I don’t want to have a lower amount of data used in training.</p>
How to add additional information for backward?

mathkobe: <p>Hi, I’m new learner of pytorch, I tried to add very simple information for backward() from layer to layer besides output_grad. However, it wouldn’t work even for the following extremely simple code.  Can someone help me on how to fix this ? Thanks a lot!</p>
<pre><code>import torch
import torch.autograd as autograd
import torch.nn as nn

class MyFun(torch.autograd.Function):
    def forward(self, inp):
        return inp

    def backward(self, grad_out, P):
        grad_input = grad_out.clone()
        print('Custom backward called!')
        return grad_input, P-1

class MyMod(nn.Module):
    def forward(self, x):
         return MyFun()(x)

mod1 = MyMod()

y = autograd.Variable(torch.randn(1), requires_grad=True)
z = mod1(y)
print(z.type)
P = autograd.Variable(torch.ones((1,1)))
#z.backward(P.data)
#z.backward(P)
z.backward()
</code></pre>
<p>TypeError: backward() missing 1 required positional argument: ‘P’</p>

colesbury: <ol>
<li>Your backward won’t be called because you return the original input ‘inp’. (This is changed in 0.3, but until then return <code>inp.clone()</code>)</li>
<li>Your backward takes in too many arguments. It should only take in grad_out because the forward only returns a single argument.</li>
<li>Your backward returns too many values. It should only return a single value because the forward only takes in a single value.</li>
</ol>

mathkobe: <p>Thanks a lot ! If I just want to backprop some P which doesn’t need to do anything for forward(). Do you have any suggestions on where I should add ? Is it better to just define another function in the nn.Module ? Thanks again for your help.</p>

philippwitte: <p>Any updates on whether this is possible? The context is the following: I want to implement a custom backward function for an invertible layer. Instead of saving the activations during the forward pass, the backward function takes the original output <code>y</code> as and additional input argument, from which the input <code>x</code> is recomputed and used to compute gradients. I therefore need my custom backward function to not only accept the gradient as an input, but also the original output (in the example of <a class="mention" href="/u/mathkobe">@mathkobe</a> this is the additional argument <code>P</code>).<br>
I added the additional argument to the backward functions in <code>torch/tensor.py</code> and <code>torch/autograd/__init__.py</code>. But after that, the functions step into the C code and I cannot follow them with the Python debugger.<br>
Any help or pointers on how to extend this would be great! Thanks a lot!</p>
Combining input streams in a proper way

ivankrylatskoe: <p>Hello!<br>
Say, we have an intermediate layer of a neural network, which gets two inputs:</p>
<ol>
<li>Output generated by a previous layer in the following format: N=2000 elements, Cin=10 channels, H=W=100 pixels</li>
<li>Another input in the following format: N=2000 elements, Cin=1 channel, H=W=100 pixels</li>
</ol>
<p>And we need to combine these two inputs into one and then apply a convolution to the combined data, which will have the following format: N=2000 elements, Cin = 10+1 = 11 channels, H=W=100 pixels</p>
<p>What is the most efficient way to do this?</p>
<p>I’ve considered several options, but all of them look bad.</p>
<ol>
<li>
<p>In the given layer construct a new tensor which will have for each input element 10 channels from the first input stream and 1 channel from the second one. This approach will require resizing of input tensor on the fly, which is not very efficient.</p>
</li>
<li>
<p>Use 11 input channels in all previous operations, but somehow restrict the operations to use only 10 of them to avoid adding unused weights. Unfortunatelly, I didn’t found how to implement this approach.</p>
</li>
<li>
<p>Use 11 input channels in all previous operations, and do not restict them from using 11th channel. This will generate redundand connections (weights) between neurons.</p>
</li>
</ol>
<p>Do you have any ideas?</p>

googlebot: <p>If you don’t want to use torch.cat (“resizing”), you can use distributivity property:<br>
conv(cat(A,B), cat(W1,W2), bias) = conv(A,W1)+conv(B,W2, bias)<br>
and do two “conv” ops (10 to 10 and 1 to 10)</p>

ivankrylatskoe: <p>Hi! Thank you for your answer. Eventually, I’ve used torch.cat.<br>
I use group=1 in conv operation (both outputs depend on both inputs), therefore the property has to be rewritten in the following way:<br>
conv(cat(A,B), cat(W1,W2)) = cat(conv(A,cat(W1,W2)), conv(B,cat(W1,W2))<br>
And the implementation of this technique seems to be even less effective than a single torch.cat.</p>

googlebot: <p>groups don’t matter, the principle is the same as with matrix multiplication:</p>
<p>with concatenated inputs you have<br>
batch x 11 @ 11 x 10 = batch x 10</p>
<p>equivalent with split matrices:<br>
batch x 10 @ 10 x 10 + batch x 1 @ 1 x 10</p>
<p>Now, two separate matmul/conv ops with smaller tensors are likely to be slower than one with concatenated inputs. The only upside of avoiding cat is memory savings (also note that inplace summation is possible).</p>
Quantized Conv Pseudo Code

vlad123: <p>What is the pseudo code for a quantized conv1d ?  I haven’t been able to work out the requantization and bias portion.</p>
<p>At the bottom is a conv1d model with 1x1 filter and [1] input, if that helps.</p>
<p>In the forums, I’ve seen the requantization scale param is defined as</p>
<pre><code class="lang-auto">requant_scale = input_scale * weight_scale / output_scale
</code></pre>
<p>Even with all zero_points=0, the following equation is close but not exactly correct</p>
<pre><code class="lang-auto">output = (input * weight * requant_scale + bias_quant) * model.conv.scale
</code></pre>
<p>Simple Model:</p>
<pre><code class="lang-auto">M(
  (quant): Quantize(scale=tensor([0.0160]), zero_point=tensor([127]), dtype=torch.quint8)
  (conv): QuantizedConv1d(1, 1, kernel_size=(1, 1), stride=(1, 1), scale=0.0021364481654018164, zero_point=0)
  (dequant): DeQuantize()
)
input=tensor([[[-2.0260]]])
input_quant=tensor([[[0]]], dtype=torch.uint8)
weight_quant=tensor([[[-128]]], dtype=torch.int8)
weight=tensor([[[-0.6016]]], size=(1, 1, 1), dtype=torch.qint8,
       quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.0047], dtype=torch.float64), zero_point=tensor([0]),
       axis=0)
bias=tensor([-0.9427], requires_grad=True)
output=tensor([[[0.2756]]])

</code></pre>
<p><a class="mention" href="/u/supriyar">@supriyar</a>  <a class="mention" href="/u/jianyuhuang">@jianyuhuang</a></p>
Questions about facebookresearch/XLM

pascal_notsawo: <p>I started experimenting with the XLM framework myself from <a href="https://github.com/facebookresearch/XLM" rel="nofollow noopener">https://github.com/facebookresearch/XLM</a>.</p>
<p>I have the impression that this documentation is only for those who have implemented it, since there is no other documentation available (in case of an error you just have to unlock it yourself).</p>
<p>Nevertheless I have some questions. I have my own data (monolingual and paralingual) in the txt files, and I’d like to apply the decu preprocessing (fastBPE…) until I have the data understandable by the train.py script (BERT, XLM…).</p>
<p>At the same time, I’d like to understand all the formats (pth, …) used in the official website.</p>
<p>Each time I used the data from wikipedia and … wastes a lot of time</p>
Increasing both loss and F1 Score when using weighted BCE_Logits

ed-fish: <p>I have some unusual behavior when I train my Multi-Label Classifier using BCE_Logits_Loss with the pos_weights calculated using the following function.</p>
<pre><code class="lang-auto">def calculate_pos_weights(class_counts):
    pos_weights = np.ones_like(class_counts)
    neg_counts = [len(data)-pos_count for pos_count in class_counts]
    print(neg_counts)
    for cdx, (pos_count, neg_count) in enumerate(zip(class_counts,  neg_counts)):
      pos_weights[cdx] = neg_count / (pos_count + 1e-5)
    print(pos_weights)
    return torch.as_tensor(pos_weights, dtype=torch.float)

criterion = nn.BCEWithLogitsLoss(pos_weight=calculate_pos_weights(weight_balance).to(device))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0062c6c7986c0d63df6bcf0199363b7ccc3387b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/0062c6c7986c0d63df6bcf0199363b7ccc3387b4" title="Screenshot from 2020-11-04 14-56-38"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0062c6c7986c0d63df6bcf0199363b7ccc3387b4_2_690x276.png" alt="Screenshot from 2020-11-04 14-56-38" data-base62-sha1="3pCUDWCIEPqcSjJqGKwLS8kNkU" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0062c6c7986c0d63df6bcf0199363b7ccc3387b4_2_690x276.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0062c6c7986c0d63df6bcf0199363b7ccc3387b4_2_1035x414.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0062c6c7986c0d63df6bcf0199363b7ccc3387b4_2_1380x552.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-11-04 14-56-38</span><span class="informations">1734×696 68.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Notice how the Recall starts at 1 and is reducing while precision is increasing (therefore improving the F1 score), but the loss is going up, indicating that the model is over-fitting.</p>
<p>When I train this model without the pos_weight I do not have this issue but I believe I can get better performance with a weighted loss.</p>
<p>Can anyone find and explain the issue?</p>
<p>Thanks!</p>