akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
AE and Unsupervised Clustering

Andrea_Grioni: <p>Hi all!</p>
<p>I am working on a dataset of ~300 samples with ~5000 data-points each - ranged between 0 and 1. I am interested in:</p>
<ol>
<li>Group samples for similarity;</li>
<li>Find the differences between groups;</li>
</ol>
<p>Would make sense to train an <strong>autoencoder</strong> to reduce the dimensionality to N points. Take the output of the encoder and use it as the input of an unsupervised algorithm (KNN, DBSCAN)?</p>
<p>if so, is it correct to use a sigmoid and relu activation for the encoder and decoder, respectively?</p>
<p>The AE architecture:</p>
<pre><code class="lang-auto">class AE(nn.Module):
    def __init__(self):
        super().__init__()

        self.encoder_hidden_layer_1 = nn.Linear(in_features=4979 , out_features=3000)
        self.encoder_hidden_layer_2 = nn.Linear(in_features=3000, out_features=1500)
        self.encoder_output_layer = nn.Linear(in_features=1500, out_features=10)
        
        self.decoder_hidden_layer_1 = nn.Linear(in_features=10, out_features=512)
        self.decoder_hidden_layer_2 = nn.Linear(in_features=512, out_features=2000)
        self.decoder_output_layer = nn.Linear(in_features=2000, out_features=4979)

    def forward(self, features):
        x = self.encoder_hidden_layer_1(features)
        x = F.relu(x)
        x = self.encoder_hidden_layer_2(x)
        x = F.relu(x)
        x = self.encoder_output_layer(x)
        encoded = F.sigmoid(x)
        
        x = self.decoder_hidden_layer_1(x)
        x = F.relu(x)
        x = self.decoder_hidden_layer_2(x)
        x = F.relu(x)
        x = self.decoder_output_layer(x)
        decoded = F.relu(x)
        
        return decoded, encoded
</code></pre>
Ghost batch norm / Virtual batch size

Maximal: <p>Hi,</p>
<p>I was trying to replicate some experiments done in TF and noticed that they use something called virtual batch size. Some papers have shown that the per device batch size and the accuracy of batch norm estimates that comes with it can matter and is often a reason why large batch size training does not perform as well as training with smaller batch sizes. At the same time, training with larger batches, especially on lower dimensional data (eg 32x32 images) often yield better GPU utilization. Is there a way to replicate this ghost batch norm in Pytorch, eg can I have a batch norm layer that automatically subdivided the batch into smaller micro-batches and computes statistics on each individual one? Right now per device batch size is coupled to total batch size and number of GPUs I am using which makes it hard to experiment with it, eg if I want to use a total bs of 1024 and a virtual batch size of 64 I need to use 16 GPUs.<br>
I found one repo that does that but they actually split the batch and perform multiple forward passes which is super inefficient.</p>
<p>Thank’s for your help,</p>
I need just one example of tensor for multiclass object detection

Stihl: <p>Hi<br>
If anybody could send me just one example of torch.tensor for multicalss detection training I would be very thankfull.<br>
From pytorch tutorial I found only oneclass detection dataset.<br>
But if I have several classes:<br>
What should be the box for image if there is no object with this label on it?<br>
Probably it should be [0,0,0,0] but I’m not sure…<br>
Correct me if I’m not right, labels should be one-hot-encoded for multiclass detection?<br>
And I don’t understand how boxes tensor should look like if there are several objects for 1 label on the picture?<br>
Please help, pytorch tutorial is completely useless</p>
One hot encoding for BCE loss?

Michael_Lempart: <p>Hi all,</p>
<p>Iam wondering if I need to one-hot encode labels for a segmentation task when using BCEwithoutLogits?<br>
The pixels in my segmentation masks are either 0 (background) or 1(foreground).</p>
<p>In some code examples I saw that labels where on-hot encoded, in others the output channel of the model is equal to 1.</p>
<p>So I now Iam somewhat confused…</p>
<p>Thanks in advance,</p>
<p>cheers,</p>
<p>Michael</p>

KFrank: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Michael_Lempart" data-post="1" data-topic="104357" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_lempart/48/12375_2.png" class="avatar"> Michael_Lempart:</div>
<blockquote>
<p>Iam wondering if I need to one-hot encode labels for a segmentation task when using BCEwithoutLogits?</p>
</blockquote>
</aside>
<p>Please see my final comment in this reply to your other thread:</p>
<aside class="quote quote-modified" data-post="5" data-topic="104158">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/same-output-when-using-multilabel-bce/104158/5">Same output when using Multilabel BCE</a> 
  </div>
  <blockquote>
    Hi Michael! 

Does this mean that: 


The large structure is – either in the real world and/or in your 
target data – exactly the union of the three substructures? 


The three substructures have exactly zero intersection with 
one another? 


If so, would it suffice to simply segment the three substructures, with 
the large structure being determined by post-processing that takes 
the union of the substructures, rather than directly by the model itself? 
You could argue that getting the large s…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
Modify subset of a tensor

science: <p>Hi,</p>
<p>I have a 2d tensor W and 2 boolean masks that select the wanted rows and columns of a tensor.  For example:<br>
<code>W[row_mask][:, column mask]</code><br>
returns the tensor subset of interest.  I want to modify these elements (for example, add 1 to each).  However, doing something like:<br>
<code>W[row_mask][:, column mask] += 1</code><br>
leaves the W tensor unchanged.  Is there a way I can modify only a subset of a tensor, selected by rows and columns?</p>

googlebot: <p>It is inefficient to do selective writes with cuda, so this is not well supported, though there are masked_scatter_ and scatter_add_ ops.<br>
But sequential operations like W += row_mask*column_mask or W = torch.where(mask, f(W), W) should be faster.</p>
How to build a U-Net and train it correctly?

Jingnan_Jia: <p>I used 2 methods to build a U-Net in Pytorch. But the performance of the 2 nets are different. I want to know if the 2 methods are the same or not?</p>
<p>First method to build U-Net:</p>
<pre><code class="lang-auto">class UNet_1(nn.Module):
  def __init__(self):
  ......
  def forward(self, x):
            x0 = self.conv_0(x)

        x1 = self.down_1(x0)
        x2 = self.down_2(x1)
        x3 = self.down_3(x2)
        x4 = self.down_4(x3)

        u4 = self.upcat_4(x4, x3)
        u3 = self.upcat_3(u4, x2)
        u2 = self.upcat_2(u3, x1)
        u1 = self.upcat_1(u2, x0)

        logits = self.final_conv(u1)
        return logits
  

net = UNet_1()
</code></pre>
<p>Second method to build U-Net:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
  def __init__(self):
  ......
  def forward(self, x):
    x0 = self.conv_0(x)
    x1 = self.down_1(x0)
    x2 = self.down_2(x1)
    x3 = self.down_3(x2)
    x4 = self.down_4(x3)
    return x0, x1, x2, x3, x4

class Decoder(nn.Module):
  def __init__(self):
  ......
  def forward(self, x0, x1, x2, x3, x4):
    u4 = self.upcat_4(x4, x3)
    u3 = self.upcat_3(u4, x2)
    u2 = self.upcat_2(u3, x1)
    u1 = self.upcat_1(u2, x0)

    logits = self.final_conv(u1)
    return logits


class UNet_2(nn.Module):
    def __init__(self, enc, dec):
        super().__init__()
        self.enc = enc
        self.dec = dec

    def forward(self, x):
        x0, x1, x2, x3, x4 = self.enc(x)
        out = self.dec(x4)
        return out

enc = Encoder()
dec = Decoder()
unet= UNet_2(enc, dec)
</code></pre>
<p>The training would be like:</p>
<pre><code class="lang-auto">opt = torch.optim.Adam(unet.parameters(), lr=1e-4)
x, y = dataloader()
pred = unet(x)
loss = cretera(pred, y)
opt.zero_grad()
loss.backward()
opt.step()
</code></pre>
<p>I found the second method cannot work normally according to the results. Because during the training of the second method I found the training loss decreased in the first several epochs, but increased again afterwards. finally the validation dice result is 0. I donot know why. Maybe because of <code>net.parameters()</code>? Any ideads would be appreciated.</p>

JuanFMontesinos: <p>Both should work the same way…<br>
It can be a matter of initialization and hyperparameters.</p>
Passing hidden state through LayerIntegratedGradients

devpramod: <p>I was trying to replicate the CNN text model interpretability example using an LSTM. But the LayerIntegratedGradients throws an error about the missing hidden state.</p>
<p>forward() missing 1 required positional argument: ‘hidden’</p>
<p>What would be the best way of handling this?</p>

googlebot: <p>create a function wrapper (partial function or lambda), hiding “auxiliary” parameters from captum</p>
<p>or use additional_forward_args argument (see docs)</p>
How to do symmetric padding and what is circular padding

jiang_ruofan: <p>The Conv2d function only has 4 types of padding mode which are ‘zeros’ , ‘reflect’ , ‘replicate’ or ‘circular’. I want to know how I could do symmetric padding using pytorch. For example, I want a tensor [[1, 2, 3], [4, 5, 6]] to become this one after padding. And I want to know what the circular padding mode is. Could you give me some visulization to explain the circular padding? Really appreciate your answer!<br>
[[2, 1, 1, 2, 3, 3, 2],<br>
[2, 1, 1, 2, 3, 3, 2],<br>
[5, 4, 4, 5, 6, 6, 5],<br>
[5, 4, 4, 5, 6, 6, 5]].</p>
How can I get just one batch of DataLoader.torch.utils.data with fix index each time?

Nazila-H: <p>Hi everyone,</p>
<p>I am new in PyTorch, I want to get just one batch from DataLoader and have the index of samples as well. and every time I want just to use these index ids (without changing ids). any idea?</p>
<p>Thank you in advance.</p>

osalpekar: <p><a class="mention" href="/u/nazila-h">@Nazila-H</a> Is your question in the context of using DataLoader specifically in a distributed training setting, or is it about more general DataLoader behavior? If it’s the latter, there may be folks with more insight into this area if you post in the Uncategorized topic.</p>

Nazila-H: <p>Yes, it is in distributed framework.<br>
I want to have the same indexes of the samples in three different virtual machines, can I modify shuffle in: DataLoader.torch.utils.data or define a fix number for a kind of random seed in each machine?</p>
Whether `cummax.indices` returns first, last etc. occurence is not part of the specification

patrick-kidger: <p>I’ve got a use case of the following form:</p>
<pre><code class="lang-auto">import torch
tensor = torch.tensor([False, True, False, False, True])
indices = tensor.cummax(dim=0).indices
print(indices)  # tensor([0, 1, 1, 1, 4])
</code></pre>
<p>in which I am relying on the fact that (at least based on my tests) <code>cummax.indices</code> always gives the <em>last</em> occurence of a maximum. The documentation leaves it unspecified what the indices are when there are multiple maximums – i.e. that last <code>4</code> could equally well be a <code>1</code>.</p>
<p>Just in case this changes in future versions of PyTorch I’m adding a test to my library that this behaviour remains consistent. How worried should I be about that possibility / can it be added to the specification for <code>cummax.indices</code> that it keeps this behaviour?</p>
Can the computation of n-th order derivatives be sped up?

domst: <p>Hey,</p>
<p>for my current project I’m using PyTorch to evaluate PDEs and need to compute up to fourth order derivatives from the network output w.r.t. its input. Up to right now I’m computing them very naively by just passing the previous derivatives into the next grad operator, like so:</p>
<pre><code class="lang-python">net = torch.nn.Sequential(
  torch.nn.Linear(2,50),
  torch.nn.Tanh(),
  torch.nn.Linear(50,1)
)

x = torch.randn(100,2).requires_grad_(True)

for ep in range(epochs):
  optimizer.zero_grad()
  
  z = net(x)
  dzdx = torch.autograd.grad(z, x, grad_outputs=torch.ones_like(x[:,0:1]), create_graph=True)[0][:,0:1]
  dzdxx = torch.autograd.grad(dzdx, x, grad_outputs=torch.ones_like(x[:,0:1]), create_graph=True)[0][:,0:1]
  dzdxxx = torch.autograd.grad(dzdxx, x, grad_outputs=torch.ones_like(x[:,0:1]), create_graph=True)[0][:,0:1]
  dzdxxxx = torch.autograd.grad(dzdxxx, x, grad_outputs=torch.ones_like(x[:,0:1]), create_graph=True)[0][:,0:1]

  loss = some_comb_of_derivatives()
  loss.backward()
  optimizer.step()
</code></pre>
<p>This however is really slow and consumes a lot of memory (which is also expected when creating the graph for each differentiation), especially for bigger networks and batch sizes. Is there something I’m doing wrong conceptually or is there a possibility to speed up the computation of the derivatives?</p>
Multi-gpu DDP in Jupyter Notebook

rooks: <p>I try to run the example from the DDP tutorial:</p>
<pre><code class="lang-python">import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP

def example(rank, world_size):
    # create default process group
    dist.init_process_group("nccl", rank=rank, init_method=None, world_size=world_size)
    # create local model
    model = nn.Linear(10, 10).to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))
    labels = torch.randn(20, 10).to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()

def main():
    world_size = 2
    mp.spawn(ex,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__ == '__main__':
    main()
</code></pre>
<p>I get an error<br>
<code>Exception: process 0 terminated with exit code 1</code></p>
<p>I am running this in a jupyter notebook inside a docker container.</p>
<p>When I run this as a script inside the container but outside jupyter, it seems it works fine.</p>
<p>What would be the reason it is not working in jupyter?</p>
<p>In general, what is the method to use DDP in a notebook?</p>

Alexey_Demyanchuk: <p>Perhaps this thread could help <a href="https://discuss.pytorch.org/t/distributeddataparallel-on-terminal-vs-jupyter-notebook/101404">link</a></p>
How to make a 2D tensor whose diagonal and its neighbors are ones

kkjh0723: <p>I want to make a 2D tensor whose diagonal and its neighbors are filled with ones and others are zeros.</p>
<p>For example 5x5 matrix and neighbor 1,<br>
[[1, 1, 0, 0, 0],<br>
[1, 1, 1, 0, 0],<br>
[0, 1, 1, 1, 0],<br>
[0, 0, 1, 1, 1],<br>
[0, 0, 0, 1, 1]]</p>
<p>Is there any pytorch operation for making those matrix?</p>

KaiHoo: <pre><code class="lang-auto">n = 5
torch.diagflat(torch.ones(n-1), 1) + torch.diagflat(torch.ones(n-1), -1) + torch.eye(n)
</code></pre>
Error with c++ import in python

GuillaumeAnoufa: <p>Hello,</p>
<p>I am trying to import c++ code in python following this tutorial : <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a></p>
<p>However i’m running into an error that I don’t understand :</p>
<pre><code class="lang-auto">ImportError: dynamic module does not define module export function (PyInit_test)
</code></pre>
<p>It is very simple to reproduce the error:<br>
I use two files :<br>
test.py</p>
<pre><code class="lang-auto">import os
from torch.utils.cpp_extension import load
module_path = os.path.dirname(__file__)

test = load(
    name='test',
    sources=[os.path.join(module_path, "test_cpp.cpp")],
    extra_cflags=['-O2'],
    verbose=False)
</code></pre>
<p>and test_cpp.cpp</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;

int placeholder() {
    std::cout &lt;&lt; "hello" &lt;&lt; '\n';
    return 0;
}
</code></pre>
<p>“python test.py” to reproduce the error.</p>
<p>Do you have any idea where this error could come from ?</p>

thomasjo: <p>You need to configure Python bindings using the pybind11 macros shown here: <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#binding-to-python" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html#binding-to-python</a></p>
<p>Remember to also include the extension header — <code>#include &lt;torch/extension.h&gt;</code></p>
<p>I would highly recommend reading the entire tutorial (perhaps even twice) to get an understanding of all the mechanisms in play here. Lots of moving parts, and definitely not trivial.</p>
<p>Good luck and have fun <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

GuillaumeAnoufa: <p>Thanks for your answer, it works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Have a good day !</p>
Question about Output of torchtext.data.iterator

Hou_ZeYu: <p>I try to use <strong>torchtext</strong> library to help me handle a machine learning problem.</p>
<p>I use the simple data for an example, the goal is understand how to create the data object and dataloader before I start training model.</p>
<p>The csv file look like blow, there are only one column “text” and the element is sentence.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/7769076cb9c5989726b4c2754dc27f9917610999.png" data-download-href="https://discuss.pytorch.org/uploads/default/7769076cb9c5989726b4c2754dc27f9917610999" title="2020-11-27 16-14-42 的螢幕擷圖"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/7769076cb9c5989726b4c2754dc27f9917610999.png" alt="2020-11-27 16-14-42 的螢幕擷圖" data-base62-sha1="h2lQeTnshBFUaxYjtZRruyL2gfn" width="136" height="500" data-dominant-color="343841"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020-11-27 16-14-42 的螢幕擷圖</span><span class="informations">156×570 19.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And I create a data object successfully.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffa1941923ffea0e8820d473122f665b205693ce.png" alt="2020-11-27 16-15-58 的螢幕擷圖" data-base62-sha1="AtpVZnzKAQJZZzHOYhGja6nATTg" width="265" height="489"></p>
<p>After I build the vocab, I can use iterator to show mini-batch like blow.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2be0ca85ce199a9efb259a7a032761b0af6fb600.png" alt="2020-11-27 16-17-46 的螢幕擷圖" data-base62-sha1="6ga8Vl3bkoKHEQpUmRNdIsMXe6I" width="319" height="468"></p>
<p>I can understand the first column is index of words, but I cannot understand the second column, is it frequency of words?</p>

Hou_ZeYu: <p>The size column is talk about batch size, I understand it.</p>
How to convert 2D Transformer Encoder output to 1D matrix?

Sine: <p>Hi, I am trying to train a CVAE with a Transformer encoder. The shape of the encoder output is [Batch size, Sequence length, Embedding size]. However, the input shape of the recognition model is [Batch size, Embedding size].</p>
<p>I came out with two ideas to solve this problem. One is to use linear layers. I reshape the encoder output to  [Batch size, Sequence length * Embedding size], as linear layer inputs. The other is to use 1-layer RNN. I utilize the last hidden states of the RNN, whose shape is [Batch size, Embedding size].</p>
<p>I don’t know which one is better in theory. Are there other solutions?</p>
<p>Thank you.</p>
Choose weights in a VGG16

Fabrice_Auzanneau: <p>Hello all<br>
I’m trying to build a VGG16 model to make an ONNX export using Pytorch. I want to force the model with my own set of weights and biases. But in this process my computer quickly runs out of memory.</p>
<p>Here is how I want to do it (this is only a test, in the real version I read the weights and biases in a set of files), this example only force all values to 0.5</p>
<pre><code class="lang-auto"># Create empty VGG16 model (random weights)
from torchvision import models
from torchsummary import summary

vgg16 = models.vgg16()
# la structure est : vgg16.__dict__
summary(vgg16, (3, 224, 224))

#  convolutive layers
for layer in vgg16.features:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        dim = layer.weight.shape
        print(dim)
        print(str(dim[0]*(dim[1]*dim[2]*dim[3]+1))+' params')

        # Remplacement des poids et biais
        for i in range (dim[0]):
            layer.bias[i] = 0.5
            for j in range (dim[1]):
                for k in range (dim[2]):
                    for l in range (dim[3]):
                        layer.weight[i][j][k][l] = 0.5

# Dense layers
for layer in vgg16.classifier:
    print()
    print(layer)
    if (hasattr(layer,'weight')):
        dim = layer.weight.shape
        print(str(dim)+' --&gt; '+str(dim[0]*(dim[1]+1))+' params')
        for i in range(dim[0]):
            layer.bias[i] = 0.5
            for j in range(dim[1]):
                layer.weight[i][j] = 0.5
</code></pre>
<p>When I look at the memory usage of the computer, it grows linealrly and saturates the 16GB RAM during the first dense layer processing. Then python crashes…</p>
<p>Is there another better way to do this, keeping in mind that I want to onnx export the model afterwards?<br>
Thanks for your help.</p>

ptrblck: <p>I guess you might be using an older PyTorch version, as you would get this error in the latest stable version:</p>
<pre><code class="lang-python">RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>Wrap your code into a <code>with torch.no_grad()</code> block and check, if the memory is still growing.</p>

Fabrice_Auzanneau: <p>Thanks, it better now</p>
How to get parameters in Tensorflow like Pytorch?

TinNguyenqn: <p>Hi everyone,<br>
I want to ask a question. In Pytorch, when I want to get data of a Conv2d layer, I just need to:<br>
for p in conv1.parameters():<br>
p.data…</p>
<p>But how can I do the same in Tensorflow?<br>
Thank you.</p>

ptrblck: <p>I would recommend to ask Tensorflow questions in their discussion group or StackOverflow, as you would most likely find more TF experts there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
'BCEWithLogitsLoss' object has no attribute 'backward'

svss: <p>Hello guys,<br>
I’m trying to fine-tune the Bert model i.e., bert-base-uncased for a text classification task. I’m getting a strange error while calling the backward() on loss, which is as follows</p>
<p><strong>torch.nn.modules.module.ModuleAttributeError: ‘BCEWithLogitsLoss’ object has no attribute ‘backward’.</strong></p>
<p>I can’t find any syntax error and also checked the inputs(outputs and targets) to the loss function i.e, nn.BCEWithLogitsLoss(outputs, targets) and I found then to be in the correct format. I don’t know what is causing this error. I would be great if any of you could help me with this…thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Here is the code</p>
<p>def train_loop_fn(data_loader, model, optimizer, device, scheduler = None):<br>
model.train()<br>
for bi, d in enumerate(data_loader):<br>
ids = d[“ids”]<br>
mask = d[“mask”]<br>
token_type_ids = d[“token_type_ids”]<br>
targets = d[“targets”]</p>
<pre><code>    ids = ids.to(device, dtype=torch.long)
    mask = mask.to(device, dtype=torch.long)
    token_type_ids = token_type_ids.to(device, dtype=torch.long)
    targets = targets.to(device, dtype=torch.float)

    optimizer.zero_grad()
    outputs = model(ids=ids, mask = mask, token_type_ids = token_type_ids)
    outputs = outputs.reshape(1)
    print("here is the outputs")
    print(outputs)
    print("here is the targets")
    print(targets)
    loss = nn.BCEWithLogitsLoss(outputs, targets)
    print("this is the loss")
    print(loss)
    loss.backward()
    optimizer.step()
    if scheduler is not None:
        scheduler.step()</code></pre>

ptrblck: <p>You have to create an object of the criterion first and later call it with your tensors.<br>
Change:</p>
<pre><code class="lang-python">loss = nn.BCEWithLogitsLoss(outputs, targets)
</code></pre>
<p>to</p>
<pre><code class="lang-python">criterion = nn.BCEWithLogitsLoss()
for data in loader:
    ...
    loss = criterion(output, target)
    loss.backward()
</code></pre>
Example Code for Inplace Operations

pla: <p>Hi there,</p>
<p>Novice and first post in this forum.<br>
I have search the the forum and the documentation and even tried</p>
<blockquote>
<p>inspect.getsourcelines(torch.tensor)</p>
</blockquote>
<p>Which only resulted in errors.<br>
A search for in-place operations in connection with Pytorch  results in anything but what I am looking for.</p>
<p>All the inplace-operations refer to their non-inplace counter part for documentation, which is fine, but I am missing a link to the source code.</p>
<p>In case I am suffering from an XY-problem:<br>
I ultimately want to write s.th. like <code>nn.ReLu()</code> but as a step-function so <code>y = 1. for x &gt; 0. else 0.</code><br>
<code>y = 1. for x &gt; 0. else -1.</code> would also be fine.</p>
<p>Motivation behind this is to normalize the values to reassume the value-range of the input-valules.</p>
<p>Cheers</p>
Cannot install torch-1.6.0 cuda102 by `pip install torch`

Igo312: <p>I upgrade my cuda version to <code>10.2</code> and I try to using <code>pip</code> to install <code>torch==1.6.0</code>. According to official command</p>
<blockquote>
<h1>CUDA 10.2</h1>
<p>pip install torch==1.6.0 torchvision==0.7.0</p>
</blockquote>
<p>It cannot work, and the error is</p>
<blockquote>
<p>pip install torch==1.6.0 torchvision==0.7.0<br>
ERROR: Could not find a version that satisfies the requirement torch==1.6.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch==1.6.0</p>
</blockquote>
<h2><a name="enviroment-1" class="anchor" href="#enviroment-1"></a>Enviroment</h2>
<ul>
<li>windows 10.0</li>
<li>python 3.6.2 64bit</li>
<li>Cuda compilation tools, release 10.2, V10.2.89</li>
</ul>

Igo312: <p>Right now I install a <code>torch+1.6.0+cu101</code> version by using <code>pip install torch==1.6.0 tochvision==0.6.0 -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>It worked whatever although my cuda is actually 10.2</p>
<p>really digging in here…</p>
RTX 3000 Support

TeraChad: <p>Hello, I’m getting this error whenever I try to run anything with Torch:</p>
<pre><code class="lang-auto">UserWarning: 
GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
</code></pre>
<p>Does this simply mean that 3090’s aren’t supported yet or is there something I can do to enable support. Currently using CUDA 10.2, PyTorch 1.6</p>
<p>Thanks in advance</p>
<p>TeraChad</p>

niata: <p>you may want to try it with cuda 11:<br>
pip install --pre torch torchvision -f <a href="https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html</a></p>

TeraChad: <p>That seems to have fixed it, thank you</p>

mcj: <p>Are you  solve your problem? I will buy  a 3090 and i  hesitate to  buy  it!!!</p>

stas: <blockquote>
<p>Are you solve your problem? I will buy a 3090 and i hesitate to buy it!!!</p>
</blockquote>
<p>You don’t get to use the full power of RTX-30*, but otherwise it works just fine with pytorch/cuda-11.0</p>
Is there option way to use fillcolor as nearesr values?

Nimrod_Daniel: <p>Hi,</p>
<p>It seems like when doing augmentation in Pytorch it’s possible to perform fillcolor using one predetermined tuple only. Keras allows doing that with the nearest value, as depicted here:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://keras.io/img/favicon.ico" class="site-icon" width="64" height="64">
      <a href="https://keras.io/api/preprocessing/image/" target="_blank" rel="noopener nofollow ugc">keras.io</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/239;"><img src="https://keras.io/img/logo-k-keras-wb.png" class="thumbnail" width="690" height="239"></div>

<h3><a href="https://keras.io/api/preprocessing/image/" target="_blank" rel="noopener nofollow ugc">Keras documentation: Image data preprocessing</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I’ll copy the relevant part that shows what it does:<br>
‘nearest’: aaaaaaaa|abcd|dddddddd</p>
<p>Is there any option to do the same in Pytorch?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
N-to-1 frame CNN prediction model: how to represent the data?

rifflebutter: <p>Hi all!</p>
<p>I’m relatively new to PyTorch and I have a CNN that can predict the next frame of image based on 1 input frame relatively well. Now I want this model to outputs 1 image based on a sequence of images (e.g. give it 2 frames, make it guess the next frame). But I’m not sure how to represent the data in a way that makes sense in PyTorch. Is there any known way to do this?</p>
<p>I’ve considered concatenating the input images along some dimension (height/width) but that doesn’t make intuitive sense, and I need to do extra shenanigans to make the output dimension work (currently input and output are both single images so they have the same dimension, but if input is multiple images stacked together that’ll not hold).</p>
<p>I’ve also thought about concatenating the input images along the channel, so, for example, 3 RGB images will now have 9 channels and the output which is 1 image will have 3. This, again, doesn’t make intuitive sense to me, and I’m also not sure how to do this in PyTorch.</p>
<p>Thank you in advance for any helpful input!</p>

ptrblck: <p>There are multiple possible approaches, which you could try:</p>
<ul>
<li>you could represent the sequence of images as a new temporal dimension, execute the feature extractor (conv-pool layers) in a loop, and pass the output to an RNN. Afterwards you could use the activation of the last time step and feed it to a classifier (linear layers).</li>
<li>instead of using the loop in the feature extractor, you could use <code>nn.*3D</code> modules and use the depth dimension as the temporal dimension.</li>
<li>stacking the frames in the channel dimension and using grouped convs could also work, but I think this would</li>
</ul>
<p>Maybe others have more and better ideas. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Libtorch's files conflict with glog's file?

chegnyanjun: <p>Hi,guys, i am use libtorch and glog in the same codes in c++,but there are indices some files conflict between each other. The libtorch’s logging_is_not_google_glog.h(<span class="hashtag">#define</span> LOG(n)) is conflict with glog’s logging.h(<span class="hashtag">#define</span> LOG(severity) COMPACT_GOOGLE_LOG_ ## severity.stream()).The details problem is:<br>
In file included from libtorch/include/c10/util/Logging.h:28:0,<br>
from /libtorch/include/c10/core/TensorImpl.h:17,<br>
from /libtorch/include/ATen/core/TensorBody.h:11,<br>
from /libtorch/include/ATen/Tensor.h:11,<br>
from //libtorch/include/ATen/Context.h:4,<br>
from /libtorch/include/ATen/ATen.h:5,<br>
from /libtorch/include/torch/csrc/api/include/torch/types.h:3,<br>
from /libtorch/include/torch/script.h:3,<br>
/libtorch/include/c10/util/logging_is_not_google_glog.h:96:0: warning: “LOG” redefined<br>
<span class="hashtag">#define</span> LOG(n)<br>
^<br>
In file included from /glog/linux/include/glog/logging.h:483:0: note: this is the location of the previous definition<br>
<span class="hashtag">#define</span> LOG(severity) COMPACT_GOOGLE_LOG_ ## severity.stream()<br>
^<br>
In file included from /libtorch/include/c10/util/Logging.h:28:0,<br>
from /libtorch/include/c10/core/TensorImpl.h:17,<br>
from /libtorch/include/ATen/core/TensorBody.h:11,<br>
from /libtorch/include/ATen/Tensor.h:11,<br>
from /libtorch/include/ATen/Context.h:4,<br>
from /libtorch/include/ATen/ATen.h:5,<br>
from /libtorch/include/torch/csrc/api/include/torch/types.h:3,<br>
from /libtorch/include/torch/script.h:3,<br>
/libtorch/include/c10/util/logging_is_not_google_glog.h:99:0: warning: “VLOG” redefined<br>
<span class="hashtag">#define</span> VLOG(n) LOG((-n))<br>
^<br>
In file included from /glog/linux/include/glog/logging.h:1068:0: note: this is the location of the previous definition<br>
<span class="hashtag">#define</span> VLOG(verboselevel) LOG_IF(INFO, VLOG_IS_ON(verboselevel))<br>
Does anyone can tell me how to deal with it?My environment is ubuntu16.04 pytorch1.3 libtorch libtorch-cxx11-abi-shared-with-deps-1.3.1.zip,qt5.5. tkx.</p>

wolfworld6: <p>hello,have you solve it?</p>
[jit] torch.cat Tuple and list tensor support pytorch 1.7.0

mahendrasnilagiri: <p>Hello,</p>
<p>I’m facing issue with saving model architecture [Serializing torch.nn.Modules]</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8fdba3b6d4fad94d47e7451e0acdeb6eff8a53f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8fdba3b6d4fad94d47e7451e0acdeb6eff8a53f" title="torchcat_list"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8fdba3b6d4fad94d47e7451e0acdeb6eff8a53f.png" alt="torchcat_list" data-base62-sha1="zwG1jKzrtJjZL5PB1g6qik0fPgb" width="690" height="193" data-dominant-color="2A2A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torchcat_list</span><span class="informations">1553×436 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
with list as input</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a07dd9e84de9ce9908229d389f8a8227ca5a2279.png" data-download-href="https://discuss.pytorch.org/uploads/default/a07dd9e84de9ce9908229d389f8a8227ca5a2279" title="torchcat_tuple"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a07dd9e84de9ce9908229d389f8a8227ca5a2279.png" alt="torchcat_tuple" data-base62-sha1="mTM1SbM1UFB38J2cJzLLZQtYBQt" width="690" height="143" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torchcat_tuple</span><span class="informations">1564×325 16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
with tuple as input.</p>
<p>Please hellp.</p>
BERT NER: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

Wayne_Hu: <p>I want to train my BERT NER model on colab. But following error occurs<br>
Code:</p>
<pre><code class="lang-auto">tr_logits = tr_logits.detach().cpu().numpy()
tr_label_ids = torch.masked_select(b_labels, (preds_mask == 1))
tr_batch_preds = np.argmax(tr_logits[preds_mask.squeeze()], axis=1)
tr_batch_labels = tr_label_ids.to(device).numpy()
tr_preds.extend(tr_batch_preds)
tr_labels.extend(tr_batch_labels)
</code></pre>
<p>Error Message:</p>
<pre><code class="lang-auto">Using TensorFlow backend.
Saved standardized data to ./data/en/combined/train_combined.txt.
Saved standardized data to ./data/en/combined/dev_combined.txt.
Saved standardized data to ./data/en/combined/test_combined.txt.
Constructed SentenceGetter with 25650 examples.
Constructed SentenceGetter with 8934 examples.
Loaded training and validation data into DataLoaders.
Initialized model and moved it to cuda.
Initialized optimizer and set hyperparameters.
Epoch:   0% 0/5 [00:00&lt;?, ?it/s]Starting training loop.
Epoch:   0% 0/5 [00:00&lt;?, ?it/s]
Traceback (most recent call last):
  File "/content/FYP_Presentation/python/main.py", line 102, in &lt;module&gt;
    valid_dataloader,
  File "/content/FYP_Presentation/python/utils/main_utils.py", line 431, in train_and_save_model
    tr_batch_preds = torch.max(tr_logits[preds_mask.squeeze()], axis=1)
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 412, in __array__
    return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>

Megh_Bhalerao: <p><code>CUDA</code> tensors can not be converted to numpy directly. As the error message says please do <code>tensor_name.cpu().numpy()</code> to convert this tensor to <code>numpy</code></p>
Do weight values also get copied when I append layer of one network to another?

Megh_Bhalerao: <p>Hello, I am doing this:</p>
<pre><code class="lang-auto">weights_init(F1)

if args.net == "resnet32":
    G_final = nn.Sequential(G,F1.fc1)
    F1.fc1 = nn.Identity()
    G = deepcopy(G_final)

</code></pre>
<p>What I want to do is append the initial layer of <code>F1</code> to <code>G</code>. I am initialising the weights of <code>F1</code> in a particular manner using the <code>weights_init</code> function.<br>
So, my question is when I do these steps, do the weights which have been specially initialised in <code>F1</code> also get transferred to the net <code>G_final</code> network?<br>
Thanks,<br>
Megh</p>
Single-gpu trained model used in multi-gpus setting

zhiqiangdon: <p>Hi,</p>
<p>I am trying to finetune a sigle-gpu trained model in multi-gpus. First, I specify<br>
CUDA_VISIBLE_DEVICES=0,1,2. Then I warp the defined model with torch.nn.DataParallel() and use the rmsprop optimizer as follows:</p>
<p>model = torch.nn.DataParallel(model).cuda()<br>
optimizer = torch.optim.RMSprop(model.parameters(), <a href="http://lr=opt.lr" rel="nofollow noopener">lr=opt.lr</a>, alpha=0.99,<br>
eps=1e-8, momentum=0, weight_decay=0)</p>
<p>This code works well if I train a model in multi-gpus from scratch. However, if I start from a checkpoint of a single-gpu trained model, when it runs to code</p>
<p>optimizer.step()</p>
<p>an error shows “…/python2.7/site-packages/torch/optim/rmsprop.py”, line 52, in step  state = self.state[p]<br>
KeyError: Parameter containing:<br>
( 0 , 0 ,.,.) =1.00000e-02 * 2.5088<br>
( 0 , 1 ,.,.) = 1.00000e-02 * 1.6257<br>
.<br>
.<br>
.<br>
(127,126,.,.) = 1.00000e-02 *2.5302<br>
(127,127,.,.) = 1.00000e-02 *-4.7111<br>
[torch.cuda.FloatTensor of size 128x128x1x1 (GPU 0)]"</p>
<p>Does anyone know what’s the problem here? Thanks in advance!</p>

colesbury: <p>What’s your code to load the model? Try doing something like this:</p>
<pre><code class="lang-python">model = MyNetwork()
model.load_state_dict(path_to_file)
model = torch.nn.DataParallel(model).cuda()
optimizer = torch.optim.RMSprop(model.parameters(), lr=opt.lr, alpha=0.99, eps=1e-8, momentum=0, weight_decay=0)
</code></pre>
<p>i.e. load the model before constructing the DataParallel and create the optimizer after creating the data parallel.</p>

zhiqiangdon: <p>Thanks for your reply! <a class="mention" href="/u/colesbury">@colesbury</a></p>
<p>I tried what said. It works. However, Here comes another related problem. I also need to use the optimizer state saved in the previous single-gpu training rather than create a new optimizer.</p>
<pre><code>optimizer.load_state_dict(checkpoint['optimizer'])
</code></pre>
<p>When I do this, the same error occurs. I guess something inside the saved optimizer is not consistent with the multi-gpus setting. Any suggestion to solve this? Thanks.</p>

Rafael_Valle: <p>Any update on this matter?</p>

puppylpg: <p>Try move model to gpu first, then create optimizer, loading it’s parameters from checkpoint</p>
GLIBC linker errors with libtorch nightly build

Carol_Ye_Liu: <p>I got the following errors when I try to build a minimal C++ project using libtorch nightly build:<a href="https://download.pytorch.org/libtorch/nightly/cu101/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/nightly/cu101/libtorch-cxx11-abi-shared-with-deps-latest.zip</a><br>
. The errors only occur with nightly build. Can anyone give some clues about how to fix these?<br>
…/libtorch_nightly_0107/lib/libtorch.so: undefined reference to <code>lgammaf@GLIBC_2.23' .../libtorch_nightly_0107/lib/libtorch.so: undefined reference to </code>lgamma@GLIBC_2.23’<br>
collect2: error: ld returned 1 exit status<br>
gmake[3]: *** [deployment] Error 1<br>
gmake[2]: *** [CMakeFiles/deployment.dir/all] Error 2<br>
gmake[1]: *** [CMakeFiles/deployment.dir/rule] Error 2<br>
gmake: *** [deployment] Error 2</p>

Carol_Ye_Liu: <p>The problem is solved by downloading the pre-cxx11 version.</p>

duo_ma: <p><a class="mention" href="/u/carol_ye_liu">@Carol_Ye_Liu</a>, This solution is very good.</p>
How to pad a sequence?

andy29: <p>I am trying to learn about pack_padded_sequence more and want to test it in this small dataset. I managed to merge two tensors of different sequence length but when I try to pad the sequence it gives me an error. Does anybody know how to solve this? I am trying to follow an example given in the stackoverflow comments but with an actual dataset. <a href="https://stackoverflow.com/questions/51030782/why-do-we-pack-the-sequences-in-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/51030782/why-do-we-pack-the-sequences-in-pytorch</a></p>
<p>Runtime Error: The expanded size of the tensor (8) must match the existing size (4) at non-singleton dimension 1.  Target sizes: [93, 8, 1].  Tensor sizes: [93, 4, 1]</p>
<pre><code class="lang-auto">!wget https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import torch
import torch.nn as nn
from torch.autograd import Variable
from sklearn.preprocessing import MinMaxScaler

training_set = pd.read_csv('airline-passengers.csv')

def sliding_windows(data, seq_length):
    x = []
    y = []

    for i in range(len(data)-seq_length-1):
        _x = data[i:(i+seq_length)]
        _y = data[i+seq_length]
        x.append(_x)
        y.append(_y)

    return x,np.array(y)

sc = MinMaxScaler()
training_data = sc.fit_transform(training_set)

seq_length = 8
x, y = sliding_windows(training_data, seq_length)

train_size = int(len(y) * 0.67)
test_size = len(y) - train_size

trainX = Variable(torch.Tensor(np.array(x[0:train_size])))
trainY = Variable(torch.Tensor(np.array(y[0:train_size])))

seq_length = 4
x1, y1 = sliding_windows(training_data, seq_length)

train_size = int(len(y1) * 0.67)
test_size = len(y1) - train_size

trainX1 = Variable(torch.Tensor(np.array(x1[0:train_size])))
trainY1 = Variable(torch.Tensor(np.array(y1[0:train_size])))

seq_batch = [trainX,
             trainX1]

seq_lens = [8, 4]

added_seq_batch = torch.nn.utils.rnn.pad_sequence(seq_batch, batch_first=True)
</code></pre>
Is zeroing gradient really make it zero?

Kamil_Adamczewski: <p>I set the gradient to zero (I tried both via hook and via parameter param.grad) and when I output the gradient, it does say 0, but the value of the parameters during training changes minimally in strange way. I set values to 1 and then during training all the weights go steadily  down by 0.0001, that is 1, 0.9999, 0.9998, etc (below it is after a few iteration). What does such a behavior mean, and how to make sure the parameters don’t change?</p>
<p>gradient:<br>
tensor([[[0., 0., 0., 0., 0.],<br>
[0., 0., 0., 0., 0.],<br>
[0., 0., 0., 0., 0.],<br>
[0., 0., 0., 0., 0.],<br>
[0., 0., 0., 0., 0.]]], device=‘cuda:0’)<br>
weights:<br>
tensor([[[0.9967, 0.9967, 0.9967, 0.9967, 0.9967],<br>
[0.9967, 0.9967, 0.9967, 0.9967, 0.9967],<br>
[0.9967, 0.9967, 0.9967, 0.9967, 0.9967],<br>
[0.9967, 0.9967, 0.9967, 0.9967, 0.9967],<br>
[0.9967, 0.9967, 0.9967, 0.9967, 0.9967]]], device=‘cuda:0’)</p>

JuanFMontesinos: <p>Gradient equal to zero doesn’t stop optimizer inertia (momentum and so on).<br>
make param.grad=None to avoid that.</p>
<p>In short:<br>
for SGD for example if gradient is zero there is still an update.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c75d6b7f77fd440e6bea90b0b90c8f92360c482.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c75d6b7f77fd440e6bea90b0b90c8f92360c482" title="imagen"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c75d6b7f77fd440e6bea90b0b90c8f92360c482.png" alt="imagen" data-base62-sha1="hL1FPtwAGoxQnoDIW9642kAPqPo" width="669" height="500" data-dominant-color="E7ECF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagen</span><span class="informations">678×506 28.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>