akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ZeroDivisionError: integer division or modulo by zero

username: pirate-zhang
cooked: <p>The problem shows as follows:<br>
“Traceback (most recent call last):<br>
File “train.py”, line 26, in <br>
train_net(config)<br>
File “train.py”, line 17, in train_net<br>
solver = Solver(config, data_loader=data_loader, device=“cuda”)<br>
File “/home/zhangwei/PSGAN-master/psgan/solver.py”, line 41, in <strong>init</strong><br>
self.snapshot_step = config.LOG.SNAPSHOT_STEP // torch.cuda.device_count()”<br>
Can someone tell me what   this problem is?</p>

username: ptrblck
cooked: <p>It seems that no GPU is detected on your system in:</p>
<pre><code class="lang-python">self.snapshot_step = config.LOG.SNAPSHOT_STEP // torch.cuda.device_count()
</code></pre>
<p>which would trigger a division by 0.</p>
<p>Your system either doesn’t have a GPU, in which case you should remove the <code>device_count()</code> call and fix the code appropriately, or the GPU is not detected if you’ve installed the PyTorch CPU binaries or e.g. are using a too old NVIDIA driver.</p>

username: pirate-zhang
cooked: <p>Thank u very much.I have solved my problem! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
How does PyTorch automatically free up GPU space?

username: crrrr30
cooked: <p>Consider the following code snippet.</p>
<pre><code class="lang-auto">import torch

for i in range(10):
    x = torch.randn(10000, 10000).cuda()
    print(torch.cuda.max_memory_allocated())
</code></pre>
<p>On my computer, the output is</p>
<pre><code class="lang-auto">400556032
801112064
801112064
801112064
801112064
801112064
801112064
801112064
801112064
801112064
</code></pre>
<p>It seems that PyTorch is automatically deleting tensors from GPU that will no longer be used. How does it know? Isn’t the name of the tensor <code>x</code> only a reference to the tensor?<br>
Consider the following code snippet</p>
<pre><code class="lang-auto">import torch

x = torch.randn(100).cuda()
print(x.mean().item())
y = x    # Shallow copy
del x
torch.cuda.empty_cache()
print(y.mean().item())
</code></pre>
<p>On my computer, the two output values are the same. This suggests that PyTorch does not “delete” tensors, even though explicitly ordered.<br>
My question is, how do memory allocation and deallocation work in PyTorch?</p>

username: ptrblck
cooked: <p>In your first example you are replacing the <code>x</code> tensor in each iteration, so that the “old” <code>x</code> tensor can be freed or more specifically: it’s memory can be reused.</p>
<p>PyTorch frees memory of tensors, which don’t have any valid reference pointing to them anymore.<br>
Since <code>y</code> points to the data of <code>x</code>, this data cannot be freed even after <code>x</code> was deleted.</p>
How to get embeddings from a pretrained model

username: noguide
cooked: <p>I am trying to write a <a href="https://github.com/fangpin/siamese-pytorch/blob/master/model.py" rel="nofollow noopener">Siamese network</a> of two embedding networks that share weights. I found several examples online for MNIST and other datasets of small images, but my images are much larger (around 500x1000). Please bear with me because I’m very confused, and I have several questions here!</p>
<ol>
<li>
<p>How should I modify the embedding networks to take in larger images? Should I use a model made for larger images, like ResNet?<br>
1a) And if I do that, should I resize my images to fit ResNet’s input size? Or is there a way to modify ResNet to work with larger images?</p>
</li>
<li>
<p>If I use a pretrained network, how should I modify it so that it still works with my dataset? (Do I train starting from the pretrained weights? Or do I just freeze those weights and work with that?</p>
</li>
<li>
<p>Do I need to remove the output layer of the model in order to get embeddings?</p>
</li>
<li>
<p>How would I incorporate all these things into my Siamese network code?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>You could use another model architecture or modify your current model to accept later images. It’s hard to tell which approach would work better and you would have to run some experiments.<br>
The <code>torchvision.models.resnet...</code> implementations accept larger input images, since an adaptive pooling layer is used before flattening the activation and feeding into the first linear layer of the classifier.</p>
</li>
<li>
<p>You could test both approaches for your use case. The best strategy depends on your current use case and used dataset. If your dataset is “similar” to ImageNet, freezing the layers might work fine.</p>
</li>
<li>
<p>I’m not completely sure what “embeddings” are in the posted model, but given your current implementation I would assume you want to reuse the resnet for both inputs and then add a custom linear layer to get the final output.</p>
</li>
<li>
<p>Try to replace the <code>self.conv</code> module with e.g. the modified resnet.</p>
</li>
</ol>
Adjust optimizer learning rate

username: oasjd7
cooked: <pre><code class="lang-auto">def op_copy(optimizer):
    for param_group in optimizer.param_groups:
        param_group['lr0'] = param_group['lr']
    return optimizer
</code></pre>
<pre><code class="lang-auto">lr = 1e-3
param_group = []
for k, v in net.named_parameters():
    param_group += [{'params': v, 'lr': lr * 0.1}]
for k, v in classifier.named_parameters():
    param_group += [{'params': v, 'lr': lr}]

optimizer = optim.SGD(param_group)
optimizer = op_copy(optimizer)
</code></pre>
<p>Hi, all.<br>
I’m new to pytorch.<br>
I have some questions about the above codes.</p>
<ol>
<li>
<p>What does <code>op_copy()</code> do?</p>
</li>
<li>
<p>If I change<br>
<code>optimizer = optim.SGD(param_group)</code> to<br>
<code>optimizer = optim.SGD(param_group, lr = 1e-3)</code><br>
then the learning rate for net become 1e-4 and classifier for 1e-3 ?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>It seems a new key named <code>lr0</code> is added the all param groups. If I’m not mistaken, this key is not used in PyTorch optimizers so I guess your script might be using it somewhere manually.</p>
</li>
<li>
<p>Yes, that should be the case.</p>
</li>
</ol>
Question about registering PyTorch parameters

username: Niels_PyTorch
cooked: <p>I am implementing an algorithm by Google (which was written in Tensorflow 1.0) in PyTorch. The Tensorflow implementation defines a classification head on top of a BERT model as follows:</p>
<pre><code class="lang-auto">hidden_size = output_layer.shape.as_list()[-1]
  output_weights = tf.get_variable(
      "output_weights", [hidden_size],
      initializer=tf.zeros_initializer()
      if config.init_cell_selection_weights_to_zero else _classification_initializer())
  output_bias = tf.get_variable(
      "output_bias", shape=(), initializer=tf.zeros_initializer())
</code></pre>
<p>I want to define the same in PyTorch. I defined it as follows:</p>
<pre><code class="lang-auto">class net(nn.Module):

    def __init__(self, config):   

        super().__init__(config)

        # classification head
        if config.init_cell_selection_weights_to_zero: 
            self.output_weights = nn.Parameter(torch.zeros(config.hidden_size)) 
        else:
            self.output_weights = nn.Parameter(torch.empty(config.hidden_size))
            nn.init.normal_(self.output_weights, std=0.02) # here, a truncated normal is used in the original implementation
        self.output_bias = nn.Parameter(torch.zeros([]))

    def forward(self, ...)

</code></pre>
<p>In other words, I am using <code>torch.nn.Parameter</code> as a counterpart of <code>tf.get_variable</code> to define this additional trainable layer. This adds the classification head to the parameters of the model (i.e. they are printed when I print <code>list(model.parameters())</code>), but they are not printed when I type <code>print(model)</code>.</p>
<p>Is this because they are not registered yet? Do I have to use <a href="https://pytorch.org/docs/1.5.0/nn.html#torch.nn.Module.register_parameter" rel="nofollow noopener"><code>register_parameter</code></a> in my init function?</p>

username: Niels_PyTorch
cooked: <p>Update: found it on <a href="https://stackoverflow.com/questions/54770249/why-arent-torch-nn-parameter-listed-when-net-is-printed" rel="nofollow noopener">Stackoverflow</a>.</p>
How improve the speed of training after using Transforms.Resize()?

username: StevenAzy
cooked: <p>When I used Transforms.Resize(), I found the training of the model gets slow. I created an issue in Github about it, someone gave me an answer that Resize() takes CPU resource. So I want to know how to increase the speed of training or make Resize() to use GPU resources.</p>

username: ptrblck
cooked: <p>You could use e.g. <a href="https://github.com/NVIDIA/DALI">DALI</a> for image processing on the GPU.<br>
Depending on your system, this could yield a speedup.<br>
Alternatively, you could use <code>PIL-SIMD</code>.</p>
<p>PS: I would recommend to stick to one thread, as this question is highly related to <a href="https://discuss.pytorch.org/t/transforms-resize/95879">this one</a>.</p>
CUDA OOM while using torch.distributed.launch and no OOM training without it

username: nullkatar
cooked: <p>Hello folks!</p>
<p>I’m stuck with one very strange problem: I work with recently released <a href="https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch" rel="nofollow noopener">Scene Graph Benchmark</a>  and made it train on GQA, but I have one issue with that. It trains as expected when I use the following command: <code>CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" </code> (it uses <code>batch_size = 2</code>). But when I decide to use another command with torch.distributed.launch (<code>CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10025 --nproc_per_node=1 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml"</code>) I’m getting <code>RuntimeError: CUDA out of memory</code>( it still has <code>batch_size = 2</code>). Initially I wanted to train it on  4 GPUs with <code> batch_size = 8</code>, but figured out about this problem. What can be the problem? And what should I do in order to properly  train it on 4 GPUs?</p>
<p>My set up includes 4 2080ti so It has a plenty of memory.</p>

username: agolynski
cooked: <aside class="quote no-group" data-username="nullkatar" data-post="1" data-topic="95921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nullkatar/48/25106_2.png" class="avatar"> nullkatar:</div>
<blockquote>
<p>I’m stuck with one very strange problem: I work with recently released <a href="https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch" rel="noopener nofollow ugc">Scene Graph Benchmark</a> and made it train on GQA, but I have one issue with that. It trains as expected when I use the following command: <code>CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" </code> (it uses <code>batch_size = 2</code> ). But when I decide to use another command with torch.distributed.launch ( <code>CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10025 --nproc_per_node=1 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml"</code> ) I’m getting <code>RuntimeError: CUDA out of memory</code> ( it still has <code>batch_size = 2</code> ). Initially I wanted to train it on 4 GPUs with <code> batch_size = 8</code> , but figured out about this problem. What can be the problem? And what should I do in order to properly train it on 4 GPUs?</p>
</blockquote>
</aside>
<p>Hi Leon,</p>
<p>What version of pytorch do you use?</p>

username: nullkatar
cooked: <p>Hello Alexander,</p>
<p>1.6.0,  one which installs with <code>conda install torch</code>.</p>

username: ptrblck
cooked: <p>Could you try to run the DDP command on a single node and GPU and check the memory usage?<br>
I guess the code might create unnecessary CUDA contexts on other devices, but since the repository contains a lot of files I haven’t looked through all of them.</p>
How to indexing IValue in java? (my case : tuple of [1,496,640,2])

username: SungmanHong
cooked: <p>Hi, I have java.torch.Size([1, 496, 640, 2]) &lt;- outputs</p>
<p><code>IValue[] outputs = model.forward(inputs).toTuple()</code></p>
<p>which is IValue</p>
<p>I want to get [1,:,:,1] and [1,:,:,2] by java command…<br>
how can I get this? also…what is the correspondence of the np.array in java?<br>
I want to get [1,:,:,1] and [1,:,:,2]  as float.array in java.</p>
<p>why pytorch has very few examples??</p>
Expected IValue type 10, actual type 2

username: SungmanHong
cooked: <p>I have error in java.<br>
<code>Expected IValue type 10, actual type 2</code></p>
<p>actually… what is the type10 and type2?<br>
how can i find any doc?</p>

username: SungmanHong
cooked: <p>I found type2 is tuple and 10 is double list, but don’t know any doc is exists</p>
How to add comments, docstrings and keywrods to the torchscript extension?

username: Shisho_Sama
cooked: <p>Hello everyone,<br>
Hope you are having a great day.<br>
I’m building a Python extension using both Pybind11 and TorchScript extension as explained <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html" rel="nofollow noopener">here</a>. my job with the Pybinnd11 is done and now I’m trying the torch script extension and see how it fares against Pybind11.<br>
I built one just now and it seems working however, I cant find anything regarding how to add docstring to the classes, methods, etc exposed using TorchScript. I also cant find the equivalent of  Pybind11’s <code>py::arg()</code> which is used to add keywords , so on Python part, the keywords are visible and people can know what they are supposed to pass!</p>
<p>Any help in this regard is greatly appreciated.</p>
Why running multiprocessing is slower than one process on CPU

username: anrki
cooked: <p>I have an inference function <code>func</code> that takes on average 9 sec to run. But when I try to use multiprocessing to parallelize it (even using torch.multiprocessing) each inference takes on average 20 sec why is that ?</p>
<p><strong>For info:</strong><br>
<code>func</code> is an inference function which takes in a <code>patient_name</code> and runs a torch model in inference on that patient’s data.</p>
<pre><code class="lang-auto">device = torch.device(torch.device('cpu'))

def func(patient_name):
    
    data = np.load(my_dict[system_name]['data_path'])
    model_state = torch.load(my_dict[system_name]['model_state_path'],map_location='cpu')
    
    model = my_net(my_dict[system_name]['HPs'])
    model = model.to(device)

    model.load_state_dict(model_state)
    model.eval()
    
    result = model(torch.FloatTensor(data).to(device))
    return result
</code></pre>
<pre><code class="lang-auto">from torch.multiprocessing import pool

core_cnt = 10

pool = Pool(core_cnt)
out = pool.starmap(func, pool_args)
</code></pre>

username: mrshenli
cooked: <p>Can you add some profiling to see which part is the bottleneck?</p>
<p>One reason might be PyTorch ops already launches many threads. So it possible that one process already saturates CPU resources.</p>
Loss.backward is so slow!

username: Wjk666
cooked: <p>hello, When I add some index lookup operations(just add this part of codes),the speed of <code>loss.backward()</code> is so slow. It cost 90 seconds during the <code>loss.backward()</code> while it just cost 3 seconds in the whole training epoch before. I’m confused the operation in the loss.backward(). And how to correct it or accelerate my codes!<br>
Thanks a lot in advance!</p>
<pre><code class="lang-auto">        global_entity = torch.zeros((batch_size,hidden_size))
        global_relation = torch.zeros((batch_size,hidden_size_relation))
        relation_indices = []
        global_entity_ = torch.zeros((self.num_relation,hidden_size))
        global_relation_ = torch.zeros((self.num_relation,hidden_size_relation))
        

        for relation in range(self.num_relation):
            current_global_indices = torch.LongTensor(list(Corpus_.re2entity[relation])).cuda()
            global_entity_[relation] = out_entity_1[current_global_indices].mean(dim=0)

            current_global_relation_indices = torch.LongTensor(list(Corpus_.re2path[relation])).cuda() 
            global_relation_[relation] = out_relation_1[current_global_relation_indices].mean(dim=0)

        for i,triple in enumerate(batch_inputs):
            head,relation,tail =  triple
            global_entity[i] = global_entity_[relation]
            global_relation[i] = global_relation_[relation]
            relation_indices.append(relation)
        relation_indices = torch.LongTensor(relation_indices).cuda()
        global_entity = global_entity.cuda()
        global_relation = global_relation.cuda()
</code></pre>
Mp.spawn() and wandb

username: Dalisson_Figueiredo
cooked: <p>I was wondering if anybody run into a problem while trying to train with mp.spawn() and DistributedDataParallel using two gpus (one process per gpu) where wandb gets stuck and wont allow the training to go on.</p>

username: ayalaa2
cooked: <p>Are you using wandb on both processes or just one? I would suggest having one process handle logging and collecting info from the other processes</p>

username: Dalisson_Figueiredo
cooked: <p>I am using init in both processes, I tried to run in a single process but it also gets stuck.</p>
Accuracy drops to 10% on GPU

username: Nurav_Adnab
cooked: <p>Hello All,</p>
<p>I tried running the program in <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">this</a> page on my GPU and the Accuracy of the network on the 10000 test images falls from 56% (on CPU) to 10%. I know that I should be getting the same scores irrespective of whether it is a CPU or GPU. Could someone let me know what could have gone wrong? Posting the entire code would be lengthy and I’ll post whatever relevant sections you ask for.</p>

username: kinwai_cheuk
cooked: <p>The code you shared runs on GPU by default. What do you change so that it runs on CPU?<br>
I suspect that your changes to the code causes this problem.</p>
<p>Can you share us all the changes you made?</p>

username: Nurav_Adnab
cooked: <p>The code I shared actually runs on the CPU by default. <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#training-on-gpu" rel="nofollow noopener">This</a> section provides the code for running it on a GPU.</p>
<p><a href="https://colab.research.google.com/drive/1MH2zTgrtYBF2m8rzjoSzUCYlBGZs1rg4?usp=sharing" rel="nofollow noopener">Here</a> is the code. It would be great if you could let me know the errors I’ve made. FYI, I ran this code on my system and not in Colab. I just uploaded the notebook over there for sharing it.</p>

username: kinwai_cheuk
cooked: <p>Your evaluation code seems wrong to me. You load your test data as <code>inputs</code>, but you feed forward <code>images</code>  to your model. i.e. You keep giving the model the same cat, ship, ship, plane images, but you are evaluating it with different <code>labels </code> obtained from the <code>testloader</code>.</p>
<p>Changing <code>images</code> to <code>inputs</code> should be able to solve your problem.</p>
<pre><code class="lang-auto">correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
#         images, labels = data
        inputs, labels = data[0].to(device), data[1].to(device) # You load the test images as `inputs`
        outputs = net(images.to(device)) # You feedforward `images`
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
</code></pre>

username: Nurav_Adnab
cooked: <p>Thank you very much. I just blindly replaced the code without much checking.</p>
nn.MultiHeadedAttention num parameters doesn't vary with num_heads?

username: vhalpurohit
cooked: <p>Hi<br>
I recently noticed that increasing the number of heads for a MultiHeadAttention layer doesn’t increase the number of parameters that the layer uses. I wonder why this is? Since each head is basically running self attention on the input and calculating a set of attention weights, shouldn’t increasing the number of heads increase the number of parameters used?</p>
<p>^edit: I’m using <code>[(name, layer.numel()) for name, layer in model.named_parameters()]</code> to print the layer size</p>

username: ayalaa2
cooked: <p>This is correct with how MultiHeadAttention is implemented in the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noopener">original paper</a>. Each head acts upon a reduced dimensionality of the original problem such that the computation cost (and also parameters) are comparable to that of one head.</p>
<p>There’s another way of performing MultiHeadAttention where the parameters (and thus computation) scales with the number of heads. It’ll basically mimic running several of Pytorch’s MultiHeadAttention layers, each with one head, in parallel. You’ll need to implement this or find an implementation.</p>
Consistent result in distributed environment

username: lampadephoria
cooked: <p>Hi All,<br>
I am a new user of torch. I would like to develop a dataloader module to feed data to torch C++ API. My focus is the C++ API.<br>
My dataset is distributed among multiple compute nodes. Each node owns a disjoint portion of data WRT any other nodes. Let’s say I have 5 nodes. They own 200,200,200,300,100 examples of data respectively. Is there a way to achieve the same training result from training on a single node (the node owns all 1000 rows of data) or on a 3-node cluster (the data distribution is different from 5-node cluster case)?</p>
<p>Is there a way to control/know how torch generates the indices when calling customDataset::get(index) in training and in scoring? Are the indices sequential (1,2,3,…) or random?</p>
<p>If I deploy multiple workers to load data in a node, is there a way to know the caller’s thread id from customDataset::get()?</p>
<pre><code>auto data_loader = torch::data::make_data_loader(std::move(dataset),
    torch::data::DataLoaderOptions().batch_size(kBatchSize).workers(2));
</code></pre>
<p>Thank you very much.</p>

username: lampadephoria
cooked: <p>Hello,<br>
Can someone offer any insight please?</p>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> <a class="mention" href="/u/simonw">@SimonW</a>  for dataloader<br>
cc <a class="mention" href="/u/glaringlee">@glaringlee</a> for C++ API</p>
General PyTorch Questions

username: jessieanker
cooked: <p>Hi everyone,</p>
<p>I am interested in contributing to PyTorch and had a couple of questions about the community:</p>
<ol>
<li>
<p>I was wondering about your perspective on how approachable this project is to someone who does not have experience with this software, given that I am a senior in college?</p>
</li>
<li>
<p>Do you have any idea on the tasks that I would be able to contribute to? Are there good first issues for someone with not a lot of experience with machine learning?</p>
</li>
</ol>

username: albanD
cooked: <p>Hey,</p>
<p>Welcome!</p>
<ol>
<li>It might not be the easiest project to dive in mostly because it has grown mostly organically and some parts are not very well documented. But you can ask questions either on the related issue or slack and people will be happy to help you discover the different part of the code and guide you with what needs to be modified. But in general, the codebase goes from lots of high level python code (including the domain libraries) that is very ML focused to a lot of low level code to handle fast operations and the many features pytorch provides.</li>
<li>I would recommend to take a look at <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md</a>. For good first issue, it depends a lot of what kind of problem you’re interested in. High level ML? Python typing? c++/python binding? jit? distributed? low level kernel implementations? etc</li>
</ol>
Plotting epoch loss

username: jkhor18
cooked: <p>I want to plot epoch loss curve, I’ve tried codes from <a href="https://discuss.pytorch.org/t/plotting-loss-curve/42632" class="inline-onebox">Plotting loss curve</a> but i’m getting errors like</p>
<blockquote>
<p>TypeError: ‘DataLoader’ object is not subscriptable</p>
</blockquote>
<blockquote>
<p>train(args.epochs, args.batch_size, args.lr, args.num_classes)</p>
</blockquote>
<p>This is my code:</p>
<pre><code class="lang-auto">def train(epochs, batch_size, learning_rate, num_classes):

    # fetch data
    train_loader, test_loader = get_data_loader(batch_size)

    # Loss and optimizer
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    model = LeNet(num_classes).to(device)
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

    # start train
    total_step = len(train_loader)
    losses = []
    for epoch in range(epochs):
        running_loss = 0.0
        for i, (images, labels) in enumerate(train_loader):

            
            # get image and label
            images = images.to(device)
            labels = labels.to(device)

            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)
            
            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            running_loss += loss.item() * images.size(0)
            epoch_loss = running_loss / len(train_loader['train'])
            losses.append(epoch_loss)

            if (i + 1) % 100 == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                      .format(epoch + 1, epochs, i + 1, total_step, loss.item()))
                running_loss = 0.0

        # evaluate after epoch train
        evaluate(model, test_loader, device)

    # save the trained model
    save_model(model, save_path='lenet.pth')
    plt.plot(np.array(losses), 'r')
    return model
</code></pre>

username: ahmed
cooked: <p>Could you show the code for <code>get_data_loader</code>? From the error <code>TypeError: ‘DataLoader’ object is not subscriptable</code>, it looks like there’s an issue with how you’re using your <code>DataLoader</code>. But it’s difficult to tell from just looking at your code. Could you also state at which line you are getting your errors?</p>
Autograd slow in multiprocessing

username: Meraki
cooked: <p>I came cross a problem that when I use torch.autograd.grad method in multiprocessing job, the time cost for grad function  becomes quite larger than I thought. Is this intentional?</p>

username: tom
cooked: <p>I must admit that from your description it is a bit hard for me to know what is going on. Depending on your setup, it could be because the gradient had already been computed with parallelization. This could reduce the effectiveness of multiprocessing for the backward.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Meraki
cooked: <p>The pseudo codes are like below:</p>
<pre><code class="lang-auto">def average_gradients(model):
    size = dist.get_world_size()
    for param in model.parameters():
        dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
        param.grad.data /= size

def train():
    for ep in range(1000):
        optimizer.zero_grad()
        a = torch.tensor(xxx)
        # multipulating a to derive tensor b
        c = torch.autograd.grad(b, a, create_graph=True)[0]
        loss = nn.MSEloss((b, b_target)) + nn.MSEloss((c, c_target))
        loss.backward(retain_graph=True)
        average_gradient(net)
        optimizer.step()
    
if __name__ is '__main__':
    processes = []
    size = 4
    for rank in range(size):
        p = Process(target=train,
                    args=(rank, size, lock))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>I set different value of size to test, the main time consuing part is torch.autograd.gard method. The smaller value of size, the faster for this line computing.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This looks quite ok (note that you don’t need the retain_graph in the backward call).<br>
What is problematic for you? Can you give concrete example of what is slower than you expect?</p>

username: Meraki
cooked: <p>I found the problem. When you create a new subprocess, pytorch sets the maximum number of potential useable threads by default (depends on how many cores in your current node). After I set_num_threads to 1 or 2, the speed behaves like what I expected.</p>
Deployment of 3 NLP Pytorch models to a flask app not working

username: EmreTokyuez
cooked: <p>I followed the notebook 2, 4 and 6 of this repo:<a href="https://github.com/bentrevett/pytorch-sentiment-analysis" rel="nofollow noopener">https://github.com/bentrevett/pytorch-sentiment-analysis</a> to built a sentiment analyzer. I now want to deploy them using flask but I can’t manage to get things working. Can experienced people with this task comment or dm me for me to deploy these models to flask?</p>
<p>I tried saving and loading the vocab using pickle or dill but there were errors with padx and also that the field attribute does not have an attribute stoi.</p>
<p>I am honestly stuck and trying for weeks now. I need to hand in the project next week <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
TransformerDecoderLayer AssertionError

username: Hari_Krishnan
cooked: <p>I read an interesting paper on transformer based variational autoencoder (VAE), so I tried to replicate a simpler model using <code>nn.TransformerEncoderLayer</code> and <code>nn.TransformerDecoderLayer</code> .The architecture is given code :-</p>
<pre><code class="lang-auto">embedding_size = 256
n_heads = 8
batch_size = 16
latent_dim = 32

class Transformer_VAE(nn.Module):
    
    def __init__(self, head, vocab_size, embedding_size, latent_dim, device = 'cpu', pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
        super(Transformer_VAE, self).__init__()
        self.head = head
        self.embedding_size = embedding_size
        self.vocab_size = vocab_size
        self.latent_dim = latent_dim
        self.embed = WordEmbedding(self.vocab_size, self.embedding_size)
        self.postional_encoding = PostionalEncoding(embedding_size, device)
        self.encoder = nn.TransformerEncoderLayer(self.embedding_size, head)
        self.decoder = nn.TransformerDecoderLayer(self.latent_dim, head)
        self.hidden_to_mean = nn.Linear(self.embedding_size, latent_dim)
        self.hidden_to_logvar = nn.Linear(self.embedding_size, latent_dim)
        self.out_linear = nn.Linear(self.embedding_size, vocab_size)
        
    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5*logvar)
        eps = torch.randn_like(std)
        return mu + eps*std
        
    def forward(self, x):
        batch_size, maxlen = x.size()[:2]
        src = self.embed(x)
        x = self.postional_encoding(src)
        x = self.encoder(x)
        print(x.size())
        mean, logvar = self.hidden_to_mean(x), self.hidden_to_logvar(x)
        z = self.reparameterize(mean, logvar)
        out = self.decoder(src, z)
        out = self.out_linear(out)
        return mean, logvar, out
</code></pre>
<p>I’m getting an <code>AssertionError:</code>, when I run the <code>self.decoder(src, z)</code> line. From what I understood from the documentation the <code>TransformerDecoderLayer</code> takes the target value and the output of the encoder as its inputs. Since this is a VAE, the target will be the same as the input and the encoder outputs have been mapped to the latent dimension. So these two values are input to the decoder layer.</p>

username: Hari_Krishnan
cooked: <p>My bad, Solved it by upscaling the latent dimension to embedding size before passing it to the <code>TransformerDecoderLayer</code>.</p>
Setting the top 1 element of a tensor to be 1

username: Silvaurus
cooked: <p>Hi! Given a 1D tensor, I would need to find its biggest number and turn that index into value 1, and all the rest of the indices are all set to 0.</p>
<p>For example I have [1,2,4,6,2,1], and I would like to convert it into [0,0,0,1,0,0].</p>
<p>I’ve used<br>
idx = torch.topk(tensor,1)[1]<br>
out = torch.zeros_like(tensor)<br>
out[idx] = 1</p>
<p>But I’m not sure if there are better ways.</p>
<p>Thank you!</p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>You can try this as well.</p>
<pre><code class="lang-auto">import torch

tensor = torch.tensor([1,2,4,6,2,1])
idx = torch.topk(tensor,1)[1]
out = (tensor &gt;= tensor[idx]).type(torch.uint8)
</code></pre>
<p>Thanks</p>

username: Silvaurus
cooked: <p>Yeah, much neater! Thanks!</p>
ParameterList and list of buffers in the same module

username: 11158
cooked: <p>I want to have a Module that is parametrized by a list of tensors. In the situation, when this Module is a “leaf” module in the pipeline (i.e. its parameters are trainable parameters), then it’s pretty simple: I just have to use ParameterList to store its parameters.</p>
<p>The question is, what should I do, if it’s not a “leaf” module, i.e. I somehow constructing it’s parameter tensors and only after that instantiating the module.</p>
<p>Consider this toy example: I want to encapsulate a rang-1 matrix decomposition:</p>
<pre><code class="lang-python">class Rang1Matrix(nn.Module):
    def __init__(self, vectors):
        super().__init__()
        self.vectors = vectors
    
    def forward(self):
        return  t.matmul(self.vectors[0], self.vectors[1])
</code></pre>
<p>In this block of code, if I want to create a module with trainable parameters, then I just have to pass <code>ParameterList</code> as a “vectors” argument. But what should I do, if I already have tensors “vector”, through which  I want autodifferentiate?</p>
<p>In particular, I want <code>Rang1Matrix</code> to fully support functions like <code>.cuda()</code>, <code>.cpu()</code>, <code>.to()</code>, etc.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You want gradients to flow back all the way to the <code>vectors</code> that were given as input to the <code>__init__</code> function?<br>
If so, your code will work fine.<br>
But if you want it to work with <code>.to()</code>-like functions, it is harder. Because then you want these to be done in a differentiable manner, which they don’t really do. And this will create part of the graph that will be re-used between forward passes and that can cause your backward to fail because some buffers have been freed.</p>
About conversion from pytorch to C++ conversion

username: Chen0729
cooked: <p>Hello, all</p>
<p>I have a pytorch model and need to convert to C++.<br>
In pytorch it is:<br>
m=layers[i]<br>
x_f = F.relu(m(x))</p>
<p>where m is one layer, after conversion to C++, I write it as:<br>
torch::Tensor x_f = torch::nn::functional::relu(layers[i]-&gt;as <a>torch::nn::Sequential</a> ()-&gt;forward(x));</p>
<p>I just would like to make sure they are equivalent. Any suggestions ?</p>
<p>Thanks.</p>
Compare pytorch and tensorflow models

username: Ahmed_Abdelaziz
cooked: <p>Hi, what would you do if you have PyTorch and TensorFlow models that are supposed to be replicas of each other and you wanted to test that they are exact replicas in terms of layer, weight initialization, loss functions, etc? I converted a TF model into PyTorch but PyTorch accuracy is far off from that of TensorFlow. I thought about using one sample or any kind of seed to force the same sample for both models, same seed for weight initialization and then observe the values for the tensors and see where they diverge? Do you think this is a good idea and is it doable? or would you have done something else?</p>
<p>Two challenges I will face in that case:</p>
<ol>
<li>How to seed both models to act exactly the same?</li>
<li>debugging is a big challenge in tf compared to PyTorch</li>
</ol>
<p>I would appreciate your input.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I wouldn’t try to use seeds in order to initialize both models etc., as the implementation of the random number generator might be different in the worst case and you would waste a lot of time.</p>
<p>If you want to compare both models, I would start with the good performing model (i.e. the TF model in your case), create a random input and compute the output.<br>
Once this is done, store all data in as numpy arrays (i.e. the input, output, model parameters and buffers).<br>
Afterwards create the corresponding PyTorch model and load the parameters and buffers manually via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layer.parameter.copy_(torch.from_numpy(param))
    ...
</code></pre>
<p>perform the forward pass next and compare the outputs of the TF model and your current PyTorch implementation.<br>
If they show a large difference, you could try to narrow down the layer, which is creating this error.</p>

username: Ahmed_Abdelaziz
cooked: <p>Yes theoretically this is a very good idea but I will have a hard time mapping the parameter names from TF to Pytorch since they don’t share the same names and I think I will have to do the mapping manually. My model is huge so I can imagine that this will take sometime. But that’s an idea worth trying, thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using RNNs without packed sequences

username: mtanti
cooked: <p>What is the recommended approach to use padded sequences (not packed) in RNNs in PyTorch? The solution must allow one to extract the final RNN state conveniently and must work for bi-directional RNNs. Is such a thing not the PyTorch way?</p>
<p>Also, will the act of packing and unpacking sequences before and after using RNNs cause a performance hit?</p>
TypeError in dist_autograd.backward(cid, [loss])

username: Khairul_Mottakin
cooked: <p>Hi, I am using the code from <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="noopener nofollow ugc">IMPLEMENTING A PARAMETER SERVER USING DISTRIBUTED RPC FRAMEWORK</a> tutorial, which should be straightforward to implement. However, I am receiving the TypeError (see following screenshot) while executing <strong>dist_autograd.backward(cid, [list])</strong> and can not get rid of it after trying a lot. Is this because the tensors need to be scalar?<br>
Your help will be much appreciated.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00a79573dad7c22b8eb7663dd2f0e0359249e05a.png" data-download-href="https://discuss.pytorch.org/uploads/default/00a79573dad7c22b8eb7663dd2f0e0359249e05a" title="Screenshot from 2020-09-07 15-48-06"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00a79573dad7c22b8eb7663dd2f0e0359249e05a.png" alt="Screenshot from 2020-09-07 15-48-06" data-base62-sha1="5N2Tg5iavnAWzy2rlk87X0uChQ" width="690" height="126" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-07 15-48-06</span><span class="informations">815×149 17.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/khairul_mottakin">@Khairul_Mottakin</a>, looks like you are using PyTorch v1.4? The cid arg is added in v1.5 IIRC. Could you please try upgrade to the latest release v1.6?</p>
<p>cc <a class="mention" href="/u/rvarm1">@rvarm1</a></p>

username: Khairul_Mottakin
cooked: <p>Yah, it is working locally after updating pytorch version. Many many thanks <a class="mention" href="/u/mrshenli">@mrshenli</a> and <a class="mention" href="/u/rvarm1">@rvarm1</a> for your contribution and helping us to implement distributed system in our own ways.<br>
While started training from remote worker, it says “RuntimeError: […/third_party/gloo/gloo/transport/tcp/pair.cc:769] connect [127.0.1.1]:14769: Connection refused”. The same issue had been raised by <a class="mention" href="/u/oleg_ivanov">@Oleg_Ivanov</a> in <a href="https://discuss.pytorch.org/t/strange-behaviour-of-gloo-tcp-transport/66651">here</a>. I am not sure whether it has been solved.  Should I use "os.environ[‘GLOO_SOCKET_IFNAME’]=‘nonexist’ " ?</p>
<p>Can you suggest any tutorial for building such smaller cluster (2-3 remote workers with 1 master PS) to implement the Parameter Server using RPC of PyTorch?</p>
<p>Thank you very much once again.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/khairul_mottakin">@Khairul_Mottakin</a>, can you try printing out <code>GLOO_SOCKET_IFNAME</code>, <code>MASTER_ADDR</code>, and <code>MASTER_PORT</code> immediately before where <code>init_rpc</code> is called on all processes? And args did you pass to <code>init_rpc</code>?</p>
What is the appropriate way to use BCE loss with ResNet outputs?

username: seankala
cooked: <p>Hello. The title explains the overall problem, but for some elaboration:</p>
<p>I’m using <code>torchvision.models.resnet18()</code> to run an anomaly detection scheme. I initialize the model by doing:</p>
<pre><code class="lang-auto">net = torchvision.models.resnet18(num_classes=2)
</code></pre>
<p>since in my particular setting <code>0</code> equals normal samples and <code>1</code> equals anomalous samples.</p>
<p>The output from my model is of shape <code>(16, 2)</code> (batch size is 16) and labels are of size <code>(16, 1)</code>. This gives me the error that the two input tensors are of inappropriate shape.</p>
<p>In order to solve this, I tried something like:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; new_output = torch.argmax(output, dim=1)
</code></pre>
<p>Which gives me the appropriate shape, but running <code>loss = nn.BCELoss(new_output, labels)</code> gives me the error:</p>
<pre><code class="lang-auto">RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>
<p>What is the appropriate way for me to approach this issue? Thanks.</p>

username: chetan_patil
cooked: <p>Hi, you can use handle this problem in two ways,<br>
One way is to get the model’s output of shape <code>(16,1)</code> , and use a <code>nn.BCELoss()</code>.<br>
Or get the model’s output to be shaped <code>(16,2)</code> and use <code>nn.CrossEntropyLoss()</code></p>
<p>In your case, you have decided to have shape <code>(16,2)</code> , so you need to use <code>nn.CrossEntropyLoss()</code> instead of <code>nn.BCELoss()</code></p>

username: seankala
cooked: <p>Hi. I’ve tried <code>nn.CrossEntropyLoss</code> as well with <code>nn.CrossEntropyLoss(output, label)</code> and <code>nn.CrossEntropyLoss(output, label.flatten())</code> and both cases still return:</p>
<pre><code class="lang-auto">RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>

username: chetan_patil
cooked: <p>Considering two classes, do the <code>label</code> span between 0 and 1 ?</p>
<p>I have a minimal code which works, make sure this code and yours are similar.</p>
<pre><code class="lang-auto">logits = torch.from_numpy(np.random.randn(16,2)).float()
target = torch.from_numpy(np.random.randint(0,2, size = 16)).long()

loss = nn.CrossEntropyLoss()
z = loss(logits, target)
</code></pre>

username: chetan_patil
cooked: <p>Hey, you just need to initialize the conditons inside the <code>nn.CrossEntropyLoss()</code>.<br>
You must pass the actual <code>predictions</code> and <code>targets</code> after you instantiate the <code>loss</code> object.</p>

username: xian_kgx
cooked: <p>You need to instantiate the loss function object first. Then only you can use it.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
loss = criterion(input, targe)
</code></pre>
How to compare a torch::tensor shape against some other shapes?

username: Shisho_Sama
cooked: <p>Hello every one,<br>
I’m trying to compare a torch::Tensors sizes agains something else, but it seems I’m doing it wrong!<br>
I tried</p>
<pre><code class="lang-cpp">    auto t = torch::ones({ 3,3 }).sizes();
    std::cout &lt;&lt; c10::IntArrayRef{ 3,3 } &lt;&lt; std::endl;
    std::cout &lt;&lt; (t.equals(c10::IntArrayRef{ 3,3 })) &lt;&lt; std::endl;
</code></pre>
<p>which always returns false!<br>
I also tried</p>
<pre><code class="lang-cpp">t == c10::IntArrayRef{ 3,3 };
</code></pre>
<p>which also returns false!<br>
Whats wrong here?</p>

username: dambo
cooked: <p>This is just an example:</p>
<pre><code class="lang-auto">torch::Tensor out_tensor = module.forward({input_tensor}).toTensor();
std::cout &lt;&lt; "dimensions:" &lt;&lt; out_tensor.ndimension() &lt;&lt; "\n";
std::cout &lt;&lt; "v_size1:" &lt;&lt; out_tensor.size(1) &lt;&lt; "\n";
std::cout &lt;&lt; "v_size2:" &lt;&lt; out_tensor.size(2) &lt;&lt; "\n";
</code></pre>

username: Shisho_Sama
cooked: <p>Thanks but it doesnt answer the question. I’m specifically after comparing the shapes of the tensors. I’m specifically want to avoid comparing each dimension by hand.</p>

username: John_Grabner
cooked: <p>Is this what you are looking for?</p>
<blockquote>
<p>torch.equal(torch.Tensor([60,16,32,64]), torch.Tensor(list(out.shape)))<br>
==&gt;True</p>
</blockquote>

username: Shisho_Sama
cooked: <p>Thank you very much<br>
I need to add  that the normal <code>==</code> should work out of the box, and my case was a special occurance where this wasnt wroking under the debug build while it was working under the release for some unknown reasons. I spent a lot of time investigating this to no luck! I ultimately concluded as not a torch issue but maybe a VS2019 /VC++/weird project configuration issue.</p>