akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Automatic rank assignment in init_process_group

starphoenix: <p>I am trying to setup distributed training and encountered some problems with initialization of process group.<br>
Since I have a shared file-system between nodes, I chose initialization with <code>file://</code>. But I got this error:</p>
<blockquote>
<p>ValueError: Error initializing torch.distributed using file:// rendezvous: rank parameter missing</p>
</blockquote>
<p>Then I found in documentation, that “automatic rank assignment is not supported anymore”, although  documentation for <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc">init_process_group</a> imply otherwise.</p>
<p>Is there a way to not tell <code>init_process_group</code> rank explicitly. And what is the point of <code>init_process_group</code> if I have to pass rank explicitly.</p>

pietern: <p>Good point.</p>
<p>This used to be possible and was not reinstated when we moved to c10d for PyTorch 1.0. I created an issue on GitHub to bring this functionality back: <a href="https://github.com/pytorch/pytorch/issues/22128" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/22128</a>.</p>
Problems in implementing of U-net

lidaryani: <p>Hello<br>
I want to implement the U-net with Pytorch but I have several questions.<br>
The first question is: The input of U-net is a slice of the abdominal CT scan and The organs must be segmented at the output, should I set the number of U-net output classes to the number of organs?<br>
every slice has a size like 1x512x512</p>
<p>I set U-net input size to one, and output size(number of classes) to number of organs. I mean, is it right?</p>
<p>Second question: how many dimensional should Grand truth be?  Suppose we have 10 classes(10 organs). Should I create a tensor, which, in every dimension, is one of the organs maintained by Grand truth(10x512x512)? Or do I create a tensor with one dimension and identify each organ with an independent value(1x512x512)?</p>

ptrblck: <p>If you are using <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code> as your criterion, the output of your model should have the shape <code>[batch_size, nb_classes, height, width]</code>, while your target should be a <code>LongTensor</code> in the shape <code>[batch_size, height, width]</code> containing the class indices.<br>
If you are not dealing with a background class, the number of classes should correspond to the number of organs.</p>

lidaryani: <p>Thank you so much for replying, and another question came to me that is: if I change criterion from <code>nn.CrossEntropyLoss</code> to Dice( or other criterion functions) the shape of the target is important?<br>
Now I am using <code>nn.CrossEntropyLoss</code> but the output after 500 epochs is not satisfied me and it’s not like the output of papers with same situations, so I thought maybe my code is wrong, so I doubted the size of the target and model. So the size of the Target and the model can cause the problem of not learning?</p>

ptrblck: <aside class="quote no-group" data-username="lidaryani" data-post="3" data-topic="48758">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/a5b964/48.png" class="avatar"> lidaryani:</div>
<blockquote>
<p>so I doubted the size of the target and model.</p>
</blockquote>
</aside>
<p>Which size did you double? The spatial size or the number of channels?</p>
<p>If you are using another loss function, e.g. dice loss, you have to make sure to pass the output and target in a specific shape. Have a look at <a href="https://github.com/pytorch/pytorch/issues/1249#issuecomment-339904369">this post</a> to see an implementation of multi-class dice loss.</p>

lidaryani: <p>The post you mentioned me solved my problem.Thank you.</p>
Parallel processing samples that can't be orgnized as batches

zhangyuygss: <p>I have a task that every sample have different sizes (or different modules of network to forward), so I can’t put them in batches. But train the samples one by one is very inefficient. How can I paralleling the process?</p>
<p>I think <code>torch.multiprocessing</code> might be one solution. But I’m still not sure how to use it after reading the docs.</p>

pietern: <p>A common approach is to pad the inputs to the biggest shape. You still have to make sure that your model works well with padded inputs of course.</p>
Multiple threads on GPU not working?

Usama-Zafar: <p>Hey, I am trying write a custom synchronization scheme with Pytorch in python. I am training a mode and at the same time spawning a thread that waits for synchronization requests from other parallel processes. But the problem is my main process get stuck at dist.send() call.</p>
<p>A equivalent code is as follows:</p>
<pre><code class="lang-auto">def run(rank_local, rank, world_size):
    print("I WAS SPAWNED ", rank_local, " OF ", rank)
    
    tensor_1 = torch.zeros(1)
    tensor_1 += 1

    while True:
        print("I am spawn of: ", rank, "and my tensor value before receive: ", tensor_1[0])
        nt = dist.recv(tensor_1)
        print("I am spawn of: ", rank, "and my tensor value after  receive from", nt, " is: ", tensor_1[0])
    
def communication(tensor, rank):

    if rank != 0:
        tensor += (100 + rank)
        dist.send(tensor, dst=0)
    else:
        tensor -= 1000
        dist.send(tensor, dst=1)
    print("I AM DONE WITH MY SENDS NOW WHAT?: ", rank)


if __name__ == '__main__':
    
    # Initialize Process Group
    dist.init_process_group(backend="mpi", group_name="main")
    
    # get current process information
    world_size = dist.get_world_size()
    rank = dist.get_rank()
    
    #torch.cuda.set_device(rank%2)

    # Establish Local Rank and set device on this node
    p = ThreadWithReturnValue(target=run, args=(0, rank, world_size))  #mp.Process(target=run, args=(0, rank, world_size))
    p.start()

    tensor = torch.zeros(1)
    communication(tensor, rank)
    
    p.join()

</code></pre>
<p>Please note that if I remove the line <code>torch.cuda.set_device(rank%2)</code> the code works perfectly fine. Any thoughts? Why am I not able to achieve the same behavior with CUDA?</p>

pietern: <p>In your snippet <code>dist.send</code> and <code>dist.recv</code> will race. Since you’re using blocking calls, either path will not continue until it has finished. There must be some interaction between MPI and CUDA that means these calls are now really blocking. You could try explicitly sequencing them and see if that changes anything. In general, it is not encouraged to use Python threading in combination with a single distributed context. It is not written to be thread safe.</p>
Scalar type Long; but got CUDAType instead

stone30: <p>Hi, I  am trying to learn with pytorch exercises  and I’m getting the error</p>
<p>Expected tensor for argument <span class="hashtag">#1</span> ‘indices’ to have scalar type Long; but got CUDAType instead (while checking arguments for embedding)</p>
<pre><code class="lang-auto">import torch.nn as nn

class RNN(nn.Module):
    
    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5):
        """
        Initialize the PyTorch RNN Module
        :param vocab_size: The number of input dimensions of the neural network (the size of the vocabulary)
        :param output_size: The number of output dimensions of the neural network
        :param embedding_dim: The size of embeddings, should you choose to use them        
        :param hidden_dim: The size of the hidden layer outputs
        :param dropout: dropout to add in between LSTM/GRU layers
        """
        super(RNN, self).__init__()
        # TODO: Implement function
        
        # set class variables
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        self.output_size = output_size
        
        # define model layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        print(self.embedding)

        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=dropout, batch_first=True)
        self.dropout = nn.Dropout(dropout)

        self.fc = nn.Linear(hidden_dim, output_size)
    
    
    def forward(self, nn_input, hidden):
        """
        Forward propagation of the neural network
        :param nn_input: The input to the neural network
        :param hidden: The hidden state        
        :return: Two Tensors, the output of the neural network and the latest hidden state
        """</code></pre>

ptrblck: <p>Are your model and data both on the GPU?<br>
Based on the error message it looks like the one of these instances is still on the CPU while the other is on the GPU.</p>

stone30: <p>The GPu should be enabled</p>
<blockquote>
<p>print(“torch.cuda.is_available()   =”, torch.cuda.is_available())<br>
print(“torch.cuda.device_count()   =”, torch.cuda.device_count())<br>
print(“torch.cuda.device(‘cuda’)   =”, torch.cuda.device(‘cuda’))<br>
print(“torch.cuda.current_device() =”, torch.cuda.current_device())<br>
torch.cuda.get_device_name(0)</p>
</blockquote>

ptrblck: <p>You would still need to manually push the model and data to the GPU using:</p>
<pre><code class="lang-python">model = MyModel()
device = 'cuda'
model = model.to(device)
data = data.to(device)
...
</code></pre>
Transfer data to GPU doubled in distributed training

Antsypc: <p>I test execution time for these two lines in <a href="https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/engine/trainer.py#L64" rel="nofollow noopener">engine/trainer.py#L64</a>. I called it as todevice time</p>
<pre><code class="lang-python">        images = images.to(device)
        targets = [target.to(device) for target in targets]
</code></pre>
<p>I used 2 nodes, each node has 8 GPUs. Each GPU processes 2 images. I run cammand on first host (second host just replace with --node_rank=1):</p>
<pre><code class="lang-auto">export NGPUS=8
python -m torch.distributed.launch --nproc_per_node=$NGPUS \
--nnodes=2 --node_rank=0 --master_addr="172.17.61.2" --master_port=22876 \
tools/train_net.py --config-file "configs/e2e_faster_rcnn_R_50_FPN_1x.yaml" MODEL.RPN.FPN_POST_NMS_TOP_N_TRAIN 2000 SOLVER.IMS_PER_BATCH 32 SOLVER.BASE_LR 0.04 SOLVER.STEPS "(30000, 40000)" SOLVER.MAX_ITER 50000 TEST.IMS_PER_BATCH 16 OUTPUT_DIR models/tmp-2n8g
</code></pre>
<p>todevice time in 2 nodes(each 8GPUs) is doubled compared to 1 node 8GPUs. I also test other time such as data_time, backbone time, rpn time, backward time, step() update params time. All these time is so closed to one node with 8 GPUs.</p>
<p>I also test 2 nodes with 16GPUs on each. It’s the same that todevice time is twice than one node with 16GPUs, each GPU processes 2 images.</p>
<p>I am very confused. Each gpu processes the same number of images in both 2 situations. But time increased in distributed mode.</p>

pietern: <p>That is weird indeed. Can you isolate the problem to data loading (i.e. don’t train a model, just iterate over the data set)? Due to asynchronous nature of CUDA, wall clock time that attribute to data transfer is in fact caused by asynchronous execution of for example autograd, the optimizer, etc.</p>
How to calculate meters in Pytorch1.1 & DistributedDataParallel()?

StuChen: <p>I want to use model parallel and data parallel at the same time, and have read many docs and tutorials from official website.<br>
One confusing problem I faced is how to collect all kinds of meter values in each Process?</p>
<p><strong>Question1</strong>: In the <a href="https://github.com/pytorch/examples/blob/5b1f45057dc14a5e2132b45233c258a1dc2a0aab/imagenet/main.py#L286" rel="nofollow noopener">official tutorial</a>, they just record meters value in each Process.<br>
But in my code, I print loss value in each process, they are different. So, I think the value of other meters are also different.<br>
<strong>Is that tutorial wrong?</strong> In my opinion, I think the right way should synchronize loss, acc and other meters first, then all processes maintain the same values, after that I just need to print meters information in one Process.</p>
<p><strong>Question2</strong>: In the <a href="https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html#initialize-model" rel="nofollow noopener">official tutorial</a>, they say ‘the DistributedDataParallel module also handles the averaging of gradients across the world, so we do not have to explicitly average the gradients in the training step’.<br>
But, because of question1, <strong>does the API actually work as what the tutorial said?</strong>  Because each of the processes has a different loss value, although they start from the same init weights, <strong>will model weights in each process be optimized in different directions?</strong></p>

pietern: <p>Hi <a class="mention" href="/u/stuchen">@StuChen</a>,</p>
<ol>
<li>The losses are different because different processes see different inputs with different labels and therefore produce different losses. If you’re looking for a global top1 or top5, you can use distributed primitives from <code>torch.distributed</code> to average them.</li>
<li>The model in each process will be optimized in the same way, because after calling <code>loss.backward()</code> the resulting gradients are identical across processes. In combination with the initial weights being identical, the resulting weights after optimization are also identical.</li>
</ol>
Best practice for training & validating on multiple GPUs?

TDHTTTT: <p>Several configuration I could think of:</p>
<ol>
<li>Train and validate on all possible same GPUs (not able to set different batch_size for train/validate)</li>
<li>Train and validate on different GPUs (can set different batch_size)</li>
<li>Train on all GPUs and save the model per epoch, later run the model on validation data. (not able to use early stopping on validation loss)</li>
</ol>
<p>What is the best practice?<br>
Any other thoughts and suggestions will be appreciated.</p>

pietern: <p>All depends on your goals. If you want to maximize validation throughput, you’ll want to use as as many devices as you can. If you don’t care and want to keep your code simple, you can choose to use just one.</p>
Training becomes slower when using nn.DataParallel with custom convolution layer

Mr.Z: <p>In my layer class, there is a value tensor, an index tensor, and a kernel tensor. In the forward function I use scatter_add to add the value to kernel according to the index. Then the kernel is used as the convolution kernel to perform convolution, my layer class looks like this:</p>
<pre><code class="lang-auto">class MyLayer(nn.Module):
  def __init__(self, C_in, C_out):
    super(MyLayer, self).__init__()

    self.C_in = C_in
    self.C_out = C_out

    self.value = nn.Parameter(...)
    self.register_buffer('inds', ...)
    self.register_buffer('kernel', torch.zeros(self.C_in * self.C_out * 1 * 1))

  def forward(self, x, p):
    value = self.value * p
    kernel = self.kernel.scatter_add(0, self.inds, value)
    kernel = kernel.view(self.C_in, self.C_out, 1, 1)

    out = F.conv2d(x, kernel, stride=1)

    return out
</code></pre>
<p>However, when I wrap my network with nn.DataParallel and training on 2 GPUs, I observe a doubled forward time compared with single GPU.  Could someone tell me why my layer becomes even slower with multi-GPU, and how to modify it to work with nn.DataParallel?</p>

pietern: <p>Nothing in the code you pasted looks particularly slow. Perhaps the larger model you’re using contains many small layers/kernels? The <code>nn.DataParallel</code> wrapper replicates a module to N devices and runs <code>forward</code> on each of them. This overhead can dominate the runtime if your model is very small or has many very small kernels.</p>

Mr.Z: <p>Thanks for your advice. Yes, there are many small kernels in my network, actually I found it extremely slow when the batch size is small. Looks like with small batch size the runtime of scatter_add becomes the bottleneck (which can not be accelerated by nn.DataParallel), and when the batch size increases the nn.DataParallel begins to speed up training.</p>
Accumulate gradients in DDP

tetratrio: <p>Hey,</p>
<p>Is there any easy way to accumulate gradients in a DistributedDataParallel model?<br>
From what I see the only way to do this would be to copy gradients to a separate buffer before the next forward/backward?</p>
<p>Any plans on adding functionality for this to Pytorch? DataParallel gives too much overhead for me otherwise I would use that.</p>

pietern: <p>This was merged very recently in <a href="https://github.com/pytorch/pytorch/pull/21736" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/21736</a>.</p>
Safely removing a Module from DDP

mdlockyer: <p>Does anyone have any thoughts on the safest way to remove the DistributedDataParallel wrapper from a Module? Currently I’m just doing something like:</p>
<pre><code class="lang-auto"># Model in this case has already been wrapped in DDP
model = model.module
</code></pre>
<p>In the docs for DDP, it mentions hooks that are being registered in the module’s params:</p>
<blockquote>
<p>when wrapping up your model with DistributedDataParallel, the constructor of<br>
DistributedDataParallel will register the additional gradient<br>
reduction functions on all the parameters of the model itself at the<br>
time of construction</p>
</blockquote>
<p>I take it those hooks are still there if I just grab the module attribute from the DDP instance right?</p>

mdlockyer: <p>Based on a <a href="https://github.com/pytorch/pytorch/issues/21344" rel="nofollow noopener">recent issue</a> opened for PyTorch, it is in fact the case currently (v1.1.0) that the module will retain the reduction functions and new ones will be added each time the model is wrapped in DDP</p>

pietern: <p>This has been fixed and will be available in PyTorch 1.2 (and is already available in the nightly builds).</p>
Why do workers end up with different loss values?

jerinphilip: <p>From <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">docs</a>,</p>
<blockquote>
<p>Constructor, forward method, and differentiation of the output (or a function of the output of this module) is a distributed synchronization point. Take that into account in case different processes might be executing different code.</p>
</blockquote>
<p>I’m trying to print loss from each worker, and I’m getting the following output:</p>
<pre><code class="lang-auto">| distributed init (rank 2): tcp://localhost:1947
| distributed init (rank 3): tcp://localhost:1947
| distributed init (rank 0): tcp://localhost:1947
| distributed init (rank 1): tcp://localhost:1947
| initialized host gnode03 as rank 3
| initialized host gnode03 as rank 1
| initialized host gnode03 as rank 2
| initialized host gnode03 as rank 0
rank 0 loss 920.7410278320312
rank 1 loss 1102.2825927734375
rank 3 loss 765.515869140625
rank 2 loss 642.1211547851562
rank 2 loss 950.1659545898438
rank 1 loss 863.4507446289062
rank 3 loss 1053.586669921875
rank 0 loss 551.5623168945312
rank 0 loss 679.0967407226562
rank 2 loss 970.89892578125
rank 1 loss 1246.443359375
rank 3 loss 1169.9415283203125
rank 0 loss 798.79833984375
</code></pre>
<p>Does this mean I have to explicitly aggregate and average the total loss by total batch size? Or is this handled internally? The segment which prints the above looks like this:</p>
<pre><code class="lang-auto">        self.model.train()
        self._optimizer.zero_grad()
        sample = move_to(sample, self.device)
        loss, logging_outputs = self.model(sample)
        loss.backward()
        clip_grad_norm_(self._model.parameters(), args.max_grad_norm)
        self._optimizer.step()
        return loss.item()
</code></pre>

pietern: <p>Ah! This is indeed what you meant in <a href="https://discuss.pytorch.org/t/distributeddataparallel-loss-compute-and-backpropogation/47205" class="inline-onebox">DistributedDataParallel loss compute and backpropogation?</a>. Pasting my answer there here as well for posterity and the indexers.</p>
<p>Each process computes its own output, using its own input, with its own activations, and computes its own loss. Then on <code>loss.backward()</code> all processes reduce their <em>gradients</em>. As <code>loss.backward()</code> returns, the gradients of your model parameters will be the same, and the optimizer in each process will perform the exact same update to the model parameters.</p>
<p>Note that this is <strong>only</strong> the case if you use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener"><code>torch.nn.parallel.DistributedDataParallel</code></a>. If you don’t, you’ll need to take care of gradient synchronization yourself.</p>
Torch.distributed: Send and Gather on GPU

spaceinvader: <p>Hello!</p>
<p>I want to write a distributed program and run it on a cluster with several multi-GPU nodes which is managed using slurm.</p>
<p>The program should have one master process, which sends (equal to MPI_Send / MPI_Recv) different data to other processes and then collect the results (equal to MPI_Gather).</p>
<p>Could you please tell me if my task can be solved using torch.distributed? In the official docs (<a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html</a>) I found only question marks for send/recv MPI operations for GPU.</p>
<p>I also tried Horovod but found no wrappers around send/recv functions.</p>

pietern: <p>The question marks mean that it depends whether or not your MPI distribution is compiled with CUDA support or not. If it is, send/recv of GPU tensors works. If it doesn’t, you’ll have to copy GPU tensors to CPU before you can pass them to send/recv. (see <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.cpu" rel="nofollow noopener"><code>torch.Tensor.cpu</code></a>).</p>
FusedAdam optimizer in Nvidia AMP package

jia_lee: <p>What is the difference between FusedAdam optimizer in Nvidia AMP package with the Adam optimizer in Pytorch?</p>

mcarilli: <p>The Adam optimizer in Pytorch (like all Pytorch optimizers) carries out optimizer.step() by looping over parameters, and launching a series of kernels for each parameter.  This can require hundreds of small launches that are mostly bound by CPU-side Python looping and kernel launch overhead, resulting in poor device utilization.  Currently, the FusedAdam implementation in Apex flattens the parameters for the optimization step, then carries out the optimization step itself via a fused kernel that combines all the Adam operations.  In this way, the loop over parameters as well as the internal series of Adam operations for each parameter are fused such that optimizer.step() requires only a few kernel launches.</p>
<p>The current implementation (in Apex master) is brittle and only works with Amp opt_level O2.  I’ve got a WIP branch to make it work for any opt_level (<a href="https://github.com/NVIDIA/apex/pull/351" rel="nofollow noopener">https://github.com/NVIDIA/apex/pull/351</a>).  I recommend waiting until this is merged then trying it.</p>

sbsky: <p>Just wondering, wouldn’t it be possible to use pytorch multiprocessing to parallelise the Adam loop?  Or CUDA streams?</p>

pietern: <p><a class="mention" href="/u/sbsky">@sbsky</a> Either technique comes with its own overhead. If the time it takes to launch one of these kernels is &gt;&gt; the time it takes to execute it, you’ll have to optimize the launch itself. If you decide to do this with multiprocessing, you’ll need to move the references to those tensors between processes, which isn’t free. The alternative is to launch fewer kernels, which is what <a class="mention" href="/u/mcarilli">@mcarilli</a> did in AMP.</p>
torch.nn.DataParallel(model) fails with different batch size

ricky123: <p>I am facing a very starnge problem with <code>torch.nn.DataParallel()</code>. I have a system with 8 GPUs and I want to use multiple GPUs for training my model. Now when I wrap the model with nn.DataParallel, it works only for batch_size 10! This is very odd because for any batch size other than 10 ( even smaller), the execution just gets stuck. When I am not using parallelism and running on single GPU, it is working properly. But for batch_size more than 16, cuda is running out of memory because my input vectors are very large and model is very big. So I am unable to take advantage of multiple GPUs. Any soltuion out there? Thank you in advance… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pietern: <p>Batch size 10 is odd indeed. I would have expected it to only work with a multiple of 8, if you’re using 8 devices. What kind of model are you trying to parallelize?</p>

ricky123: <p>A transformer model. It has 12 layers of encoder blocks.</p>

pietern: <p>If the batches are asymmetric in size then it is possible that some devices can handle 2 examples where others can’t. Not much that can be done about this, save for memory profiling to prove that this is what’s happening.</p>
Unequal GPU amount in nodes for distributed training

briankosw: <p>Hypothetically, if I have 2 GPUs in node 0 and 3 GPUs in node 1, how would I configure it to support that? All the examples in the documentation as well as example codes perform <code>word_size = gpus_per_node * args.world_size</code>, which assumes from <code>gpus_per_node</code> that there is an equivalent amount of GPUs per node.</p>

pietern: <p>That expectation is built into the <code>torch.distributed.launch</code> utility but not elsewhere. You can start 5 processes (1 per GPU) and use <code>world_size=5</code> where you have 2 processes on one machine and 3 processes on the other machine. It’s not very common to have this situation, so I’m not surprised most of the examples you see assume a symmetric contribution across machines. That said, you can still make it work, but will have to adapt those examples or start from scratch with <code>torch.nn.parallel.DistributedDataParallel</code>.</p>
How to learn distributed data parellel?

CHENCHAO0526: <p>I didn’t learn basic knowledge of computer before, can’t understand communication, concurrent, multiprocessing, what do I need to learn for understanding DDP?</p>

pietern: <p>You can start with the tutorial. I suggest you study and research the concepts you don’t know thoroughly, if that’s what your goal is. The documentation for <a href="https://docs.python.org/3.7/library/multiprocessing.html" rel="nofollow noopener">Python multiprocessing</a> is very thorough, for starters.</p>
What is multi-node in ''Multi-Node multi-process distributed training mean?

CHENCHAO0526: <p>In the DDP tutorial, the author user 'multi-node ’ In one computer, I don’t understand why using it, I want to know when should I use multi-node?</p>

ptrblck: <p>Could you post a link to the tutorial?<br>
A node usually refers to a host, so that I’m not sure what multi-node on a single computer means.</p>

CHENCHAO0526: <p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a><br>
I guess that the author just want to show how to use multi-node, and let us generize to another enviroment?</p>

pietern: <p>You can replace it with “multi-process” and it will still be valid. It’s common to use a single PyTorch process per GPU device in your system. Running 8 processes across 8 machines won’t be different from running 8 processes on a single machine (provided it has 8 GPUs), except for performance.</p>
Must I define a model from very begining to use torch.jit.scriptmodule?

coincheung: <p>Hi,</p>
<p>I am trying to use libtorch. According to the docs <code>torch.jit.trace</code> cannot handle cases with control flow, I mean to use <code>torch.jit.scriptmodule</code>. This is my code:</p>
<pre><code class="lang-auto">class Resnet18(nn.Module):
    def __init__(self):
          ...
    def forward(self, inten):
        ...

class SerializeModule(torch.jit.ScriptModule):
    def __init__(self):
        super(SerializeModule, self).__init__()
        save_pth = 'res/model_final_naive.pth'
        self.model = Resnet18(n_classes=10, pre_act=True)
        state_dict = torch.load(save_pth)
        self.model.load_state_dict(state_dict)
        self.model.eval()

    @torch.jit.script_method
    def forward(self, inten):
        out = self.model.forward(inten)
        return out

def main():
    serial = SerializeModule()
    serial.save('res/model_serial.pt')

if __name__ == "__main__":
    main()

</code></pre>
<p>This gives the error message of like this:</p>
<pre><code class="lang-auto">File "serialize.py", line 24, in &lt;module&gt;
    main()
  File "serialize.py", line 20, in main
    serial = SerializeModule()
  File "/home/coin/build/miniconda3/envs/py36/lib/python3.6/site-packages/torch/jit/__init__.py", line 1047, in init_then_register
    _create_methods_from_stubs(self, methods)
  File "/home/coin/build/miniconda3/envs/py36/lib/python3.6/site-packages/torch/jit/__init__.py", line 1012, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
RuntimeError: 
attribute lookup is not defined on python value of type 'Resnet18':
@torch.jit.script_method
def forward(self, inten):
    out = self.model.forward(inten)
          ~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    return out

</code></pre>
<p>What is the good way to use torchscript please? And what is the reason that brings about this error?</p>
Explanation autograd, chain rule and Variables

lorenzo_fabbri: <p>I’m trying to understand how <code>autograd</code> works. Suppose I have the following code:</p>
<pre><code class="lang-auto">x = Variable(torch.arange(0, 4).double(), requires_grad=True)

y = Variable(x * 2, requires_grad=True)

z = y * x

z.backward(torch.ones(y.size()).double())
y.sum().backward()

print(x.grad, 2*x)
</code></pre>
<p>It seems to be working correctly. The problem is that if I change the 3rd line to <code>z = Variable(y*x, requires_grad=True)</code>, then it does not work anymore. What’s the difference? Thank you.</p>

rasbt: <p>Is your error something along the lines of?</p>
<pre><code>RuntimeError: Trying to backward through the graph a second time ...
</code></pre>
<p>?</p>
<p>That’s because the graph gets deleted after it’s been evaluated via .backward(). If you want to backpropate through the graph a second time. You can set the argument "<code>retain_graph=True</code> in your first backward call.</p>
<p>Also, please note that Variable has been deprecated since PyTorch 0.4 (multiple versions ago) and it’s better to use the torch.tensor constructor. E.g., the following should work:</p>
<pre><code class="lang-python">x = torch.tensor(torch.arange(0, 4).double(), requires_grad=True)

y = x.detach().requires_grad_(True)*2

z = y * x

z.backward(torch.ones(y.size()).double(), retain_graph=True)

y.sum().backward()

print(x.grad, 2*x)
</code></pre>
Tensor info like np.info() does it exist?

blackbirdbarber: <p>I would like to get info about a tensor just like I can do that for np array.</p>
<pre><code class="lang-auto">import numpy as np
n = np.array([[1, 2, 3], [4, 5, 6]])
np.info(n)
</code></pre>
<p>Is this possible?</p>
<p>Also can you comment why there is difference in stride:</p>
<pre><code class="lang-auto">t = torch.tensor(np.array([[1, 2, 3], [4, 5, 6]]))
print(t.stride())
np.info(t.data.numpy())
###
(3, 1)
class:  ndarray
shape:  (2, 3)
strides:  (12, 4)
itemsize:  4
aligned:  True
contiguous:  True
fortran:  False
data pointer: 0x20ad4068950
byteorder:  little
byteswap:  False
type: int32

</code></pre>
<p>Is it because PyTorch tensors are packed differently.</p>
How to start contributing to pytorch

ASHUTOSH_CHANDRA: <p>I’m looking for open source projects these days and want to contribute to any.</p>
<p>I looked to pytorch project on github and it seemed to me a good place to start. I’m using pytorch from last 1-2 months, so I don’t think it’d be very hard to understand the codebase.</p>
<p>I can’t figure out how to start. I saw the bugs and issues and nothing seemed to me to get started with.</p>
<p>Also I tried joining the pytroch slack channel, but they’re not accepting my proposal. Can someone help me get started.</p>

ptrblck: <p><a class="mention" href="/u/ezyang">@ezyang</a>’s <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="nofollow noopener">post on PyTorch internals</a> might be a good starter to get some ideas about how to contribute.<br>
Also <a class="mention" href="/u/tom">@tom</a> recently published a post and video about <a href="http://lernapparat.de/fixing-your-first-pytorch-bug/" rel="nofollow noopener">fixing your first PyTorch bug</a>.</p>
<p>I would generally recommend to have a look at the usability / simple-fixes category in the <a href="https://github.com/pytorch/pytorch/projects/5" rel="nofollow noopener">issues</a>.</p>
Pytorch SIGILL bug

Peter_Featherstone: <p>On my shitty laptop, I get a SIGILL signal when running some simple pytorch code. I ran it throught GDB and get the following:</p>
<p>Thread 1 “python3.6” received signal SIGILL, Illegal instruction.<br>
0x00007fffb62cc5c4 in Sleef_expf8_u10fma4 () from /usr/local/lib/python3.6/dist-packages/torch/lib/libcaffe2.so</p>
<p>I checked, my laptop doesn’t support FMA instructions. Is that it? Do I need better hardware?  It seems some runtime checking of supported instructions would have been a good idea…</p>

Peter_Featherstone: <p>It looks like other people have had similar issues. The answer seems to be to build from source. Even then the cmake file isn’t working correctly. I suggest run time cpu feature detection for fma, sse, avx and co.</p>
Why Final Fully connected layer is needed in a CNN?

rpalsaxena: <p>While building an image classifier, we use a set of convolution layers and then put a final fully connected layer as the final layer.<br>
If building a 10-class classifier, Can’t we directly use a final convolution layer with output matrix’s dimension as batch_size * w * h*n_channels and transpose it to get the results then use softmax activation function to get probabilities of each class?</p>
<p>It would be great if someone can share the reason of using the final FC layer</p>

Krish: <p>It will work exactly as a FC layer.<br>
As a side note in Pytorch the outputs are in order <em>(batch_size, channels, height, width)</em> contrast to the Numpy way of <em>(height, width, channels)</em></p>

JuanFMontesinos: <p>It makes no sense. Convolutions keep spatial structure. Flattening everything would lead to a worse solution as you should fine a kernel such that applied to a tensor and flattened provides good features. That’s harder than applying fully connected layers.</p>
<p>Sometimes ppl pass max pooling features or average pooling features as classification scores.</p>

Krish: <p>But aren’t we just trying to find that kernel while doing backprop?</p>
Question relate to in_channel and out_channel

Kai_Qu1: <p>I am a beginner in Pytorch. When I was studying the pytorch example for beginners, I found that the output_channel in the previous layer was equal to the output channel to the next layer. Why was that? Also, the input features for fc1 was 800. How to derive that number? Thanks a lot!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/42f535328ae38b592d959b0d94739bd722226277.png" data-download-href="https://discuss.pytorch.org/uploads/default/42f535328ae38b592d959b0d94739bd722226277" title="25%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/42f535328ae38b592d959b0d94739bd722226277_2_410x500.png" alt="25%20PM" data-base62-sha1="9ykRPUzv49D3yksIFCLwX32O9mf" width="410" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/42f535328ae38b592d959b0d94739bd722226277_2_410x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/42f535328ae38b592d959b0d94739bd722226277_2_615x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/42f535328ae38b592d959b0d94739bd722226277_2_820x1000.png 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">25%20PM</span><span class="informations">1612×1964 450 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Krish: <p>Copy-pasting the Conv2d definition from Pytorch Documentation <em>(shrinked)</em>:<br>
<code>nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)</code></p>
<p>So, 20 in line 14 is the number of input channels to that layer. In Pytorch, you do have to state explicitly the input size of the layer.</p>
<p>Coming to that <em>800</em> number, the MNIST images are of size 28x28 and the order of operations is defined in the forward function. The sizes for successive layers are:<br>
<strong>self.conv1(x)</strong>: ((28 - 5 + 1), (28 - 5 + 1)) = (24, 24)<br>
<em>F.max_pool2d(x)</em>: ((24/2), (24/2)) = (12, 12)<br>
<strong>self.conv2(x)</strong>: ((12 - 5 + 1), (12 - 5 + 1)) = (8, 8)<br>
<em>F.max_pool2d(x)</em>: ((8/2), (8/2)) = (4, 4)</p>
<p>Now the number of channels in the output convolutional layer is 50, so the number of values are 50 * 4 * 4 = 800</p>
<p>If you are not familiar with convolutional arithmetic, I will suggest this:<br>
<a href="https://arxiv.org/abs/1603.07285" rel="nofollow noopener">A Guide to Convolutional Arithmetic</a></p>
Transform matrix with set of function per row

LeanderK: <p>I have a matrix where I want to transform each row element-wise with a different function. It’s not a simple linear function, but a non-linear one. Is this possible in pytorch?</p>
<p>My set of functions is not that small (it depends on the dimensionality of the input).</p>

abhayraw: <p>I think you could achieve this by using <code>torch.unfold</code> on the last dimension and perform your operations using the python’s <code>map</code> function. You would need to the concatenate the output of the mapping.</p>
<pre><code class="lang-auto">funcs = [torch.cos, torch.sin, torch.tan, torch.log]
x = torch.tensor(np.arange(20).astype('f')).view(4,5)
xu = torch.unbind(x, dim=0)
torch.stack([funcs[i](x) for i, x in enumerate(xu)], dim=0)
</code></pre>
<p>If you have some functions that need more than one arguments, you could always wrap them up inside lambda functions (Not always the best thing to do), but that depends on what kinds of functions you’re going to be using.</p>

LeanderK: <p>All the functions are calculations for some density (in a statistical sense). It might be that I have to go higher-order if the functions take more than a scalar as an input.</p>
<p>Thanks! Interesting Idea</p>
Importing trained PyTorch models into SageMaker

KeepingItClassy11: <p>I read the announcement a few days ago that AWS SageMaker now supports PyTorch. I already have a trained PyTorch model, I just want to import it into SageMaker and deploy it that way. Is there a quick &amp; easy reference for this, or could someone who’s done this give me some pointers? All the references I’ve found so far are about training a PyTorch model in SageMaker.</p>

Rohit_Agarwal: <p>Have you been able to find a resolution to your issue? If yes, pls share.</p>
Question about how PyTorch handles Regularization

DuttaAbhigyan: <p>So lets say I have a regularisation term which varies with input. But since PyTorch does things in a batch how to communicate to PyTorch we want the regularisation for a single example and not between all entries of the matrix.</p>
<p>For example, if we have an <strong>input dependent</strong> co-adaptation regularizer but now if we pass this as a scalar over batch_size * (2D reg_cost__matrix), how do I ensure that the co-adaptation is reduced in the 2D matrix (i.e <strong>intra matrix</strong> optimisation) and not between the 2D matrices of the batch size (i.e <strong>inter matrix</strong> optimisation).</p>
Indexing a 1-dimensional tensor with another 2-dimensional tensor

mtbler: <p>Hi there</p>
<p>I am trying to index a one-dimensional tensor with a two-dimensional index.</p>
<p>In <code>numpy</code>, the following works:</p>
<pre><code class="lang-python">a = np.array([5 6 7 8 9])
idx = 
[[1 2]
 [4 4]
 [3 3]
 [0 1]
 [2 0]]

# such that
a[idx] = 
[[6 7]
 [9 9]
 [8 8]
 [5 6]
 [7 5]]
</code></pre>
<p>How do I achieve the same thing in pytorch 1.0?</p>
<p>Doing</p>
<pre><code class="lang-python">a = torch.tensor(a)
idx = torch.tensor(idx)
a[idx]
</code></pre>
<p>yields:</p>
<pre><code class="lang-python">IndexError: too many indices for tensor of dimension 1
</code></pre>

ptrblck: <p>Your code just works fine:</p>
<pre><code class="lang-python">a = torch.tensor([5, 6, 7, 8, 9])
idx = torch.tensor([[1, 2],
                    [4, 4],
                    [3, 3],
                    [0, 1],
                    [2, 0]])

a[idx]
&gt; tensor([[6, 7],
        [9, 9],
        [8, 8],
        [5, 6],
        [7, 5]])
</code></pre>

mtbler: <p>Uh yeah, forgot to convert the <code>idx</code> to a <code>torch.tensor</code> in my actual code. It does not seem to work if <code>idx</code> is a <code>numpy</code> array, though. Anyway, thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Train deeplabv3 on your own dataset

Vishrut10: <p>I am using models.segmentation.deeplabv3_resnet101(pretrained=False, num_classes=12, progress=True) as model to train my own dataset.<br>
Dataset consists of jpg and annotation in png(12 classes)</p>
<p>I transformed both to tensors using transforms.ToTensor()</p>
<p>training:<br>
model.train()<br>
outputs = model(inputs)<br>
loss = criterion(outputs[‘out’], labels)</p>
<p>I get the following error<br>
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument <span class="hashtag">#2</span> ‘target’</p>
<p>I checked the dimension for output:<br>
outputs[‘out’].size()<br>
torch.Size([2, 12, 512, 512])</p>
<p>But the target has dimension:<br>
labels.size()<br>
torch.Size([2, 1, 512, 512])</p>
<p>How do I resolve this ? Thanks in advance</p>

ptrblck: <p>How did you create the target?<br>
If you are using <code>nn.CrossEntropyLoss</code> as your criterion, your target should be a <code>LongTensor</code>, have the shape <code>[batch_size, height, width]</code>, and contain the class indices in the range <code>[0, nb_classes-1]</code>.</p>

Vishrut10: <p>Thanks a lot for your reply.<br>
Now I am stuck on the following:</p><aside class="quote" data-post="1" data-topic="15828">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e0b2c6/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/question-about-output-and-label-channels-in-semantic-segmentation/15828">Question about output and label channels in semantic segmentation</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I use some nets,FCN8 ,SegNet for semantic segmentation .The trouble follow: 
all of the nets I used,The last layers of this net output the  feature maps is (1,22,256,256),why not (1,3,256,256)? and another question is that the labels size is (1,1,256,256),why not(1,3,256,256)? 
please help me, I a new gay
  </blockquote>
</aside>

<p>Essentially converting the [batch_size, nb_classes, height, width] from output with 12 classes.</p>
<p>predictions[‘out’]<br>
Out[76]:<br>
tensor([[[[-0.0465, -0.0465, -0.0465,  …, -0.0556, -0.0556, -0.0556],<br>
[-0.0465, -0.0465, -0.0465,  …, -0.0556, -0.0556, -0.0556],<br>
[-0.0465, -0.0465, -0.0465,  …, -0.0556, -0.0556, -0.0556],<br>
…,</p>
<p>predictions[‘out’].size()<br>
Out[77]: torch.Size([1, 12, 300, 400])</p>

ptrblck: <p>I’ve answered in the other thread.</p>

Vishrut10: <p>Yes, thank you for the quick responses</p>
How to replace one type layers by another type layers in pretrained network?

MariosOreo: <p>As the topic, if I want to change all ReLU activation function to ReLU6 or others in pretrained networks(e.g. resnet18), how can I achieve it in a convenient way?</p>
<p><strong>(maybe in for loops instead of using the index of the target layer)</strong></p>
<p>The structure of pretrianed model includes <code>nn.Model</code> and <code>nn.Sequential</code>, so I can not change them easily.</p>
<p>Could you give me some hints? Any advice would be appreciated!</p>

ptrblck: <p>If you just would like to swap the <code>ReLU</code> activation for another one, you could just access <code>self.relu</code> directly:</p>
<pre><code class="lang-python">model = models.resnet18()
model.relu = nn.ReLU6()
</code></pre>
Commented models in a model do change the output. why?

falmasri: <p>During some test I have a model this way :</p>
<pre><code class="lang-auto">class Downsampler(nn.Sequential):
    def __init__(self, nFeat, nBlock):
        super().__init__()
        self.nBlock = nBlock

        self.RDBS = nn.ModuleList()
        for i in range(nBlock):
            self.RDBS.append(RDB(nFeat, nBlock, 16))

        # # global feature fusion (GFF)
        self.GFF_1x1 = nn.Conv2d(nFeat * nBlock, nFeat, kernel_size=1, padding=0, bias=False)
        self.GFF_3x3 = nn.Conv2d(nFeat, nFeat, kernel_size=3, padding=1, bias=False)

        self.filter = nn.Sequential(
            nn.Conv2d(nFeat, nFeat, kernel_size=7, padding=3, bias=False, groups=nFeat)
        )

        self.Gauss = GaussianLayer(layers=3, k=21, sigma=3)

    def forward(self, x):

        # RDB_outs = []
        # for i in range(self.nBlock):
        #     out = self.RDBS[i](x)
        #     RDB_outs.append(out)
        # FF = torch.cat(RDB_outs, 1)
        # FdLF = self.GFF_1x1(FF)

        filterd = self.Gauss(x)

        DFdLF = nn.functional.interpolate(filterd, scale_factor=0.5, mode='bicubic', align_corners=False)


        # FGF = self.GFF_3x3(DFdLF)
        # FDF = FGF + x

        return DFdLF
</code></pre>
<p>As you can see I kept them not commented in the model but in the forward. After the I found that the model work better with this implementation I deleted all the non commented models in the model and the commented lines.  To have this :</p>
<pre><code class="lang-auto">class Downsampler(nn.Sequential):
    def __init__(self, nFeat, nBlock):
        super().__init__()
        self.nBlock = nBlock

        self.Gauss = GaussianLayer(layers=3, k=21, sigma=3)

    def forward(self, x):

        filterd = self.Gauss(x)

        DFdLF = nn.functional.interpolate(filterd, scale_factor=0.5, mode='bicubic', align_corners=False)

        return DFdLF
</code></pre>
<p>I found that the results are different after deleting these lines even if they exist earlier but not used.</p>

ptrblck: <p>Most likely you just see some effects of different parameter initializations.<br>
Since the modules were initialized in <code>__init__</code>, the random number generator was called, thus sampled other values for all following modules.<br>
Even if you set a seed, you’ll see the effect.</p>
<p>Could you remove the unused modules, run the experiment for a few different seeds, and compare the results?</p>

falmasri: <p>This is possible, but I have to mentioned that outputs are really different. I’ll check with random seed and update the post</p>