akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is this a correct way to combine mpi and nccl in distributed training

huoge: <p>Hello everyone. I used the mpi to run multiprocess and use the nccl backend with DDP, Is this a correct way that I use the mpi and nccl? I’d appreciate if anybody can help me! Thanks in advance!<br>
here is my sample code:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def dist_train(rank, size):
   
    local_rank = int(os.environ['OMPI_COMM_WORLD_LOCAL_RANK'])
    if args.gpu:
        torch.cuda.set_device(local_rank)

    # set torch device
    device = torch.device("cuda" if args.gpu and torch.cuda.is_available() else "cpu")
    model = model.to(device)
    model = DDP(model, device_ids=[local_rank])

    '''training code......'''


def init_process(rank, size, fn, backend='gloo'):
    dist.init_process_group(backend, init_method='tcp://master_ip:port', rank=rank, world_size=size)

    fn(rank, size)

world_size = int(os.environ['OMPI_COMM_WORLD_SIZE'])
world_rank = int(os.environ['OMPI_COMM_WORLD_RANK'])
init_process(world_rank, world_size, dist_train, backend='nccl')
</code></pre>
<p>My running command is : mpirun -np ${totals} -H ${slots} ${COMMON_MPI_PARAMETERS} python demo.py</p>

mrshenli: <p>This should be fine, did you see any error?</p>
In DDP training, how to do gradient synchronization among part of DDP nodes?

Asta: <p>As far as I know, when in DDP(DistributedDataParallel), <code>loss.backward()</code>will synchronize gradient for <strong>all nodes</strong> in the group automatically through <code>Reducer</code>. However, If I do want to synchronize and update model parameters among <strong>part of nodes</strong> in some epochs, how can I manage to do that?</p>
<p>I would appreciate you for any hints or concrete code sample <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

mrshenli: <p>Hey <a class="mention" href="/u/asta">@Asta</a></p>
<p>I see two options:</p>
<p>Option 1: create two DDP instances on each process and construct then using different <code>ProcessGroup</code> instances. One DDP instance can use the global <code>ProcessGroup</code> which will synchronize across all nodes, and another DDP instance can use a different <code>ProcessGroup</code> of a sub-group which is created using the <code>new_group</code> API.</p>
<p>Option 2: Use the <a href="https://github.com/pytorch/pytorch/issues/39272" rel="nofollow noopener">DDP comm hook</a> [<a href="https://github.com/pytorch/pytorch/blob/5d608d45cfd5c42b374c9ac6d2f8bb6ddd1c6e7f/torch/nn/parallel/distributed.py#L615" rel="nofollow noopener">code and example</a>]. This is still a prototype feature and might change in the future.</p>
<p>One thing to mention is that, when you do this (sync gradients in subgroup), it will create gradient inconsistency across processes (as some process didn’t participate in some iteration). This would then lead to inconsistency in model replicas on different processes. DDP only broadcasts model in its ctor. To keep all model replicas consistent, it relies on the assumption that all processes see the same gradient in all iterations. So, if you do partial sync, you might also need to manually broadcast model to bring all processes back to sync, otherwise the result might be numerically incorrect.</p>
How to generate following tensor?

thyr: <p>Whats the most efficient way to generate a tensor like below?</p>
<p>[[0, 0],<br>
[0, 1],<br>
[0, 2],<br>
…<br>
[0, X],<br>
[1, 0],<br>
…<br>
[1, X],<br>
…<br>
…<br>
[X, X]]</p>

shartzog: <p>Not at my computer to test, but maybe something like this?</p>
<pre><code class="lang-auto">tens_list = list()
for x in range(10):
    for y in range(100):
        tens_list.append((x, y))
tens = torch.tensor(tens_list, [data types if needed])
</code></pre>

thyr: <p>I was trying to avoid for-loops if possible.</p>

shartzog: <p>For aesthetic or functional reasons?  If aesthetics, the following is equivalent to the for loop implementation above:</p>
<pre><code class="lang-auto">tens_list = [(x, y) for x in range(10) for y in range(100)]
</code></pre>
<p>If functional, I’m not sure what the fastest method would be.  I’d guess that creating a “template” object with something like the above and then generating a new tensor from it as needed elsewhere in your code would be a contender for “fastest”.  Unfortunately, I’m pretty new at all this stuff, so I could be way off… <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

thyr: <p>I mean I was thinking if I can use some builtin pytorch functions rather than looping myself. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

user_123454321: <p>Maybe something like this ?</p>
<pre><code class="lang-auto">l = torch.arange(X+1)
r = torch.arange(X+1)

l = l.view(X+1, 1).repeat(1, X+1).view((X+1) * (X+1))
r = r.view(1, X+1).repeat(X+1, 1).view((X+1) * (X+1))

m = torch.stack([l, r], 1)
</code></pre>
Can't overwrite __len__() in Dataset

ebelle: <p>I’ve created a LazyDataset but I’d like to skip the first line of the file, so I’m shortening the <strong>len</strong>() by 1 (and then adding 1 to the index in the __get__item(). However, I can’t overwrite the <strong>len</strong>. I’ve even tried:</p>
<p>def <strong>len</strong>(self):<br>
return 12</p>
<p>and it’s not working. Full code:</p>
<p>class LazyDataset(Dataset):</p>
<pre><code>def __init__(self, filepath, source_vocab, target_vocab, task):
    self.source_vocab = source_vocab
    self.text_init = self.source_vocab.init_token
    self.text_eos = self.source_vocab.eos_token
    self.target_vocab = target_vocab
    self.target_init = self.target_vocab.init_token
    self.target_eos = self.target_vocab.eos_token
    self.filepath = filepath
    # get total file length
    self._total_data = sum(1 for _ in open(self.filepath, "r"))
    self.task = task
    
def __len__(self):
    "Denotes the total number of samples"
    # subtract 1 from the file len to skip header in getitem
     return self._total_data - 1

def tokens_to_idx(self, text, target):
    # TODO: add arguments to make init &amp; eos optional
    # add init and eos tokens
    if self.task == "translation":
        text = [self.text_init] + text + [self.text_eos]
        target = [self.target_init] + target + [self.target_eos]
    # tokens to indices
    text = [self.source_vocab.vocab.stoi[t] for t in text]
    target = [self.target_vocab.vocab.stoi[t] for t in target]

    return text, target

def __getitem__(self, index):
    "Generates one sample of data"
    # normally you need +1 since linecache indexes from 1
    # here, we skip the header by adding +2 instead of +1
    print(index)
    line = linecache.getline(self.filepath, index+2)
    text, target = line.split("\t")
    # string to list, tokenizing on white space
    text, target = text.split(), target.split()
    text, target = self.tokens_to_idx(text, target)
    text_lens = len(text)
    return text, target, text_lens</code></pre>

user_123454321: <p>What is the error you get ? And shouldn’t <code>__len__</code> return the value like this…</p>
<pre><code class="lang-auto">def __len__(self):
    "Denotes the total number of samples"
    # subtract 1 from the file len to skip header in getitem
     return self._total_data - 1
</code></pre>

ebelle: <p>Yeah, sorry I edited that last minute. I had it as return 12 so I quickly edited it and missed the return. I’ll fix it now.</p>
<p>The error is that it’s pulling from an empty line (the line beyond the last line of text in the file). I want the max index that goes in to be N-1 and the max index it’s getting is still N. But even if I set it to “return 12”, it’s still returning 23,037. If I overload it to “return 12” it should return 12, no?</p>

user_123454321: <p>It returns 23,037 even if you return 12 ? Weird. Can you show how you are instantiating the dataset and getting the length ?</p>

ebelle: <p>I figured it out! It’s the batch loader I’m using. Thank you so much for your time. I’m sorry for asking a question where you wouldn’t even be able to see the answer. Have a great day!!!</p>
Actor network loss in Proximal Policy Optimization

Kimonili: <p>Hi everyone,</p>
<p>I have implemented a PPO algorithm where the actor and the critic are completely different networks. During training there are some times where the actor loss is getting negative. Loss is considered “optimal” when it is equal to 0, meaning that the network could not do better than that. How negative actor loss can be explained logically and if its not weird, should my goal be to get the most negative loss possible and not just 0?</p>
<p>As for the critic loss, the minimum loss is always &gt;= 0.</p>
<p>Thank you in advance!</p>
Converting torchscript models to version 1 instead of version 3?

joeyballentine: <p>Amazon Elastic Inference supports PyTorch and currently uses a modified version of 1.3.1. However, the JIT it uses seems to be a really old version as it says it requires version 1 torchscript models. The models that I’m able to create locally are version 3, and it seems to be that way no matter what version of PyTorch I use. When trying to script() or trace() the models on AWS, it seems to not actually work.</p>
<p>Ideally, I’d be able to convert the models I have locally to a version 1 torchscript models ao I could copy them to my AWS server and run inference with them. Does anybody know how to do that?</p>
In a torch.autograd.Function class, the model loaded with torch.load cannot get a result with (requires_grad = True)

Lusica: <p>Hi,</p>
<p>When I run the codes, I found the output of the model get a False requires_grad.</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.autograd import Function

class RouteNet(torch.nn.Module):
    def __init__(self):
        super(RouteNet, self).__init__()
        self.encode = torch.nn.Sequential(
            torch.nn.Conv2d(3, 32, 9, 1, 4),
            torch.nn.MaxPool2d(2, 2, 0, 1),
            torch.nn.Conv2d(32, 64, 7, 1, 3),
            torch.nn.MaxPool2d(2, 2, 0, 1),
            torch.nn.Conv2d(64, 32, 9, 1, 4))
        self.decode = torch.nn.Sequential(
            torch.nn.Conv2d(32, 32, 7, 1, 3),
            torch.nn.ConvTranspose2d(32, 16, 9, 2, 4, 1),
            torch.nn.Conv2d(16, 16, 5, 1, 2),
            torch.nn.ConvTranspose2d(16, 4, 5, 2, 2, 1),
            torch.nn.Conv2d(4, 1, 3, 1, 1))
 
    def forward(self, x):
        encode_out = self.encode(x)
        # res = conv_out.view(conv_out.size(0), -1)
        out = self.decode(encode_out)
        return out

class CongestionComputeFunction(Function):

    @staticmethod
    def forward(ctx):
        net = RouteNet()
        net.load_state_dict(torch.load("./net_H_epoch_39_batch_15_param.pkl"), strict=False)
        inp = torch.rand(2, 3, 100, 100, requires_grad=True)
        out = net(inp)
        print(out.requires_grad)    #False
    
    @staticmethod
    def backward(ctx):
        return None


class CongestionCompute(nn.Module):
    def __init__(self):
        super(CongestionCompute, self).__init__()

    def forward(self):
        return CongestionComputeFunction.apply()

if __name__=='__main__':
    congestion = CongestionCompute()
    congestion.forward()

</code></pre>
<p>I want to get the grad in this model.</p>
<p>Looking forward to your reply.</p>

albanD: <p>Duplicate of <a href="https://discuss.pytorch.org/t/grad-is-none-even-when-i-set-requires-grad-true/93106/10" class="inline-onebox">Grad is None even when I set requires_grad=True</a><br>
Answered there.</p>
Using GPUS in linux virtual machine

msilvaferreira: <p>Hello,</p>
<p>I was trying to use dataParallel on windows but it doesn’t work. I got the error: Pytorch is not compiled with NCCL support</p>
<p>I think the reason for this is that is not supported by windows.</p>
<p>So I try to run the script in ubuntu windows application, however GPUS are not detected.</p>
<p>I type nvidia-smi on the ubuntu command window and it says that it is not able to communicate with NVIDIA driver.</p>
<p>I installed nvidia-utils-440 and nvidia-driver-440 on the ubuntu virtual machine but still it gives the same error.</p>
<p>Everyone knows why?</p>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>You will need to set up your virtual machine to have full access to the GPU so that the linux machine can use it.<br>
I am not sure you can actually do that all the time, you will need to check the configuration of your virtual machine to set it up properly.</p>
How to transition to functions not being allowed to have member variables

pytorcher: <p>I used to have a torch.autograd.Function that had self variables that could be modified both within the forward and backward calls, and outside the function and the values would carry over correctly.  Now I instead have created a new class that just stores these values.  I can pass them all into the forward function, save them with  ctx.save_for_backward, and use them in backward.  However it looks like if I change them outside the function (especially between forward and backward) the values seen in backward don’t change and if I change them within backward the values outside backward don’t change.  Is there a recommended method for something like this now that torch.autograd.Function are meant to be static?</p>

albanD: <p>Hi,</p>
<p>You only want to save either inputs or outputs with <code>save_for_backward</code>. And they are saved in a slightly complex way.<br>
If you just want to have a state to pass around, you can give a dictionnary to your forward:</p>
<pre><code class="lang-auto">class YourFn(Function):
  @staticmethod
    def forward(ctx, arg1, arg2, my_state):
      # This assumes that my_state is NOT a Tensor
      # If it is, you have to use ctx.save_for_backward()
      # or you will see a memory leak
      ctx.my_state = my_state
      # compute the output
      return output

  @staticmethod
  def backward(ctx, grad_output):
    my_state = ctx.my_state
    # compute grad1, grad2
    return grad1, grad2, None

fn_state = {}
output = YourFn.apply(arg1, arg2, fn_state)
</code></pre>

pytorcher: <p>Looks like that works!  Any additional input for how to make a dictionary work in conjunction with dataParallel to use multiple GPUs?  I’m now getting “RuntimeError: The size of tensor a (8) must match the size of tensor b (16) at non-singleton dimension 0” Which I’m guessing is because some values are being split across GPUs and others are not.</p>
<p>asked in a new thread here: <a href="https://discuss.pytorch.org/t/resetting-dataparallel-after-initialization-and-using-it-in-a-dictionary/62660" class="inline-onebox">Resetting Dataparallel after initialization, and using it in a dictionary</a></p>

pytorcher: <p>Googling to try to solve my problem (<a href="https://discuss.pytorch.org/t/memory-leaks-from-custom-function/93028" class="inline-onebox">Memory leaks from custom function</a>) I came across this, which is how i was doing it originally which was working haha. Did this answer not take memory leaks into account or has something changed since last year?  Or upon further research was this answer assuming the state was normal variables and not other tensors?</p>

albanD: <p>Yes, from the question I guess I assumed that the extra arguments were not Tensor. And so that wouldn’t be a problem.<br>
Let me edit the answer to avoid any confusion in the future <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Loss for retrieval problem

Seo: <p>Hello everyone. I am working on retrieval problem where I need to retrieve a key response K from a pool of 100 candidates based on a given query Q. The output of my network is a tensor of scores that represents the matching between Q and Ki, for each Ki in the pool.<br>
My ground truth is a tensor filled with 0’s except for a single entry that contains 1 representing the correct K to retrieve. (e.g., [1, 0, 0, 0, …, 0] means K1 is the correct response to be retrieved).</p>
<p>To compute the loss I am using the BCE but I was wondering since I have the constraints of always having at least a 1 in my results (so the configuration with all 0’s is not accepted), is there any better loss for this type of task?<br>
I ask you this because the final tensor will be a list of 100 scores and 99 of them will be forced to be 0, so the model will try to predict all 0’s to minimize the loss. Is <code>pos_weight</code> the only thing I can try?</p>
<p>Thank you</p>
Question about how many convolution layer should there be

Archy_dragon: <p>Hi just wonder are there formula to determined how many convolution layer should there be for CNN im not sure if i put too much that will be over fit?</p>

BaruchG: <p>I’m not sure that anyone can really answer that for you, there are all types of architectures with all types of use cases, an no formula that will give you an exact answer as to how much you should use. With that said, you should probably look at the commonly used networks which are available for fine-tuning in pytorch to get a feel for what is standard.  You can find them at<br>
<a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html</a> and you can import them as shown in that page in the docs.  You can then just <code>print</code> out the model and it will print out it’s architecture.</p>

Archy_dragon: <p>thank you so much will check it out</p>
First time deployment questions about TorchScript, C++, and prototyping

RyanPatterson: <p>So I finally got my model working well enough to consider moving to a prototype and realized it isn’t as easy as I had initially thought. I just want to make sure I understand the steps of the process before I put in a lot of time learning TorchScript and C++. The model is a binary(possibly more classes later) semantic segmentation of cracked roads. Also I know OpenCV has C++ capabilities but are there similar libraries like numpy, pandas. Any recommendations on hardware to use?<br>
Steps</p>
<ol>
<li>Use TorchScript to map/compile just the Pytorch model to C++</li>
<li>Then use CodeBlocks to write a C++ program that will call the model</li>
<li>Use OpenCV and other addons in program for video filtering and matrix operations</li>
<li>Install the program on a hardware device. Any recommendations?</li>
</ol>
Sampling function for UV space to vertex space conversion

ForrestPi: <p>I am looking for a Differentiable Renderer library with  sampling function for UV space to vertex space conversion</p>

ForrestPi: <p><a href="https://gist.github.com/peteflorence/a1da2c759ca1ac2b74af9a83f69ce20e" rel="nofollow noopener">https://gist.github.com/peteflorence/a1da2c759ca1ac2b74af9a83f69ce20e</a><br>
I got it !</p>
`torch.cuda.is_available()` returns `False`, but `torch.version.cuda` returns `10.2

navid_mahmoudian: <p>Hello,<br>
I am trying to install Pytorch 1.6.0 via <strong>Pip</strong> package (<code>pip install torch torchvision</code>) on a computing grid in which I don’t have permission to install any software by myself. The <code>torch.cuda.is_available()</code> returns <code>False</code>, but <code>torch.version.cuda</code> returns <code>10.2</code>.<br>
Here is the output of <code>nvcc --version</code> :</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
</code></pre>
<p>, and this is the output of <code>nvidia-smi</code>:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:3B:00.0 Off |                    0 |
| N/A   23C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>How can I fix the problem?</p>

navid_mahmoudian: <p>I should add that other versions of CUDA are available on the computing grid, including CUDA 10.0 and CUDA 9.0, but unfortunately according to PyTorch installation none of these CUDA versions are appropriate to install Pytorch via Pip (currently PyTorch with CUDA 9.2, 10.1, and 10.2 can be installed via Pip). Is there any way to solve this problem?</p>

ptrblck: <p><code>torch.version.cuda</code> indicates which CUDA runtime was shipped in the binaries (or used to build PyTorch from source). It does not tell you, if CUDA can be used as seen by the output of <code>torch.cuda.is_available()</code>.</p>
<p>For CUDA10.2.89 you would need an NVIDIA driver &gt;= 440.33 as seen <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">here</a>.</p>
<p>You could update the driver or, if that’s not possible, use the binaries with CUDA9.2, which should work using your driver.</p>

navid_mahmoudian: <p>Thank you for the answer.</p>
Docker torch.cuda.is_avaiable returns false and nvidia-smi is not working

aslu98: <p>I’m trying to build a docker image that can run using GPUS, this my situation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29bac96b31c1aec0fee727608be2d53caf15bd91.png" data-download-href="https://discuss.pytorch.org/uploads/default/29bac96b31c1aec0fee727608be2d53caf15bd91" title="Schermata 2020-08-09 alle 12.08.24"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29bac96b31c1aec0fee727608be2d53caf15bd91.png" alt="Schermata 2020-08-09 alle 12.08.24" data-base62-sha1="5X9Lrdl1nqLMu4whrg6JxlJyRk5" width="493" height="500" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermata 2020-08-09 alle 12.08.24</span><span class="informations">607×615 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have python 3.6 and I am starting from image nvidia/cuda:10.0-cudnn7-devel.<br>
Torch does not see my GPUs.</p>
<p>nvidia-smi is not working to, returning error:</p>
<blockquote>
<p>Failed to initialize NVML: Unknown Error<br>
The command ‘/bin/sh -c nvidia-smi’ returned a non-zero code: 255</p>
</blockquote>
<p>I installed nvidia toolkit and nvidia-smi with</p>
<blockquote>
<p>RUN apt install nvidia-cuda-toolkit -y<br>
RUN apt-get install nvidia-utils-410 -y</p>
</blockquote>

ptrblck: <p>How did you execute the container?<br>
Note that you would need the nvidia docker runtime to be able to use GPUs inside the container.<br>
Older <code>docker</code> versions used:</p>
<pre><code class="lang-python">nvidia-docker run container
</code></pre>
<p>while newer ones can be started via:</p>
<pre><code class="lang-python">docker run --gpus all container
</code></pre>

aslu98: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="92156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>docker run --gpus all container</code></p>
</blockquote>
</aside>
<p>I used this command.</p>
<p>I solved my problem and forgot to take a look at this question, the problem was that it is not possible to check the avaiability of the GPUs during building of an image as I was doing in the image above.<br>
Once I tried to start the image and use command torch.cuda.is_avaiable() it returned True and I figured out the problem was not the installed packages but the way I was checking if they were working.</p>
Image Normalization for pretrained network

terro: <p>Hi all,<br>
I’m currently using alexnet pretrained network for my experiments. I need to preprocess the images performing normalization. I have images with values in the range [0,255]. I have divided by 255 and I have performed normalization using <code>normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])</code>. However after the normalization the values are not in the range [0,1]. Should I compute for each image the mean and standard deviation and then subtracting the mean and divide by standard deviation?</p>
<p>thanks for your help</p>

RaLo4: <aside class="quote no-group" data-username="terro" data-post="1" data-topic="93196">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/terro/48/12375_2.png" class="avatar"> terro:</div>
<blockquote>
<p>However after the normalization the values are not in the range [0,1].</p>
</blockquote>
</aside>
<p>This is normal.</p>
<aside class="quote no-group" data-username="terro" data-post="1" data-topic="93196">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/terro/48/12375_2.png" class="avatar"> terro:</div>
<blockquote>
<p>Should I compute for each image the mean and standard deviation and then subtracting the mean and divide by standard deviation?</p>
</blockquote>
</aside>
<p>This is exactly what <code>torchvision.transforms.Normalize</code> does.</p>
<p>for example one popular thing to do would be, using <code>torchvision.transforms.ToTensor</code> to make your image of range [0,255] to a tensor of range [0,1] and then use <code>torchvision.transforms.Normalize</code> with mean and std of 0.5 over all channels. Since (0 - 0.5) / 0.5 = -1 and (1 - 0.5) / 0.5 =1, this would normalize your images in a range of [-1,1].</p>
<p>Since you are using a pretrained model you can also use the mean and std the data was normalized with when being trained. Since you are probably using alexnet pertained on imagenet, that would be the mean and std of the imagenet dataset.<br>
Which is the one in your code: <code>mean = [0.485, 0.456, 0.406]</code>, <code>stds = [0.229, 0.224, 0.225]</code></p>
Hi, i have some problems related to concatenation. Please help

Blizzard_boi2020: <p>Actually I have a 3D dataset, with each 3D image has got 2 GT labels. So, when I am creating a list of both folders, the number of list items is changed. I have got 6 3D images, whereas for the GTs they are 8. There may be 2 extra for one set of varying shapes?</p>
<p>Will padding will work? like this in TF we have</p>
<pre><code class="lang-auto">Data=tf.data.Dataset.from_tensor_slices([image_paths],[label_paths]) #hold true with equal list size, but what should be done if image, and label list sizes are different?

</code></pre>

ptrblck: <p>Could you post a code snippet for the desired concatenation and explain what exactly is failing?</p>

Blizzard_boi2020: <p>I solved it! Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Image normalization

terro: <p>I’m following the tutorial <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a> and I did not know how to perform normalization of images that has values 0-255. I should create a custom Normalization transformation to normalize images for using to train the pretrained alexnet network.</p>

mariosasko: <p>Usually, the images are scaled to the [0, 1] interval first (<code>images = images / 255)</code>. Then, to normalize them, you can use <code>torchvision</code>'s <code>transforms.Normalize</code> and this is a <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6">link</a> to the post where it’s explained how to compute the mean and std of the images.</p>

RaLo4: <p>using <code>torchvision.transforms.ToTensor</code> you can:</p>
<blockquote>
<p>Convert a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0] if the PIL Image belongs to one of the modes (L, LA, P, I, F, RGB, YCbCr, RGBA, CMYK, 1) or if the numpy.ndarray has dtype = np.uint8</p>
</blockquote>
<p>to normalize this you can then use torchvision’s <code>transforms.Normalize</code> as mentioned by <a class="mention" href="/u/mariosasko">@mariosasko</a>.<br>
This lets you normalize your tensor using mean and standard deviation. The formular is image = (image - mean) / std.<br>
Popular would be using 0.5 for all channels of both mean and std, since this would normalize your tensors in a range between -1 and 1 ( (0 - 0.5) / 0.5 = -1 and (1 - 0.5) / 0.5 = 1).</p>
Torch save without explicitly defining the model class

Aryaman_Sriram: <p>So I downloaded a trained model’s weights and loaded it using torch.load(). But I am unable to directly run inference on it since it says it is unable to reference the class of the model. Keep in mind I am not loading the state dict of the model but the whole model itself. Why do I still need to explicitly define the model class name? Just curious</p>

ptrblck: <p>I think this is a limitation of the Python <code>pickle</code> module e.g. as described <a href="https://wiki.python.org/moin/UsingPickle">here</a> at the bottom of the page.</p>
Converting custom Detectron2 model with torchserve?

r5sb: <p>Hi everyone,</p>
<p>I have trained a model in Detectron2 using the <a href="https://github.com/facebookresearch/detectron2/blob/master/configs/Misc/cascade_mask_rcnn_R_50_FPN_1x.yaml" rel="nofollow noopener">cascade_mask_rcnn_R_50_FPN_1x.yaml</a> config file. From my understanding, the model_final.pth file saved after training is a state_dict? I want to use torchserve to serve this model as a REST endpoint.</p>
<p>The first step to that is running <code> torch-model-archiver</code> which creates a .mar file from the saved model file (model_final.pth). In eager mode, this also needs a model.py file as an argument which contains the model architecture. I am not sure what to pass here since it was trained using Detectron2 config file and no explicit model.py file with the defined architecture.</p>
<p>Another alternative is to pass a torchscript version of the model instead of the model.py file. I am not sure how to do this for the Detectron2 model (model_final.pth). Can I just call torch.jit.script() after loading the model state_dict?</p>
<p>If anyone has any pointers or has been able to use torchserve with a Detectron2 trained model please let me know! Thank you!</p>

tkaleczyc: <p>Bump - does anybody know the answer to this?</p>

Nakul_Joshi: <p>torch-model-archiver --model-name xyz --version 1.0 --serialized-file model.pth --handler handler.py</p>
<p>This is how you can convert a detectron model into a mar file.</p>
<p>NOTE: In your handler.py load config with model_zoo</p>
Difference between Cuda:0 vs Cuda with 1 GPU

n4tman: <p>If I only have one gpu does doing either of the below mean that the same gpu will be used?</p>
<pre><code class="lang-auto">device = torch.device('cuda:0')

device = torch.device('cuda')
</code></pre>
<p>Thanks!</p>

LeviViana: <p>Yes ! The device index is only useful when you have multiple gpus.</p>

n4tman: <p>Thanks and if I do torch.device(‘cuda’) with multiple GPUs this is the same as doing torch.device(‘cuda:0’)?</p>

PiedPiper: <p>I think so. By default,  torch.device(‘cuda’) refers to GPU index 0. Similarly, tensor.cuda()  and model.cuda() move the tensor/model to “cuda: 0” by default if not specified.</p>

n4tman: <p>Right, so by default doing torch.device(‘cuda’) will give the same result as torch.device(‘cuda:0’) regardless of how many GPUs I have?</p>

PiedPiper: <p>Yes and if you want using multi GPUs you shall check with model.DataParallel() and model.to(device).</p>

n4tman: <p>And assuming I have at least 1 GPU. By default it doesn’t matter if I use  <code>device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')</code>  or  <code>device=torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')</code>  the same GPU will be used regardless of how many GPUs I have?</p>
How to maintain zero for the zero vectors in Linear layer

ppgxs: <p>For example, my data is like this:</p>
<pre><code class="lang-python">tensor_1 = torch.tensor([
    [1,2,3], 
    [2,3,4], 
    [0,0,0]
])
</code></pre>
<p>I defined a linear layer like:</p>
<pre><code class="lang-python">layer = torch.nn.Linear(3,10)
output = layer(a.float())
</code></pre>
<p>and the output would be like:</p>
<pre><code class="lang-auto">tensor([[-1.6636, -2.1729, -1.6064,  0.4921,  0.5485, -0.5628,  0.8863,  1.1453,
          0.6818, -0.2417],
        [-2.6192, -3.2155, -1.9747,  0.6968,  0.9965, -0.5925,  0.7474,  1.9600,
          0.8981, -0.3384],
        [ 0.4313, -0.0326, -0.2426,  0.1669, -0.4219, -0.1127,  0.3835, -0.2567,
          0.4994, -0.0233]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>Is there a way to maintain zeros for output in the last raw?<br>
I can imagine a “mask” method that “mask_filled” the output. Is there another way?</p>

ptrblck: <p>You could remove the <code>bias</code> from the linear layer or alternatively multiply the output with a mask.<br>
What is you use case that you would like to pass a sample with all zeros and expect an all-zero output?</p>

corypaik: <p>Forgive me if I didn’t understand you correctly, but here is one option if you’re trying to maintain an all-zero output.</p>
<pre><code class="lang-python">output *= (tensor_1.sum(dim=1) &gt; 0).view(-1, 1)
</code></pre>

ppgxs: <p>Thanks for replying. I’m using a <a href="https://github.com/airsplay/py-bottom-up-attention" rel="nofollow noopener">bottom-up feature</a> which may lead to a situation that no bounding box is detected in one single image sometimes, and as for the images that have zero bounding box, I choose zero features to replace them. So of course for the zero features, I want to maintain zero in the whole model.</p>

ppgxs: <p>Thanks for replying. This is a workable solution.</p>
Simultaneous reads of the same torchvision.datasets object

kkonstantinidis: <p>Consider the following piece of code to fetch a data set for training from <code>torchvision.datasets</code> and to create a <code>DataLoader</code> for it.</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms

training_set_mnist = datasets.MNIST('./mnist_data', train=True, download=True)
train_loader_mnist = torch.utils.data.DataLoader(training_set_mnist, batch_size=128,
                                                 shuffle=True)
</code></pre>
<p>Assume that several Python processes have access to the folder <code>./mnist_data</code> and execute the above piece of code simultaneously; in my case, each process is a different machine on a cluster and the data set is stored in an NFS location accessible by everyone. You may also assume that the data is already downloaded in this folder so <code>download=True</code> should have no effect. Moreover, each process may use a different seed, as set by <code>torch.manual_seed()</code>.</p>
<p>I would like to know whether this scenario is allowed in PyTorch. My main concern is whether the above code can change the data folders or files in <code>./mnist_data</code> such that if ran by multiple processes it can potentially lead to unexpected behavior or other issues. Also, given that <code>shuffle=True</code> I would expect that if 2 or more processes try to create the <code>DataLoader</code> each of them will get a different shuffling of the data assuming that the seeds are different. Is this true?</p>

ptrblck: <p>Since you are only reading the files, there shouldn’t be any interactions between the processes.<br>
Yes, different seeds should result in a different shuffling.</p>
Running BatchNorm: "lerp_cuda" not implemented for 'Half'

Martin_cb: <p>I am trying to implement <a href="https://www.google.com/search?rlz=1C1CHBF_en-GBGB773GB773&amp;sxsrf=ALeKk026TNsRAWAAzkilJUSd3xHxhiUhrQ:1597527828867&amp;q=Running+BatchNorm+generalized&amp;spell=1&amp;sa=X&amp;ved=2ahUKEwjgq4j2lp7rAhURQUEAHYZWD8AQBSgAegQIDhAo" rel="nofollow noopener"> <em><strong>Running BatchNorm</strong></em> </a> from here:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://forums.fast.ai/uploads/default/optimized/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5_2_32x32.png" class="site-icon" width="16" height="16">
      <a href="https://forums.fast.ai/t/simplified-runningbatchnorm-in-lesson-11/56340" target="_blank" rel="nofollow noopener" title="07:23PM - 11 October 2019">Deep Learning Course Forums – 11 Oct 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:108/85;"><img src="https://forums.fast.ai/uploads/default/optimized/3X/f/b/fb46e1b024a8a9934b8f6e1550982018c46d2d46_2_1024x808.jpeg" class="thumbnail" width="108" height="85"></div>

<h3><a href="https://forums.fast.ai/t/simplified-runningbatchnorm-in-lesson-11/56340" target="_blank" rel="nofollow noopener">Simplified RunningBatchNorm in Lesson 11</a></h3>

<p>At the beginning of Lesson 11, at 2:18 Jeremy shows a simplified (finalized?) RunningBatchNorm class in an updated version  of the 07_batchnorm.ipynb notebook. See the screenshot below. I couldn’t find this notebook in the fastai repo.  How can we...</p>

  <p><span class="label1">Reading time: 1 mins 🕑</span>
    <span class="label2">Likes: 1 ❤</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>However, as I am on FP16, I am getting the following error:<br>
<code>"lerp_cuda" not implemented for 'Half'</code></p>
<ul>
<li>Any suggestions how to get around this?</li>
</ul>

ptrblck: <p><code>lerp</code> is currently only dispatched for <code>float</code> and <code>double</code> as seen <a href="https://github.com/pytorch/pytorch/blob/ba98c0e38ca76a8e03d27e20a1d2bfc324d502ca/aten/src/ATen/native/cuda/Lerp.cu#L19">here</a>. I don’t know, if <code>half</code> would not yield stable results or if there are other limitations.<br>
Maybe <a class="mention" href="/u/ngimel">@ngimel</a> knows the reason.</p>
<p>As a workaround, you could transform your data back to a <code>FloatTensor</code>, apply <code>lerp</code>, and transform it back, if you are using a manual approach. If you are using <code>torch.cuda.amp</code>, you could disable <code>autocast</code> for this particular operation.</p>
On the clearing of tensorboard history + running of tensorboard after killing it

brenth: <p>Hi all,</p>
<p>2 questions from me regarding tensorboard:<br>
1)<br>
I previously got tensorboard to run but after killing it via the windows command prompt, am unable to restart it again. How can I restart it successfully without running into ’ localhost refused to connect./<br>
2)<br>
Also, would like to delete history on tensorboard. I understand from some forums the directory has to be deleted + process killed. Not too sure how the former works, would appreciate any advice on this.<br>
Where can I find the directory? Within the tensorboard package in windows explorer?</p>
<p>Thanks</p>

brenth: <p>An update for anyone who might find themselves with the same problem:</p>
<p>I decided to end my misery over tensorboard and have switched allegiance to visdom - it’s been an absolute breeze.</p>
<p>Linking a very useful github which helped me quickly set up visdom for the plotting of accuracy and loss graphs:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/noagarcia/visdom-tutorial" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/32582066?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/noagarcia/visdom-tutorial" target="_blank" rel="nofollow noopener">noagarcia/visdom-tutorial</a></h3>

<p>A simple tutorial to use Visdom to plot your PyTorch training graphs - noagarcia/visdom-tutorial</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Runtime error when testing out torchvision C++ frontend

yshi: <p>So I was following <a href="https://github.com/pytorch/vision/tree/master/examples/cpp/hello_world" rel="nofollow noopener">https://github.com/pytorch/vision/tree/master/examples/cpp/hello_world</a> to test out the C++ frontend for torchvision. Compilation was successful with cmake.</p>
<p>But when I run <code>./hello-world</code> I got a <code>error while loading shared libraries: libc10_cuda.so: cannot open shared object file: No such file or directory</code>, and I checked I do have <code>libc10_cuda.so</code> under <code>path/to/libtorch/lib</code>.  I’m on <code>ubuntu 20.04</code> with <code>torch 1.6</code>, <code>cuda 10.2</code> and <code>gcc 9.3</code>.</p>
<p>What would be a likely cause for this error?</p>

yshi: <p>solved by <code>export LD_LIBRARY_PATH=absolute/path/to/libtorch/lib</code></p>
How to request data for the whole batch instead of each sample?

BadTimeManagement: <p>As the title, is it possible to request data in batches instead of request sample 1 by 1?</p>

ptrblck: <p>If you are using a <code>Dataset</code> and <code>DataLoader</code> (as explained in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>), you can set the <code>batch_size</code> to the desired value.<br>
On the other hand, if you are directly working with tensors, you could slice the tensors directly to create the batch.</p>
Values after each epoch not getting stored

alameer: <p>The code below calculates the MSE and MAE values but I have an issue where the values for MAE and MSE don’t get stored at each epoch in store_MAE and store MSE. It seems to use the values of the last epoch only. Any idea what i need to do in the code to save the total values for each epoch i hope this make since</p>
<pre><code class="lang-auto">       # logging:
        loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)
        
        global_step += 1
        store_MAE = 0
        store_MSE = 0
        mean_test_error = mean_test_error / len(loader_test)
        store_MAE += mean_test_error
        mse = math.sqrt(loss / len(loader_test))
        store_MSE +=mse
        print("Test count error: %f" % mean_test_error)
        print("MSE: %f" % mse)

    if mean_test_error &lt; best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)

print("MAE Total: %f" % store_MAE)
print("MSE Total: %f" % store_MSE)
model_mae= store_MAE /epoch
model_mse= store_MSE /epoch
print("Model MAE: %f" % model_mae)
print("Model MSE: %f" % model_mse)</code></pre>

ptrblck: <p>It seems you are reinitializing <code>store_MAE</code> and <code>store_MSE</code> inside the logging loop?<br>
Could you move this code</p>
<pre><code class="lang-python">        store_MAE = 0
        store_MSE = 0
</code></pre>
<p>before executing the loop?</p>

alameer: <p>Many Thanks for your help!</p>
Ways to Create computational graph image?

jamesbond007: <p>I would like to know about the ways to create a computational graph image from PyTorch code?</p>
<p>One way is using the SummaryWriter in Tensorboard.<br>
Is there any other way to create this? Are any other packages/libraries available?</p>

ptrblck: <p><a href="https://github.com/szagoruyko/pytorchviz">torchviz</a> can also be used to visualize the computation graphs.</p>
How to train the fasterrcnn_resnet50_fpn in torchvision

a_d: <p>Hello all.<br>
I am importing the model as <code> frcnn = fasterrcnn_resnet50_fpn(pretrained=False,                                 num_classes=6,                                 pretrained_backbone=True,                                 progress=True)</code></p>
<p>In the docs it is written that I have to pass a list of images and targets which would return the cls_loss and regression_loss,<br>
I am sorry this might be a silly question, but we have to propagate the loss backward, or is the loss.backward() or optim.step() taken care of internally?</p>
<p>TIA</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">Object Detection Finetuning Tutorial</a> might be a good starter for your use case.<br>
In this tutorial the <a href="https://github.com/pytorch/vision/blob/27278ec8887a511bd7d6f1202d50b0da7537fc3d/references/detection/engine.py#L13-L55"><code>train_one_epoch</code></a> utility function is used, which takes care of the <code>backward</code> and update step.<br>
However, you could of course look at the linked source code and apply these steps manually.</p>
How to determined out_channels

Archy_dragon: <p>Hi I try to develop CNN and im not sure how to determine  out_channels for conv2d:</p>
<pre><code class="lang-auto">torch.nn.Conv2d(in_channels, out_channel, kernel,stride,Padding)
</code></pre>
<p>I know in_channel depend on you use colour or Gray scale, kernel, <em>stride</em> and padding is try and error but how does out_channel come from ? is it just guess? for example</p>
<p><code>self.conv1 = ConvBlock(3, 64, 3, 2, 1)</code></p>
<p>how does 64 come from</p>

mariosasko: <p><code>out_channels</code> represents the number of output channels or feature maps. Usually they are chosen by intuition or empirically (there are guides on how to do that on other sites, e.g. <a href="https://stats.stackexchange.com/questions/196646/what-is-the-significance-of-the-number-of-convolution-filters-in-a-convolutional" rel="nofollow noopener">link</a>).</p>