akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Keep model in memory between CLI calls?

ZimoNitrome: <p>Hi,<br>
I am using a pretty bodged method of saving images from a 3rd party application, calling my Python script for inference referencing those images, and then importing the result into the 3rd party application again. These calls are made through Javascript with a script that only lives for a few milliseconds. So each time I call the python script it needs to load my model into memory again.</p>
<p>Is there any way to keep a model in memory between my calls to inference script?</p>
<p>Maybe it could be possible to call one script that loads the model into memory and another one that performs inference (and can somehow access that model)?</p>

ZimoNitrome: <p>The best option seems to be to create a local webserver.</p>
AssertionError: Torch not compiled with CUDA enabled error

Gopikrishnan_R: <p>I’m not able to run pytorch with cuda even after installing with cuda enabled<br>
I used this command for installing<br>
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch<br>
nvidia-smi is showing GPU with cuda 10.2 version.<br>
I’ve tried uninstalling and installing again but no luck</p>

ptrblck: <p>Could you create a new virtual environment and reinstall PyTorch?<br>
The error message suggests that the currently found installation doesn’t ship with the CUDA runtime, so you might have multiple (old) installs in the current env.</p>

Gopikrishnan_R: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thankyou. That worked</p>
Applying Performance metric like Inception score or FID

Nawaf_Khan: <p>Hello, everyone, I hope you all will be good I need some help regarding apply the inception score and FID in this GitHub repository[<a href="https://github.com/odegeasslbc/Progressive-GAN-pytorch?fbclid=IwAR0s8K41vVc0SoXNj6LHo49dcEWPzrMuqwic1ae9ykG52SqSLgQ7PnBh7oc" rel="noopener nofollow ugc">https://github.com/odegeasslbc/Progressive-GAN-pytorch</a>] can someone help out it will be very helpful for me.</p>

ptrblck: <p>Where are you currently stuck and what have you implemented so far?</p>

Nawaf_Khan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  I have implemented this pggan code and I wanna apply performance metric to it can you help me with that how to that here is the link <a href="https://github.com/odegeasslbc/Progressive-GAN-pytorch?fbclid=IwAR0s8K41vVc0SoXNj6LHo49dcEWPzrMuqwic1ae9ykG52SqSLgQ7PnBh7oc" rel="noopener nofollow ugc">https://github.com/odegeasslbc/Progressive-GAN-pytorch?fbclid=IwAR0s8K41vVc0SoXNj6LHo49dcEWPzrMuqwic1ae9ykG52SqSLgQ7PnBh7oc</a></p>

ptrblck: <p>I would recommend to test some already implemented approaches such as <a href="https://github.com/mseitzer/pytorch-fid">pytorch-fid</a> and check if they would be working for your use case. Also, I think you might be able to find some implementations for these metrics in this forum. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Nawaf_Khan: <p>I tried a lot but I don’t know why it’s hard for me to implement these metrics by the way thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Manual arithmetic on embedding weights

PLarsen79: <p>I am looking for an efficient way of manual editing embedding weights, in between forwards-backprop-iterations.</p>
<p>Say I have three embeddings, example:</p>
<pre><code class="lang-auto">self.tok_embA = nn.Embedding(config.vocab_size, config.n_embd)
self.tok_embB = nn.Embedding(config.vocab_size, config.n_embd)
self.tok_embC = nn.Embedding(config.vocab_size, config.n_embd)
</code></pre>
<p>Then I train tok_embA as usual in the model, and perhaps tok_embB too. Using an idx array of tensor indexes.</p>
<p>But then I want to do manual arbitrary arithmetic on tok_embC, using the same tensor indexes, and the weights from the trained tok_embA and tok_embB. Like a simple loop through the current tensors for the current input prompt, but the arbitrary arithmetic comes with lots of conditional stuff so simple self.tok_embA.sub(self.tok_embB) etc over all tensors won’t help.</p>
<p>How is a preferable way to loop through the current idx embedded weigths, and perform manual arithmetic on them?</p>

ptrblck: <p>The easiest way would be to apply the loop and manipulate the parameter using all conditions.<br>
It’s hard to tell if and how these operations might avoid the loop without seeing the condition, but you might be able to use e.g. a mask.</p>

PLarsen79: <p>Thanks! I actually had the mask option in mind, and now I am more confident in that being the way forward. Now I need to somehow convert my logic into masked matrix-operations … but that’s another story <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Zero loss values for resnet segmentation

jihyeon: <p>I am training a resnet model for image segmentation (two classes, image and binary mask for training). While my accuracy values are very high (~0.99), both my training and validation losses are zeroes throughout all epochs. What might be the issue? I have attached a section of my code for training the model below, thank you in advance!</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    #zero out epoch based performance variables 
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    all_acc = {key: 0 for key in phases} 
    all_loss = {key: torch.zeros(0).to(device) for key in phases}
    cmatrix = {key: np.zeros((2,2)) for key in phases}

    for phase in phases: #iterate through both training and validation states

        if phase == 'train':
            model.train()  # Set model to training mode
        else: #when in eval mode, we don't want parameters to be updated
            model.eval()   # Set model to evaluate mode
        running_loss = 0.0
        running_corrects = 0
        for ii , (X, y, y_weight) in enumerate(dataLoader[phase]): #for each of the batches
            optim.zero_grad()
            X = X.to(device)  # [Nbatch, 3, H, W]
            y_weight = y_weight.type('torch.FloatTensor').to(device)
            y = y.type('torch.LongTensor').to(device)  # [Nbatch, H, W] with class indices (0, 1)
            with torch.set_grad_enabled(phase == 'train'): #dynamically set gradient computation, in case of validation, this isn't needed
                                                            #disabling is good practice and improves inference time

                prediction = model_ft(X)  # [N, Nclass]
                y = y[:, 0,0]
                loss = criterion(prediction, y)
                _, preds = torch.max(X, 1)
                preds = preds[:,0,0]
                if phase=="train": #in case we're in train mode, need to do back propogation
                    
                    loss.mean().backward()
                    optim.step()
                
                #all_loss[phase]=torch.cat((all_loss[phase],loss.detach().view(1,-1)))

                
        #all_acc[phase]=(cmatrix[phase]/cmatrix[phase].sum()).trace()
        #all_loss[phase] = all_loss[phase].cpu().numpy().mean()
            running_loss += loss.data[0]
            running_corrects += torch.sum(preds == y)
        epoch_loss = running_loss / len(dataLoader[phase].dataset)
        epoch_acc = running_corrects.double() / len(dataLoader[phase].dataset)

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))

        # deep copy the model
        if phase == 'val' and epoch_acc &gt; best_acc:
           best_acc = epoch_acc
           best_model_wts = copy.deepcopy(model.state_dict())
        if phase == 'val':
           val_acc_history.append(epoch_acc)

    print()

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

model.load_state_dict(best_model_wts)
</code></pre>
Calculating Eucledian Distance Between Two tensors

Bidhan: <p>How do we calculate Eucledian distance between two tensors of same size. The tensors have size of [1,1, 512,1]?</p>

tr_arun: <p>Reshape it to 1 d and then find the euclidean distance.</p>
<pre><code class="lang-auto">a = torch.randn(1, 1, 512, 1)

b = troch.randn(1, 1, 512, 1)

euclidena_dist = sum(((a - b)**2).reshape(512))
</code></pre>

Bidhan: <p>This worked for me. One more question:<br>
What if the size of the tensors are [3,1,512,1] and [1,1,512,1]?</p>

chetan_patil: <p>In this case, your (1,1,512,1) shaped Tensor will copy itself to match the target dimension is (3,1,512,1), a technique known as Broadcasting.<br>
You just need to write the code for Eucledian distance, Pytorch will perform Broadcasting inherently.<br>
Here’s the code</p>
<pre><code class="lang-auto">A= torch.randn(1,1,512,1).reshape(-1,512)
B = torch.randn(3,1,512,1).reshape(-1,512)

Distance = ((A-B)**2).sum(axis=0)

</code></pre>

sigma_x: <p>Another way:</p>
<p><code>torch.cdist(a,b)**2</code></p>
Downstream Task Evaluation for Siamese Nets

grizzlycoder: <p>Hello!</p>
<p>So I have a self supervised Siamese net for which I have saved the train and test feature vectors for each input. As input, I take two human tracks (so cropped bounding box régions from a video, and output their interaction label 1 or 0).</p>
<p>Now, I want to perform a downstream evaluation task for human interaction recognition. How should I go about this when I have a pair of inputs? I just want to create a MLP network that learns to classify the interaction classes in a fully supervised setting.</p>
<p>Thank you</p>
Split CNN feature map to patches

yoad.tewel: <p>Hey,<br>
So Im working on a computer vision model, and in a part of my code I’ve got 2 images, A and B:<br>
I want to run A and B through a cnn (lets say vgg16), get the feature map of both images from specific layer (<strong>Until now it’s simple to do</strong>).</p>
<p>Now that Iv’e got ft_A and ft_B (the feature maps of both images), I want to split each feature map to patches/blocks. <strong>For example</strong>:</p>
<p>If ft_A is 128x96x96 (CxWxH), I want to split it to patches of size 16x16, So I will have 6*6 patches of size 128x16x16, in a tensor of size 36x128x16x16 (num_PATCHESxCxW_PATCHxH_PATCH).</p>
<p>Now I will do some operation on each patch, for example take each patche’s gram matrix, which will turn each patch to 36x128x128.</p>
<p>Finally my loss function will be MSE between <code>gram_matrix_A</code> of size 36x128x128 and <code>gram_matrix_B</code> of size 36x128x128 .</p>
<p><strong>My question</strong><br>
How can I do that? Is there a fast wat to do these operations? Specifically, in tensorflow there is the function <a href="https://www.tensorflow.org/api_docs/python/tf/space_to_batch_nd" rel="noopener nofollow ugc">space_to_batch_nd</a> that turns the feature map to patches, is there such thing in pytorch, or a good way to implement it?</p>
<p>Thanks!</p>

InnovArul: <p>you could try with <code>torch.nn.functional.unfold</code> method.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html" target="_blank" rel="noopener">Unfold — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">import torch, torch.nn as nn, torch.nn.functional as F
x = torch.randn(1, 128, 96, 96).cuda() 
o = F.unfold(x, kernel_size=16, stride=16) # 1, 32768, 36 
o = o.view(1, 128, 16, 16, 36)  # 1, 128, 16, 16, 36
o = o.permute(0, 4, 1, 2, 3)  # 1, 36, 128, 16, 16
</code></pre>

yoad.tewel: <p><a class="mention" href="/u/innovarul">@InnovArul</a> I think it’s what I need, thank you!<br>
I will update if it works or if I face problems!</p>
Loss and Metric Clarification

moreshud: <p>From the implementation of the dice metric, I have seen repositories using a threshold - 0.5 on their metric with a Sigmoid final activation layer. e.g <a href="https://github.com/eisen-ai/eisen-core/blob/master/eisen/ops/metrics/dice.py" rel="noopener nofollow ugc">Eisen Framework Dice Metric</a> In case one is doing multiclass, do I need to use this threshold knowing fully well that that final activation layer is now Softmax?</p>
<p>Also, when training 2D Unet or general deep learning model, does lower training loss compared to validation necessarily need to give higher evaluation metric in training compared to validation? In addition, where do I have my best performing model that generalizes well on the testing dataset, where I have the lowest score (on training/validation) or the highest metric?</p>
<p>Finally, using a single model for two segmentation tasks with criteria which is the best or advisable approach for backward propagation:</p>
<pre><code class="lang-auto">final_loss = lossA + lossB
final_loss.backward()
</code></pre>
<p>or</p>
<pre><code class="lang-auto">lossA.backward(retrain_graph =True)
lossB.backward()
</code></pre>
<p>?</p>
<p>Thanks for your contributions in advance</p>
RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient

Sayak_Paul: <p>Hello.</p>
<p>I am trying to convert the <a href="https://arxiv.org/abs/2005.11129" rel="noopener nofollow ugc">Glow TTS model</a> from PyTorch to ONNX. I am running into the following issue during the export step -</p>
<pre><code class="lang-auto">/content/TTS_repo/TTS/tts/layers/glow_tts/glow.py in forward(self, x, x_mask, reverse, **kwargs)
    261                     self.weight) * (c / self.num_splits) * x_len  # [b]
    262 
--&gt; 263         weight = weight.view(self.num_splits, self.num_splits, 1, 1)
    264         z = F.conv2d(x, weight)
    265 

RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
Tensor:
-0.4500 -0.2462 -0.0506 -0.4333
-0.3748 -0.0181  0.0442  0.3575
-0.2864  0.7973 -0.0522 -0.1719
 0.1037  0.2156 -0.9611  0.2521
[ torch.FloatTensor{4,4} ]
</code></pre>
<p>Here’s my Colab Notebook for the full reproduction of the issue - <a href="https://colab.research.google.com/gist/sayakpaul/96f00d33385cbb2c97e1befedcf2e3cd/copy-of-glow_tts.ipynb" rel="noopener nofollow ugc">https://colab.research.google.com/gist/sayakpaul/96f00d33385cbb2c97e1befedcf2e3cd/copy-of-glow_tts.ipynb</a>. Any directions to solve this issue would be greatly helpful.</p>
Giving custom data input to a CNN model built using Pytorch framework

Sandeep_Kumar_GITAM: <p>Dear friends, I have a train data X_train which is a numpy array with dimension of (36000,15,15,30), where 36000 is number of image samples, 15 is height of image,15 is width of image and 30 is the number of channels in a image. y_train is the integer array of shape (36000,1) that contains the class labels of corresponding train images. I am new to PyTorch. Can any one please suggest me the procedure for giving this numpy arrays as input to the CNN model and train the model. I would also request to suggest me how to evaluate the model after the training process.</p>

ptrblck: <p>You can transform the numpy array to a PyTorch tensor via <code>tensor = torch.from_numpy(array)</code> and would need to permute it afterwards, as PyTorch layers expect the input in the channels-first memory layout via: <code>tensor = tensor.permute(0, 3, 1, 2)</code>.<br>
If you are dealing with a multi-class classification, your target should have the shape <code>[batch_size]</code> without any additional dimension, so you could use <code>target = torch.from_numpy(target_array).squeeze(1)</code> to remove <code>dim1</code>.<br>
Once this is done, you can directly pass the input data to the model or use a <code>TensorDataset</code> or a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>
<p>I would generally recommend to take a look at the <a href="https://pytorch.org/tutorials">tutorials</a> for a general introduction. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Dataloader can not be run in subprocess

Jingnan_Jia: <p>I want to have 2 independent dataloader running in different subprocess.</p>
<pre><code class="lang-auto">def inifite_generator(dataloader, dataloader2, keys = ("image", "label")):
    # dataloader2 = copy.deepcopy(dataloader)
    dataloader = iter(dataloader)
    dataloader2 = iter(dataloader2)
    d = next(dataloader)  # it works well because it is in the main process
    data_queue = []
    data_q = Queue(maxsize=10)
    data_q2 = Queue(maxsize=10)

    def start_dataloader(dl, q):
        while True:
            try:
                data = next(dl)
            except StopIteration:
                data = next(dl)
            x_pps = data[keys[0]]
            y_pps = data[keys[1]]
            for x, y in zip(x_pps, y_pps):
                x = x[None, ...]
                y = y[None, ...]
                data = (x, y)
                if q==1:
                    data_q.put(data, timeout=100)
                else:
                    data_q2.put(data, timeout=100)

    p1 = multiprocessing.Process(target=start_dataloader, args=(dataloader, 1, ))
    p2 = multiprocessing.Process(target=start_dataloader, args=(dataloader2, 2, ))
    p1.start()
    p2.start()

    use_q2 = False
    while True:
        if len(keys)==2:
            if data_q.empty() and data_q2.empty():
                continue
            else:
                if data_q.empty() or use_q2:
                    q = data_q2
                    use_q2 = True
                else:
                    q = data_q
                    use_q2 = False
            data = q.get(timeout=100)
            yield data
</code></pre>
<p>Error shows:</p>
<pre><code class="lang-auto">Process Process-13:
Traceback (most recent call last):
  File "/home/jjia/.conda/envs/py37/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/jjia/.conda/envs/py37/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jjia/data/monai/train_mtnet.py", line 308, in start_dataloader
    data = next(dl)
  File "/home/jjia/.conda/envs/py37/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/jjia/.conda/envs/py37/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1069, in _next_data
    idx, data = self._get_data()
  File "/home/jjia/.conda/envs/py37/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1030, in _get_data
    raise RuntimeError('Pin memory thread exited unexpectedly')
RuntimeError: Pin memory thread exited unexpectedly

</code></pre>

ptrblck: <p>Is this still an issue or would it be solved by <a href="https://discuss.pytorch.org/t/how-to-have-a-fluent-dataloader/106557/2">this answer</a> from your other thread?</p>
No option to change gpus when using torch.distributed.init_process_group

saeed: <p>When we run <a href="https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py" rel="noopener nofollow ugc">the NVIDIA code</a> using the following:</p>
<pre><code class="lang-auto">$ python -m torch.distributed.launch --nproc_per_node=n main_amp.py -a resnet50 --b 224 --workers 4 --opt-level O2 ./
</code></pre>
<p>as soon as <code>torch.distributed.init_process_group(backend='nccl', init_method='env://')</code> is executed, Pytorch spawns n processes on n gpus as the following argument makes it to do so <code>torch.distributed.launch --nproc_per_node=n</code>. All these processes started from the <code>0-th</code> index up to <code>n-1</code>. Unfortunately, there is no way to choose the indices of gpus starting them from anything else rather than zero or chose optional gpus.<br>
I have also tried to use the following:</p>
<pre><code class="lang-auto">$ CUDA_VISIBLE_DEVICES=4,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 main_amp.py -a resnet50 --b 224 --workers 4 --opt-level O2 ./
</code></pre>
<p>and I get <code>torch._C._cuda_setDevice(device) RuntimeError: CUDA error: invalid device ordinal</code>.</p>
<p>Can you please help me with this?</p>

ptrblck: <p>We recommend to use the native mixed-precision training via <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples"><code>torch.cuda.amp</code></a> as well as the native DDP implementation, as described <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">here</a>.</p>
'NoneType' object has no attribute 'data'

Peter_Ham: <p>I’m trying to perform gradient backward propagation like this,</p>
<pre><code>    this_out = model(this_inp)
    model.zero_grad()
    this_diff = torch.randn(1,1)
    this_diff[0] = 1.0
    this_diff = this_diff.double()
    this_out.backward(this_diff)
    for f in model.parameters():
            _ = f.data.add_(f.grad.data * learning_rate)
</code></pre>
<p>and it says “f.grad” data is a NoneType</p>

Peter_Ham: <p>Sorry I just found that when I define the network, there are some layers not used in forward pass (which means they are not included in the network). Delete these unused layers helps to solve the problem. But why should we use all layers defined in the init function? Won’t PyTorch automatically ignore these layers?</p>

zjsong: <p>Thank you for sharing the error information and solution, which really helped me to solve my problem.</p>

ahsueh1996: <p>Worked for me too! Thanks!</p>

Winnie_Yang: <p>Hi Thank you for share your experience! It helped me solve my problem! So happy!!!</p>

Rokia: <p>thanks for your question and answer</p>
How to check if a small tensor is inside anthor large tensor

111429: <p>a = torch.tensor([[1,1],[2,2]])<br>
I want to know if tensor([1,1]) is inside in a (return one bool)</p>
<blockquote>
<blockquote>
<blockquote>
<p>a.eq(torch.tensor([1,1]))<br>
tensor([[ True,  True],<br>
[False, False]])<br>
 → which should return a  True to my case.<br>
a.eq(torch.tensor([1,2]))<br>
tensor([[ True, False],<br>
[False,  True]])<br>
 → which should return a  false to my case.<br>
Any suggestion?</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">a = torch.tensor([[1,1],[2,2]])
print(a.eq(torch.tensor([1, 1])).all(dim=1).any())
&gt; tensor(True)

print(a.eq(torch.tensor([1,2])).all(dim=1).any())
&gt; tensor(False)
</code></pre>
Set requires_grad=True to stop tracking history?

Yingfan: <p>Well, I am not sure whether I understand the tutorila right or not. In the AUTOGRAD: AUTOMATIC DIFFERENTIATION tutorial, it says that " If you set its attribute <code>.requires_grad</code> as <code>True</code> , it starts to track all operations on it." However, in the later tutorial, it says " You can also stop autograd from tracking history on Tensors with <code>.requires_grad=True</code> either by wrapping the code block in <code>with torch.no_grad():</code>"  To stop autograd from tracking history on tensors, don’t we should set the requires_grad = False? Is it a conflict?</p>

ptrblck: <p>No, you won’t be able to set <code>requires_grad=False</code> for any non-leaf tensor and will get the following error:</p>
<pre><code class="lang-python">RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
</code></pre>
<p>Use the <code>torch.no_grad()</code> block, which will make sure that Autograd won’t track any operations in this block, or <code>detach()</code> the tensor as described in the tutorial.</p>
Loss function for array of integers where rank matters

jack_jill: <p>My required output from a model is an integer array and the order of those integers in the array are important. As an example,</p>
<p>Let [10, 7, 8, 3, 2] be the expected result.</p>
<p>Then [10, 8, 7, 3, 2] should have a low loss while [8, 1, 5, 6, 7] should have a high loss.</p>
<p>Please let me know a reasonable loss function for this since I don’t think that square loss or cross entropy loss grasp the idea</p>
Should I calculate dataset mean over just training data, or testing data as well?

FuriouslyCurious: <p>This might be somewhat splitting hair, but I am wondering about this:</p>
<p>When you calculate the dataset mean and std, do you compute it over just the training data or do you compute it over the entire dataset (train + test)?</p>
<p>What are the underlying mathematical considerations when making this decision?</p>
<p>Thank you!</p>

InnovArul: <p>Ideally, we will not have access to test data except the assumption that test data will come from same distribution. Thus, we have to calculate mean and std only over train data.</p>
How should I interpret these gradient histograms?

seankala: <p>Hi. I’m currently working on a personal implementation of the Transformer architecture. The code I’ve written as <a href="https://github.com/seanswyi/transformer-implementation" rel="noopener nofollow ugc">here</a>.</p>
<p>The problem that I’m facing is that I believe my model isn’t training properly and I’m not sure what kind of measures I should take to fix that. I’ve come to this conclusion after using Weights &amp; Biases to visualize the model’s gradient histograms and they look something like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36852bc6e3a3670c3a1db8e571b9719035380aa3.png" data-download-href="https://discuss.pytorch.org/uploads/default/36852bc6e3a3670c3a1db8e571b9719035380aa3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36852bc6e3a3670c3a1db8e571b9719035380aa3_2_690x323.png" alt="image" data-base62-sha1="7Mj6g6qDwjy4ZPjuf9zrvVWmKZl" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36852bc6e3a3670c3a1db8e571b9719035380aa3_2_690x323.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36852bc6e3a3670c3a1db8e571b9719035380aa3_2_1035x484.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36852bc6e3a3670c3a1db8e571b9719035380aa3_2_1380x646.png 2x" data-dominant-color="F4F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1822×854 40.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The gradients seem to quickly converge to zero. There is a portion of code that contains a feedforward neural network that uses ReLU activation, and I changed this to Leaky ReLU under the suspicion that dying ReLU’s may be the problem. However, using Leaky ReLU’s doesn’t help and just prolongs the zero-convergence.</p>
<p>Any feedback on what else I may try is appreciated.</p>
Dataloader for sequence of images each with 100 frames

Thabang_Lukhetho: <p>I have a sequence of images of length 100 extracted from a video and saved locally. I’m trying to do frame to frame prediction for arbitrary steps in the future. How do i create a dataloader that loaders a sequence of images?</p>

InnovArul: <p>You might have to write a custom dataset for your case.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" target="_blank" rel="noopener">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 1.7.1...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>You can have a look at the code in this repo:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/MohsenFayyaz89/PyTorch_Video_Dataset/blob/master/GeneralVideoDataset.py" target="_blank" rel="noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/MohsenFayyaz89/PyTorch_Video_Dataset/blob/master/GeneralVideoDataset.py" target="_blank" rel="noopener">MohsenFayyaz89/PyTorch_Video_Dataset/blob/master/GeneralVideoDataset.py</a></h4>
<pre><code class="lang-py">"""
A Simple PyTorch Video Dataset Class for loading videos using PyTorch
Dataloader. This Dataset assumes that video files are Preprocessed
 by being trimmed over time and resizing the frames.


If you find this code useful, please star the repository.
"""

from __future__ import print_function, division

import os
import pickle

import cv2
import numpy as np
import torch
from torch.utils.data import Dataset


</code></pre>

  This file has been truncated. <a href="https://github.com/MohsenFayyaz89/PyTorch_Video_Dataset/blob/master/GeneralVideoDataset.py" target="_blank" rel="noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Embedding in PyTorch

vymao: <p>The first parameter to input into the initialization is to determine the size of the embedding dictionary. Is there a guide on how big this should be? Does every element we want to embed have to be smaller than the size of the dictionary? Or are the elements hashed?</p>

ptrblck: <p>The <code>num_embeddings</code> is defines as the number of indices you would like to pass into it in the range <code>[0, num_embeddings-1]</code>. E.g. if you are dealing with 100 different indices in <code>[0, 99]</code>, <code>num_embeddings</code> would be set to 100.<br>
You can pick the value for <code>embedding_dim</code> and check, what would work the best for your use case.</p>

vymao: <p>Yes, though the values I am trying to embed are sparsely scattered between 1 and 10000. So if I set the first parameter to 10000, it seems like a lot of wasted space.</p>

ptrblck: <p>Would it be possible to shift the values to a contiguous range?</p>
$ nvcc --version nvcc: command not found

Mona_Jalal: <p>I installed PyTorch from its original Website using conda</p>
<pre><code class="lang-auto">(base) mona@mona-ThinkPad-X1-Extreme-Gen-3:~/Downloads/debian$ python
Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.backends.cudnn.version()
7605
&gt;&gt;&gt; quit()
(base) mona@mona-ThinkPad-X1-Extreme-Gen-3:~/Downloads/debian$ nvcc --version
nvcc: command not found
(base) mona@mona-ThinkPad-X1-Extreme-Gen-3:~/Downloads/debian$ nvidia-smi
Fri Dec 18 14:37:44 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 165...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   42C    P8     3W /  N/A |    837MiB /  3911MiB |     21%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       974      G   /usr/lib/xorg/Xorg                           133MiB |
|    0      1569      G   /usr/lib/xorg/Xorg                           365MiB |
|    0      1822      G   /usr/bin/gnome-shell                         320MiB |
|    0      2068      G   /usr/libexec/gnome-initial-setup               1MiB |
|    0     10971      G   /usr/lib/firefox/firefox                       1MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>However, nvcc is not recognizer. It says CUDA is installed but I don’t have it installed here:<br>
/usr/local/cuda/bin</p>
<p>I am using Ubuntu 20.04</p>

InnovArul: <p>I think you should install cuda from NVIDIA website.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>: Does the NVIDIA binaries shipped with pytorch contain <code>nvcc</code> as well?</p>

ptrblck: <p>No, the CUDA compiler is not part of the binaries.<br>
<a class="mention" href="/u/mona_jalal">@Mona_Jalal</a> you would need to install a CUDA toolkit in the matching version as used by your PyTorch binaries.</p>
Size of checkpoint.pth file

snowe: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have two CNNs out of which I save a checkpoint each, as a .pth file. The first model has a regular forward method whereas the second one has a forward method where I loop over each channel in an RGB image to process them separately.</p>
<p>I noticed that the .pth file from the CNN with a loop is roughly 3x smaller compared to the regular one but both models have the same number of parameters… Can anyone explain this to me?</p>
<p>Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>
Why trace method used insted of script method in graph creation?

tam: <p>Why we use,</p>
<pre><code class="lang-auto">            trace = torch.jit.trace(model, args)
            graph = trace.graph
</code></pre>
<p>for creating graph for tensorboard insted of script method. Wont it be better for capturing networks, say with conditional statements in the forward?<br>
Refr :</p>
<pre><code class="lang-auto">         https://github.com/pytorch/pytorch/blob/master/torch/utils/tensorboard/_pytorch_graph.py
```
lines : 286:289</code></pre>
How to expand a tensor

111429: <p>Hi,<br>
x=torch.zeros((3,112,112),dtype=torch.float32)<br>
x.size() <span class="hashtag">#torch</span>.Size([3, 112, 112])<br>
I want to repeat and stack this tensor 25 times as a tensor with shape torch.Size([25, 3, 112, 112])<br>
what are some neat ways to do it?<br>
Thanks.</p>

user_123454321: <p>If I am not mistaken you need the repeat function ? Here is an example:</p>
<pre><code class="lang-auto">a = torch.tensor([
                     [1., 2.]
                     [3., 4.]
                 ])
b = a.repeat([2, 1, 1]) # repeat twice
print(b)
tensor([[[1., 2.],
         [3., 4.]],

        [[1., 2.],
         [3., 4.]]])
</code></pre>

111429: <p>Thanks, I will try it. looks like the api I am looking for.</p>
MathJax plugin in forum

Nikronic: <p>Hi <a class="mention-group" href="/groups/admins">@admins</a></p>
<p>I wonder can we have a way to directly write/render latex?</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://d11a6trkgmumsb.cloudfront.net/optimized/3X/b/3/b33be9538df3547fcf9d1a51a4637d77392ac6f9_2_32x32.png" class="site-icon" width="32" height="32">
      <a href="https://meta.discourse.org/t/discourse-math-plugin/65770" target="_blank" rel="noopener nofollow ugc" title="03:42PM - 25 October 2020">Discourse Meta – 25 Oct 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/426;"><img src="https://d11a6trkgmumsb.cloudfront.net/original/3X/e/1/e1fc9fe4ca601099872448b293c5ce34e1e1b86e.png" class="thumbnail" width="690" height="426"></div>

<h3><a href="https://meta.discourse.org/t/discourse-math-plugin/65770" target="_blank" rel="noopener nofollow ugc">Discourse Math Plugin</a></h3>

<p>Summary: The math plugin uses MathJax (default) or KaTeX to render maths.  🔗 GitHub:  GitHub - discourse/discourse-math: Official MathJax support for Discourse  ➡  Install: Follow the plugin installation guide.  Features    You can render blocks of...</p>

  <p><span class="label1">Reading time: 25 mins 🕑</span>
    <span class="label2">Likes: 53 ❤</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://www.discourse.org/a/img/favicon.ico" class="site-icon" width="128" height="128">
      <a href="https://www.discourse.org/plugins/mathjax.html" target="_blank" rel="noopener nofollow ugc" title="04:43PM - 18 December 2020">Discourse - Civilized Discussion – 18 Dec 20</a>
  </header>
  <article class="onebox-body">
    <img src="https://discourse.org/a/img/favicon.png" class="thumbnail onebox-avatar" width="221" height="221">

<h3><a href="https://www.discourse.org/plugins/mathjax.html" target="_blank" rel="noopener nofollow ugc">MathJax</a></h3>

<p>Display mathematical notation in posts using LaTeX.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Bests</p>
What is the best way to load a large numpy array as PyTorch dataset?

Weixin_Jin: <p>I want to use PyTorch to train a ConvNet. But my data is one single large numpy <code>.npy</code> file, with shape like <code>(100000, 5, 200, 200)</code>, rather than traditional images. The first dimension is sample size. The next three dimensions can be regarded as the channels, width and height of images.</p>
<p>Is there any “standard” way to load large training/test dataset into PyTorch, especially when the input is a large <code>.npy</code> file? And do I need to preprocess the data and save it as another file? Thank you!</p>
<p>I could use <code>mmap_mode='r'</code> to load it lazily. But if I want to preprocess the data, such as normalization, training/test set split, I still need to load all data into memory.</p>
Multiple forward from single input using for loop

Juna: <p>Hi,<br>
down below, I coded smooth gradient function<br>
in order to implement the interpretation method ‘smooth grad’.</p>
<pre><code class="lang-auto">    def smooth_grad(inputs, labels, target_layer, args, by_label=True):
        iterations = args.smooth_num
        alpha = args.smooth_std
        for i in range(iterations):
            inputs_noise = inputs + alpha*torch.randn(inputs.shape).cuda()
            activation_output = prediction(inputs_noise)
            if i == 0:
                R = simple_grad(activation_output, labels, target_layer, args, no_R_process=True).detach()
            else:
                # If you want to train the model by using LR with smooth grad, then you need to remove detach() function.
                R += simple_grad(activation_output, labels, target_layer, args, no_R_process=True).detach()

        return R
        
        # Do something with R
</code></pre>
<p>simple_grad function is a function that calculates the gradient of activation_output with respect to inputs_noise with “forward propagation”. and at the end, I’m gonna backpropagate with the result I’m going to make with R.<br>
What I found is that even though I’m using vgg16, 8x3x224x224 size of input and<br>
there are multiple forward props(16 of them), it takes less gpu memory than 11GB.</p>
<p>Question</p>
<ol>
<li>How is this possible?</li>
<li>Is there any problem in the way that I used for loop like this? Will backprop be able to flow to all those 16 forward path?</li>
</ol>
Gradient of layers before softmax

RajaGanapathyM: <p>Gradients of variables/parameters before softmax becomes zero,</p>
<p>for eg.<br>
import torch</p>
<p>a1 = torch.rand([4, 4], requires_grad=True)<br>
b1 = a1.squeeze(0)**2<br>
b1=F.softmax(b1,dim=1)<br>
b1.sum().backward()<br>
print(a1.grad)</p>
<p>Gives<br>
tensor([[0., 0., 0., 0.],<br>
[0., 0., 0., 0.],<br>
[0., 0., 0., 0.],<br>
[0., 0., 0., 0.]])</p>
<p>How to deal with this in case i need to use softmax in between layers in my model?<br>
Thanks</p>

user_123454321: <p>I am not sure what you want to do here exactly, but the answer is theoretically correct.<br>
You are taking a 4 x 4 matrix (a1), then squaring each value. Then you are taking the softmax along the columns. This results in each row having values that <strong>sum to 1</strong>. When you then sum this matrix, you basically get the result as 4.<br>
Thus, this means, however you change the values of a1, the sum remains 4. This essentially means your are taking derivative of a constant which should be 0.</p>
Loss is much higher after resume the training

111439: <p>Good day,<br>
I am trying to resume the training. When the training was performed on one gpu, the training continues - the loss is nearly the same as it was before, as well as the evaluation score. When I am trying to continue the training of the model that was trained on multiple gpus, the loss is much higher than it was (0.99 in comparison to 0.34).</p>
<p>Could you please help me, what is the correct way to load the previous weights of the model trained on multiple gpus?</p>
<p>I use <a href="https://github.com/wolny/pytorch-3dunet/blob/62e10674c5cf9f44e252297203213b8c7f23c5f7/pytorch3dunet/unet3d/trainer.py#L91" rel="noopener nofollow ugc">pytorch-unet</a>. Can the error be in <a href="https://github.com/wolny/pytorch-3dunet/blob/62e10674c5cf9f44e252297203213b8c7f23c5f7/pytorch3dunet/unet3d/utils.py#L65" rel="noopener nofollow ugc">these lines</a>?</p>
<pre><code class="lang-auto">state = torch.load(checkpoint_path, map_location='cpu')
model.load_state_dict(state['model_state_dict'])
</code></pre>
<p>The problem shouldn’t be in the optimizer because its state is also loaded</p>
TorchScript trace size()

churchillmic: <p>I want to add preprocessing to a wrapper class around my forward model, and trace with TorchScript to be able to use in a C++ code. The normalization of input data, etc. works fine, but I want to be able to seamlessly handle inputs with different channel numbers (padding with 0’s if 1 channel). I tried doing this with <code>if input.size()[0]==1</code>, but get a warning that makes me think this might be ignored:</p>
<blockquote>
<p>create_torchscript_model.py:37: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.</p>
</blockquote>
<p>I try keeping everything in tensors:</p>
<pre><code class="lang-auto">        if torch.tensor(input.size())[0]==torch.tensor(1):
            input = F.pad(input,(0,0,0,0,0,0,1,0))
</code></pre>
<p>but get similar warnings. Is there a better way to do this, or should I just keep it in the C++ code?</p>

ptrblck: <p>Tracing is using the currently provided inputs and uses it as constants as given in the warning.<br>
Could you try to script the module instead?</p>

churchillmic: <p>Ah, I see better from your comment and reading the documentation closer also (<a href="https://pytorch.org/docs/stable/jit.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/jit.html</a>), control flow has to be done with scripting not tracing, because tracing will all depend on the input you give it when tracing. Thanks, I switched this over to scripting using:</p>
<pre><code class="lang-auto">traced_script_module = torch.jit.script(wrapper)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">traced_script_module = torch.jit.trace(wrapper,example)
</code></pre>
<p>I have some related issues with mixing CPU/GPU here, <a href="https://discuss.pytorch.org/t/mixing-gpu-cpu-code-in-torchscript-traced-model/106490" class="inline-onebox">Mixing GPU/CPU code in TorchScript traced model</a>, but in general I’m seeing scripting is answering some of those questions (e.g. <code>with torch.no_grad</code> throws an error, saying its not allowed). I’m still not certain about <code>.cuda()</code> and <code>.cpu()</code> mixing in a scripted module, and if I can have separate locations for some parameters, but this answers this question, and I’ll await answers to the others in the other post.</p>
Torch.get_num_threads() gives half the number of threads

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’ve been trying to use PyTorch’s cpu threading capabilities and I’ve noticed that PyTorch’s command <code>torch.get_num_threads</code> only returns half the number of threads. I’m currently working on a computer with 12 threads but <code>torch.get_num_threads</code> returns 6. Is there any reason for this?</p>
<p>If I run <code>lscpu</code> I get the following information</p>
<pre><code class="lang-auto">CPU(s):              12
On-line CPU(s) list: 0-11
Thread(s) per core:  2
Core(s) per socket:  6
Socket(s):           1
</code></pre>
<p>Is it perhaps returning the number of cores instead? (for some reason)? I’m using PyTorch 1.6.</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>It returns the number of thread that can be used to perform multithreading (note that you can change that value if you want).<br>
We don’t put it too high by default to avoid over-subscription that is very detrimental in terms of runtime. But this value will need to be tuned if you want to get the best performances out of your system for sure.</p>

AlphaBetaGamma96: <p>Hi,</p>
<p>Thanks for the explanation! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>