akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Mask_rcnn_resnet50_fpn() 'min_size & 'max_size' arguments: what do they do?

username: xGrouchx
cooked: <p>Hi all,</p>
<p><strong>High-level explanation:</strong><br>
I am confused by what the min and max size argument actually does - and how it should be optimized for both training and inference. I notice that after training my mask rcnn model, I can basically tinker with those two values and get dramatically different results on my test data.</p>
<p><strong>More details:</strong><br>
I trained a mask r-cnn for ship detection. During training I instantiated my model like so:</p>
<p>def get_model_instance_segmentation(num_classes):<br>
# load an instance segmentation model pre-trained pre-trained on COCO<br>
model = torchvision.models.detection.maskrcnn_resnet50_fpn(<br>
min_size=256,<br>
max_size=512<br>
)</p>
<p>I did this because my training chips varied in size from 256x256 to 512x512. By default the min_size and max_size are 800, 1333 respectively. In the end I found this doesnt seem to matter for your chip size - but can impact your results during inference regardless of chip/input image size.</p>
<p>The min_size and max_size arg are passed as kwargs to torchvision.models.detection.maskrcnn() as per the source code here: <a href="https://pytorch.org/vision/stable/_modules/torchvision/models/detection/mask_rcnn.html#maskrcnn_resnet50_fpn" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.detection.mask_rcnn — Torchvision 0.12 documentation</a></p>
<p>What it says in the source code comments:<br>
min_size (int): minimum size of the image to be rescaled before feeding it to the backbone<br>
max_size (int): maximum size of the image to be rescaled before feeding it to the backbone</p>
<p>Digging deeper into the source code, the min and max size do some sort of a transform. But there is no explanation of what it does - and how it can impact your training and inferencing. Any help would be greatly appreciated.</p>
<p>Thank you everyone - I really appreciate your time in reading my request.<br>
Alex</p>
RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)

username: RigCor7
cooked: <p>I encountered the following problem, when I set a larger world_size</p>
<pre><code>File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/__init__.py", line 195, in init_rpc

    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/__init__.py", line 229, in _init_rpc_backend

    rpc_agent = backend_registry.init_backend(

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/backend_registry.py", line 106, in init_backend

    return backend.value.init_backend_handler(*args, **kwargs)

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/backend_registry.py", line 315, in _tensorpipe_init_backend_handler

    api._all_gather(None, timeout=rpc_constants.DEFAULT_RPC_TIMEOUT_SEC)

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper

    return func(*args, **kwargs)

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/api.py", line 204, in _all_gather

    rpc_sync(

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper

    return func(*args, **kwargs)

File "/root/anaconda3/envs/final/lib/python3.9/site-packages/torch/distributed/rpc/api.py", line 765, in rpc_sync

    return fut.wait()

RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
</code></pre>
<p>And this is my code:</p>
<pre><code>import os

import time

from entity.Server import Server

import torch.multiprocessing as mp

from torch.distributed import rpc

from utils.options import args_parser

import torch.distributed as dist

SERVER_NAME = "Server"

CLIENT_NAME = "Client{}"

os.environ['MASTER_ADDR'] = 'localhost'

os.environ['MASTER_PORT'] = '29500'

def run(rank, args):    

    if rank == 0:

        rpc.init_rpc(SERVER_NAME, rank=rank, world_size=args.world_size)

        # server = Server(args)
    else:

        rpc.init_rpc(CLIENT_NAME.format(rank), rank=rank, world_size=args.world_size)

    rpc.shutdown()

if __name__ == "__main__":

    args = args_parser()

    mp.spawn(

        run,

        args=(args, ),

        nprocs=args.world_size,

        join=True

    )
</code></pre>
<p>When I set the world_size value above 25, the code fails.</p>
<p>I think this may be a problem with the linux system configuration, does anyone know how to configure or fix this?</p>

username: Yanli_Zhao
cooked: <p>cc <a class="mention" href="/u/pbelevich">@pbelevich</a> for configuring RPC</p>
Weighted re-sampling for multi-label classification

username: srikanthram
cooked: <p>Hello,</p>
<p>I have been trying to look for help on something similar to weighted resampling but in the multi-label object detection scenario for my highly imbalanced dataset.</p>
<p>Quick overview:<br>
My dataset consists of images that are broadly classified as good images and bad images (images with defects). Furthermore, the bad images are further classified into seven different (labels 1 through 7) based on the type of defects they have. The good ones which do not have any defects are encoded as all zeros [0,0,…,0] and an image with defects 1 and 2 for example is multi-hot encoded as [1,1,0,…,0] .</p>
<p>Problem:<br>
The good ones are a lot in number and the ones with defects are very less. This calls for weighted re-sampling. I have looked at a discussion thread here: <a href="https://discuss.pytorch.org/t/unbalanced-data-in-multiple-object-detection/88104" class="inline-onebox">Unbalanced data in multiple object detection</a> and here: <a href="https://discuss.pytorch.org/t/help-for-sampling-training-data-for-multi-label-classification-task/7794/2" class="inline-onebox">Help for Sampling training data for multi-label classification task?!</a>. Was wondering if anyone has a sample implemented. Thanks.</p>

username: duddal
cooked: <p><a class="mention" href="/u/srikanthram">@srikanthram</a> Did you find anything?</p>

username: mmitjans
cooked: <p><a class="mention" href="/u/srikanthram">@srikanthram</a> I’m in the same situation. <a class="mention" href="/u/duddal">@duddal</a> did you find a solution somewhere else?</p>
Are Distributed Optimizers supported for CUDA?

username: asnq99
cooked: <p>In the PyTorch documentation page for <a href="https://pytorch.org/docs/master/distributed.optim.html" rel="noopener nofollow ugc">Distributed Optimizers</a>, the webpage says:</p>
<pre><code class="lang-auto">WARNING: Distributed optimizer is not currently supported when using CUDA tensors
</code></pre>
<p>Link to snapshot of the documentation page captured by web archive at the time of writing this post: <a href="http://web.archive.org/web/20220511133806/https://pytorch.org/docs/master/distributed.optim.html" rel="noopener nofollow ugc">Distributed Optimizers — PyTorch master documentation (archive.org)</a></p>
<p>However, the tutorial on Distributed optimizers (<a href="https://pytorch.org/tutorials/recipes/zero_redundancy_optimizer.html" rel="noopener nofollow ugc">Shard Optimizer States with ZeroRedundancyOptimizer — PyTorch Tutorials 1.11.0+cu102 documentation</a>) uses models on CUDA devices with no issues at all.</p>
<p>I tried using the <code>ZeroRedundancyOptimizer</code> with my model too and it does seem to be working even when models are on CUDA/GPUs. Is the documentation page inaccurate/out of date, or am I misinterpreting what the warning means?</p>
<p>TIA.</p>

username: Yanli_Zhao
cooked: <p>Distributed optimizers are a little bit different, it is often used with distributed rpc system.</p>
<p>The <a href="https://pytorch.org/tutorials/recipes/zero_redundancy_optimizer.html" rel="noopener nofollow ugc">ZeroRedundancyOptimizer</a> is a sharded optimizer often used with DDP, it can support CUDA tensors</p>
Support for tensor_split with ONNX support V13

username: nirmalthomas2609
cooked: <p>From what I understand, I believe there is no support for tensor_split or any dynamic splitting operator while exporting a model to ONNX.<br>
Is there a workaround for this? Or, can someone point me to the source code where I could add support for this?</p>
<p>Thanks!</p>
How does Pytorch function architecturally?

username: valya
cooked: <p>As with most other auto-differentiation libraries, pytorch doesn’t require you to explicitly define a graph variable which records computations (computational graph).</p>
<p>I was wondering how does pytorch autograd accomplish this? Does a tensor produced as a result of a tensor operation store references to its children tensors? Or is there a global variable that holds all tensors which tensor handles can point to? Or some other way?</p>
<p>I don’t have a good knowledge of C++, so I can’t look into the codebase to understand it, but maybe there’s a good conceptual explanation that’s independent of the implementation language?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Parallelize embedding process in the forward method

username: lkc1
cooked: <p>Hi,</p>
<p>I’m working on a textual modeling problem. There are three steps in <code>forward</code> method as shown below.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.ModuleDict()
        self.layers['layer_0'] = nn.Linear(in, out)
        self.layers['relu_0'] = nn.ReLU()

    def forward(self, inp):
        
        with torch.no_grad():
            # Step 1: tokenize input
            tokenized_inp = tokenize(inp)
    
            # Step 2: convert out of tokenization to embeddings on cuda device
            x = generate_embedding(tokenized_inp)

        # Step 3: run the network that follows
        for layer_name, layer in self.layers.items():
            x = layer(x)

        return x
</code></pre>
<p>Steps 1 and 2 do not include any trainable parameters. I’m using huggingface tokenizer and embedding model. Can I parallelize these steps before executing step 3 which is the real ‘training’ step of the network?</p>
<p>I want to do this because my GPU memory is not fully utilized. With a batch size of 64, I only use 20% of the GPU memory.</p>
<p>Step 1 includes tokenization that is done serially on the CPU and step 2 includes moving data from CPU to CUDA. These steps scale linearly with the batch size. So if I try to utilize more GPU memory with a higher batch size, I lose on speed at these two steps.</p>
<p>Is it possible to perform tokenization and embedding in parallel to take advantage of the GPU memory capacity?</p>
<p>Thanks</p>

username: tom
cooked: <p>I think what you describe is part of the reason that people don’t usually do the tokenization in the <code>nn.Module</code> but move it to the dataset. This allows the DataLoader to use several processes (if you pass <code>num_workers</code>). Then you can increase the batch size.<br>
Doing embeddings on the GPU should be fast, so I would not worry about it too much.</p>
<p>Best regards</p>
<p>Thomas</p>

username: lkc1
cooked: <p>That makes sense. Thanks for the suggestion!</p>

username: lkc1
cooked: <p>DataLoader documentation says the following:</p>
<p>It is generally not recommended to return CUDA tensors in multi-process loading because of many subtleties in using CUDA and sharing CUDA tensors in multiprocessing (see CUDA in multiprocessing). Instead, we recommend using automatic memory pinning (i.e., setting pin_memory=True), which enables fast data transfer to CUDA-enabled GPUs.</p>
<p>Does that mean when the embeddings are generated as CUDA tensors, I shouldn’t return them if I’m using DataLoader?! Would I need to copy them back to CPU before returning and setting automatic memory pinning to True? It seems redundant to copy to CPU to only copy back to the GPU.</p>

username: tom
cooked: <p>Yes, so the standard idea is to have the parallelized CPU part (the tokenization) in the dataset, move the tokens to the CPU in the loop (so after the dataloader batched them) and the embedding in the model on the GPU.</p>
<p>Best regards</p>
<p>Thomas</p>

username: lkc1
cooked: <p>I see, that makes sense. Thanks</p>
Only use libtorch 1.10.2 can detect GPU

username: Tumb1eweed
cooked: <p>Very strange thing.<br>
In my device, with cuda 11.1, only libtorch1.10.2 + cu102 can use “torch.devce_count” to detect GPU, higher(1.11.x) or lower version(1.9.x) can’t detect either, what’s the problem?</p>
How to convert numpy.linalg.norm to LibTorch codes

username: vujadeyoon
cooked: <p>Dear all,</p>
<p>I have tried to convert numpy.linalg.norm to LibTorch, not PyTorch.<br>
I mange to find the LibTorch API [1]-[2], but I cannot apply the LibTorch API because I don’t know some required specific data type of the parameter (e.g. const optional &amp;opt_ord in [1]).</p>
<p>If you are familiar with the LibTorch or know the above API usage, could you explain how to convert below numpy codes to the corresponding LibTorch codes?<br>
I attach some Python3 and Cpp-LibTorch codes below.</p>
<pre><code class="lang-python">import numpy as np


if __name__=='__main__:
    ndarr_temp = np.asarray([[1, 2], [3, 4]])
    ndarr_norm = np.linalg.norm(ndarr_temp, axis=0) 
    print(ndarr_norm) # array([3.16227766, 4.47213595])
</code></pre>
<pre><code class="lang-cpp">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/Functions.h&gt;


int main(int argc, const char* argv[]) {
    torch::Tensor tensor_temp = torch::tensor({{1, 2}, {3, 4}});
    
    // torch::Tensor tensor_norm = torch::linalg::norm(tensor_temp, 0); // This code is wrong.
    // Todo: Please implement the LibTorch codes corresponding to the above python codes.

    std::cout &lt;&lt; tensor_norm &lt;&lt; std::endl;

    return 0;
}
</code></pre>
<p>Best regards,<br>
Vujadeyoon</p>
<p>[1] <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1_1linalg_1abf0f7c9aeab6eb62ab93be852cb7ab38.html" class="inline-onebox" rel="noopener nofollow ugc">Function torch::linalg::norm(const Tensor&amp;, const optional&lt;Scalar&gt;&amp;, OptionalIntArrayRef, bool, optional&lt;ScalarType&gt;) — PyTorch master documentation</a></p>
<p>[2] <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1_1linalg_1a024aff0fe586f3c9a43de98ee8b82e5c.html" class="inline-onebox" rel="noopener nofollow ugc">Function torch::linalg::norm(const Tensor&amp;, std::string, OptionalIntArrayRef, bool, optional&lt;ScalarType&gt;) — PyTorch master documentation</a></p>

username: vujadeyoon
cooked: <p>Dear all,</p>
<p>Unfortunately, I haven’t been able to find the solution that I hope.<br>
However, I make a function, <em>linalg_norm</em> that does exactly the same thing as the <em>numpy.linalg.norm</em>.<br>
Thus, the entire Cpp-LibTorch codes using it are as follows:</p>
<pre><code class="lang-cpp">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/Functions.h&gt;


torch::Tensor linalg_norm(torch::Tensor *ptr_tensor, int dim = 0, bool keepdim = false) {
    return torch::sqrt(torch::sum(torch::pow(*ptr_tensor, 2), {dim}, keepdim));
}


int main(int argc, const char* argv[]) {
    torch::Tensor tensor_temp = torch::tensor({{1, 2}, {3, 4}});
    torch::Tensor tensor_norm = linalg_norm(&amp;tensor_temp, 0, true);
    std::cout &lt;&lt; tensor_norm &lt;&lt; std::endl;
    /*
    3.1623  4.4721
    [ CPUFloatType{1,2} ]
    */

    return 0;
}
</code></pre>
<p>Best regards,<br>
Vujadeyoon</p>
The C++ equivalent of numpy.linalg.norm?

username: dambo
cooked: <p>Hello,<br>
In Pytorch/Python I am generating Resnet34 feature vectors,e.g. a vector of size 512 for a single image which I normalise using:<br>
<code>numpy.linalg.norm</code>.</p>
<p>What is the equivalent form in libtorch? This (libtorch 1.2) does not seem to work.</p>
<pre><code class="lang-auto">torch::Tensor out_tensor = module.forward({input_tensor}).toTensor();
const auto dimensions = out_tensor.ndimension();
const auto v_size = out_tensor.size(1); //Should be 512
//std::cout &lt;&lt; out_tensor.norm().item() &lt;&lt;std::endl;
std::cout &lt;&lt; torch::norm(out_tensor) &lt;&lt;std::endl;
</code></pre>
<p>The code in PyTorch:</p>
<pre><code class="lang-auto">f =features[i].cpu().detach().numpy()
f= f / np.linalg.norm(f)
</code></pre>
<p>Thanks,</p>

username: vujadeyoon
cooked: <p>Dear dambo,</p>
<p>I had the same concerns as you, and designed a cpp function, <em>linalg_norm</em> [1] using the LibTorch that performs the functions of the <em>numpy.linalg.norm</em>.<br>
Actually, the LibTorch also provides <em>Function torch::linalg::norm()</em> [2], but I cannot use it because I don’t know the required data types for the function.</p>
<p>I hope this reply is helpful to you.</p>
<p>Best regards,<br>
Vujadeyoon</p>
<p>[1] <a href="https://discuss.pytorch.org/t/how-to-convert-numpy-linalg-norm-to-libtorch-codes/151592" class="inline-onebox">How to convert numpy.linalg.norm to LibTorch codes</a></p>
<p>[2] <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1_1linalg_1abf0f7c9aeab6eb62ab93be852cb7ab38.html" class="inline-onebox" rel="noopener nofollow ugc">Function torch::linalg::norm(const Tensor&amp;, const optional&lt;Scalar&gt;&amp;, OptionalIntArrayRef, bool, optional&lt;ScalarType&gt;) — PyTorch master documentation</a></p>
One optimizer for generator and discriminator in GAN

username: Shawn_Zhuang
cooked: <p>Hi,<br>
recently I am learning the implementation of GAN with pytorch.</p>
<p>In the official version of <a href="https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/basic-gan.html" rel="noopener nofollow ugc">PYTORCH LIGHTNING BASIC GAN TUTORIAL</a>, two optimizers are used to train generator and discriminator as following.</p>
<blockquote>
<pre><code>    opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
    opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))
</code></pre>
</blockquote>
<p>What happen if only one Adam is used as following,</p>
<blockquote>
<p>opt_g_d = torch.optim.Adam(<br>
{‘params’: self.generator.parameters()},<br>
{‘params’: self.discriminator.parameters()}, lr=lr, betas=(b1, b2))</p>
</blockquote>

username: William_Justin
cooked: <p>I have same question here. Hope someone will reply. Thanks.</p>
<p>Here’s my code.</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self):
        pass
    
    def forward(self, x):
        pass

class Discriminator(nn.Module):
    def __init__(self):
        pass
    
    def forward(self, x):
        pass

class GAN(nn.Module):
    def __init__(self):
        super().__init__()
        self.G = Generator()
        self.D = Discriminator()
    
    def forward(self, x):
        pass

model = GAN()
optimizer = torch.optim.Adam(model.parameters())
</code></pre>
Can not use class method or member variables in `@torch.jit.script` decorated method

username: vermont_waters
cooked: <p>I am working on a custom model <code>MY_MODEL_CLASS</code> (inhereting from Pytorch nn.module).</p>
<p>I can convert the model to TorchScript via:</p>
<blockquote>
<p>torchscript_model = torch.jit.trace(torch_model, input)</p>
</blockquote>
<p>However, there is some control flow that needs to be handled, so I want to use a <code>@torch.jit.script</code> decorator for a method <code>MY_SCRIPT_DECORATED_CLASS_METHOD</code> of this model. In this method, another class method <code>USED_IN_SCRIPTED_CLASS_METHOD</code> of <code>MY_MODEL_CLASS</code>  is called.</p>
<p>When I now try to convert the model with tracing plus the decorator, I get the following error:</p>
<blockquote>
<p>‘Tensor (inferred)’ object has no attribute or method ‘USED_IN_SCRIPTED_CLASS_METHOD’.:</p>
</blockquote>
<p>I see that the same thing happens for member variables of the model that are used in this scripted function. I am a bit lost in this case what causes this.<br>
I am also not sure what <code>'Tensor (inferred)'</code> refers to, since I was expecting this to rather see the name <code>MY_MODEL_CLASS</code> here.</p>
<p>Do you now what causes this?<br>
How could I debug this?</p>

username: Misarov
cooked: <p>The use of your method is just not supported in the scripting case. You will have to use the tracing functionality.</p>
LSTM different results

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>When I apply my LSTM to obtain two outputs (one is symmetric to the other), my LSTM does not perform the same in both outputs, one is clearly better than the other.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2814d37e0b64fbffece22f5a2f676bbd3fdb255.png" alt="image" data-base62-sha1="pt814ajfin8XQGlHmci4tjJXJMF" width="408" height="268"></p>
<p>Has anyone had the same problem?</p>

username: Michal_Bogacz
cooked: <p>Hi,<br>
on what data was it trained?</p>

username: Andre_Amaral_IST
cooked: <p>My dataset consists on:<br>
92 different dimensions of transistors<br>
For each dimension, I have 9 datasets with 2 input waves and 2 output waves.</p>
<p>The objective is to mimic the behaviour of the circuit for all the dimensions, in other words it need to pe capable to, given a dimension and the input waves, produce the correct two output waves</p>
WeightedRandomSampler for custom image dataloader

username: siddharth_MV
cooked: <p>I am trying to solve class imbalance by using Weighted Random Sampler on a custom data loader for multiclass image classification. I can’t seem to find the best way to implement this. The images are in a folder and labels are in a csv file. The dataloader code without the weighted random sampler is given below.</p>
<pre><code class="lang-auto">class CassavaDataset(Dataset):
    def __init__(self, df, data_root, transforms=None, output_label=True):
        super().__init__()
        self.df = df.reset_index(drop=True).copy() # data
        self.transforms = transforms
        self.data_root = data_root
        self.output_label = output_label
 
    def __len__(self):
        return self.df.shape[0] # or len(self.df)
    
    def __getitem__(self, index: int):
        
        # get labels
        if self.output_label:
            target = self.df.iloc[index]['label']
          
        path = "{}/{}".format(self.data_root, self.df.iloc[index]['image_id'])
        
        img  = get_img(path)

        if self.transforms:
            img = self.transforms(image=img)['image']
           
        # do label smoothing
        if self.output_label == True:
            return img, target
        else:
            return img
</code></pre>
<p>What will be the best way to get weights of each class and feed it to the sampler before augmentation? Thanks in advance!</p>

username: tom
cooked: <p>I know it is hacky, but I usually just enlarge the dataset to repeat indices that should have higher weight.<br>
Or you could have to “leak” the weights from the dataset to the sampler.</p>
<p>Best regards</p>
<p>Thomas</p>

username: siddharth_MV
cooked: <p>Hey Thomas thanks for the reply but I’m not sure I understand. Could you elaborate or give an example of some sort?</p>

username: tom
cooked: <p>I don’t know much about your dataset, but here is a very simple two-class example:</p>
<p>Say you have class 1 at indices 1…9_999 and class 2 at indices 10_000…10_048, you could do the following at the top of getitem:</p>
<pre><code class="lang-python">index = index if index &lt; 10_000 else (index - 10_000) % 49
</code></pre>
<p>and then set the <code>__len__</code> to return 20_000. This would give you a dataset with class 2 to be oversampled to the weight of class 1.</p>
<p>Best regards</p>
<p>Thomas</p>
Truncate tensors after a specific token

username: Millon_Madhur_Das
cooked: <p>I have a batch of sequences(tensors). Each sequence is of the form &lt;|sos|&gt; … &lt;|eos|&gt; … &lt;|eos|&gt; . I want to truncate each sequence after the first  token.<br>
Thanks in advance.</p>

username: tom
cooked: <p>This is not that easy a problem. One thing you can do use <code>arange</code> to get a tensor of indices and then <code>where</code> to mask the tensor of indices to be 2000 (or &gt; seqlen). Then the minimum of these masked indices will be the first token you did not mask.<br>
Then you can use these to get a slice of the tensor. Note that if you do work on the GPU, you will have a synchronization point because the slicing needs the index on the CPU.</p>
<p>Best regards</p>
<p>Thomas</p>
Weights and outputs become nan

username: Nemfor
cooked: <p>Hi! I face the following problem:<br>
I try to use custom loss (RMSLE)</p>
<pre><code class="lang-auto">class RMSLELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss(reduction='none')
    def forward(self, pred, actual):
        return torch.sqrt(self.mse(torch.log(pred + 1), torch.log(actual + 1)))
</code></pre>
<p>during training I use loss.mean() to average the results</p>
<pre><code class="lang-auto">output = self._model(batch).squeeze(dim=1)
target = batch['target'].to(self._device)
loss = loss_func(output, target)
loss = loss.mean()                    
</code></pre>
<p>So i just average the loss by myself, but after some iterations weights and outputs become nans.<br>
But if i replace in init of RMSLE class to just MSELoss() with reduction as default- so it avarages by itself, all problems will go away. Why it happens?<br>
If i use just MSELoss(), not my custom, the problem doesnt show again.</p>

username: AlphaBetaGamma96
cooked: <p>Are your <code>pred</code> and <code>actual</code> Tensors strictly positive? the Log of a negative number is undefined and will lead to a <code>nan</code>.</p>

username: Nemfor
cooked: <p>I can chech it but the question is why the are becoming probably negative only in the case when I avarage my loss by myself?<br>
And as I sad, the weights of the layears are always become to <code>nans</code></p>

username: eqy
cooked: <p>I’m not sure the expressions are identical as the first computes the mean inside the sqrt while the second does it on the outside so they could have different numerical properties.<br>
e.g.,<br>
d/dx((sqrt(x) + sqrt(y))/2) is 1/(4sqrt(x))<br>
whereas<br>
d/dx(sqrt((x + y )/ 2)) is 1/(2sqrt(2)sqrt(x+y)）</p>
PyTorch pack_padded_sequence is really slow

username: cabezon
cooked: <p>I am building a GRU-based architecture. Before, I was just padding the batches of sequences and passing it to the GRU. Obviously, that was introducing some small error in the results because it’s not quite the 100% correct thing to do (the GRU doesn’t know to stop when it reaches the padding elements).</p>
<p>Thus I switched out the naive batch of 2d padded sequences for pack_padded_sequence, so that I’m not passing extraneous padding items to the GRU. The training time increased by at least 3x. I am doing the pack_padded_sequence on GPU, so I need to check if perhaps it’s just inefficient to do on GPU? When doing pack_padded_sequence with word embeddings, is it preferable to embed first and then pack, or pack first and then embed? I have been doing the former and not sure if it’s contributing to a slowdown.</p>
<p>Any suggestions would be appreciated!</p>

username: tom
cooked: <p>Usually, packing should be fast enough, but here are a couple of considerations:</p>
<ul>
<li>Re “it’s not quite the 100% correct thing to do” given that output and final state are the same, you could just read off the <code>input_len - 1</code>th output of each sequence,</li>
<li>Usually, packing should not be that slow, but I would only expect a speedup when using CuDNN. One of the things to know is that you probably want pre-computed lengths and absolutely not compute them on the fly from the padding.</li>
<li>I’d probably pack first and then embed for efficiency reasons, but it may be that you need to do the application of the embedding manually (There are many threads on “elementwise” operations on packed sequences.)</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: cabezon
cooked: <p>Thanks Thomas! How would you handle the bidirectional RNN case? I realized I could take input_len - 1 without issue for the forward pass, but the padding in the other direction would most definitely corrupt the results (would see a bunch of padding as the first couple of inputs to the backward RNN). I tried building a separate dedicated backward RNN to complement the forward RNN, but it is not efficient and not preferred.</p>
<p>I will check if CuDNN is properly installed on the machine.</p>
<p>I am familiar with how to apply the embed on a packed sequence, it’s a little hacky but you essentially need to rebuild the packed sequence after modifying the packed sequence’s .data attribute via embed(…) call</p>

username: tom
cooked: <p>As far as I remember, the bidirectional case is tricky without <code>pack_padded_sequence</code>, unfortunately. I could imagine using JIT + a mask, but one would have to see exactly how to get the JIT to generate good code for it and it would be dependent on the PyTorch version.</p>
<p>Best regards</p>
<p>Thomas</p>
How to put everything into a Sequential container

username: Samuele_Bolotta
cooked: <p>I have this code that works and does what I want it to do. However, I would also need the Sequential container to contain not only the linear layers, but also the one hot encoding. How could I achieve this? Thanks!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

preprocessing_layers = {}

## Input
obs_space = torch.Tensor([0, 1,2,3])
obs_space = obs_space.to(torch.long)

## Sequential
layers = [torch.nn.Linear(obs_space.size(-1), 5)]
preprocessing_layers["direction"] = nn.Sequential(*layers)

## One hot
obs_space = F.one_hot(obs_space,4)
obs_space = obs_space.to(torch.float32)

## Forward
a = preprocessing_layers["direction"](obs_space)


</code></pre>

username: tom
cooked: <p>You could use the one hot encoding as the forward of a module and then stick that into the sequential, so this might go in:</p>
<pre><code class="lang-python">class OneHot(torch.nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.dim = dim
    def forward(self, x):
        return F.one_hot(x, self.dim)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
How to initialize one hot encoding in a class in PyTorch

username: Samuele_Bolotta
cooked: <p>I’m trying to convert to PyTorch this TF code.</p>
<pre><code class="lang-auto">preprocessing_layers[scalar_name] = tf.keras.models.Sequential(
        [multigrid_networks.one_hot_layer(scalar_dim),
         tf.keras.layers.Dense(scalar_fc)])
</code></pre>
<p>This is how I would write it in PyTorch:</p>
<pre><code class="lang-auto">## Sequential
layers = [F.one_hot(scalar_dim), torch.nn.Linear(obs_space.size(-1), 5)]
preprocessing_layers[scalar_name] = nn.Sequential(*layers)
</code></pre>
<p>The problem is that if I do so, PyTorch is throwing this error at me:</p>
<pre><code class="lang-auto">TypeError: one_hot(): argument 'input' (position 1) must be Tensor, not int
</code></pre>
<p>My question is: how do I put hot encoding in a Sequential container in the <strong>init</strong> method of a class, without inserting the input (which is what I’ll do in the forward method)?</p>

username: Andrei_Cristea
cooked: <p>Try this:</p>
<pre><code class="lang-auto">scalar_dim = 3
num_classes = 10

F.one_hot(torch.LongTensor([scalar_dim]), num_classes=num_classes)

Output:
tensor([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0]])
</code></pre>

username: Samuele_Bolotta
cooked: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>This is how I rewrote the code following your piece of advice:</p>
<pre><code class="lang-auto">import torch.nn.functional as F
import torch.nn as nn
import torch

preprocessing_layers = {}

## Input
obs_space = torch.LongTensor([0,1,2,3])

## Sequential
scalar_dim = 3
num_classes = 4

layers = [F.one_hot(torch.LongTensor([scalar_dim]), num_classes=num_classes), torch.nn.Linear(obs_space.size(-1), 5)]
preprocessing_layers["direction"] = nn.Sequential(*layers)
</code></pre>
<p>But now I get this error:</p>
<pre><code class="lang-auto">TypeError: torch.LongTensor is not a Module subclass
</code></pre>

username: Andrei_Cristea
cooked: <p>You’ll have to wrap it in a Module child class, like this:</p>
<pre><code class="lang-auto">class ToOneHot(nn.Module):
    def __init__(self, num_classes):
        super(ToOneHot, self).__init__()
    def forward(self, x):
        return F.one_hot(torch.LongTensor([x], num_classes=num_classes))

nn.Sequential(ToOneHot(num_classes), torch.nn.Linear(obs_space.size(-1), 5))

Output:
Sequential(
  (0): ToOneHot()
  (1): Linear(in_features=4, out_features=5, bias=True)
)
</code></pre>

username: Samuele_Bolotta
cooked: <p>Brilliant, thank you!</p>
What it means when model is unable to overfit small batch of examples, even though it is not small?

username: The_Snail
cooked: <p>I am trying to train RNN model to classify sentences into 4 classes, but it doesn’t seem to work. I tried to overfit 4 examples (blue line) which worked, but even as little as 8 examples (red line) is not working, let alone the whole dataset.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38f416dcea766803676e08ac370f657ab3a2d81c.png" data-download-href="https://discuss.pytorch.org/uploads/default/38f416dcea766803676e08ac370f657ab3a2d81c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38f416dcea766803676e08ac370f657ab3a2d81c_2_302x500.png" alt="image" data-base62-sha1="87PHekwC4tCyuwhHu46MO5WHZJq" width="302" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38f416dcea766803676e08ac370f657ab3a2d81c_2_302x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/8/38f416dcea766803676e08ac370f657ab3a2d81c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/8/38f416dcea766803676e08ac370f657ab3a2d81c.png 2x" data-dominant-color="F9F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">437×723 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I tried different learning rates and sizes of hidden_size but it doesn’t seem to help, what am I missing? I know that if the model is not able to overfit small batch it means the capacity should be increased but in this case increasing capacity has no effect.<br>
The architecture is as follows:</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size=1, hidden_size=256, num_classes=4):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        #x=[batch_size, sequence_length]
        x = x.unsqueeze(-1)  #x=[batch_size, sequence_length, 1] 
        _, h_n = self.rnn(x)  #h_n=[1, batch_size, hidden_size]
        h_n = h_n.squeeze(0)
        out = self.fc(h_n)  #out=[batch_size, num_classes]
        return out
</code></pre>

username: The_Snail
cooked: <p>The issue was due to the vanishing gradient, it got resolved by using batch normalization.</p>

username: Andre_Amaral_IST
cooked: <p>That can be the problem also <a href="https://discuss.pytorch.org/t/lstm-different-results/151311">here</a>?</p>
Index to matrix transform

username: Mohammadreza_Nazari
cooked: <p>I am wondering what is the most efficient way of converting a list of indices into a matrix. For example, I want to convert<br>
ind : [[1,3],[1,2,3]]<br>
to the form of<br>
torch.tensor([[0,1,0,1],[0,1,1,1]])</p>
<p>Thanks</p>

username: Hong
cooked: <p>You can do something like:</p>
<pre><code class="lang-auto">a = [[1,3],[1,2,3]]
tensors = []
for l in a:
    tensors.append(torch.zeros(4).scatter_(0, torch.tensor(l), 1))
result = torch.stack(tensors, 0)
</code></pre>

username: Mohammadreza_Nazari
cooked: <p>Thanks, this works. I was wondering whether there is more efficient pytorch function which can do it without the for loop. It seems that not.</p>

username: Konstantin_Burlachen
cooked: <p>I have created a code snippet when you may want to calculate modified Cross Entropy loss with sum reduction where you can weight samples, not classes.</p>
<p>It seems that 2-dim tensors can be indexed by a tuple of two elements - where first element is denoted for indicies of elements from first axis, and second list or iteratable element is denoted to index elements from second axis.</p>
<pre><code class="lang-auto">#!/usr/bin/env python3

import torch
import torch.nn as nn

# input.shape[0] -- number of samples
# input.shape[1] -- number of classes with linear scores for each class

loss = nn.CrossEntropyLoss(reduction="sum")
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)
output = loss(input[0:2,:], target[0:2])

e = input.exp()
eSum = torch.sum(e, dim = 1)
eSumRepeat = eSum.repeat((input.shape[1],1)).T
eSumNomralized = -torch.log(e/eSumRepeat)

mask = torch.zeros(eSumNomralized.shape) 
mask[range(input.shape[0]),target] = 1.0

weight=torch.Tensor([1.0,1.0,0.0])
weightRepeat = weight.repeat((input.shape[1],1)).T

loss = (eSumNomralized * mask * weightRepeat).sum()

print(output, loss)

</code></pre>
<p>p.s. Cross-Entropy in PyTorch terminology means usual Cross-Entropy CE(P,Q) with two extra assumptions:</p>
<ul>
<li>Before computing CE the Q is not necessary p.m.f. but it’s arbitrarily scores that are converted into probability simples with typical symmetric logistic transformation.</li>
<li>And CE in PyTorch assume that P is discrete delta function, where in signal processing and Machine Learning it is denoted as one-hot vector.</li>
</ul>
Is overfitting a good idea for testing and debuging?

username: knoriy
cooked: <p>Hi,</p>
<p>I’m training a model on text-image pairs. I wondered if overfitting to a small subset of your training data is a valid method of knowing if your model will perform the task?</p>
<p>Thank you in advance.</p>

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>My technique when training a model is to overfit it to the train set to see if my model is capable enough to learn the data. However, after I see it, I will try to mitigate the overfitting effect to make my model more general to unseen data.</p>

username: knoriy
cooked: <p>Hi <a class="mention" href="/u/andre_amaral_ist">@Andre_Amaral_IST</a></p>
<p>Thank you for your answer.</p>
<p>Do you train your model on the entire training set or a single batch?</p>
<p>I’ve trained my model on a single batch, and loss shows that the model converges, but when evaluating the model on the same data as it was trained on, it shows nothing significant (my model is not able to classify any labels)</p>

username: Andre_Amaral_IST
cooked: <p>In my case since I have 90 million of data, I train my model using batches. To see if the model is learning the training data I evaluate the model using the train data and if it is not good enough I change the hyperparameters unit it is perfect. After that, I use the test data and check whether exists or not the overfit.</p>
<p>In your case it seems that your model is not learning at all, it might be a problem of the model structure (layers or activation functions) not being complex enough for your data.<br>
The other problem might be the hyperparameters, such as the learning rate, number of neurons, etc…</p>

username: knoriy
cooked: <p>Thank you, that’s insightful. I’ll try modifying the hyperparameters.</p>
<p>I didn’t understand if you train your model on the full 90M or a subset (e.g. 1000 samples) to overfit?</p>
<p>I’ve only tested it on a single batch on 16 text-image pairs.</p>

username: Andre_Amaral_IST
cooked: <p>I feed the model with 90M of data, however, it is fed in batches of 10.000, otherwise, my computer would not be capable to do it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>16 pairs seems a bit low, but my suggestion is for you o test the hyperparameters and see if with that only 16 examples your model is capable to learn something.</p>

username: knoriy
cooked: <p>Thanks so much for your advice. I’ll give that a try <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Andre_Amaral_IST
cooked: <p>You’re welcome, if the model keeps not learning just use this post and the community and I will try to help you!<br>
Keep up the good work!</p>
How to combine (results) of multiple pytorch models?

username: elan1337
cooked: <p>Hello pytorch-community,</p>
<p>in pytorch (&amp; yolov5) i was able to train two models (one detects cars, the other detects bikes).</p>
<p>I am loading two different models (<a href="https://last.pt" rel="noopener nofollow ugc">last.pt</a> files) with the following commands:</p>
<pre><code class="lang-auto">model_c = torch.hub.load('ultralytics/yolov5', 'custom', path='/last.pt', force_reload=True)

model_b = torch.hub.load('ultralytics/yolov5', 'custom', path='/last.pt', force_reload=True)
</code></pre>
<p>I use an image for the model to detect cars or bikes in:</p>
<p><code>img = os.path.join('data', 'images', 'traffic.jpg')</code></p>
<p>The results command can show the model detection for cars:</p>
<pre><code class="lang-auto">results = model_c(img)

results.print()
</code></pre>
<p>…<strong>or</strong> bikes:</p>
<pre><code class="lang-auto">results = model_b(img)

results.print()
</code></pre>
<p><em>Question 1:</em></p>
<p><strong>How can the results of cars and bikes detections be shown in the same image at the same time?</strong></p>
<p><em>Question 2:</em></p>
<p><strong>Is there a way to combine two model pt.files into one?</strong></p>
<p>Thank you very much in advance.</p>

username: azhanmohammed
cooked: <p>It is better if you train a new model which predicts the two classes at the same time. As for your questions, they can be solved in the following manner.</p>
<ol>
<li>Viewing the results of cars and bike detections in same image at the same time.<br>
You can first pass the image through model 1 and detect cars, and then pass the output image through the second model, this will detect the bikes.</li>
<li>I have never seen if there is a way to combine the two model files in to one. A much better option would be to train a new network which performs both the classifications at the same time.</li>
</ol>
LSTM Results why?

username: Andre_Amaral_IST
cooked: <p>Hey,<br>
Can the num_works make the model perform worst? Or it does not affect the performance but only the computational speed?</p>
<p>Regards<br>
Andre</p>

username: Michal_Bogacz
cooked: <p>Hello,<br>
No, it only defines the number of sub-processes which will be run in order to load data.</p>

username: Andre_Amaral_IST
cooked: <p>So what might be causing <a href="https://discuss.pytorch.org/t/lstm-different-results/151311">these results</a>? Do you have any idea?</p>
Deploying a saved Pytorch Deep Learning model

username: Lohith_Kumaar
cooked: <p>Hey Guys,<br>
I am working on image classification problem with my custom dataset of plant disease and I want to deploy it a web application but unable to find a proper website or video.<br>
I am unable to know how to deploy a saved model onto the web.<br>
Can anyone help me deploy my trained model to the web?<br>
I have the model from model zoo and saved it after training.</p>

username: ptrblck
cooked: <p>You could check out <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html">this tutorial</a> but I’m unsure how useful this would be for your use case. Also, <a href="https://pytorch.org/serve/"><code>torchserve</code></a> might be interesting for your use case.</p>

username: azhanmohammed
cooked: <p>Hi <a class="mention" href="/u/lohith_kumaar">@Lohith_Kumaar</a> I have written a detailed tutorial for serving PyTorch models using Flask, please refer to the <a href="https://github.com/sheikhazhanmohammed/Transfer-Learning-Model-hosted-on-Heroku-using-React-Flask" rel="noopener nofollow ugc">repository</a> here. Let me know if you need any more help with the serving part. Would love to help.</p>
Torch.fx.wrap for custom cuda kernel

username: fyzirder
cooked: <p>Hello,</p>
<p>I have my own quantization operator written in cuda (according to <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 2.1.1+cu121 documentation</a>) and it works fine. But I need to use ASP (automatic sparsity package) which uses torch.fx, which throws error at my quantization operator</p>
<blockquote>
<p>TypeError: fpquantizer_(): incompatible function arguments. The following argument types are supported:<br>
1. (arg0: at::Tensor) → None<br>
Invoked with: Proxy(getattr_1)</p>
</blockquote>
<p>So far as I understood from torch.fx manual(hopefully correctly), I need to decorate my operator with torch.fx.wrap.</p>
<blockquote></blockquote>
<p>PYBIND11_MODULE(fpquantize, m)<br>
{<br>
py::object fx_wrap = py::module::import(“torch.fx”).attr(“wrap”);<br>
m.def(“fpquantizer_”, &amp;fp_conversion_, “inplace quantization and storing in a float array”);<br>
m.attr(“fpquantizer_”) = fx_wrap(m.attr(“fpquantizer_”));<br>
}</p>
<p>which seems to do job of decorating operator, but torch.fx.wrap have problem with c++ function itself, it doesn’t have</p>
<pre><code class="lang-auto">__code__
</code></pre>
<p>attribute and also</p>
<pre><code class="lang-auto">isinstance(fpquantize.fpquantizer, types.FunctionType)
</code></pre>
<p>is False, so torch.fx.wrap throws an error</p>
<pre><code class="lang-auto">ImportError: AssertionError: fn_or_name must be a global function or string name
</code></pre>
<p>Is there any way how to handle it? Is my approach correct?</p>
<p>Thanks a lot</p>

username: fyzirder
cooked: <p>The problem was that using just PYBIND11 doesn’t make c++ function torch operator. One need to use procedure from <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops" class="inline-onebox" rel="noopener nofollow ugc">Extending TorchScript with Custom C++ Operators — PyTorch Tutorials 2.1.1+cu121 documentation</a> . i.e.</p>
<blockquote>
<p>TORCH_LIBRARY(fpquantize, m)<br>
{<br>
m.def(“fpquantizer_”, &amp;fp_conversion_);<br>
m.impl(“fpquantizer_”, torch::kCUDA, &amp;fp_conversion_);<br>
}</p>
</blockquote>