akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use a 1D tensor as an index to slice a 2D tensor?

username: ddavo
cooked: <p>Normally, you can slice a 2D tensor like this <code>slice = t[:, :k]</code> where k is an integer. Is it possible to do something like this but with k being a 1-dimensional vector of integers with the number of items that I want to obtain for each row?</p>
<p>Masking the items with 0’s or NaN would also be fine.</p>
<p>For example:</p>
<pre><code class="lang-python">k = torch.Tensor([1,3,2])
t = torch.Tensor([1,1,1], [2,2,2], [3,3,3])

# perform some operations and the result should be
# 1 - -
# 2 2 2
# 3 3 -
</code></pre>

username: InnovArul
cooked: <p>I could think of the following way:</p>
<pre><code class="lang-auto">k = torch.Tensor([1,3,2])
t = torch.Tensor([[1,1,1], [2,2,2], [3,3,3]])

h, w = t.shape

# create a mask of ones with size h, w+1
mask = torch.ones(h, w+1)

# set other elements to 0, depending on length of each row
mask[torch.arange(h), k.long()] = 0.
mask = mask.cumprod(dim=1)

# multiply the mask
result = t * mask[:, :-1]
print(result)
</code></pre>
<pre><code class="lang-auto"> tensor([[1., 0., 0.],
        [2., 2., 2.],
        [3., 3., 0.]])
</code></pre>

username: ddavo
cooked: <p>I ended up doing something similar. But, instead of multiplying, I just did <code>t[mask] = 0</code></p>
Is beta a trainable parameter in nn.Softplus Layer?

username: Tony_Gracious
cooked: <p>In the source code, it is shown as a constant, as follows.</p>
<pre><code>__constants__ = ['beta', 'threshold']
beta: int
threshold: int

def __init__(self, beta: int = 1, threshold: int = 20) -&gt; None:
    super().__init__()
    self.beta = beta
    self.threshold = threshold
</code></pre>

username: Paplham
cooked: <p>Hello Tony,<br>
Beta is indeed a constant and is not trainable in the standard definition.</p>
<p>I think that this <a href="https://discuss.pytorch.org/t/learnable-parameter-in-softplus/60371">post</a> answers your question. You can define a custom operation where the beta is learnable.</p>
Unit tests with DistributedDataParallel

username: thorstenwagner
cooked: <p>Hi,</p>
<p>when I run unit tests with DistributedDataParallel components I always end up with the following exception:</p>
<pre><code class="lang-auto">[W socket.cpp:436] [c10d] The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use).
[W socket.cpp:436] [c10d] The server socket has failed to bind to 0.0.0.0:29500 (errno: 98 - Address already in use).
[E socket.cpp:472] [c10d] The server socket has failed to listen on any local network address.
</code></pre>
<p>This is probably the relevant piece of code:</p>
<pre><code class="lang-auto">os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '29500'
tdist.init_process_group(backend='gloo', rank=rank, world_size=world_size)
</code></pre>
<p>I tried different ports, but I always run into that issue. Outside unittests it works.</p>
<p>Any ideas?</p>
<p>Best,<br>
Thorsten</p>
<p>PS: I’m using the latest nightly</p>

username: wconstab
cooked: <p>this can happen if the port is already in use.  Often the case is that you have launched the same script previously and failed to kill it completely, leaving the port occupied.  Also sometimes the case is that another user is using the port.  In any case, you could use <code>ps -ef | grep python</code> or <code>htop</code> to view your processes and look for any command lines from your previous runs, and kill them.  Or, use netstat and figure out what is using the port.</p>
Loss and accuracy spikes when shuffling dataset per epoch

username: DIMENSION
cooked: <p>Hello everyone,</p>
<p>I’m currently working on a simple ImageNet classifier and I’ve encountered an issue.</p>
<p>I’ve been following the standard practice of shuffling my data at the beginning of each epoch during the training process, but I’ve noticed some strange spikes in both the loss and accuracy (not in validation).</p>
<p>The strange part is, when I deviate from this standard practice and instead randomly select images with replacement (so no epochs), both my loss and accuracy graphs become stable:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfa33dabd6012631d4d9876acf9b68fc4e1df0d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/bfa33dabd6012631d4d9876acf9b68fc4e1df0d2" title="erhgyzsdrfsefsefz"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bfa33dabd6012631d4d9876acf9b68fc4e1df0d2_2_442x500.png" alt="erhgyzsdrfsefsefz" data-base62-sha1="rliWd2UU7NsXFsRT6iuzLFxbVJM" width="442" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bfa33dabd6012631d4d9876acf9b68fc4e1df0d2_2_442x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bfa33dabd6012631d4d9876acf9b68fc4e1df0d2_2_663x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bfa33dabd6012631d4d9876acf9b68fc4e1df0d2_2_884x1000.png 2x" data-dominant-color="FBFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">erhgyzsdrfsefsefz</span><span class="informations">1608×1818 215 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any help or insights would be greatly appreciated, thank you!</p>
PyTorch DataLoader Returning List Instead of Tuple

username: Imahn
cooked: <p>Hi,</p>
<p>My issue is similar to <a href="https://discuss.pytorch.org/t/calling-enumerate-on-dataloader-gives-list-instead-of-tensor/124783">this solved one</a>, but I am wondering about why the DataLoader returns a list instead of a tuple.</p>
<p>Code for reproducing (PyTorch 2.0.1):</p>
<pre><code class="lang-python">from typing import Any, Optional

import torch 
from torch.utils.data import Dataset, DataLoader 

class ImageDataset(Dataset): 
    def __init__(self, imgs: torch.Tensor, targets: torch.Tensor, img_transform: Optional[Any] = None) -&gt; None:
        """Image dataset class.""" 
        super().__init__()
        self.imgs = imgs
        self.targets = targets
        self.img_transform = img_transform
    
    def __len__(self) -&gt; int:
        assert len(self.imgs) == len(self.targets), "Number of images and targets must be equal"
        return len(self.imgs)

    def __getitem__(self, idx: int) -&gt; Tuple[torch.Tensor, torch.Tensor]:
        img = self.imgs[idx]
        if self.img_transform is not None:
            img = self.img_transform(img)

        target = self.targets[idx]

        return img, target


if __name__ == "__main__": 
    imgs = torch.rand(256, 3, 32, 32)
    targets = torch.randint(0, 10, (256,))
    dataset = ImageDataset(imgs, targets)
    dataset[0]
</code></pre>
<p>The type of <code>dataset[0]</code> is <code>tuple</code>, just as I would expect, but putting the dataset into a DataLoader yields:</p>
<pre><code class="lang-python">data_loader = DataLoader(dataset, batch_size=1, shuffle=False)
for idx, data in enumerate(data_loader):
    print(data)
    break 
</code></pre>
<p>Why is the type of <code>data</code> a <code>list</code> now? I would still expect a tuple…</p>
<p>Best,<br>
Imahn</p>

username: ptrblck
cooked: <p>The default <code>collate_fn</code> seems to return <code>list</code>s for <code>Sequence</code>s as seen <a href="https://github.com/pytorch/pytorch/blob/2156f0434c51e492316400f27c1733be799390cb/torch/utils/data/_utils/collate.py#L133C10-L148">here</a>. If you want to return a <code>tuple</code> you might need to implement a custom <code>collate_fn</code>.</p>
Initialized weights method of model in pytroch

username: avishai_weizman
cooked: <p>what is the initialized weights method of a model in Pytroch?</p>

username: ptrblck
cooked: <p>Each module has its own <code>reset_parameters</code> method responsible to reset and initialize the parameters. You can find this method for e.g. <code>nn.Linear</code> layers <a href="https://github.com/pytorch/pytorch/blob/2156f0434c51e492316400f27c1733be799390cb/torch/nn/modules/linear.py#L103-L111">here</a>.</p>
Repeat with creating new last dimensions in a nice way

username: vadimkantorov
cooked: <p>How do I repeat along last dimensions?</p>
<p>e.g. from a <code>torch.rand(4)</code> I would like to obtain a tensor of shape [4, 100, 5]</p>
<p>E.g. <code>torch.rand(4).repeat(100, 5, 1).movedim(-1, 0).contiguous()</code> or <code>torch.rand(4)[..., None, None].expand(4, 100, 5).contiguous()</code>  works, but I wonder if there is a more colloquial version?</p>

username: tom
cooked: <p><code>torch.randn(4, 1, 1).repeat((1, 100, 5))</code><br>
I’d use the expand and contiguous version if only because repeat used to not be very efficient, so <code>torch.randn(4, 1, 1).expand(-1, 100, 5).contiguous()</code> if you need the contiguous.</p>
<p>Best regards</p>
<p>Thomas</p>

username: vadimkantorov
cooked: <p>One problem is that if <code>[100, 5]</code> can be an arbtirary shape, we need to unsqueeze a dynamic number of times that hurts…</p>

username: tom
cooked: <p>You mean <code>torch.randn(4, *[1 for _ in s]).expand(-1, *s)</code> ?</p>

username: vadimkantorov
cooked: <p>Yeah, this kind. Thanks for this snippet.</p>
Can a tokenizer and then model be trained on same data?

username: Tornike
cooked: <p>Hi, my question is can a tokenizer and then for example NLP summarization or something be trained on the same data?</p>

username: Tornike
cooked: <p>One more question: how can i train a language that has non latin alphabet ? when i train the tokenizer can not read the sentence. it returns something like this :</p>
<blockquote>
<p>[‘
’, ‘áĥ¥’, ‘áĥĲáĥłáĥĹáĥ£áĥļáĥĺ’, ‘Ġáĥĺáĥ¡’, ‘áĥ¬áĥĲáĥķáĥļ’, ‘áĥĶ’, ‘?’, ‘
’]</p>
</blockquote>
<hr>
<p>should i map each char to a number manually ? like a:0 b:1 and etc… but a and b in my language is ა, ბ.</p>
<p>Talking about the Huggingface Tokenizer</p>

username: Miro_Mlynarik
cooked: <p>Yes, you can train tokenizer and model on the same data.</p>
Libtorch's LSTM input shape

username: Jesus
cooked: <p>Hello everyone. Please someone to explaine me the shape of LSTM input " tensor of shape (L,Hin) for unbatched input, (L,N,Hin) when <code>batch_first=False</code> or (N,L,Hin​) when <code>batch_first=True</code> containing the features of the input sequence." I want to know the difference between these two shapes: (L,N,Hin) , (N,L,Hin​).</p>

username: Miro_Mlynarik
cooked: <p>They have just switched dimensions between L and N. See documentation:</p>
<p><em><strong>batch_first</strong> – If <code>True</code>, then the input and output tensors are provided as (batch, seq, feature) instead of (seq, batch, feature).</em></p>

username: Jesus
cooked: <p>Thank for the reply. I know that they just switched the dimensions. If i want to fed a sequence to a LSTM which one should i use concretely ?</p>

username: Miro_Mlynarik
cooked: <p>You can use any of those two variants. You just have to align your inputs with <code>batch_first</code> setup.</p>

username: Jesus
cooked: <p>OK, I see. Thank you</p>
Create a Feature extractor from pretrained model and export to ONNX

username: Marco_Franzoi
cooked: <p>Hello!</p>
<p>I need a suggestion from the community. I’ve a pretrained classification model (let’s take resnet50 as a reference). I want use such pretrained model as feature extractor, doing some stuff with extracted features, and export it as ONNX.</p>
<p>I’ve tried several approaches, but all of them brutally failed.</p>
<p>Code example:</p>
<pre><code class="lang-auto">
weights = torchvision.models.ResNet50_Weights.IMAGENET1K_V2
model = torchvision.models.resnet50(weights=weights)
preprocess = weights.transforms()
img = torch.zeros(1, 3, 512, 1024).to("cuda")


class prova(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model.to("cuda")
        self.outputs = ()

        def hook(module, input, output):
            outputs = self.outputs + (output.detach(),)

        self.model.layer1.register_forward_hook(hook)

    def forward(self, x):
        with torch.no_grad():
            print("| embedding extract | test |")
            result = self.model(x)
            embedding_vectors = self.outputs[0]
        return (result,) + (embedding_vectors,)


feat_extractor = prova(model)

feat_extractor(img)

torch.onnx.export(
    feat_extractor,  # --dynamic only compatible with cpu
    img,
    "prova.onnx",
    verbose=False,
    opset_version=13,
    do_constant_folding=True,  # WARNING: DNN inference with torch&gt;=1.12 may require do_constant_folding=False
    input_names=["images"],
)
</code></pre>
<p>Fail with:<br>
<code>[ CUDAFloatType{1,256,128,256} ]) of traced region did not have observable data dependence with trace inputs; this probably indicates your program cannot be understood by the tracer.</code></p>
<p>Do you have any suggestion on how to reach the goal? It would be important not modifying the forward pass of the original model as it is embedded in a complex flow.</p>
Step by step understanding approximate joint training method

username: saeed_ni
cooked: <p>i don’t understand exactly approximate joint training method.<br>
i know RPN and detector merged as a one network during training.<br>
the forward path is started pre trained conv network and pass from RPN and finally arrives to fast rcnn layers. loss is computed :</p>
<blockquote>
<p>RPN classification loss + RPN regression loss + Detection classification loss + Detection bounding-box regression loss.</p>
</blockquote>
<p>but where is it from the backpropagation path? is it from detector and RPN and finally pretrained convnet?<br>
in this case how derivation performed in decoder section in RPN? offcets produced with 1x1 reg-conv layer in RPN is translated to proposals in decoder.</p>
Conversion of my trained model.pt into torchscript file

username: Arpan_Gyawali
cooked: <p>I trained a model for medical imaging segmentation task using Monai label and saved it as model.pt.<br>
Now when i am trying to deploy it it says i need a torchscript file.<br>
error: monai.deploy.exceptions.ItemNotExistsError: A predictor of the model is not set.<br>
I tried tracing and scripting but couldn’t do it.<br>
Can someone suggest me how to convert my model.pt into torchscript file.<br>
Any help would be very appreciable.</p>

username: Paplham
cooked: <p>Please, refer to this <a href="https://discuss.pytorch.org/t/conversion-of-pytorch-pt-model-file-into-torchscript-ts-file/185671">topic</a>.</p>
Can't trace model using torch.jit.trace()

username: Ahmed_Louati
cooked: <p>I tried to script a model using <strong>torch.jit.trace()</strong> but I always have this error:</p>
<pre><code class="lang-auto">TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!
</code></pre>
<p>I used model.eval() like suggested in other similar problems but the problem persists.</p>
<p>I am using :</p>
<ul>
<li>torch 2.0.0+cu117</li>
<li>Python 3.10.6</li>
</ul>

username: Paplham
cooked: <p>Hello Ahmed,<br>
please provide us with your code, otherwise, it is impossible to help.</p>
<p>Which function are you trying to trace?</p>
Batched matrix multiplication speed

username: honkamj
cooked: <p>I am doing a matrix multiplication of two relatively large matrices and changing the batch size to 2 significantly increases the execution time (20 times). I am wondering why that is, and if something can be done about it. Here are the two examples:</p>
<pre><code class="lang-auto">start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
matrix = torch.randn(1, 2, 10000000, device='cuda:0')
start_event.record()
torch.bmm(matrix, matrix.transpose(1, 2))
end_event.record()
torch.cuda.synchronize()
print(f"Time elapsed: {start_event.elapsed_time(end_event)}")

# Time elapsed: 1.6721919775009155
</code></pre>
<pre><code class="lang-auto">start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
matrix = torch.randn(2, 2, 10000000, device='cuda:0')
start_event.record()
torch.bmm(matrix, matrix.transpose(1, 2))
end_event.record()
torch.cuda.synchronize()
print(f"Time elapsed: {start_event.elapsed_time(end_event)}")

# Time elapsed: 34.95014572143555
</code></pre>
<p>Pytorch version: 1.13.1<br>
GPU: NVIDIA GeForce RTX 3090 24GB</p>
How to ensemble multiple jit models?

username: jerrybai1995
cooked: <p>I’m trying to deploy an ensemble model that consists of 5 base models. Sources like <a href="https://pytorch.org/docs/master/func.migrating.html" rel="noopener nofollow ugc">this</a> and <a href="https://pytorch.org/functorch/stable/notebooks/ensembling.html" rel="noopener nofollow ugc">this</a> provide great examples of building ensemble PyTorch models, but is there a way for us to either:</p>
<ul>
<li>JIT (i.e., jit script and optimize_for_inference) an ensemble model (that is based on vmap); or</li>
<li>ensemble multiple JIT models?</li>
</ul>
<p>I tried the former, which led to an error saying “Compiled functions can’t take variable number of arguments or use keyword-only arguments with defaults”. The latter approach led to an error saying attribute “self.training” does not exist in these jit scripted models.</p>
<p>Any help is appreciated!</p>

username: Paplham
cooked: <p>Hi jerrybai1995, could you provide us with the source code which you tried to use for the ensemble? What models are you trying to ensemble? Are they already JIT scripted, are they instances of <code>nn.Module</code>, or something else?</p>
<p>I personally ensemble JIT scripted models in the following way:</p>
<pre><code class="lang-auto">class Ensemble(torch.nn.Module):
    """
    Ensemble model, wrapping multiple sub-models in separate torch jit threads.
    Allows for parallel computation of the ensemble forward passes. 
    """

    def __init__(self, models: List[torch.nn.Module]):
        super().__init__()
        self.models = torch.nn.ModuleList(models)

    def forward(self, x: torch.Tensor) -&gt; Dict[str, List[torch.Tensor]]:
        futures = [torch.jit.fork(model, x) for model in self.models]
        results = [torch.jit.wait(fut) for fut in futures]
        result = torch.mean(torch.stack(results), dim=0)
        return result
</code></pre>
<p>and I instantiate the ensemble with already JIT scripted models, e.g.,</p>
<pre><code class="lang-auto">    models = []
    for m in ['model_1.pt', 'model_2.pt', 'model_3.pt']:
        model = torch.jit.load(m)
        model.eval()
        models.append(model)


    ensemble = Ensemble(models)

    # Script and save ensemble
    scripted = torch.jit.script(ensemble)
    torch.jit.save(scripted, 'ensemble.pt')
</code></pre>
Debugging variables

username: secrettoad
cooked: <p>Hello,</p>
<p>I am vectorizing a set of documents using a torch.nn module that is part of the huggingface transformers library and getting an error that asks me to set CUDA_LAUNCH_BLOCKING and TORCH_USE_CUDA_DSA  in order to get a coherent stack trace.</p>
<p>I cannot find anywhere where and when exactly these variables must be set in order to enable debugging. Can CUDA_LAUNCH_BLOCKING be set after torch installation but before import?<br>
What value must TORCH_USE_CUDA_DSA be set to and can it also be done post-installation?</p>
<p>Thank you in advance for your help.</p>

username: ptrblck
cooked: <p>These are environment variables, which should be set before you launch your Python script. You could also try to set them inside your code via the <code>os</code> package, but I would strongly recommend to set them in your terminal via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>.</p>

username: secrettoad
cooked: <p>Thanks for your reply. Am I understanding correctly that you are saying TORCH_USE_CUDA_DSA can be set after CUDA installation and torch installation? It can be set just before the python process initiates?</p>
<p>Also, should it be set to 1? or true?</p>
<p>Thanks again</p>

username: ptrblck
cooked: <p>No, <code>TORCH_USE_CUDA_DSA</code> is an env variable used during the build and irrelevant during the runtime. To enable it you would need to build PyTorch from source with this env variable.</p>

username: secrettoad
cooked: <p>Okay, makes sense. Thank you.</p>
<p>Last question: should the env var be set to 1 or some other value?</p>

username: ptrblck
cooked: <p>Both env variables should be set to <code>1</code> to enable them.<br>
From my previous post:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
Transforms.Resize()

username: StevenAzy
cooked: <p>After adding the transforms.Resize(), Why the training of model gets slowly?</p>

username: ptrblck
cooked: <p><code>Resize</code> is executed on the CPU. If you see a significant slowdown, your CPU might suffer from this additional workload although I would assume the operation isn’t particularly expensive (but might be wrong).</p>

username: pytorch_fun
cooked: <p>Is <code>Resize</code> always executed on the CPU?  It seems common to create a list of transforms which includes a <code>Resize</code> and pass it to a method (e.g. torchvision.datasets.CIFAR10) to create a dataset.  Then pass the dataset to a <code>torch.utils.data.DataLoader</code>.  Lastly we iterate over the dataloader object and often move batches <code>.to(device)</code>.<br>
Seems like this paradigm always has the transforms running on CPU.  Is this what we normally want?  Is it not faster to run the transforms on the GPU?</p>

username: ptrblck
cooked: <p>It depends on your use case, but the standard approach is to use multiprocessing allowing the workers to load and process the next batch(es) while the GPU is busy training the model. If your GPU is underutilized you could check e.g. <a href="https://developer.nvidia.com/dali">DALI</a>, which also provides GPU loading and processing.</p>
How to run inference in parallel on a single GPU with a single copy of model?

username: ritwik.m07
cooked: <p>I have a relatively simple model. it is a classifier finetuned with a pretrained encoder from huggingface (transformers). It takes a text as input and produces a number between 0 to 1. We classify based on a threshold.</p>
<p>I trained it on multiple GPUs using DDP. But now I have a long list of examples (test_list) on which I need to run inference. I am aware of the method where I can use DDP again and divide the test_list onto multiple GPUs (<a href="https://discuss.pytorch.org/t/how-do-i-run-inference-in-parallel/126757/2">like this</a>). But downside of this method is that if I have n GPUs then I can partition the test_list into at max n partitions. Moreover, each GPU will have a copy of model running on them.</p>
<p>Another thing I can do is to divide the test_list into N (&gt;&gt;n) smaller lists and for each smaller list I can run inference using N bash shells. Like this (one command on each bash shell)</p>
<pre><code class="lang-auto">CUDA_V_D=0 python file.py --partition 1
CUDA_V_D=0 python file.py --partition 2
...
CUDA_V_D=0 python file.py --partition N
</code></pre>
<p>But this method will also create N copies of model on the single GPU.</p>
<p>Is there a way by which I can create a single copy of model on a single GPU but run inference in parallel?</p>

username: tom
cooked: <p>You don’t really want to this if you don’t have to. It is better to do async processing + request batching to run the model. This is because sharing the GPU can be rather costly. If you use the torch jit (as long as it works, alas), you also escape the GIL that way. We do have a demo of that in Part III of our book.</p>
<p>Best regards</p>
<p>Thomas</p>