akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use resized_crop_mask?

deeperlearner: <p>Hi,<br>
I want to randomly adding rectangle masks to image by using transform for data augmentation.<br>
I found <a href="https://github.com/pytorch/vision/tree/main/torchvision/transforms/v2" rel="noopener nofollow ugc">transform.v2</a> has resized_crop_mask. But I can’t upgrade my torchvision for some reason. I want directly implement the function. How can I do it? Thanks a lot!</p>
<pre><code class="lang-auto">def resized_crop_mask():
    ...
    return

class MyDataset(Dataset):
    def __init__(self, root_dir):
        super(MyDataset, self).__init__()
        self.transform = transforms.Compose(
            [transforms.ToPILImage(),
             transforms.RandomHorizontalFlip(),
             resized_crop_mask(),
             transforms.ToTensor(),
             transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
             ])
        self.root_dir = root_dir
        self.img...

    def __getitem__(self, index):
        sample = self.img[index]
        label...
        if self.transform is not None:
            sample = self.transform(sample)
        return sample, label

    def __len__(self):
        return len(self.img)
</code></pre>
<p>This is ChatGPT <a href="https://chat.openai.com/share/f1f6d834-6d56-430f-8d7f-66827919e991" rel="noopener nofollow ugc">suggestion</a>. But not sure if it is well coded.</p>

ptrblck: <aside class="quote no-group" data-username="deeperlearner" data-post="1" data-topic="202489">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeperlearner/48/41843_2.png" class="avatar"> deeperlearner:</div>
<blockquote>
<p>This is ChatGPT <a href="https://chat.openai.com/share/f1f6d834-6d56-430f-8d7f-66827919e991">suggestion</a>. But not sure if it is well coded.</p>
</blockquote>
</aside>
<p>The posted code does not show any implementation, but just a template how transformations are used in a custom <code>Dataset</code>.</p>
<p>You can reuse the <code>torchvision</code> implementation from <a href="https://github.com/pytorch/vision/blob/947ae1dc71867f28021d5bc0ff3a19c249236e2a/torchvision/transforms/v2/functional/_geometry.py#L2169-L2178">here</a> by copy/pasting all needed parts since you cannot update for some reason.</p>
What is cudagraphs backend for torch compiler?

schinmayee: <p>Torch.compile page <a href="https://pytorch.org/docs/stable/torch.compiler.html" rel="noopener nofollow ugc">here</a> mentions a cudagraphs backend. If I understand correctly, inductor also uses cudagraphs and aotautograd: <a href="https://hud.pytorch.org/benchmark/compilers" class="inline-onebox" rel="noopener nofollow ugc">PyTorch CI HUD</a>. What is the cudagraphs backend and how is it different from the inductor backend with cudagraphs?</p>

schinmayee: <p>Is it that inductor backend with cudagraphs emits triton code but just cudagraphs doesn’t to triton on gpus? What code does the cudagraphs backend emit? TORCH_COMPILE_DEBUG doesn’t show an output file for cudagraphs backend.</p>

ptrblck: <p>A similar question was asked <a href="https://discuss.pytorch.org/t/memory-footprint-of-eager-mode-vs-torch-compiled-default-with-cuda-graph-and-cuda-graph-dynamic/199958">here</a> but wasn’t clarified yet.</p>
[Off-topic] Accelerate Model Training with PyTorch 2.x

Maicon_Melo_Alves: <p>Hello everyone! My name is Maicon Melo Alves and I’m a High Performance Computing (HPC) system analyst specialized in AI workloads.</p>
<p>I would like to announce that my book “Accelerate Model Training with PyTorch 2.X: Build more accurate models by boosting the model training process” was recently launched by Packt.</p>
<p>This book is for intermediate-level data scientists, engineers, and developers who want to know how to use PyTorch to accelerate the training process of their machine-learning models.</p>
<p>You can get it at Amazon on this link: <a href="https://a.co/d/iTADiEF" rel="noopener nofollow ugc">https://a.co/d/iTADiEF</a></p>
<p>If you think this book can help other professionals, please share this post with your community! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<p>Thank you very much!</p>
How to create a tensor based on another one

VictorVidigal: <p>I’m studying IA using PyTorch and implementing some toy examples.<br>
First, I created a one-dimensional tensor (X) and a second tensor (y), derived from the first one:</p>
<pre data-code-wrap="python"><code class="lang-python">X = torch.arange(0, 100, 1.0).unsqueeze(dim=1)
y = X * 2
</code></pre>
<p>So I have something like</p>
<pre><code class="lang-auto">X = tensor([[0.], [1.], [2.], [3.], [4.], [5.], ...
y = tensor([[ 0.], [ 2.], [ 4.], [ 6.], [ 8.], [10.], ...
</code></pre>
<p>Then, I trained a model to predict y and it was working fine.</p>
<p>Now, I would like something different. The X will be 2D and y 1D. y is calculated by an operation in the elements of X:<br>
<code>If x[0] + x[1] &gt;0? y = 10: y -10 </code></p>
<pre><code class="lang-auto">X = tensor([[ 55.5348, -97.7608],
            [ 29.0493, -52.1908],
            [ 47.1722, -43.1151],
            [ 11.1242, -62.8652],
            [ 44.8067,  80.8335],...
y = tensor([[-10.], [-10.], [ 10.], [-10.], [ 10.],...
</code></pre>
<p>First question, Is it make sense in terms of Machine Learning?</p>
<p>Second one…<br>
I’m generating the tensors using numpy. Could I do it in a smarter way?</p>
<pre data-code-wrap="python"><code class="lang-python"># Criar X valores de entrada para testes
X_numpy = np.random.uniform(low=-100, high=100, size=(1000,2))
print("X", X_numpy)

#y_numpy = np.array([[ (n[0]+n[1]) &gt;= 0 ? 10:-10] for n in X_numpy])
y_numpy = np.empty(shape=[0, 1])
for n in X_numpy:
    if n[0] + n[1] &gt;= 0:
        y_numpy = np.append(y_numpy, [[10.]], axis=0)
    elif n[0] + n[1] &lt; 0:
        y_numpy = np.append(y_numpy, [[-10.]], axis=0)
</code></pre>
How to distribute sequence length instead of batch size?

weight_theta: <p>I am working on frames that are part of a video sequence, and was curious about whether torch has a method to distributed the sequence length instead of the batch size, similar to DDP ?</p>
<p>The primary reason is that I can only fit a batchsize of 32 on a single GPU for a still image object detection task. That is 32 images per batch <code>batchsize, image dims</code> . However since we are working on sequence we know have a sequence length dimensions. That is <code>batchsize, image dims, sequence length</code>.</p>
<p>I am assuming that I will be able to fit a single batch with a sequence length of 32 on to GPU. Now with DDP I can increase the effective total batchsize, however 32 frames per video for a video length of 420 frames leads to a loss of temporal information. I would like to increase the sequence length to 128 without running into CUDA memory allocation errors.</p>
<p>I am already using the following strategies to reduce memory allocation:</p>
<ol>
<li>mixed precision</li>
<li>checkpointing</li>
</ol>
<p>Is there away to distribute sequence length across GPUs or so ?</p>
How to use torch.load(f, map_location="cpu") within multiprocessing

KukumavMozolo: <p>I am trying to prefetch multiple serialized objects including but not limited to tensor object using multiprocessing.<br>
The objects are serialized with pickle, to avoid tensors beeing deserialized on gpu I have to use the custom Unpickler <code>CPU_Unpickler</code> that sets the map_location to ‘cpu’.</p>
<pre><code class="lang-auto">def data_gen(files: List[Bytes]):
    for file in files:
        result = get(file)
        yield result


def get(obj: Bytes) -&gt; Any:
    unpickler = CPU_Unpickler(obj)
    result = unpickler.load()
    return result


class CPU_Unpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module == "torch.storage" and name == "_load_from_bytes":
            def closing_bytes(b: bytes):
                with io.BytesIO(b) as f:
                    res = torch.load(f, map_location="cpu")
                return res
            return lambda b: closing_bytes(b)
        else:
            return super().find_class(module, name)

with ParallelGenerator(
        data_gen(files), max_lookahead=1
) as g:
    for data in g:
</code></pre>
<p>Here ParallelGenerator is a multiprocessing wrapper like(<a href="https://pypi.org/project/multiprocessing-generator/" class="inline-onebox" rel="noopener nofollow ugc">multiprocessing-generator · PyPI</a>) that Prefetches from the generator <code>data_gen</code> in the background so that what ever happens in the for loop that iterates over the generator doesnt have to wait for the results.</p>
<p>Unfortunatelly this does not work and produces the following cryptic errors:</p>
<pre><code class="lang-auto">  File ".../lib/python3.11/multiprocessing/queues.py", line 244, in _feed
  File ".../lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
  File ".../python3.11/site-packages/torch/multiprocessing/reductions.py", line 568, in reduce_storage
  File ".../python3.11/site-packages/torch/storage.py", line 304, in wrapper
  File ".../python3.11/site-packages/torch/storage.py", line 374, in _share_fd_cpu_
RuntimeError: unable to open shared memory object &lt;/torch_128493_2623044343_478&gt; in read-write mode: Too many open files (24)
</code></pre>
<p>I am quite confident this happens in the <code>result = unpickler.load()</code> call and i am guessing that it is related to <code>torch.load(f, map_location="cpu")</code> becouse of the error messages.<br>
So any clue to why this might happen or some better way of solving the problem to prefetch data that is heavily io-bound in batches like i am trying to do here?</p>

KukumavMozolo: <p>Found a workaround by concatenating all tensors into one, however this problem does not appear in a non multiprocessing context, so there seems to be an issue.</p>
Nan values appear during training of five folds

Noorulain_Islam: <p>I am facing this error when I train my model with five folds. If I individually train my data which is separated in third fold, my model training goes smoothly. When I train my model with five folds, sometimes this error occurs during evaluation appear in  third fold, sometimes it appears in second fold. Could anyone help to resolve this issue.</p>
<p>Traceback (most recent call last):<br>
File “/home/exp/src/main.py”, line 273, in <br>
train(model, optimizer, train_data, fold)<br>
File “/home/exp/src/main.py”, line 97, in train<br>
tst_auc, tst_aupr = test_epoch()<br>
File “/home/exp/src/main.py”, line 76, in test_epoch<br>
test_auc = roc_auc_score(y_true, y_score_test, average=‘macro’)<br>
File “/home/.conda/envs/fo/lib/python3.10/site-packages/sklearn/utils/_param_validation.py”, line 211, in wrapper<br>
return func(*args, **kwargs)<br>
File “/home/.conda/envs/fo/lib/python3.10/site-packages/sklearn/metrics/_ranking.py”, line 605, in roc_auc_score<br>
y_score = check_array(y_score, ensure_2d=False)<br>
File “/home/.conda/envs/fo/lib/python3.10/site-packages/sklearn/utils/validation.py”, line 959, in check_array<br>
_assert_all_finite(<br>
File “/home/.conda/envs/fo/lib/python3.10/site-packages/sklearn/utils/validation.py”, line 124, in _assert_all_finite<br>
_assert_all_finite_element_wise(<br>
File “/home/.conda/envs/fo/lib/python3.10/site-packages/sklearn/utils/validation.py”, line 173, in _assert_all_finite_element_wise<br>
raise ValueError(msg_err)<br>
ValueError: Input contains NaN.</p>
How can I change the slope of the sigmoid function?

songsong0425: <p>Greetings, I have a question about the slope of the sigmoid function.<br>
I’m trying to use the sigmoid function to convert the scores to the probability in the link prediction task.<br>
But most of the edges had more than 0.9 value after going through the sigmoid function.<br>
To deal with this problem, I tried to change the slope of the sigmoid, but I couldn’t find any parameter about it in the basic sigmoid function (Now I’m using it as <code>tensor.sigmoid()</code> form).<br>
<a href="https://pytorch.org/docs/stable/generated/torch.sigmoid.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.sigmoid.html</a></p>
<p>In this case, 1) is it a reasonable approach to change the slope of the sigmoid function? 2) are there any function to change scores to the probability (0~1)?<br>
Thank you for reading this question.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/songsong0425">@songsong0425</a>,</p>
<p>If you want to change the slope of the sigmoid function, you could just multiple the input to the sigmoid function by a scalar (or you just use a <code>nn.Parameter</code> if you want to vary the slope given a specific input).</p>
<pre><code class="lang-auto">slope = nn.Parameter(1)
probabilities = torch.sigmoid(slope * x)
</code></pre>
<p>You could also enforce that the slope is strictly positive via a <code>nn.ReLU</code> function as a negative slope will flip your probability values.</p>
Llearning to write and train GPT in PyTorch?

Mah_Neh: <p>Is there any recommended Tutorial that I could take to learn how to write simple gpt architectures using pythorch and could be trained in a standard Laptop?</p>

vdw: <p>Maybe this <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/3-neural-nlp/Section%204.1%20-%20Transformer%20(Basic%20Architecture).ipynb" rel="noopener nofollow ugc">notebook</a> and this <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/3-neural-nlp/Section%204.3%20-%20Generative%20Pretrained%20Transformer%20(GPT).ipynb" rel="noopener nofollow ugc">notebook</a> could be useful.</p>

konan009: <p>This <a href="https://www.youtube.com/watch?v=kCc8FmEb1nY&amp;t=4901s&amp;ab_channel=AndrejKarpathy" rel="noopener nofollow ugc">youtube video</a> from Andrej Karpathy is good. If you find it hard to understand definitely you can start watching the <a href="https://www.youtube.com/watch?v=VMj-3S1tku0&amp;list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&amp;ab_channel=AndrejKarpathy" rel="noopener nofollow ugc">makemore series</a>.</p>
How to enable DDP to train a structure-changing model?

seigeweapon: <p>My goal is to accelerate 3D-Gaussian-Splatting (3DGS) model training by using distributed tech.</p>
<p>While the DDP’s API is concise and elegant, I find it quite hard to move from MNIST DDP sample to enabling DDP on 3DGS. The biggest hurdle I think is this:</p>
<p>Unlike most cases, where the model’s hyperparams (structure) are fixed, 3DGS model’s equavelant parameters are its 3d-points, which needs to be densified &amp; pruned during training. This is the reason I call it “structure-changing model” in the title.</p>
<p>It seems to me, what matters is how to correctly sync multiple diverged models on multiple gpus. Any advise to get me started with?</p>
Is there a way to use torch indcutor to compile an optimized fx graph?

jasonsun: <p>Hi there,</p>
<p>I am wondering if there is a way use torch inductor to compile a customed optimized fx graph after i use torch dynamo custom backend to first optimize the fx graph like this:</p>
<pre><code>def toy_backend(gm, sample_inputs):
   def my_compiler(gm, sample_inputs):
    # do some graph rewrtie for fx graph in aten ir 
    matches = subgraph_rewriter.replace_pattern(gm, gelu_pattern, gelu_tanh_replacement)
    return gm

    return aot_module_simplified(
        gm,
        sample_inputs,
        fw_compiler=my_compiler
    )
</code></pre>
<p>fn = torch.compile(backend=toy_backend, dynamic=True)(model)</p>
<p>Thanks!</p>
Fine-tuning and retraining of the model

andergalvan: <p>Hi,</p>
<p>Does anyone know of a method to retrain a model with some images of people and then test it with images of the same and different people? I mean, it is possible to keep the feature extraction that the model implements, but retraining the model to better detect some people? I have retrained the model with images of some people and it detects them more reliably through cosine similarity, but with images of people who do not participate in the retraining more false positives appear.</p>
<p>Thank you in advanced,<br>
Ander</p>
How to Obtain the Mapping between Python API and C++ Implementation in PyTorch

yenming: <p>I’ve learned that PyTorch provides bindings for C++ and Python. After looking up the PyTorch documentation and various posts, I found that the two files<code>pytorch/tools/autograd/gen_python_functions.py</code> and <code>pytorch/aten/src/ATen/native/native_functions.yaml</code> are related to the bindings .<br>
However, it appears that I need to manually inspect <code>pytorch/aten/src/ATen/native/native_functions.yaml</code> and *.h files to deduce the mapping between Python API and C++ implementation.<br>
I’m curious if there’s a more automated method to obtain this mapping in PyTorch. For example, given <code>torch.zeros_like</code> , can we easily determine its implementation location in C++, such as <code>pytorch/aten/src/ATen/native/TensorFactories.cpp</code> ?</p>
Simple cumulative sum of bag of words in sparse format?

Matthew_Kennel: <p>I’m trying to compute something which I think should be fairly simple but is apparently unobvious except in the inefficient explicit way.</p>
<p>I have a sequence of tokens (from something other than a natural language).</p>
<p>I wish to make an efficient sparse encoding of the cumulative count, bag of words, for a sliding (or non-sliding if necessary but cumulative), for example c_i where c_i is the number of times the token index i occurred previously at this position.</p>
<p>I want to apply nonlinear-but zero preserving functions to this, such as log(c+1) maintaining sparsity and then that goes into a predictive model (which is dense after dimensionality reduction from the large word space).</p>
<p>There is a batch dimension as well of course.</p>
<p>This seems like an elementary problem that people have addressed in NLP or sequence prediction but I haven’t seen a clear answer.    Conv1d does not support sparse tensors.</p>
<p>What are the preferred ways to solve this?</p>
Disable TracerWarnings for known constants

tjoseph: <p>Hi everyone,<br>
is there a way to deactivate some TracerWarnings for when I know the input size will not change?</p>
<p>My code is written such that it work with variable input sizes, but actually a single run always uses constant sized input. I do not want to switch to TorchScript because distributions are not properly supported and also because I do not need the functionality, since my inputs are actually constant.</p>
<p>For example the following gives me a warning:</p>
<pre><code class="lang-auto">all(s == x.shape[0] for s in x.shape)

TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!

</code></pre>
<p>I know why the warning makes sense, however I am wondering whether I can deactivate it since I know that in a single run the condition will be constant.</p>

ptrblck: <p>You could try to set <code>check_trace=False</code> or if that doesn’t help you could try to remove them using <code>warnings.filterwarnings</code>.</p>

tjoseph: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="124634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>u</p>
</blockquote>
</aside>
<p>Thanks. <code>check_trace</code> doesn’t work for me, but I will look into <code>warnings.filterwarnings</code>!</p>

AlbertZeyer: <p>The warning happens when calling <code>torch.onnx.export</code> or when JIT tracing in general. However, you would not want to filter all those warnings for the <code>torch.onnx.export</code> call, but only for some of those specific checks. How would you do that? Using <code>filterwarnings</code> would again also filter all such warnings which is not what you would want. So maybe using it together with <code>warnings.catch_warnings</code>? But is this really the best way? It looks quite ugly.</p>

Axel-Jacobsen: <p>Here’s what I’ve done:</p>
<pre><code>    import warnings
    from torch.jit import TracerWarning

    with warnings.catch_warnings():
        warnings.simplefilter("ignore", category=TracerWarning)
        if self.foo:
            blah()
</code></pre>
Deterministic Differentiable Rotation on GPU

SoulofAkuma: <p>Hi,</p>
<p>I’m working on creating exactly reproducible results when backpropagating a loss in a vision transformer back to its input. To clarify, I am modifying the input and not the model weights. Before the forward pass, some transformations are applied to the image, through which the gradient is ultimately backpropagated. These transformations currently contain a rotation with kornia, but I am changing it to the torchvision rotation. The issue is, that both of these implementations use the pytorch function <code>grid_align</code> of which the backwards cuda implementation is not deterministic. My current function to trigger the determinism is the following:</p>
<pre data-code-wrap="py"><code class="lang-py">def activate_determinism_with_seed(seed: int):
    torch.backends.cudnn.benchmark=False
    torch.backends.cudnn.deterministic=True
    torch.use_deterministic_algorithms(True, warn_only=True)
    random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
</code></pre>
<p>After calling this function and backprogpating, I indeed get a warning when using either of the kornia or the torchvision rotation functions. Is there a deterministic alternative (such that no warning is raised by <code>torch.use_deterministic_algorithms(True)</code>), that runs on the GPU and performs a rotation on a Tensor of shape <code>B x C x H x W</code> around its center or do you have recommendations on how to implement one?</p>
Why is the loss superclass protected?

pmeier: <p>I wanted to write a custom loss function by subclassing the <code>_Loss</code> superclass. At first I didn’t pay much attention to the underscore until the following import failed:</p>
<p><code>from torch.nn.modules import _Loss</code></p>
<p>After that I noticed that <code>_Loss</code> is not included in the imports within the <code>__init__.py</code>.</p>
<p>While this can easily be circumvented by importing it directly</p>
<p><code>from torch.nn.modules.loss import _Loss</code></p>
<p>I was left wondering what the intentions might be.</p>
<p>Shall custom loss functions not subclass <code>_Loss</code> but rather <code>module</code> directly? If not, why protect and ‘hide’ it?</p>

AreTor: <p>I am still wondering about this design choice, but I have not found an answer yet.</p>
Distribute C++ Pytorch app with cuda

mihtsys: <p>Hi!</p>
<p>I’m trying to build a C++ GUI app that runs several models and heavy math in libtorch under the hood. I’m using Qt for GUI and I don’t want the source code to be visible. I was able to build and distribute it with the CPU version by simply using the <code>windeployqt</code> tool that comes with Qt. Now I want to add GPU compatibility. What would be the best way to distribute the app so that all the CUDA dependencies are taken care of?</p>

QFuser: <p>Well you need to make sure all have the same cuda installation and minimum spec but first obviously you need to make sure everything works on 1 machine NVTX can be a problem. The rest can be build into an executable .</p>
<p>The rest really depends on you target requirements OS windows , linux. Simple user or expert.<br>
You maybe able to create an docker image for it for pytorch itself there exist one too.</p>
Partially mask gradient with scaling hyper parameter

ZSheikhb: <p>I have this function to block the gradient based on the mask</p>
<pre><code class="lang-auto">class blocked_grad(torch.autograd.Function):

    @staticmethod
    def forward(ctx, x, mask):
        ctx.save_for_backward(x, mask)
        return x

    @staticmethod
    def backward(ctx, grad_output):
        x, mask = ctx.saved_tensors
        return grad_output * mask, mask * 0.0
</code></pre>
<p>If the mask values are 0, 1 then I am wondering how I could partially mask gradient with some scaling parameter?</p>
What workflow for converting from Pytorch to tflite model with QUANTIZATION?

_lovepytorch: <p>My board supported tflite model (it is mandatory) for classification task. If I develops with Tensorflow, so deployment tfite on edge devices is straightforward: Tensorflow model (.h5) ==&gt; <strong>Quantize</strong> and save to tflite mode. Tensorflow supprort many quantization techniques and it is easy to use.</p>
<p>But now, community widely use Pytorch as Framework for Deep Learning, and I want to develop with Pytorch to get better float32 model. I have read in forums that workflow of deployment tfite on edge devices is as follow<br>
<code>.pt model ==&gt; .onnx model =&gt; .tflite model</code>, I call it as flow 2.<br>
I have a question. With Pytorch flow, where can I apply quantization (PTQ or QAT) in flow 2 to get better performance of tflite model?<br>
Thank you so much.</p>

_lovepytorch: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Please help me. Thank you.</p>

ptrblck: <p>I’m not familiar enough with your quantization workflow, but moved your topic to the <code>quantization</code> category.</p>
Speeding up PyTorch model testing/inference

spacemeerkat: <h2><a name="premise-1" class="anchor" href="#premise-1"></a>Premise</h2>
<hr>
<p>I have a rather simple/small, trained convolutional autoencoder model.</p>
<p>I want to put my test data through the model but I have a lot of test data. (No seriously… I’m talking ~280 million inputs to test, so it’s slow… even with batches).</p>
<p>My rather vague question is this:</p>
<p><em>What are the various clever PyTorch options for speeding up the model inference stage?</em></p>
<hr>
<h3><a name="details-2" class="anchor" href="#details-2"></a>Details</h3>
<p>Straight off the bat there are some basic/easy ones that we all know:</p>
<ul>
<li>Batch loading inputs during testing</li>
<li>Using a GPU in testing mode (if the .to(device) step is not a limiting factor)</li>
</ul>
<p>But are there more approaches which are more focused on deeper PyTorch functionalities?</p>
<p>e.g. If there was a big red button that allowed you to duplicate the model across the maximum GPU/CPU memory allocation and then fire batches off in an efficient way, or some similar approaches… that’s the sort of thing I’m angling at.</p>
<p>Alternatively, if that big red button doesn’t exist, what would be the next best thing? I assume running in CPU mode and instantiating multiple models across cores in a parallelized way would also work (but that would be an MPI thing rather than a PyTorch thing).</p>
<hr>
<p>Sorry it’s such a vague question, but hopefully it sparks a small discussion and others can use this as a place to see ways to speed up their model inference if they become stuck too!</p>
<p>Many thanks in advance</p>
How pytorch support python [] (__getitem__) ops backpropogation?

sanbuphy: <p>I would like to ask about a tensor. Its operation <code>[]</code> , i.e., <code>__getitem__</code> , supports backpropagation when indexing or slicing. However, I do not know how this is supported. Is there an overload or some code that can show how this operation supports both forward and backward propagation? I would like to know how it is registered and how different backward codes are overloaded under different conditions. Thank you.</p>
<p>supplement:<br>
like this, we can know its grad_fn is selectBackward<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7d792599183ed7731249d2d94fca914a45de248.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f7d792599183ed7731249d2d94fca914a45de248" title="8d03cb6b55e335903fef0aaa3f9878b"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7d792599183ed7731249d2d94fca914a45de248_2_690x447.jpeg" alt="8d03cb6b55e335903fef0aaa3f9878b" data-base62-sha1="zmvNjuyahIsgLTdThUsWSH9HeA0" width="690" height="447" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7d792599183ed7731249d2d94fca914a45de248_2_690x447.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7d792599183ed7731249d2d94fca914a45de248.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7d792599183ed7731249d2d94fca914a45de248.jpeg 2x" data-dominant-color="482D1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">8d03cb6b55e335903fef0aaa3f9878b</span><span class="informations">795×516 51.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>but i don’t know why this situation  a[0][1][1] ,the  ‘<span class="chcklst-box fa fa-square-o fa-fw"></span><span class="chcklst-box fa fa-square-o fa-fw"></span><span class="chcklst-box fa fa-square-o fa-fw"></span>’ will be select op in pytorch , who<br>
register it?</p>

QFuser: <p>You showing python examples but what I can with my limited knowledge is it part of the Tensor structure/object(in opposite to the pure mathematical term) and yes they have their custom operation overloads so the tensor “saves” what has been done with it .</p>
<p>if you want to know more you need to dive deep into the code in github</p>
Gradient of a matrix with respect to its rows

sherlock.h: <p>Hey guys<br>
I have a matrix M(X) with shape <code>(N, F)</code>, where X is my dataset, also <code>(N, F)</code>. I am interested in calculating the gradient of M_ij with respect to X_i.<br>
Currently, in Pytorch, I am doing it this way:</p>
<pre data-code-wrap="python"><code class="lang-python">grad = torch.stack([
    torch.stack([
        torch.autograd.grad(M[i,j], X, retain_graph=True)[0][i] for j in range(X.shape[1])
    ]) for i in range(X.shape[0])
])
</code></pre>
<p>This gives back a matrix whose shape is <code>(N, F, F)</code>.<br>
However, this is painfully slow. Can you guys think of a way to vectorize this?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/sherlock.h">@sherlock.h</a>,</p>
<p>You’ll want to have a look at the <code>torch.func</code> namespace. You can compose <code>torch.func.grad</code> and <code>torch.func.grad</code> to efficiently vectorized over your gradient computation.</p>
<p>The docs for <code>torch.func</code> can be found here: <a href="https://pytorch.org/docs/stable/func.html" class="inline-onebox" rel="noopener nofollow ugc">torch.func — PyTorch 2.3 documentation</a></p>
Understanding Multiple Backward Passes for a Single Forward Pass for a CNN Model with Multiple Outputs

Med_Amine_99: <p>Hi PyTorch community,</p>
<p>I’m seeking clarification on the inner workings of the forward and backward processes when dealing with a single forward pass and multiple backward passes and how gradients are computed.</p>
<p><strong>Context :</strong><br>
I am currently working on a CNN model with <strong>multiple outputs</strong> (num outputs = num bounding boxes)<br>
I am trying to compute the gradients of a specific detection’s top_score wrt sub-module/layer 's output using module backward hooks.</p>
<p><strong>Importantly, these gradients are intended for debugging purposes and enhancing model explainability, rather than for training or updating model params, thus, I will not be using optimizers and inputs will be non-batched (batch_size = 1)</strong></p>
<p>My specific questions are:</p>
<ol>
<li>
<p>When and how are gradients computed exactly in PyTorch during the forward and backward passes?</p>
</li>
<li>
<p>If I perform one forward pass and multiple backward passes (e.g., <code>top_scores[bbox_index].backward()</code> for <code>i</code> in range(num_detections)), how does PyTorch handle this internally?</p>
<pre><code class="lang-auto">for bbox_index in range(num_boxes):
    
    # the model outputs a scores tensor of shape (n_boxes, n_classes) 
    # top_scores is a tensor computed from the model output tensor 
    # so it should be attached to the computation graph 
    # it contains the top score for each bbox
    top_scores[bbox_index].backward(retain_graph=True)
</code></pre>
</li>
</ol>
<ul>
<li>
<p>Do gradients accumulate if I perform <code>top_scores[1].backward(retain_graph=True)</code> followed by <code>top_scores[2].backward(retain_graph=True)</code> with <code>retain_graph=True</code>? Does <code>retain_graph</code> keep gradient values or just the computation graph ?</p>
</li>
<li>
<p>Do I need to set gradients to 0 after computing each <code>top_scores[i].backward(..)</code>? If so, when is the appropriate time to do this?</p>
</li>
</ul>
<p>I’d appreciate any insights !</p>