akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch Conference: Fine Tuning Mini Summit

kyliewd: <p><strong>Attending the PyTorch Conference? Checkout the Fine tuning mini summit on the 18th!</strong></p>
<p>Fine-tuning has become key to personalizing LLMs, and the summit brings together technical content and some of the best experts on topics ranging from parameter-efficient fine-tuning, quantization and performance at scale to reproducible evaluations, multimodal LLMs and agents.</p>
<p>For more details, checkout: <a href="https://events.linuxfoundation.org/pytorch-conference/program/fine-tuning-mini-summit/" rel="noopener nofollow ugc">Fine Tuning Mini-Summit | LF Events</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6dbda742d681a8f32632ed7ae1286c5ff2c9927.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d6dbda742d681a8f32632ed7ae1286c5ff2c9927" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6dbda742d681a8f32632ed7ae1286c5ff2c9927_2_690x361.jpeg" alt="image" data-base62-sha1="uEJd9YWnBbrlTa9H4UZJLR9mDVZ" width="690" height="361" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6dbda742d681a8f32632ed7ae1286c5ff2c9927_2_690x361.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6dbda742d681a8f32632ed7ae1286c5ff2c9927_2_1035x541.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6dbda742d681a8f32632ed7ae1286c5ff2c9927.jpeg 2x" data-dominant-color="3F242B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1200×628 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to Download timm models to specified location.

17swagat: <p>Hi everyone,<br>
I’m trying to download a timm model to a user-specified location on my local machine using the following code:<br>
<code>model = timm.create_model(model_name, pretrained=True)</code></p>
<p>The code above downloads the model to the default location: <code>C:\Users\17swa\.cache\torch\hub\checkpoints</code>, which I don’t want. I want the model to be stored in my user specified location.</p>
<p>However, I couldn’t find a way to specify the download location. Is it possible to do this? If yes, could someone please guide me on how to achieve it?</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/hub.html#where-are-my-downloaded-models-saved">docs</a>:</p>
<pre data-code-wrap="python"><code class="lang-python">The locations are used in the order of

    * Calling hub.set_dir(&lt;PATH_TO_HUB_DIR&gt;)

    * $TORCH_HOME/hub, if environment variable TORCH_HOME is set.

    * $XDG_CACHE_HOME/torch/hub, if environment variable XDG_CACHE_HOME is set.

    * ~/.cache/torch/hub
</code></pre>
<p>so you could change it via the Python calls or the mentioned env variables.</p>
Do pytorch/pytorch container images comply with Anaconda's Terms of Service?

davidxia: <p>I see the pytorch Dockerfile downloads Miniconda from <code>repo.anaconda.com</code> and <a href="https://github.com/pytorch/pytorch/blob/b0171c39209acadec0ff07f584a176ea1bbd5039/Dockerfile#L41" rel="noopener nofollow ugc">installs software from Anaconda here</a>. According to Anaconda’s Terms of Service (ToS)</p>
<blockquote>
<p>Unless expressly agreed by Anaconda, You may not:  (a) Make, sell, resell, license, sublicense, distribute, rent, or lease any Offerings available to anyone other than You or Your Users, unless expressly stated otherwise in an Order, Custom Agreement or the Documentation or as otherwise expressly permitted in writing by Anaconda… If You would like to redistribute or embed any Offering in any product You are developing, please contact the Anaconda team for a third party redistribution commercial license.</p>
</blockquote>
<blockquote>
<p>Your registration, download, use, installation, access, or enjoyment of all Anaconda Offerings on behalf of an organization that has two hundred (200) or more employees or contractors (“<strong>Organizational Use</strong> ”) requires a paid license of Anaconda Business or Anaconda Enterprise… Utilizing Miniconda to pull package updates from the Anaconda Public Repository without a commercial license (if required by the conditions set forth in Section 2 of this Terms of Service) is considered a violation of the Terms of Service.</p>
</blockquote>
<h2><a name="questions-1" class="anchor" href="#questions-1"></a>Questions</h2>
<ol>
<li>Do the resulting <code>pytorch/pytorch</code> container images comply with Anaconda’s ToS?</li>
<li>Does use of these images (as-is or modified, like <code>FROM pytorch/pytorch</code>) in organizations with 200 or more employees require a paid license?</li>
</ol>
<h2><a name="context-2" class="anchor" href="#context-2"></a>Context</h2>
<blockquote>
<p>Software maker Anaconda Inc has sued Intel in Delaware federal court, accusing the chipmaker of misusing its software for developing artificial-intelligence platforms, according to a lawsuit made public on Friday.</p>
</blockquote>
<p>— <a href="https://www.reuters.com/legal/litigation/intel-sued-copyright-infringement-over-ai-software-2024-08-09/" rel="noopener nofollow ugc">https://www.reuters.com/legal/litigation/intel-sued-copyright-infringement-over-ai-software-2024-08-09/</a></p>

davidxia: <p>I couldn’t put more than two links above. Here’s the other relevant links</p>
<ul>
<li><a href="https://legal.anaconda.com/policies/en/#terms-of-service" rel="noopener nofollow ugc">Anaconda Tos</a></li>
<li><a href="https://hub.docker.com/r/pytorch/pytorch" rel="noopener nofollow ugc">pytorch/pytorch images</a></li>
</ul>
Torch 2.4 still not provided for intel macs not in pypi and not conda

milad_aghajohari: <p>I have tried to install torch2.4 with conda and pip but I think prebuilt wheels are not provided for mac os x86 macs for torch2.4. Is there any reason?</p>

ptrblck: <p>Mac on x86 is deprecated as described <a href="https://discuss.pytorch.org/t/why-no-macosx-x86-64-build-after-torch-2-2-2-cp39-none-macosx-10-9-x86-64-whl/204546/2">here</a>.</p>

milad_aghajohari: <p>Is there any solution other than building from the source (which is very time taking and also possibly will result in bugs as it is not tested on those versions?).</p>
<p>While I understood the reasoning behind deprecation, but still I think this is very limiting for researchers who test codes locally on their macs. Many researchers I know have an intel mac still, and as purchasing a new mac is a 4000$ purchase, I think this is very limiting.</p>

ptrblck: <p>No, I’m not aware of other approaches besides source builds and I also don’t know if this would even work for PyTorch 2.4 as I’m not using MacOS for PyTorch.</p>
Index_copy vs index_copy_ dtype mismatch

apaz-cli: <p>Why does index_copy_() only take a LongTensor as indices? The out of place impl index_copy() (which is very poorly documented as it seems) takes int32 or int64. Is this an oversight in the API, or is it this way because of legacy code, or is it a bug?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.index_copy.html#torch.Tensor.index_copy" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.index_copy.html#torch.Tensor.index_copy</a></p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.index_copy_.html#torch.Tensor.index_copy_" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.index_copy_.html#torch.Tensor.index_copy_</a></p>
Cannot run Demand forecasting with the Temporal Fusion Transformer

JustNow: <h3><a name="demand-and-forecasting-with-the-temporal-fusion-transformer-1" class="anchor" href="#demand-and-forecasting-with-the-temporal-fusion-transformer-1"></a>Demand and forecasting with the Temporal Fusion Transformer</h3>
<p>Hi everyone,<br>
I hope you’re doing well!</p>
<ol>
<li>I’m new to both PyTorch and this forum, but I’ve been drawn to the <strong>Temporal Fusion Transformer (TFT)</strong> model because of its potential and usefulness.</li>
<li>For the past two weeks, I’ve been trying to run the basic example from the tutorial here: <a href="https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html#" class="inline-onebox" rel="noopener nofollow ugc">Demand forecasting with the Temporal Fusion Transformer — pytorch-forecasting documentation</a>.</li>
<li>Unfortunately, despite my best efforts, I haven’t been able to get it to work.</li>
<li>I’m using <strong>Windows</strong>, <strong>CUDA 12.4</strong>, <strong>Python</strong>, and <strong>Visual Studio Code</strong>.</li>
<li>I’ve tried different combinations of Python versions (3.8, 3.10, 3.12) with Conda and Pip, and even Google Colab, but faced issues at various points.</li>
<li>I’m starting to wonder if I’m missing something fundamental or if the example code has issues.</li>
<li>Could someone kindly provide a simple, step-by-step guide for setting up the environment (specific versions of packages, installation commands, etc.)? Detailed instructions would be really helpful!</li>
<li>I’ve encountered multiple errors, mostly package collisions and import issues, so I’ve been unable to get a clean run of the example.</li>
<li>I’ve searched this forum and online resources, but while I’ve found fixes for individual issues, new ones keep arising with each step.</li>
<li>I’m feeling stuck at this point, so any advice or suggestions would be greatly appreciated!</li>
</ol>
<p>Thank you so much in advance for your help!<br>
Regards from New York</p>
PyTorch 2 is too slow on AWS EC2 and SageMaker

blackberry: <p>Hi all,</p>
<p><strong>TLDR:</strong> PyTorch 2 (including the latest 2.4 release) is much slower than PyTorch 1.13.1 on AWS EC2/SageMaker. Solutions provided on the web help, but it is still 3-5 times slower.</p>
<p><strong>Details:</strong> Whether I get one of the available PyTorch 2 Ubuntu AMIs or whether I re-install PyTorch 2 using conda/pip, the training and inference is much slower compared to PyTorch 1.13.1 (same code, same instance, different PyTorch versions in different containers). I have been using AWS p3/p4/g5 instances (so probably unrelated to instance type).</p>
<p>I tracked it down to DataLoader. People say it is due to openmp (and CPU affinity), which is used to multi-threaded/multi-process data loading. Following the recommended solutions (e.g., installing intel-openmp instead of llvm-openmp, export KMP_AFFINITY=disabled, importing torch before numpy) seem to help, but it is still 3-5 times slower.</p>
<p>What is it that was changed in PyTorch 2 DataLoader that causes this huge difference in speed?</p>
<p>Anyone experienced this on AWS EC2/SageMaker and solved it? Or any ideas about the root cause and potential solutions?</p>
Data modelling for a Graph Conv + RNN model

ml-gg: <p>I’m out of my depth here and could use some guidance. I have a dataset consisting of multiple files where the data nicely fits into a graph structure. The data can also be segmented in a time-wise manner.</p>
<p>I was considering use graph convolutional layers combined with an RNN variant in PyTorch and PyTorch Geometric to hopefully capture spatial and temporal features. The ultimate goal is graph classification.</p>
<p>I’ve already created a homogeneous multigraph in a geometric data object per file. In this graph, only edges with a specific feature value are directional, which introduces a sense of sequence. All other edges are bidirectional, representing relationships where the direction doesn’t matter. I also noted down the timestamps of the nodes in the time attribute.</p>
<p>Is my current data modeling approach sufficient? What additional considerations should I be aware of? I’m aware of geometric_temporal, but that seem more focused on analyzing a single time series, while my case involves dealing with one temporal signal per file.</p>
<p>Any insights are welcome.</p>
Why are there some pytorch functions not listed in the documentation?

jack_S: <p>For instance, I can’t find <code>Tensor.new</code> in the documentation. I also tried</p>
<pre><code class="lang-auto">from torch import Tensor
print(Tensor.new.__doc__)
</code></pre>
<p>in my console, but no words appeared. Besides, I remember there are some other pytorch functions that are without documentation, but I can’t recall any name of them right now so I just take <code>Tensor.new</code> as an example.</p>

ptrblck: <p>Some methods are not properly documented as these are mainly used in internal code. E.g. the <code>new()</code> call returns a tensor in a new shape with uninitialized memory:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(1)
print(x)
# tensor(1)
y = x.new(2)
print(y)
# tensor([3616445622929465956, 6066685342842367284])
</code></pre>
<p>but also accepts a <code>storage</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x.untyped_storage()
#  1
#  0
#  0
#  0
#  0
#  0
#  0
#  0
# [torch.storage.UntypedStorage(device=cpu) of size 8]

x.new(x.untyped_storage())
# tensor([1])
</code></pre>
<p>User code should explicitly call e.g. <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.new_empty.html#torch-tensor-new-empty"><code>new_empty</code> </a>.</p>

jack_S: <p>Thanks for your immediate reply! I am used to writing <code>new_empty</code>, but when I read other users’ codes with <code>new</code>, I may get confused since there’s no documentation.</p>
What is the difference between WeightNormSparsifier from torch.ao and torch.nn.utils.prune.l1_unstructured?

Mayank_Kumar: <p>Relevent links:</p>
<ol>
<li><a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.l1_unstructured.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.utils.prune.l1_unstructured — PyTorch 2.4 documentation</a></li>
<li><a href="https://github.com/pytorch/pytorch/blob/32e057636c7593f8bb4820395e2a55eea471a145/torch/ao/pruning/sparsifier/weight_norm_sparsifier.py#L21" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/pruning/sparsifier/weight_norm_sparsifier.py at 32e057636c7593f8bb4820395e2a55eea471a145 · pytorch/pytorch · GitHub</a></li>
</ol>
Using Pytorch autograd to adjust model weights

Hummus_Akemi: <p>Hello,</p>
<p>I currently have a classification task, except I use an object detection model. The model outputs detections for all objects it locates in the image, and assigns a confidence to each. However, I am only interested in classifying the entire image as one class, so I will need to perform some postprocessing.</p>
<p>I first have a vector of K threshold values, one for each class, and I threshold all confidences such that if a confidence is less than the corresponding class threshold, I clamp it to zero (essentially deleting the detection).</p>
<p>I also have a vector of K class weights, one for each class, and I multiply all confidences by the corresponding value. This is used to “re-order” the confidences so to improve the accuracy. For example, a certain class could be detected with low confidences by the model, but is a high priority for the user (such that when it appears, the user would always want the image to be classified as it).</p>
<p>I currently have the mathematical problem where I want to find the optimal values for both the thresholds and the weights, such that with these new parameters, I can alter the confidences such that the highest accuracy is achieved (the highest confidence detection, after thresholding/weighting, in each image coincides with the ground truth).</p>
<p>I know that this scheme runs a lot of risks with regards to overfitting, or that accuracy is not the best metric when the dataset is well-balanced, etc. but I am purely interested in this as a mathematical question.</p>
<p>Currently, for the weight optimization, I am running a gradient descent algorithm using a binary cross-entropy loss. I use binary cross entropy because I cannot handle the “None” (no detections) class using regular cross-entropy, as far as I know (please correct me if I’m wrong). This works with decent results.</p>
<p>But I am not sure how to optimize the thresholds, as it’s non-differentiable, short of using a brute-force or heuristic algorithm like genetic algorithm or grid search. I have tried using an iterative local search but it is slow and is slow to improve.</p>
<p>I’ve been struggling with this problem for a while, and any help would be greatly appreciated.</p>
<p>Thanks!</p>
Pytorch lightning, profiler assertion error when running trainer.test after training

LewsTherin: <p>I’m using this code for training an X3D model:</p>
<pre><code class="lang-auto">from lightning.pytorch.profilers import PyTorchProfiler
from pytorch_lightning.callbacks import ModelCheckpoint, LearningRateMonitor, StochasticWeightAveraging, BackboneFinetuning

schedule = torch.profiler.schedule(skip_first=20, wait=5, warmup=1, active=5)
profiler = PyTorchProfiler(filename="profile", schedule=schedule)
trainer_kwargs = {
        "max_epochs": 2,
        "precision": 16,  
        "deterministic": True,
        "benchmark": True,
    }

callbacks = list()
callbacks.append(LearningRateMonitor(logging_interval="epoch"))
callbacks.append(ModelCheckpoint(monitor="val/accuracy", mode="max", save_top_k=1))
callbacks.append(GenerateMetricImages())

model_module = _model_init_func(**model_kwargs, **data_kwargs, **trainer_kwargs)
data_module = _dataset_init_func(videos_dir, train_file, val_file, test_file, model_params, **data_kwargs)
trainer = pytorch_lightning.Trainer(
        logger=[tb_logger, profiler=profiler, callbacks=callbacks, **trainer_kwargs
    )

trainer.fit(classification_module, data_module)
trainer.test(classification_module, data_module)
</code></pre>
<p>The training finishes without issues, but when running the trainer.test I get this error:</p>
<pre><code class="lang-auto">File "/home/my_user/anaconda3/envs/point_outcome_updated/lib/python3.11/site-packages/torch/profiler/profiler.py", line 191, in events
    assert self.profiler
           ^^^^^^^^^^^^^
AssertionError
Exception ignored in: &lt;function Profiler.__del__ at 0x72cac877eca0&gt;
Traceback (most recent call last):
  File "/home/my_user/anaconda3/envs/point_outcome_updated/lib/python3.11/site-packages/lightning/pytorch/profilers/profiler.py", line 147, in __del__
  File "/home/my_user/anaconda3/envs/point_outcome_updated/lib/python3.11/site-packages/lightning/pytorch/profilers/pytorch.py", line 561, in teardown
  File "/home/my_user/anaconda3/envs/point_outcome_updated/lib/python3.11/site-packages/lightning/pytorch/profilers/pytorch.py", line 545, in _delete_profilers
  File "/home/my_user/anaconda3/envs/point_outcome_updated/lib/python3.11/site-packages/lightning/pytorch/profilers/pytorch.py", line 537, in _cache_functions_events
  File "/home/my_user/anaconda3/envs/point_outcome_updated/lib/python3.11/site-packages/torch/profiler/profiler.py", line 191, in events
AssertionError: 
</code></pre>
<p>I found a discussion <a href="https://github.com/Lightning-AI/pytorch-lightning/issues/9450" rel="noopener nofollow ugc">here</a>, stating that “the training phase seems to kill the pytorch profiler, that therefore doesn’t exist anymore for the test phase”, but it seems the issue should’ve been solved in version 1.5.5, and I’m using pytorch-lightning==2.3.3.</p>
<p>Any suggestions?</p>

ptrblck: <p>Is this issue specific to PyTorch-Lightning? If so you might want to cross-post this question into their discussion board.</p>
No cuda kernel image available for execution on device Custom cuda kernel

KukumavMozolo: <p>Hi there,<br>
i wrote a custom cuda kernel for pytorch 2.3.0 and cuda 12.1,<br>
my nvcc-version:</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:34:21_PDT_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0
</code></pre>
<p>and compiled it via setuptools: e.g.</p>
<pre data-code-wrap="python"><code class="lang-python">from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension



setup(
    name='test',
    ext_modules=[
        CUDAExtension('test_k', [
            'test.cpp',
            'test_cuda_kernel.cu'
        ])
    ],
    cmdclass={
        'build_ext': BuildExtension
    })

</code></pre>
<p>The kernel was compiled on an rtx3090, however when trying to run it on an a100 i get the following error:<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
What am i doing wrong?</p>

ptrblck: <p>Compile your kernel for <code>sm_80</code> by specifying it via <code>TORCH_CUDA_ARCH_LIST="8.0;8.6"</code>.</p>

KukumavMozolo: <p>Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Is there a way to manually clear the memory occupied by activation

nastyapple: <p>I want to implement a fault tolerance mechanism in the distributed training. I wander if there is a way to fully clear all the activation created in a forward pass as there might exist some data that cause OOM in one process and if I clear activation manually, the training process can recover and ready to process other data.</p>

AlphaBetaGamma96: <p>I believe this should be freed by default, but if you want to, you could try using a forward hook (after the function has been triggered) and then manually delete the activations, and see if this helps.</p>
<p>The documentation on the forward hooks are here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_hook" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 2.4 documentation</a></p>

nastyapple: <p>I think you are right. The memory cannot be automatically released only when I wrapped the model with DeepSpeed and it’s all fine with original Python.</p>
How to add a variable to the Options argement for ProcessGroup function

MyByte0: <p>I have a modified on pytorch2. add a std::string variable on Types.hpp::BarrierOptions, and i do add py::class_&lt;::c10d::BarrierOptions&gt; the def_readwrite. After finish compile and install, python report that not find variable member on BarrierOptions.<br>
I use  <code>USE_SYSTEM_NCCL=1 USE_DISTRIBUTED=1 USE_CUDA=1 USE_NCCL=1 BUILD_TEST=0 python setup.py develop</code> cmd.<br>
What the other code I need modify?</p>
Help wanted: How to pad and mask correctly for variable length label sequences

Andre1: <p>Hello,</p>
<p>i implemented a transformer-encoder which takes some cp_trajectories and has to then create a fitting log mel spectrogram for those. Because input as well as labels are variable in length i use a custom_collate_fn to pad them like this</p>
<pre><code class="lang-auto">import torch
from torch.nn.utils.rnn import pad_sequence

def pad_and_mask(batch):
    # Assuming each element in 'batch' is a tuple (sequence, label)
    sequences = [torch.tensor(item[0]) for item in batch]
    labels = [torch.tensor(item[1]) for item in batch]

    # Pad the sequences to have the same length
    sequences_padded = pad_sequence(sequences, batch_first=True, padding_value=0)
    labels_padded = pad_sequence(labels, batch_first=True, padding_value=0)

    # Create attention masks for sequences
    attention_masks = torch.zeros((sequences_padded.size(1), len(batch)), dtype=torch.float32)
    for i, seq in enumerate(sequences):
        attention_masks[i, :len(seq)] = 1

    # Create label masks for labels
    label_masks = torch.zeros_like(labels_padded, dtype=torch.float32)
    for i, label in enumerate(labels):
        label_masks[i, :len(label)] = 1

    return sequences_padded, attention_masks, labels_padded, label_masks```

i use the attention_masks in the transformer and i believe they work fine if i input them into key_padding_mask. Now i want to caltulate the loss as a MSELoss but i dont want the loss to be skewed because i dont mask the output of my transformer. How would i implement that? Or is that simply a completely wrong approach. 

Thanks for the help :D</code></pre>

huynn: <p>Hello my friend, I have a same problem. And I found the information in pytorch, we can use:</p>
<ul>
<li>from torch.nn.utils.rnn import pad_sequence to pad data and</li>
<li>from torch.nn.utils.rnn import pack_padded_sequence to mask data</li>
<li>However, when using pack_padded_sequence, the order of sequences in input data and labels is changed. I afraid that thís loses the correspondence between input data - label pairs.<br>
Did you solve this problem ?<br>
I also post my question here <a href="https://discuss.pytorch.org/t/padding-and-packing-both-input-data-and-label/208389">Link</a>.</li>
</ul>
Padding and packing both input data and label

huynn: <p>Hi every one. I have a seq2seq problem. My data include input data and corresonding labels. Lengths of sequences in input data are different and lengths of sequences in labels are also different.<br>
<em><strong>Ex:</strong></em><br>
<em>- Data 1: input data length: 1, label length: 5</em><br>
<em>- Data 2: input data length: 8, label length: 3</em><br>
<em>- Data 3: input data length: 2, label length: 7</em><br>
<em>- Data 4: input data length: 5, label length: 5</em><br>
<em>- …</em></p>
<p>As a result, I want to use padding and packing when training. However, when I use torch.nn.utils.rnn.pack_padded_sequence <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_padded_sequence.html#torch.nn.utils.rnn.pack_padded_sequence" rel="noopener nofollow ugc">Link</a>, it always sort the data in decreasing order of length.<br>
<em><strong>For example:</strong></em></p>
<pre><code class="lang-auto">x_packed = pack_padded_sequence(xx_pad, x_lens, batch_first=True, enforce_sorted=False)
y_packed = pack_padded_sequence(yy_pad, y_lens, batch_first=True, enforce_sorted=False)
</code></pre>
<p>After packing, both input data (x) and labels (y) are sorted. It causes the correspondence between x, y pairs to be lost.</p>
<p>Can you help me please ! Thank you so much !</p>
Why is my profiler results are duplicated?

YuA: <p>I am using torch.profiler.profile for profiling my training.<br>
I am sure that prof.step() is properly called; however, duplicated (slightly different though) results are saved. You can see what is happening in the screenshot below.<br>
I have checked my code many times, but could not identify the cause. Can you guess possible cuases?</p>
<p>(Update; I checked pt.trace.json files carefully, and I found that almost the same results were saved by two different processes. For example, in rank0, two processes with pid  395230 and pid 0 wrote the same step records. It seems that the first one (pid=395230) is the process on CPU, and the other may be GPU process, I guess.)</p>
<p>(Picture: the same step numbers appear, such as 'step16, step16, step17, step17,…).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/683917032befebfc0b085aed7f18dc42e1243e86.png" data-download-href="https://discuss.pytorch.org/uploads/default/683917032befebfc0b085aed7f18dc42e1243e86" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683917032befebfc0b085aed7f18dc42e1243e86_2_690x235.png" alt="image" data-base62-sha1="eRZYehhRpYDYsFGn6lZVe4aWwvQ" width="690" height="235" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683917032befebfc0b085aed7f18dc42e1243e86_2_690x235.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683917032befebfc0b085aed7f18dc42e1243e86_2_1035x352.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683917032befebfc0b085aed7f18dc42e1243e86_2_1380x470.png 2x" data-dominant-color="B3BEE3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1391×474 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Change mask branch dimensions for Mask R-CNN

FiReTiTi: <p>Hi,<br>
I use Mask R-CNN to detect and segment large objects (about 750x750 pixels) in a very large image. The detection works really well, but the segmentation seems a little bit approximative on the borders.<br>
Mask R-CNN has a segmentation branch that segment a <code>m x m</code> mask for each ROI. This <code>m x m</code> mask is usually small (about 28x28 if I’m right) and after passing through the ROI_Align it has to represent/segment a big 750x750 ROI.<br>
Is there a way to modify the dimensions of the small <code>m x m</code> mask to make it bigger?</p>
Converting augmentation of Keras into Pytorch

jawi289p: <p>I am reproducing a code (which is in tensorflow/keras) in which the following augmentations are applied.</p>
<pre><code class="lang-auto">ImageDataGenerator(
    zoom_range=0.1,
    fill_mode='reflect',
    width_shift_range=0.4,
    height_shift_range=0.4)
</code></pre>
<p>How can I use the same augmentation methods in Pytorch.</p>

Eduardo_Lawson: <p>I would suggest to take a look at <a href="https://pytorch.org/vision/0.9/transforms.html" rel="noopener nofollow ugc">torchvision.transforms</a> or at <a href="https://albumentations.ai/docs/" rel="noopener nofollow ugc">Albumentations</a></p>

jawi289p: <p>there are not available in both</p>

jawi289p: <p>may be <a class="mention" href="/u/ptrblck">@ptrblck</a> can guide better on this</p>

Soumya_Kundu: <p>You can make custom transforms: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>
<p>For width and height shift range: <a href="https://discuss.pytorch.org/t/how-shift-an-image-vertically-or-horizontally/117074" class="inline-onebox">How shift an image vertically or horizontally?</a></p>
<p>For zoom: <a href="https://discuss.pytorch.org/t/pytorch-randomresizedcrop-vs-zoom-range-in-datagenerator/161434" class="inline-onebox">Pytorch randomresizedcrop vs zoom_range in datagenerator</a></p>
<p>Reflect: <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.pad — PyTorch 2.4 documentation</a> (I guess)</p>
I don't understand WHY don't just use nn.Identity()?

nnurseultan07: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a862c97df8ef34c25ddadc1af9d45012d15f7311.png" alt="image" data-base62-sha1="o1BREv3Q8v3kdyCPzXUF1wKZQ2d" width="386" height="393"></p>
<p>Here in <a href="https://github.com/huggingface/pytorch-image-models/blob/main/timm/layers/patch_embed.py#L26" rel="noopener nofollow ugc">pytorch-image-models/timm/layers/patch_embed.py at main · huggingface/pytorch-image-models (github.com)</a> people in timm set norm_layer = None<br>
Than write:</p>
<pre><code class="lang-auto"> self.norm = norm_layer(hidden_features) if norm_layer is not None else nn.Identity()
</code></pre>
<p>Question is why don’t just set norm_layer to nn.Identity()?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/nnurseultan07">@nnurseultan07</a>,</p>
<p>This seems to be a general fallback if a particular norm layer isn’t specified, so if you pass <code>norm_layer = nn.Identity()</code>, you’d get the same behavior as setting it to <code>norm_layer = None</code>. I’d assume this is just a way of having a ‘default’ <code>norm_layer</code> (which does just the identity) in the case no <code>norm_layer</code> is selected, but you’d get the same behavior if you pass <code>norm_layer = nn.Identity()</code>.</p>
How does loss.backward() work with batch size more than 1?

L_Z: <p>I understand the back propagation. But I am just confused about the case where batch size is greater than 1. let’s say I have broken down the dataset as batch_size = 32. So my outputs will be in length of 32 (outputs = model(sequence)). However, when I apply loss = criterion(outputs, labels), this loss is a float number instead of 32 losses and then I apply loss.backward(). So I am treating the whole batch as a whole, where this affect my model’s generalization. Like I do not want to related the data file in one batch.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/l_z">@L_Z</a>,</p>
<p>When you call <code>loss = criterion(outputs, labels)</code>  you’re implicitly summing over all your data. If you want all 32 losses, you can change the reduction method on the loss to <code>None</code>. In the case of the cross-entropy loss function (docs below), you can see the <code>reduction</code> kwarg, which specifies how to reduce your data to an individual loss value. If you want all individual loss values (across the entire batch), then pass <code>reduction = 'None'</code>, when you instantiate the loss function object.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a></p>

L_Z: <p>Thank you for getting back to my questions. I am not sure what you mean by saying summing over all my data. I am just wondering, in general case, where we want to treat the training data separately, it is fine for as to summing over all data? Or another way to solve this is to let batch_size = 1? Will that also work?</p>

AlphaBetaGamma96: <p>Let’s take the MSELoss (docs below) function as the example (as you can’t have <code>reduction='none'</code> for the cross-entropy function,</p>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss(size_average=None,
                             reduce=None,
                             reduction='mean')
</code></pre>
<p>As can be seen, the default option for <code>reduction</code> is <code>mean</code>, so it will return the <code>loss</code> as the average over all individual loss values. If you set this to <code>'none'</code>, it’ll return all individual loss values.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#mseloss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#mseloss</a></p>
<p>For example,</p>
<pre><code class="lang-auto">import torch
criterion = torch.nn.MSELoss() #defaults to reduction='mean'
criterion_ps = torch.nn.MSELoss(reduction='none') 
x=torch.randn(4,)
y=torch.randn(4,)

criterion(x,y) #returns tensor(0.6103)

criterion_ps(x,y) #returns tensor([0.3331, 1.2524, 0.0025, 0.8534])
</code></pre>
<p>If we take the mean of the <code>criterion_ps</code> function, we get the same results as <code>criterion</code>, i.e. <code>criterion_ps(x,y).mean()</code> returns <code>tensor(0.6103)</code></p>
Torch compile error

Konsthr: <p>Hi!  I have troubles using the torch.compile with any compilation attempt resulting with the following error.</p>
<blockquote>
<p>BackendCompilerFailed: backend=‘inductor’ raised:<br>
CppCompileError: C++ compile error</p>
<p>Command:<br>
g++ /tmp/torchinductor_kharitok/ry/cryqhwcqbu2kfchsyxzmhfsmaxu7ez3e3ulu6xddcanfcyfqrsp4.cpp -shared -fPIC -Wall -std=c++17 -Wno-unused-variable -Wno-unknown-pragmas -D_GLIBCXX_USE_CXX11_ABI=0 -I/gpfs/exfel/data/scratch/kkonstantin/Envs/Torch 2.4/lib/python3.11/site-packages/torch/include -I/gpfs/exfel/data/scratch/kkonstantin/Envs/Torch 2.4/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/gpfs/exfel/data/scratch/kkonstantin/Envs/Torch 2.4/lib/python3.11/site-packages/torch/include/TH -I/gpfs/exfel/data/scratch/kkonstantin/Envs/Torch 2.4/lib/python3.11/site-packages/torch/include/THC -I/gpfs/exfel/data/scratch/kkonstantin/Envs/Torch 2.4/include/python3.11 -L/gpfs/exfel/data/scratch/kkonstantin/Envs/Torch 2.4/lib/python3.11/site-packages/torch/lib -lgomp -lc10 -O3 -DNDEBUG -ffast-math -fno-finite-math-only -fno-unsafe-math-optimizations -ffp-contract=off -march=native -fopenmp -D TORCH_INDUCTOR_CPP_WRAPPER -D C10_USING_CUSTOM_GENERATED_MACROS -o /tmp/torchinductor_kharitok/ry/cryqhwcqbu2kfchsyxzmhfsmaxu7ez3e3ulu6xddcanfcyfqrsp4.so</p>
<p>Output:<br>
In file included from /tmp/torchinductor_kharitok/ry/cryqhwcqbu2kfchsyxzmhfsmaxu7ez3e3ulu6xddcanfcyfqrsp4.cpp:2:<br>
/tmp/torchinductor_kharitok/sk/cskh5dx62fglpphcrl6723dnmowdabouerrzy3dmqcngbxwfa7bv.h:17:10: fatal error: ATen/NumericUtils.h: No such file or directory<br>
17 | <span class="hashtag-raw">#include</span> &lt;ATen/NumericUtils.h&gt;<br>
|          ^~~~~~~~~~~~~~~~~~~~~<br>
compilation terminated.</p>
</blockquote>
<p>Do you have any advises on how I can solve it?</p>

AlphaBetaGamma96: <p>Could you try a clean install?</p>
Pass keyword arguments into jacrev's func

skumar_ml: <p>I tried to reproduce the minimal example from <code>torch.func.jacrev</code>’s documentation, but with a small change in the function g (where it’s given a keyword):</p>
<pre><code class="lang-auto">import torch

x = torch.randn(5)

def f(x):
    return x.sin()

def g(x, constant=1):
    result = f(x) * constant
    return result, result

jacobian_f, f_x = torch.func.jacrev(g, has_aux=True)(x, constant=2)
</code></pre>
<p>Running this gives me the following error:</p>
<p><code>TypeError: g() got an unexpected keyword argument 'constant'</code></p>
<p>What I am doing wrong? If I don’t specify <code>constant=2</code> and just pass in <code>2</code>, then it works, but I want to specify keyword args (for readability and for when I vmap around this).</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/skumar_ml">@skumar_ml</a>,</p>
<p>I don’t believe you need to pass kwargs to the input of <code>jacrev</code> (or any other <code>torch.func</code> function), as I think <code>torch.func</code> expects just the inputs.</p>
<p>Also, you should wrap you <code>torch.func.jacrev</code> call within a <code>torch.func.vmap</code> call otherwise you’ll be computing all the off-diagonals, which are zero by definition.</p>
<p>Here’s an example,</p>
<pre><code class="lang-auto">import torch
from torch.func import jacrev, vmap

x = torch.randn(5)

def f(x):
    return x.sin()

def g(x, constant=1):
    result = f(x) * constant
    return result, result

jacobian_f, f_x = vmap(jacrev(g, argnums=(0,), has_aux=True), in_dims=(0,None))(x, 2)
print(jacobian_f, f_x)
#returns: 
(tensor([ 0.0822,  1.5112, -0.8985,  1.6367, -0.7803]),) tensor([ 1.9983, -1.3101, -1.7868,  1.1494,  1.8415])

#Without 'vmap' returns: 
(tensor([[ 1.7465,  0.0000,  0.0000,  0.0000, -0.0000],
        [ 0.0000,  1.9095,  0.0000,  0.0000, -0.0000],
        [ 0.0000,  0.0000,  1.6745,  0.0000, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  1.9700, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  0.0000, -1.6554]]),) tensor([ 0.9746, -0.5948,  1.0937,  0.3451,  1.1223])
</code></pre>
Parametrizations and Gradient Accumulation?

Avelina9X: <p>I have a training loop which will be running multiple forward passes for gradient accumulation; are there any special measures required for parametrizations (e.g. orthogonal) to ensure identical behaviour between a single forward pass at larger batch size vs multiple forward passes at smaller batch size?</p>
<p>For orthogonal I don’t think it should make a difference, but I wouldn’t be surprised if other parameterisations (especially custom ones) would behave differently when forward passed multiple times before performing an optimizer step.</p>
<p>I’m asking because I’m unsure how pytorch handles the parametrization process, especially when compared to something like Tensorflow Addons which will absolutely perform the process multiple times even if the parameters remain unchanged.</p>

KFrank: <p>Hi Avelina!</p>
<aside class="quote no-group" data-username="Avelina9X" data-post="1" data-topic="208209" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/avelina9x/48/72186_2.png" class="avatar"> Avelina9X:</div>
<blockquote>
<p>are there any special measures required for parametrizations (e.g. orthogonal) to ensure identical behaviour between a single forward pass at larger batch size vs multiple forward passes at smaller batch size?</p>
</blockquote>
</aside>
<p>I haven’t tested this, but I believe that, with the exception of expected<br>
floating-point round-off error, gradient accumulation will give you the<br>
same result as a single larger batch even with parametrizations.</p>
<p>As I understand it, parametrizations are just an “automatic” framework<br>
for interposing your parametrization function between the actual<br>
trainable parameter and the rest of the network and behave no<br>
differently, including with gradient accumulation, than if you had put<br>
in the parametrization function by hand.</p>
<p>Best.</p>
<p>K. Frank</p>

Avelina9X: <p>Hi, thanks for the response!</p>
<p>Unfortunately we’ve run into a second problem in that when caching is enabled the gradients are not retained between backward calls. So we’ve switched to manually computing the <code>Q</code> matrix every forward, although this is significantly slower.</p>

AlphaBetaGamma96: <p>One potential thing that comes to mind (at least for me), is to see if using <code>torch.func.vmap</code> (with the <code>chunk_size</code> option) would get consistent behavior for varying batch sizes, but if the gradient accumulations emerge strictly from OOM issues of a single forward pass (from storing multiple forward passes), this may not be practical.</p>
F.one_hot error: class values must be non-negative

marcomatta: <p>Hi everyone! I’m performing a NER task on a custom dataset using transformers (Roberta-based language model). Due to an imbalanced training set I decided to use the DiceLoss function loss, directly from the official code on github (<a href="https://github.com/ShannonAI/dice_loss_for_NLP/blob/master/loss/dice_loss.py" rel="noopener nofollow ugc">dice_loss_for_NLP</a>).<br>
My task has 38 labels and the model deals with special tokens (used as sentence separators and padding labels) by setting their labels to -100 “so they are automatically ignored in the default loss function” (cross entropy). This is the part of the <em>dice_loss.p</em>y code where I got the “class values must be non-negative” exception:</p>
<pre><code class="lang-auto">def _multiple_class(self, input, target, logits_size, mask=None):
        flat_input = input
        flat_target = F.one_hot(target, num_classes=logits_size).float() if self.index_label_position else target.float()
</code></pre>
<p>Is there a way to make the one_hot() ignore the -100 labels? I tried to cope with this by applying a mask to the tensor labels, which basically replaces -100 with 0 but I actually have a class 0, wouldn’t that affect the loss computation?</p>
<p>My “key-role” tensors are:</p>
<ul>
<li>labels, the target in the function above → shape: (batch size, num tokens in the batch)</li>
<li>logits, the input in the function above → shape: (batch size, num tokens in the batch, num labels)</li>
<li>predictions → shape: (batch size, num tokens in the batch)</li>
</ul>
<p>I can feed the DiceLoss either with logits or predictions.<br>
Sorry if it sounds stupid but I’m new to PyTorch and transformers in general <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:"><br>
Thanks for the help!</p>

mMagmer: <p>hi,<br>
i found an implementation of dice loss <a href="https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/losses/dice.py" rel="noopener nofollow ugc">here</a>, which has <code>ignore_index</code> option.<br>
but it’s for image segmentation.<br>
you should be able to reshape your model output and target for using it directly.(NxCxW —&gt; NxCxWx1)</p>

mMagmer: <p>I checked  the code, and I think it’s wrong.<br>
Don’t use it. Sorry. It is replacing  <code>ignore_index</code> with zero.</p>

akshayklr057: <p><a class="mention" href="/u/marcomatta">@marcomatta</a> Did you find any solution to the problem?</p>

Timm: <p>One hot does not support negative values, but you could convert the negative values to a positive one and ignore them in the loss with the mask.</p>
<p>Something like:</p>
<pre><code class="lang-auto">def nonnegative_multiple_class(self, input, target, logits_size):
    sample_row, neg_indices = torch.where(target &lt; 0)
    target[sample_row, neg_indices] = num_classes

    mask = torch.ones_like(input)
    mask[sample_row, neg_indices] = 0

    self._multiple_class(input, target, logits_size, mask=mask)

</code></pre>
How can a tensor without NaNs and Infs have -inf mean and inf standard deviation?

alexj: <p>Hi, I have a function</p>
<pre><code class="lang-auto">def tensorStory(name, tnsr): 
    print (f"{name}: {tnsr.shape}, dtype: {tnsr.dtype}, device: {tnsr.device}, " + \
           (" has grad, " if tnsr.requires_grad else " no grad, ") + \
            f"has {torch.isnan(tnsr).sum().item()} NaNs and {torch.isinf(tnsr).sum().item()} Infs " +\
            f"avg {torch.mean(tnsr)} dev {torch.std(tnsr)}") 
</code></pre>
<p>Which gives me the following:</p>
<pre><code class="lang-auto">long_matrix: torch.Size([315, 4]), dtype: torch.float32, device: cpu,  has grad, has 0 NaNs and 0 Infs avg -inf dev inf
</code></pre>
<p>How is it even possible???<br>
long_matrix is a result of matrix multiplication involving a sparse COO matrix.</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="alexj" data-post="1" data-topic="208310" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexj/48/70445_2.png" class="avatar"> alexj:</div>
<blockquote>
<pre><code class="lang-auto">long_matrix: torch.Size([315, 4]), dtype: torch.float32, device: cpu,  has grad, has 0 NaNs and 0 Infs avg -inf dev inf
</code></pre>
<p>How is it even possible???</p>
</blockquote>
</aside>
<p>Presumably your tensor in question contains elements that are close<br>
enough to being outside of the range of a <code>float32</code> that the computations<br>
for <code>mean()</code> and <code>dev()</code> overflow.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.4.0'
&gt;&gt;&gt; def tensorStory(name, tnsr):
...     print (f"{name}: {tnsr.shape}, dtype: {tnsr.dtype}, device: {tnsr.device}, " + \
...            (" has grad, " if tnsr.requires_grad else " no grad, ") + \
...             f"has {torch.isnan(tnsr).sum().item()} NaNs and {torch.isinf(tnsr).sum().item()} Infs " +\
...             f"avg {torch.mean(tnsr)} dev {torch.std(tnsr)}")
...
&gt;&gt;&gt; tFloat = -2.e38 * torch.ones (2, requires_grad = True)
&gt;&gt;&gt; tFloat
tensor([-2.0000e+38, -2.0000e+38], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; tensorStory ('tFloat', tFloat)
tFloat: torch.Size([2]), dtype: torch.float32, device: cpu,  has grad, has 0 NaNs and 0 Infs avg -inf dev inf
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Weight parameters with 8, 14 bit precisions?

Alf_manto: <p>Hi, I need to train a small classifier on MNIST using, during training, weights of different precisions (more in detail, 8bit and 14bit). Is this possible to do in Pytorch in an easy way? That is, to simply take the code of an existing classifier online (from the many tutorials), and change a couple of lines to obtain so?</p>

Soumya_Kundu: <p><a href="https://pytorch.org/tutorials/recipes/quantization.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/quantization.html</a></p>
<p>This is a nice gist as well: <a href="https://gist.github.com/martinferianc/d6090fffb4c95efed6f1152d5fde079d" class="inline-onebox" rel="noopener nofollow ugc">Quantisation example in PyTorch · GitHub</a></p>

Alf_manto: <p>Thanks! However, this does not allow to train the model on the precisions I need. It is for inference only. Are you aware of any other ways of doing this?</p>

Soumya_Kundu: <p>Is quantisation aware training something different from what you want to achieve? Or Do you want quantisation in the backward pass too? I imagine this is not an easy feat.</p>