akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Transfer learning v.s. end-to-end training

xblackdavid: <p>Hello everyone,</p>
<p>I’m an ADAS engineer and not an AI major, nor did I graduate with an AI-related thesis, but my current work requires me to start utilizing AI technologies.</p>
<p>My tasks currently involve Behavioral Cloning, Contrastive Learning, and Data Visualization Analysis. For model validation, I use metrics such as loss curve, Accuracy, Recall, and F1 Score to evaluate performance on the training, validation, and test sets. So far, I’ve managed to achieve results that align with some theoretical expectations.</p>
<p>My current model architecture is relatively simple: it consists of an Encoder for static feature extraction (implemented with an MLP - Multi-Layer Perceptron), coupled with a Policy Head for dynamic feature capturing (GRU - Gated Recurrent Unit combined with a Linear layer and Softmax activation).</p>
<p>Question on Transfer Learning and End-to-End Training Strategies<br>
I have some questions regarding the application strategies for Transfer Learning and End-to-End Learning. My main concern isn’t about specific training issues, but rather, I’d like to ask for your insights on the best practices when training neural networks:</p>
<p>Direct End-to-End Training: Would you recommend training end-to-end directly, either when starting with a completely new network or when the model hits a training bottleneck?</p>
<p>Staged Training Strategy: Alternatively, would you suggest separating the Encoder and Policy Head? For instance, initially using Contrastive Learning to stabilize the Encoder, and then performing Transfer Learning to train the Policy Head?</p>
<p>Flexible Adjustment Strategy: Or would you advise starting directly with end-to-end training, and if issues arise later, then disassembling the components to use Contrastive Learning or Data Visualization Analysis to adjust the Encoder, or to identify if the problem lies with the Dynamic Feature Capturing Policy Head?</p>
<p>I’ve actually tried all these approaches myself and generally feel that it depends on the specific situation. However, since my internal colleagues and I have differing opinions, I’d appreciate hearing from all experienced professionals here.</p>
<p>Thanks for your help!</p>
Disabling guards generation using dynamo based export

Himalayjor: <p>How can I disable guards generation when using torch.onnx.export(dynamo=True) with dynamic_shapes.  Guards generated suggests fix which is wrong and doesn’t work.</p>
<p>In my case, it suggests following fixes which is wrong. Model works with any number of frames in torchscript.</p>
<pre><code class="lang-auto">Suggested fixes:
  _frames = Dim('_frames', max=200)
  batch = 1
  frames = 16*_frames - 4
</code></pre>

Hamza_Javaid: <aside class="quote no-group" data-username="Himalayjor" data-post="1" data-topic="221094">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/himalayjor/48/49474_2.png" class="avatar"> Himalayjor:</div>
<blockquote>
<pre><code class="lang-auto">Suggested fixes:
  _frames = Dim('_frames', max=200)
  batch = 1
  frames = 16*_frames - 4
</code></pre>
</blockquote>
</aside>
<p>When you use <code>dynamo=True</code> in ONNX export, PyTorch tries to be helpful by suggesting constraints on your dynamic dimensions, but as you’ve discovered, these suggestions can be overly restrictive or simply incorrect.</p>
<p>Try this first</p>
<pre><code class="lang-auto"># Reset dynamo state
torch._dynamo.reset()

# Disable ALL shape specialization
torch._dynamo.config.automatic_dynamic_shapes = False
torch._dynamo.config.specialize_int = False
torch._dynamo.config.assume_static_by_default = False
torch._dynamo.config.guard_nn_modules = False

# Now export your model
torch.onnx.export(
    model,
    example_input,
    "model.onnx",
    input_names=['input'],
    output_names=['output'],
    dynamic_axes={'input': {1: 'frames'}},  # Assuming frames is dimension 1
    dynamo=True,
    export_options=ExportOptions(dynamic_shapes=True)
)
</code></pre>
<p>Also you can try a more legacy approach</p>
<pre><code class="lang-auto">torch.onnx.export(
    model,
    example_input,
    "model.onnx",
    input_names=['input'],
    output_names=['output'],
    dynamic_axes={'input': {1: 'frames'}},
    opset_version=17,  # Use a recent opset
    # Note: no dynamo=True here
)
</code></pre>
Segmentation error on MacOS because of OpenMP

lorentzenchr: <p>Dear PyTorch community</p>
<p>There is a severe bug reported in <a href="https://github.com/pytorch/pytorch/issues/121101" class="inline-onebox" rel="noopener nofollow ugc">Segmentation error for torch==2.2.1 on MacOs · Issue #121101 · pytorch/pytorch · GitHub</a> that causes segfaults basically after the import statement(s), so worst case for users.<br>
Because the underlying problem stems from different OpenMP runtime libraries (and the fact that PyTorch always ships it without name mangling), this merits some community coordination (lightGBM, scikit-learn and XGBoost all use OpenMP), please refer to <a href="https://github.com/microsoft/LightGBM/issues/6595#issuecomment-2351398026" class="inline-onebox" rel="noopener nofollow ugc">[python-package] SegFault on MacOS when pytorch is installed · Issue #6595 · microsoft/LightGBM · GitHub</a>.</p>
<p>It would be great if you could participate in the discussion in the linked issues.</p>
<p>Best<br>
Christian</p>
Use torch_tensorrt.compiler()

Muhammad_Tarek_Refaa: <p>i try to use torch_tensorrt.compiler() with a model and i got that<br>
File “/home/cplus/projects/m.tarek_master/Image_enhancement/UW_CycleGAN/swin2sr/load_ttrt_model.py”, line 148, in <br>
trt_model_2 = compile_with_torch_tensorrt(model=model.eval(),<br>
File “/home/cplus/projects/m.tarek_master/Image_enhancement/UW_CycleGAN/swin2sr/load_ttrt_model.py”, line 117, in compile_with_torch_tensorrt<br>
trt_model = torch_tensorrt.compile(<br>
File “/home/cplus/projects/m.tarek_master/python_enviornments/virtual_env/lib/python3.10/site-packages/torch_tensorrt/_compile.py”, line 232, in compile<br>
compiled_ts_module: torch.jit.ScriptModule = torchscript_compile(<br>
File “/home/cplus/projects/m.tarek_master/python_enviornments/virtual_env/lib/python3.10/site-packages/torch_tensorrt/ts/_compiler.py”, line 161, in compile<br>
compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))<br>
RuntimeError: 0 INTERNAL ASSERT FAILED at “/pytorch/torch/csrc/jit/ir/alias_analysis.cpp”:616, please report a bug to PyTorch. We don’t have an op for aten::floor_divide but it isn’t a special case.  Argument types: int, int,</p>
<p>Candidates:<br>
aten::floor_divide(Tensor self, Tensor other) → Tensor<br>
aten::floor_divide.Scalar(Tensor self, Scalar other) → Tensor<br>
aten::floor_divide.out(Tensor self, Tensor other, *, Tensor(a!) out) → Tensor(a!)<br>
aten::floor_divide.Scalar_out(Tensor self, Scalar other, *, Tensor(a!) out) → Tensor(a!)</p>
<p>Process finished with exit code 1</p>
Torch.fft in torch_tensorrt

Tomator-2017: <p>I hava a Error when use torch_tensorrt1.4.0 speed up my model with torch.fft in Jetson agx orin(jetpack5.1). I try it in RTX4090 with the latest torch-tensorrt is ok and no similar errors were found. Why have this error in jetson? Is it a problem with lower torch_tensorrt? For me, using the latest version is difficult. So I want to know how to solve this problem。 Thank you very much.</p>
<p>ERROR: [Torch-TensorRT] - Method requested cannot be compiled end to end by Torch-TensorRT.TorchScript.<br>
Unsupported operators listed below:</p>
<ul>
<li>aten::real(Tensor(a) self) → Tensor(a)</li>
<li>aten::fft_ifft2(Tensor self, int[1]? s=None, int[1] dim=[-2, -1], str? norm=None) → Tensor</li>
<li>aten::fft_fft2(Tensor self, int[1]? s=None, int[1] dim=[-2, -1], str? norm=None) → Tensor<br>
You can either implement converters for these ops in your application or request implementation</li>
</ul>
<p>my code:<br>
script_model = torch.jit.script(model) <span class="hashtag-raw">#is</span> ok<br>
spec = {<br>
“forward”: torch_tensorrt.ts.TensorRTCompileSpec(<br>
**{<br>
“inputs”: [torch_tensorrt.Input([1, 3, 1288, 1288])],<br>
“enabled_precisions”: {torch.float},<br>
“refit”: False,<br>
“debug”: True,<br>
“device”: {<br>
“device_type”: torch_tensorrt.DeviceType.GPU,<br>
“gpu_id”: 0,<br>
“dla_core”: 0,<br>
“allow_gpu_fallback”: True,<br>
},<br>
“capability”: torch_tensorrt.EngineCapability.default,<br>
“num_avg_timing_iters”: 1,<br>
}<br>
)<br>
}<br>
trt_model = torch._C._jit_to_backend(“tensorrt”, script_model, spec)</p>

ptrblck: <p>What’s blocking you from using the latest releases?</p>

Tomator-2017: <p>I use docker ‘jetson-containers run -it dustynv/torch_tensorrt:r35.4.1’ manage my environment in <a href="https://github.com/dusty-nv/jetson-containers/tree/master/packages/pytorch/torch_tensorrt" rel="noopener nofollow ugc">jetson-containers/packages/pytorch/torch_tensorrt at master · dusty-nv/jetson-containers</a>, because I cannot find the torch_tensorrt version of arrch64. Have you some idea to use the latest releases in Jetson? Thank you.</p>

Tomator-2017: <p>sir, Could you tell me how install the latest torch_tensorrt in Jetson AGX Orin? Just use ‘python3 setup.py install --jetpack-version 5.1 --use-cxx11-abi’, I have try it, but dont success. Whats more, I find that ‘ir=dynamo’ is not supported in my torch-2.0.0+nv23.05, torch_tensorrt-1.4.0, will this be a significant reason for my failure? I would greatly appreciate any response from you，thanks.</p>

Hamza_Javaid: <p>Use this</p>
<pre><code class="lang-auto">docker pull nvcr.io/nvidia/l4t-pytorch:r35.2.1-pth2.0-py3
docker run --runtime nvidia -it --rm --network=host nvcr.io/nvidia/l4t-pytorch:r35.2.1-pth2.0-py3
</code></pre>
Call backward() twice does not incur errors

bingqing_liu: <pre><code class="lang-auto">x=torch.rand(2,2,requires_grad=True)
y=x+2
# y=x*2
z=torch.sum(y)
zz=z*2
z.backward()
zz.backward()
</code></pre>
<p>call backward() twice does not incur errors. However, if i use<code>  y=x*2</code> rather than <code>y=x+2</code>, the error occurs, why?</p>
<pre><code class="lang-auto">x=torch.rand(2,2,requires_grad=True)
#y=x+2
y=x*2
z=torch.sum(y)
zz=z*2
z.backward()
zz.backward()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e21a24d19d4e37445c62930c66f5a278967aacb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/e21a24d19d4e37445c62930c66f5a278967aacb4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e21a24d19d4e37445c62930c66f5a278967aacb4_2_689x125.png" alt="image" data-base62-sha1="wgbV76VwwJqooLjnCjCH5T03L4U" width="689" height="125" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e21a24d19d4e37445c62930c66f5a278967aacb4_2_689x125.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e21a24d19d4e37445c62930c66f5a278967aacb4_2_1033x187.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e21a24d19d4e37445c62930c66f5a278967aacb4.png 2x" data-dominant-color="F9D8D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1195×217 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

soulitzer: <p>x * 2 saves a value (2) for backward, and the first time backward happens, this value is cleared<br>
this causes the second backward to error unless retain_graph=True</p>
<p>x + 2 does not have the same problem because it doesn’t need to save any values in order to compute the backward</p>
Building the exact same docker images as seen in Docker Hub

chargeuk: <p>Hi,<br>
Recently I have been building my own docker images, but I find them to be much larger than the official ones, at around 18GB. In addition, I am getting different performance characteristics within comfyUI.</p>
<p>Can somepne confirm the exact command I need to run to produce the <a href="https://hub.docker.com/layers/pytorch/pytorch/2.7.1-cuda12.6-cudnn9-devel/images/sha256-639b8229ccfd8a3aa803cf49c33d6d6fe406750d79aaf723fe8c0eb1060d8cff" rel="noopener nofollow ugc">pytorch/pytorch:2.7.1-cuda12.6-cudnn9-devel</a> docker image?</p>
<p>I am currently building using the docker.Makefile, but recently noticed the .ci/docker/build.sh file and I’m unsure which I should use, and what commands to pass in to get the same exact images.</p>
<p>Thanks</p>

ptrblck: <p>Check the <a href="https://github.com/pytorch/pytorch/blob/2eb744c08d600e84b167dbda7daa792243a2c235/Dockerfile">Dockerfile</a> in our repository which just <code>pip install</code>s the binaries if I’m not mistaken as seen <a href="https://github.com/pytorch/pytorch/blob/2eb744c08d600e84b167dbda7daa792243a2c235/Dockerfile#L74-L77">here</a>.</p>
Can the aten ir return both in-place and non-in-place operation parameters at the same time? 

lost-city: <p>I currently have an operator’s aten ir and want to return both in-place and non-in-place operation parameters at the same time. How should I define the aten ir? Can it be defined in the following way?<br>
func: test_op(Tensor a, Tensor b, Tensor c, Tensor d, Tensor(a!) e, Tensor(b!) f, *, Tensor? g=None, Tensor? h=None) → (Tensor, Tensor, Tensor(a!), Tensor(b!))<br>
<a class="mention" href="/u/cpuhrsch">@cpuhrsch</a></p>
(Using Adadelta) After reloading checkpoint loss exhibit high rate of change compared to when it was saved

Craig_Hicks1: <pre><code class="lang-auto">@final
class Checkpoint:
    def __init__(self, epoch: int, accuracy: float, loss: float,
                 model: Net,
                 optimizer: Optimizer
) -&gt; None:
        self.epoch = epoch
        self.accuracy = accuracy
        self.loss = loss
        self.model = copy.deepcopy(model.state_dict())
        self.optimizer = copy.deepcopy(optimizer.state_dict())
        self.rng_state = torch.get_rng_state()
        self.cuda_rng_state = torch.cuda.get_rng_state() if torch.cuda.is_available() else None
        self.numpy_rng_state = np.random.get_state()
        self.python_rng_state =random.getstate()

    def restore(self, model: Net, optimizer: Optimizer):
        assert self.model
        model.load_state_dict(self.model)
        assert self.optimizer
        optimizer.load_state_dict(self.optimizer)
        # older versions didn't have the rng states
        if hasattr(self,"rng_state"):
          torch.set_rng_state(self.rng_state)
        if hasattr(self,"cuda_rng_state"):
          if torch.cuda.is_available() and self.cuda_rng_state is not None:
              torch.cuda.set_rng_state(self.cuda_rng_state)
        if hasattr(self,"numpy_rng_state"):
          np.random.set_state(self.numpy_rng_state)
        if hasattr(self,"python_rng_state"):
          random.setstate(self.python_rng_state)

</code></pre>
<p>The lr is confirmed to be the same, but the rate of change of loss function initially high, even though before saving it had reached a steady state plateau.  It as though the adadelta state is being recomputed from scratch, yet I can see that  the state is successfully reloaded.  An array of 10 parameter groups, corresponding to 5 layers, are there.  The groups each have “step”, “square_avg”, and “acc_delta” tensors.</p>
<p>Any similar experiences?</p>
What loss function should the inner loop of MAML use?

Vulcan88: <p>Hi,</p>
<p>I’m implementing MAML in the Reinforcement Learning domain. I’m planning to use PPO training for validation. However I want to know if the PPO loss function is appropriate during Meta-Training. The emphasis on gradual policy updates may not be suitable for fast adaptation to tasks in the inner loop?</p>
<p>I’ve seen some implementations use just the policy loss. However, I do not understand <em>how</em> you can calculate an accurate policy loss if you have an untrained value network (As the advantage relies on the value estimates).</p>
<p>Can someone help me to understand this?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

clark598dahl: <aside class="quote no-group" data-username="Vulcan88" data-post="1" data-topic="214371">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vulcan88/48/74543_2.png" class="avatar"> Vulcan88:</div>
<blockquote>
<p>Hi,</p>
<p>I’m implementing MAML in the Reinforcement Learning domain. I’m planning to use PPO training for validation. However I want to know if the PPO loss function is appropriate during Meta-Training. The emphasis on gradual policy updates may not be suitable for fast adaptation to tasks in the inner loop?</p>
<p>I’ve seen some implementations use just the policy loss. However, I do not understand <em>how</em> you can calculate an accurate policy loss if you have an untrained value network (As the advantage relies on the value estimates).</p>
<p>Can someone help me to understand this?</p>
</blockquote>
</aside>
<p>Hello,<br>
Using PPO with MAML for RL meta-training—wondering if PPO’s conservative updates hinder fast inner-loop adaptation. Also unclear how to compute policy loss accurately without a trained value network, since advantage estimates depend on it. Any insights? <a href="https://www.catmeowing.net" rel="noopener nofollow ugc">cat meowing</a></p>
<p>Best Regards,<br>
Clark Dahl</p>

Hamza_Javaid: <p>PPO and MAML don’t really play nice together. PPO is all about being careful and conservative with updates (that’s the whole point of the clipping), but MAML needs to adapt FAST in just a few gradient steps.</p>
<p>The Value Function Problem: how can you calculate advantages without a trained value function? Most folks just.. don’t use a value function in the inner loop. Instead, they use the actual returns from the episodes.</p>
<pre><code class="lang-auto">returns = []
R = 0
for r in reversed(rewards):
    R = r + 0.99 * R
    returns.insert(0, R)
    
# Super simple "advantage"
advantages = returns - np.mean(returns)
</code></pre>
<p>Not fancy, but it works often</p>
Copy weights from PyTorch GRU layer to TensorFlow

nada: <p>Hello,</p>
<p>I am trying to copy weights from a PyTorch GRU layer to TensorFlow.<br>
My layers def are:</p>
<p><code> layers.GRU(units=100, activation=None, dropout=0.0, reset_after=True) nn.GRU(input_size=400, hidden_size=100, bidirectional=False, dropout=0.0, batch_first=True)</code></p>
<p>I copy the weights using:</p>
<pre><code>  w_ih = pt_layer.weight_ih_l0.detach().numpy().T  # (input_dim, 3*hidden_size)
  w_hh = pt_layer.weight_hh_l0.detach().numpy().T  # (hidden_size, 3*hidden_size)
  b_ih = pt_layer.bias_ih_l0.detach().numpy()
  b_hh = pt_layer.bias_hh_l0.detach().numpy()

  # Helper funcs to reorder gates: PyTorch [z, r, n] -&gt; TensorFlow [r, z, n]
  def reorder_gru_weights(w):
      z, r, n = np.split(w, 3, axis=1)
      return np.concatenate([r, z, n], axis=1)

  def reorder_gru_bias(b):
      b_z, b_r, b_n = np.split(b, 3)
      return np.concatenate([b_r, b_z, b_n])

  w_ih = reorder_gru_weights(w_ih)
  w_hh = reorder_gru_weights(w_hh)
  b_ih = reorder_gru_bias(b_ih)
  b_hh = reorder_gru_bias(b_hh)


  reset_after = getattr(tf_layer, 'reset_after', False)

  if reset_after:
      bias = np.stack([b_ih, b_hh], axis=0)
  else:
      bias = b_ih + b_hh

  tf_shapes = [w.shape for w in tf_layer.get_weights()]
  new_weights = [w_ih, w_hh, bias]

  for i, (target, actual) in enumerate(zip(tf_shapes, [w.shape for w in new_weights])):
      if target != actual:
          raise ValueError(f"Mismatch at weight {i}: expected {target}, got {actual}")

  tf_layer.set_weights(new_weights)
</code></pre>
<p>Running the two layers, I get:<br>
TensorFlow output: [[0.4295744]]<br>
PyTorch output: [[0.44837096]]<br>
Max absolute difference: 0.018796563</p>
<p>PS: The networks contain other Linear and Conv layers. When I remove the GRU layers, that difference is significantly smaller (around 0.00002)</p>
<p>Am I missing something in how I copy the weights? Thanks</p>
Exporting model to onnx using scripted model fails

Himalayjor: <p>I am trying to export ONNX model using torch.onnx.export(torch.jit.script(model), dynamo=False)</p>
<p>How can I figure out which tuples it can not lower. Model is very complex and most of the forward method returns tuples.</p>
<p>Error:</p>
<pre><code class="lang-auto"> File "/home/himalay.joriwal/.conda/envs/uat/lib/python3.11/site-packages/torch/onnx/__init__.py", line 375, in export
    export(
  File "/home/himalay.joriwal/.conda/envs/uat/lib/python3.11/site-packages/torch/onnx/utils.py", line 502, in export
    _export(
  File "/home/himalay.joriwal/.conda/envs/uat/lib/python3.11/site-packages/torch/onnx/utils.py", line 1564, in _export
    graph, params_dict, torch_out = _model_to_graph(
                                    ^^^^^^^^^^^^^^^^
  File "/home/himalay.joriwal/.conda/envs/uat/lib/python3.11/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
    graph = _optimize_graph(
            ^^^^^^^^^^^^^^^^
  File "/home/himalay.joriwal/.conda/envs/uat/lib/python3.11/site-packages/torch/onnx/utils.py", line 581, in _optimize_graph
    _C._jit_pass_lower_all_tuples(graph)
RuntimeError: Couldn't lower all tuples

</code></pre>
How do I install NVTOOLSEXT

sort: <p>I want to use CUDA to run my models, and I have both CUDA and the Lib Torch Directories installed. I installed the full CUDA toolkit, latest model, and everything installed well. The only thing I am missing is the NVTOOLSEXT_PATH, but I cannot figure out where that is or how to find it. I have not found it in the Nvidia GPU computing toolkit folder. If there are any articles on this error please link them because I have not been able to find them. I am operating on windows 11 and using Microsoft Visual Studio 2022. Thank you so much.</p>

random_function: <p>hey, got same problem, have you found any solutions?</p>

FeelTheOatmeal: <p>Also encountering the same problem…Anyone figure something out?</p>
<p>It looks like CUDA 12.1 onwards nvTools has just become a header file and so the folder is missing and is being searched for by the extension which it ultimately cannot find.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49d08247bc384d8b56a5e37bfa78176c1850121b.png" data-download-href="https://discuss.pytorch.org/uploads/default/49d08247bc384d8b56a5e37bfa78176c1850121b" title="Template_issue"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49d08247bc384d8b56a5e37bfa78176c1850121b.png" alt="Template_issue" data-base62-sha1="awZAbq1cxVHtbEyXSJG5iemWXqj" width="373" height="75"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Template_issue</span><span class="informations">498×100 2.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The official torch documentation for getting started with libtorch in VS provides a guide/video to try using this extension, from which this error occurs.</p>
Why the reserved memory is much larger than occupied memory?

Zoe_Zhao: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33e9f1a8abd06b5513e78c52797c9e089783b57c.png" data-download-href="https://discuss.pytorch.org/uploads/default/33e9f1a8abd06b5513e78c52797c9e089783b57c" title="torch.profiler.profile"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33e9f1a8abd06b5513e78c52797c9e089783b57c_2_690x424.png" alt="torch.profiler.profile" data-base62-sha1="7pfzz8QNbD0l3zvfXsxDpxdgnhi" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33e9f1a8abd06b5513e78c52797c9e089783b57c_2_690x424.png, https://discuss.pytorch.org/uploads/default/original/3X/3/3/33e9f1a8abd06b5513e78c52797c9e089783b57c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/3/33e9f1a8abd06b5513e78c52797c9e089783b57c.png 2x" data-dominant-color="E0CEC9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch.profiler.profile</span><span class="informations">960×591 39.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
For my understanding, the y axis is the occupied memory. The part of the allocated memory that is actually being used to store data at a given moment — like active tensors, gradients, etc. Instead of frequently asking CUDA to allocate and free memory, the allocated memory store data to be used. If I use <code>torch.cuda.empty_cache()</code> to delete the cache memory (i.e. the extra part of reserved memory), will it affect the speed?</p>

ptrblck: <aside class="quote no-group" data-username="Zoe_Zhao" data-post="1" data-topic="221063">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zoe_zhao/48/78238_2.png" class="avatar"> Zoe_Zhao:</div>
<blockquote>
<p>If I use <code>torch.cuda.empty_cache()</code> to delete the cache memory (i.e. the extra part of reserved memory), will it affect the speed?</p>
</blockquote>
</aside>
<p>Yes, clearing the cache will synchronize your code and free the cached memory (which is also slowing down your code). The next iteration will then use the slow and synchronizing <code>cudaMalloc</code> calls (assuming the default caching allocator is used) to allocate the needed memory.</p>
NVDECODE and NVENCODE APIs

Yep: <p>Does pytorch use nvidia’s gpu built-in Hardware-Based Decoder and Encoder</p>

ptrblck: <p>Yes, <code>vision</code> uses these APIs as described in the <a href="https://github.com/pytorch/vision/blob/0721867e42841171254c7acaa45fbaf8ee16d3d7/torchvision/csrc/io/decoder/gpu/README.rst#L4">decoder README</a>:</p>
<blockquote>
<p>GPU decoder depends on ffmpeg for demuxing, uses NVDECODE APIs from the nvidia-video-codec sdk and uses cuda for processing on gpu.</p>
</blockquote>
Multi-discriminator GAN: No inf checks were recorded for this optimizer

40uf411: <p>Any idea why I am getting <em><strong>AssertionError: No inf checks were recorded for this optimizer.</strong></em><br>
Here is the code snippet:</p>
<pre><code class="lang-auto">def generator_step(args, device, planes, netG, optG, scalerG, discriminators, pos_coeff, active_plane, x_3d_prev, t):
    gen_loss = 0.
    netG.zero_grad()
    x_3d = denoising_step(args, device, netG, pos_coeff, x_3d_prev, t)
    for plane in planes:
        if args.discriminator_cycle_length &gt; 0 and plane != active_plane: # type: ignore
            continue
        cur = extract_plane_slices(x_3d, plane, args.slices_per_axis)
        prev = extract_plane_slices(x_3d_prev, plane, args.slices_per_axis)
        t_exp = t.unsqueeze(1).repeat(1, args.slices_per_axis).view(-1)
        out_fake_G = discriminators[plane](cur, t_exp, prev).view(-1)
        gen_loss += F.softplus(-out_fake_G).mean() / len(planes) if args.discriminator_cycle_length &lt;= 0 else F.softplus(-out_fake_G).mean()
    scalerG.scale(gen_loss).backward()
    scalerG.step(optG)
    scalerG.update()
    if args.ema:
        optG.swap_parameters_with_ema(store_params_in_ema=True)
    return x_3d,gen_loss
</code></pre>
<pre><code class="lang-auto">def denoising_step(args, device, netG, pos_coeff, x_3d_prev, t):
    z = torch.randn(args.batch_size, args.nz, device=device)
    with autocast(dtype=torch.float16):
        x0 = netG(x_3d_prev, t, z)
    x_3d = sample_posterior(pos_coeff, x0, x_3d_prev, t).detach()
    return x_3d
</code></pre>

ptrblck: <p>You are detaching the output of <code>netG</code>:</p>
<aside class="quote no-group" data-username="40uf411" data-post="1" data-topic="221049">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/40uf411/48/55863_2.png" class="avatar"> 40uf411:</div>
<blockquote>
<pre><code class="lang-auto">    with autocast(dtype=torch.float16):
        x0 = netG(x_3d_prev, t, z)
    x_3d = sample_posterior(pos_coeff, x0, x_3d_prev, t).detach()
</code></pre>
</blockquote>
</aside>
<p>and I assume <code>optG</code> is the optimizer for <code>netG</code> which won’t do anything since the computation graph was cut and thus no gradients will be calculated in the <code>backward</code> pass.</p>

40uf411: <p>Rookie mistake, thanks.</p>
Running a transformer inside an RNN (Unparallelize a Transformer)

Stingpie_52: <p>I’m working on an RNN that is broken down into several modules, and one of these modules estimates the accuracy of a given result. The RNN has multiple different ways of computing the result, and by estimating the accuracy of a result, it is able to gauge the efficacy of the technique. I want to use a transformer for this module, for it’s strong predictive power, but I can’t figure out a way to do it without recalculating results multiple times.</p>
<p>Each timestep, the module receives the input-output pair of tensors from the rest of the RNN. The result of the module is then used in the next step to modify the technique of the RNN. The issue is, each timestep I’m having to recalculate all the previous values.<br>
i.e.</p>
<pre><code class="lang-auto">step 1: X1 -&gt; RNN -&gt; Y1; ((X1),(Y1))-&gt;Transformer-&gt;A1
step 2: X2 -&gt; RNN -&gt; Y2; ((X1,X2),(Y1,Y2))-&gt;Transformer-&gt;(A1,A2)
step 3: X3 -&gt; RNN -&gt; Y3; ((X1,X2,X3),(Y1,Y2,Y3))-&gt;Transformer-&gt;(A1,A2,A3)
</code></pre>
<p>I only need A1 from step 1, A2 from step 2, and A3 from step 3, but step 3 calculates A1, A2 <em>and</em> A3.<br>
Ideally I would want:</p>
<pre><code class="lang-auto">step 1: X1 -&gt; RNN -&gt; Y1; (X1,Y1)-&gt;Transformer-&gt;A1
step 2: X2 -&gt; RNN -&gt; Y2; (X2,Y2)-&gt;Transformer-&gt;A2
step 3: X3 -&gt; RNN -&gt; Y3; (X3,Y3)-&gt;Transformer-&gt;A3
</code></pre>
<p>This shouldn’t affect the accuracy of the model as a whole, but it’s nearly impossible to train due to the giant computation graphs it creates–my computer runs out of memory almost instantly.</p>
<p>I’ve looked at KV-caching, but that seems to be exclusively for inference, and I also don’t know how to program that.<br>
Maybe I could copy and paste one of those ‘make a transformer from scratch’ tutorials, but I’d prefer not to.</p>
<p>Anyone else have an idea how to solve this?</p>
Jagged nested tensors massively slow down a DataLoader

Matt_Pitkin: <p>The <a href="https://docs.pytorch.org/docs/stable/nested.html#construction" rel="noopener nofollow ugc">current recommendation</a> for nested tensors is to create them with a <code>torch.jagged</code> layout as opposed to the <code>torch.strided</code> layout. However, I have noticed a considerable drawback to this - if I have a nested tensor in a <a href="https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="noopener nofollow ugc"><code>Dataset</code></a> for use in a <a href="https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>DataLoader</code></a>, the dataset using jagged layouts is massively slower. For example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch


# dataset with nested tensor
class TestDatasetNested(torch.utils.data.Dataset):
    def __init__(self, N=100, dl=100, layout=torch.jagged):
        self.N = N
        self.dl = dl

        # set std deviations
        nnoise = torch.randint(1, high=5, size=(self.N,))
        sigmas = [torch.rand(n) for n in nnoise]

        self.sigmas = torch.nested.nested_tensor(sigmas, layout=layout)

    def __getitem__(self, i):
        sigmas = self.sigmas[i % self.N]

        return torch.cat([sigma * torch.randn((self.dl, 1)) for sigma in sigmas], dim=-1).sum(dim=-1)

    def __len__(self):
        return self.N


# create dataset with jagged and strided nested tensor layouts
dataset_jagged = TestDatasetNested(layout=torch.jagged)
dataset_strided = TestDatasetNested(layout=torch.strided)

# create dataloader for both cases
dl_jagged = torch.utils.data.DataLoader(dataset=dataset_jagged, batch_size=10)
dl_strided = torch.utils.data.DataLoader(dataset=dataset_strided, batch_size=10)

def nepochs(dl, n=100):
    for i in range(n):
        for _ in dl:
            pass

%timeit nepochs(dl_jagged)
4.57 s ± 17.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit nepochs(dl_strided)
322 ms ± 3.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>So, in this case using the jagged layout is ~14 times slower. Does anyone know what this is the case and if it can be improved?</p>
Model Evaluation Metrics on edge devices (Beginner Question)

Mahammad_Nabizade: <p><strong>Sorry if this sounds a bit noob — I’m still new to deploying deep learning models on edge devices.</strong></p>
<p>I’ve been reading a lot of academic papers and what I keep seeing is that, they only report latency or FPS when they talk about real-time performance on the device. But I do not see any predictive metrics like accuracy, precision, or recall reported <em>on-device</em> during deployment.</p>
<p>My question is:<br>
Why don’t we just take a small chunk of the test set (isolated before the training), run it directly on the edge device, and evaluate the predictive performance while the model is running on that hardware? That seems like it would give us the most realistic measure of the model’s actual performance in deployment. Is this approach:</p>
<ul>
<li>Not standard practice?</li>
<li>Technically difficult or even impossible?</li>
<li>Considered meaningless or unnecessary?</li>
</ul>
<p>And more generally — what <em>is</em> the standard process here?<br>
Is it:</p>
<ol>
<li>Train and test the model locally (with full evaluation metrics),</li>
<li>Deploy the model on the device,</li>
<li>Then only measure latency/FPS on-device — and nothing about predictive accuracy?</li>
</ol>

ptrblck: <p>My expectation would be to see the same predictive accuracy on the training and deployment device. I.e. even if different algorithms, kernels, etc. are used causing the expected numerical mismatches due to the limited floating point precision, I would not expect to see any difference in the actual prediction unless the implementation is broken.</p>

kafkashino: <p>This is a really insightful question — definitely not noob at all. You’re right to wonder why on-device predictive performance isn’t more commonly reported, especially since deployment environments can introduce subtle changes (e.g., quantization errors, hardware-specific ops). In practice, many researchers assume that accuracy doesn’t change post-training, especially if the model architecture and weights remain untouched. But when quantization or other optimizations are involved, it absolutely makes sense to evaluate a held-out test set <em>on the actual device</em>.</p>
<p>That said, the standard workflow does tend to separate model evaluation (done offline) and deployment benchmarking (focused on latency/FPS/power). This is mostly due to practicality and reproducibility — not because it’s meaningless to evaluate on-device accuracy. In real-world applications, teams often do exactly what you described: send a test batch through the deployed model and log the results for validation.</p>
<p>So no, it’s not meaningless — just underreported in academic papers. Great that you’re thinking about this!</p>

fizzaliaqat: <p>Absolutely agree — prediction consistency between training and deployment is a fundamental expectation. Minor floating-point differences are understandable due to hardware or kernel variations, but if the outputs are diverging beyond tolerance, it’s usually a sign of something deeper like preprocessing mismatches, model export issues, or incorrect inference configuration.</p>
No rule registered for HOP triton_kernel_wrapper_mutation

KC314: <p>At the beginning of my model training, I run a forward and backward under FlopCounterMode context manager to compute the total number of flops. I recently upgraded to torch==2.7 from torch==2.3 and I’m now getting the following error</p>
<pre><code class="lang-auto">NotImplementedError: There was no rule registered for HOP triton_kernel_wrapper_mutation and mode &lt;torch.utils.flop_counter._FlopCounterMode object at 0x7f6798e07cb0&gt;. We recommend filing an issue.
Error: SystemExit: worker with local_rank 0 exited with non-zero exitcode 1
</code></pre>
<p>I’ve determined this is due to the fact that I have a custom triton kernel in my code. Ideally, I’d like to just dispatch to the usual implementation and not bother with estimating the FLOPs of this kernel (it’s negligble anyway). I tried to use the <code>HigherOrderOperator.py_impl</code> method but I can’t manage to make it work correctly. Specifically, I’m not sure how exactly to register my implementation for FlopCounterMode (which should just call out to run my triton kernel like normal).</p>
<p>I’m running on python 3.12 with no torch.compile on an H200 GPU.</p>
Graph exported by torch.onnx.dynamo_export

April_Hu: <p>Hi all,</p>
<p>I am trying to export a model from PyTorch to Onnx using the new <code>torch.onnx.dynamo_export</code> interface. Based on <a href="https://cloudblogs.microsoft.com/opensource/2023/08/01/introducing-onnx-script-authoring-onnx-with-the-ease-of-python/" rel="noopener nofollow ugc">this article</a> and the result I got, it seems that graphs exported by <code>torch.onnx.dynamo_export</code> are composed of aten functions like what is shown in the post <a href="https://pytorch.org/docs/stable/onnx_dynamo.html#inspecting-the-onnx-model-using-gui" rel="noopener nofollow ugc">here</a>. However, what I want is a graph composed of ONNX ops instead just like what is exported by the old <code>torch.onnx.export</code> method (i.e., if I keep clicking the <code>f</code> icon on the top right corner of these aten functions in Netron, I can get the decomposed graph with only ONNX ops I want). Just wondering, is there a way to achieve a flattened ONNX graph without these hierarchy of aten funcs to ONNX ops with the new dynamo export interface?</p>

siahuat0727: <p>Same question here! Do you have any insight after that?</p>

YixuanSeanZhou: <p>Hi <a class="mention" href="/u/april_hu">@April_Hu</a> , I wonder did you end up figuring it out?</p>
<p>Thanks!</p>
How to disable triton when calling torch.compile in max-autotune mode

proshm: <p>I train my model in DDP and introduce torch.compile:<br>
model = torch.compile(model, mode=“max-autotune”)<br>
it worked until triton was installed. So I wonder how to disable triton before torch.compile in max-autotune mode</p>

ptrblck: <p>You could try to disable Triton via selecting the desired backends in <code>torch._inductor.config.max_autotune_gemm_backends</code>.</p>
Pin_memory limitation

B_S: <p>I can’t pin more than 2gb at ones, to pin more memory I need to break it to multiple chunks.<br>
for example doing this will arise an error</p>
<pre><code class="lang-auto">import torch
buffer =  torch.empty(int(3.0*1024**3), dtype=torch.uint8, pin_memory=True)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>however pinning larger memory by breaking into multiple chunks will work fine</p>
<pre><code class="lang-auto">buffer_1 = torch.empty(int(2.0*1024**3), dtype=torch.uint8, pin_memory=True)
buffer_2 = torch.empty(int(2.0*1024**3), dtype=torch.uint8, pin_memory=True)
</code></pre>
<p>I don’t know why this is happening, the size of the memory &gt; 1T and there is no limitation on the max locked memory. also it seems that it is occurring with others too, check <a href="https://github.com/deepspeedai/DeepSpeed/issues/7150" rel="noopener nofollow ugc">this discussion</a><br>
in deepspeed github repo.</p>
<p><strong>more system info</strong>:<br>
os: 24.04.1-Ubuntu<br>
gpu :  NVIDIA H100<br>
python: 3.12<br>
cuda:  12.8<br>
pytorch: 2.7.0</p>

ptrblck: <p>Not reproducible on my system, so I guess your system might disallow these large allocations:</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; buffer =  torch.empty(int(3.0*1024**3), dtype=torch.uint8, pin_memory=True)
&gt;&gt;&gt; buffer.shape
torch.Size([3221225472])
</code></pre>

python: <p>I’m seeing the same issue on my end: allocating a single pinned memory buffer larger than ~2GB throws a <code>CUDA error: invalid argument</code>, but multiple smaller chunks work fine. This points to a limitation on the size of individual contiguous pinned memory allocations, not the total pinned memory available. It may be due to driver or hardware-level constraints, especially with the H100 architecture. Since there are no locked memory limits set, the behavior likely stems from how the CUDA backend handles large pinned memory requests.</p>

B_S: <p>would be helpful if you know how to check for such constraints</p>
Optimizing CNN Models on Edge Devices for Real-time IoT Applications

aria_james: <p>Hi everyone, I’m currently working on deploying a lightweight CNN model on edge devices for an IoT-based object detection project. The tricky part is optimizing PyTorch-trained models to run efficiently on constrained hardware, such as ESP32, in conjunction with external processors. While going through resources, I found a helpful overview about IoT basics <a href="https://www.theengineeringprojects.com/2023/06/top-iot-starter-kits-for-the-beginners-to-learn-programming.html" rel="noopener nofollow ugc">https://www.theengineeringprojects.com/2023/06/top-iot-starter-kits-for-the-beginners-to-learn-programming.html</a> which nicely explains the broader IoT setup I’m using, but it doesn’t cover the machine learning side in depth. Has anyone here successfully converted and deployed a model for real-time IoT applications? Would love any practical tips or recommended workflows.</p>
Torch.matmul calls nvjet_tst kernel in torch 2.7.1

TerrenceZhangX: <p>Hi, I’m testing torch 2.7.1 matmul performance with cuda 12.6 on H100 GPU. Surprisingly the matmul is calling nvjet kernel instead of sm90_xmma kernel with torch 2.6.0.<br>
Does anyone know why this behaviour is happening?<br>
Thanks a lot for the kind help!</p>

ptrblck: <p>This is expected as cuBLAS selects the fastest kernel for the given workload via its heuristics.</p>

TerrenceZhangX: <p>Thanks for the clarification! Very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to know if gradients backpropagate through a pytorch function?

Anthony_Quint: <p>For the pytorch functions:</p>
<ul>
<li>torch.special.erf() which is used in torch.distributions.normal.Normal().cdf()</li>
<li>torch.special.erfinv() which is used in torch.distributions.normal.Normal().icdf()</li>
<li>torch._standard_gamma() which is used in torch.distributions.gamma.Gamma().rsample()</li>
</ul>
<p>I’m wondering if gradients can actually backpropagate through these? It appears that they each have their respective grad_fn’s, however, for something like torch._standard_gamma(“parameter”), I thought this was a completely stochastic node (statistically I’m not sure how you could reparametrize this), and so I thought can’t backpropagate through “paramete”.</p>
<p>As a more general question, I’m curious on how to know if gradients can backpropagate through a Pytorch function?</p>
<p>I appreciate any help! I’m doing some augmentations with VAEs for my thesis, so I want to make sure this works.</p>

andrewkoulo: <p>this post is very old but for anyone working with gradients for the gamma distribution in the future, here is code that verifies indeed the function has a valid backward pass:</p>
<p>alpha = torch.tensor([2.0, 3.5], requires_grad=True)<br>
x = torch._standard_gamma(alpha)<br>
print(x.grad_fn)   `</p>
Combining image and tabular data for a binary classification task

Antonio_Rossi: <p>Hi all,</p>
<p>I’m working on a binary classification task where the goal is to determine whether a tissue contains malignant cells</p>
<p>Each instance in my dataset consists of</p>
<p>*a microscope image of the tissue<br>
a small set of tabular metadata including an identifier of the imaging session and a binary feature indicating whether the cell was treated with fluorescent particles or not</p>
<p>I’m considering a hybrid neural network combining a CNN to extract features from the image<br>
and either a TabNet model or a fully connected MLP to process the tabular data</p>
<p>My idea is to concatenate the features from both branches and pass them to a shared classification head</p>
<p>My questions<br>
1 how should I handle the identifier? should I one embed it or drop it completely (overfitting)<br>
2 are there alternative ways to model the tabular branch beyond MLP or TabNet especially with very few tabular features<br>
3 any best practices when combining CNN image embeddings with tabular data?</p>
<p>Thanks in advance for any suggestions or shared experiences</p>
Issues with torch.compile in DistributedModelParallel (DMP)

pygoraz: <p>Hi!</p>
<p>I’m recently torch compiling a module before wrapping the model with DMP. The module I try to compile is not sharded. However, it seems the module is not compiled at all by inspecting the pytorch profiler trace. Does torch compile support DMP?</p>

ptrblck: <p>Could you share where <code>DMP</code> comes from and how it’s defined for your model?</p>