akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Functional.nn.grid_sample does not support the bicubic method for 5D tensor

anitrop: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/7276584510bb27cdac0fc29047e8fe2a5f538aa2.png" data-download-href="https://discuss.pytorch.org/uploads/default/7276584510bb27cdac0fc29047e8fe2a5f538aa2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/7276584510bb27cdac0fc29047e8fe2a5f538aa2_2_690x456.png" alt="image" data-base62-sha1="gkzZ1ho80woHgPgowQUHTv1j72O" width="690" height="456" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/7276584510bb27cdac0fc29047e8fe2a5f538aa2_2_690x456.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/7276584510bb27cdac0fc29047e8fe2a5f538aa2_2_1035x684.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/2/7276584510bb27cdac0fc29047e8fe2a5f538aa2.png 2x" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1093×723 66.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I need implement warp on volume feature, But ‘bilinear’ method doesn’t give me the accuracy I need.Or is there another way to implement warp on features?<br>
Thanks for your help!!!</p>
How could I share a tensor amoung processes?

coincheung: <p>Hi,</p>
<p>I launched distributed training, and there is a tensor that is identical in each process, I would like this tensor to keep only one copy in the memory and can be accessed by multi-process.</p>
<p>However, when I defined the tensor and run <code>share_memory_()</code>. It seems that all the processes still defines their own tensor and all of them moved their tensors into the shared memory which does not save memory at all.</p>
<p>Would you tell me how could I save memory in this way?</p>
Exporting libtorch model and loading in Python

botelho: <p>Is it possible to save a <code>nn::Module</code> to file in libtorch, and then later load it in python for inference with Pytorch? I’ve seen several examples of the other way around, but never a complete description of the C++ --&gt; model file --&gt; Python direction that actually works. Thanks!</p>

tom: <p>This does not seem to be something that is often requested.<br>
I would probably wrap the model in a Python extension and use that. This is safely within what is supported.</p>
<p>Or you could try to trace the model and then save it with the JIT, but that would lose the model structure (I think), as the magic that lets you keep the model structure in Python is likely missing. I must admit that I didn’t try tracing in C++ and the last time I suggested to just do as if you would in Python (for the anomaly detection) and that it should be working, it would just segfault and I had to go and implement the feature just to avoid leaving a wrong answer in the forums for posterity. No guarantee from me that this works.</p>
<p>Best regards</p>
<p>Thomas</p>

botelho: <p><a class="mention" href="/u/tom">@tom</a> thanks for the response. Could you please elaborate on your first suggestion (<em>wrap the model in a Python extension and use that</em>)? If it’s not too much trouble, could you maybe show some snippet of code? Thanks.</p>

tom: <p>Well, you know how to run your model in C++, right?</p>
<p>Then you’d imitate <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">the C++ extension tutorial</a> but instead of implementing lltm you call your model.</p>
<p>You could either make the model a static global variable or pass it around as a shared ptr.</p>
<p>Best regards</p>
<p>Thomas</p>

SeaShellAtSeaSide: <p>I hope there is an answer for this, this is actually extremely helpful for those small researchers that has no GPU locally and only uses free google colab and kaggel compute, then whenever they hit their limit they can train in local with C++ (CPU) since training with python CPU is slow specially when reading the datasets/dataloader.</p>
What caused matrix inputs to no longer be transposed in PyTorch 2.5?

kishoreganesh: <p>In PyTorch 2.4, when I do a torch.compile (with the Triton matmul backend) of a Resnet model, both matmul inputs are transposed for the resulting Triton kernels generated</p>
<p>Eg:</p>
<pre><code class="lang-auto">    M = 196
    N = 1024
    K = 256
    if M * N == 0:
        # early exit due to zero-size input(s)
        return
    stride_am = 1
    stride_ak = 196
    stride_bk = 1
    stride_bn = 256
</code></pre>
<p>In PyTorch 2.5, only the second matmul input is transposed.</p>
<p>Eg:</p>
<pre><code class="lang-auto">    M = 196
    N = 1024
    K = 256
    if M * N == 0:
        # early exit due to zero-size input(s)
        return
    stride_am = 256
    stride_ak = 1
    stride_bk = 1
    stride_bn = 256
</code></pre>
<p>As you can see, stride_am=1 in the first and stride_am=256 in the 2nd.</p>
<p>I was trying to understand which PyTorch commit/PR changed this behavior  so that  I can better understand this.</p>
Where to find the compiling scripts or the detailed build configurations used by PyTorch

LamForest: <p>Hello,</p>
<p>I am planning to compile PyTorch from source and aim to replicate the official PyTorch builds as closely as possible. To achieve this, I would like to use the <strong>exact compile options and flags</strong> that are used for the official releases.</p>
<p>Could anyone please direct me to where I can find the compiling scripts or the detailed build configurations used by PyTorch?</p>
<p>Thank you for your assistance.</p>
From RAM to GRAM

Manvel_Petrosyan: <p>Dear friends,<br>
I am using pytorch mainly for linear algebra tasks and I have question about transfer of some tensor section from RAM to GRAM. For example</p>
<pre><code class="lang-auto">for iter in range(40):
    if flag == 2:
        tmp1[:] = A[iter*1000:(iter+1)*1000,:]
        flag = 1
    elif flag == 1:
        c = c + tmp1[:]
        tmp2[:] = A[iter*1000:(iter+1)*1000,:]
        flag = 0
    elif flag == 0:
        c = c + tmp2[:]
        tmp1[:] = A[iter*1000:(iter+1)*1000,:]
        flag = 1
</code></pre>
<p>A tensor is very big and located in RAM, tmp1, tmp2 and c tensors are in GPU RAM. I want to load parts of a tensor to GPU and do some calculations with them, as you can see I want do split calculation part and loading part. And the question is it done automatically asynchronous or I need to do additional configuration?</p>

Tony-Y: <p><a href="https://pytorch.org/tutorials/intermediate/pinmem_nonblock.html#a-guide-on-good-usage-of-non-blocking-and-pin-memory-in-pytorch" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/pinmem_nonblock.html#a-guide-on-good-usage-of-non-blocking-and-pin-memory-in-pytorch</a></p>
<p>This tutorial would interest you.</p>
Torch.fx.symbolic_trace with multiple GPUs

yazanobeidi: <p>Hi there,</p>
<p>I couldn’t find any documentation around this topic.</p>
<p>I’d like to <code>torch.fx.symbolic_trace</code> a large model that exceeds a single GPU memory e.g. LLaMa 405B. I have multiple GPUs. What’s the best way to go about doing this?</p>
<p>Thank you!</p>

robotsp: <p>Very good question! Do you find any solution <a class="mention" href="/u/yazanobeidi">@yazanobeidi</a> <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
perplexity very high on wikitext for GPT2XL

Emily: <p>I am trying to compute the perplexity of pretrained GPT2XL model on wikitext dataset, but I am getting a very high value (<code>ppl=25353.77267945427</code> ), and I do not understand why it is that high. Here is the code:</p>
<pre><code class="lang-auto">import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset
from torch.utils.data import DataLoader
import math

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
gpt2_tokenizer = AutoTokenizer.from_pretrained("gpt2-xl")
gpt2_tokenizer.pad_token = gpt2_tokenizer.eos_token
gpt2_model = AutoModelForCausalLM.from_pretrained("gpt2-xl").to("mps")

all_texts = [example['text'] for example in dataset]
inputs = gpt2_tokenizer(
    all_texts,
    return_tensors='pt',
    padding=True,
    truncation=True,
    max_length=512
)

input_ids = inputs['input_ids'].to('mps')
attention_mask = inputs['attention_mask'].to('mps')

dataloader = DataLoader(
    list(zip(input_ids, attention_mask)),
    batch_size=2,
    shuffle=False
)

def calculate_perplexity(model, dataloader):
    model.eval()
    total_loss = 0
    num_batches = 0
    with torch.no_grad():
        for batch in dataloader:
            input_ids, attention_mask = batch
            outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=input_ids)
            loss = outputs.loss
            total_loss += loss.item()
            num_batches += 1
    avg_loss = total_loss / num_batches 
    perplexity = math.exp(avg_loss)
    return perplexity


gpt2_perplexity = calculate_perplexity(gpt2_model, dataloader)
print(f"GPT-2 XL Perplexity on WikiText-2: {gpt2_perplexity}")
</code></pre>
<p>Is there something wrong with my code? tokenization? Thanks</p>
Pytorch riscv vector support?

Rubiczhang: <p>Hello everyone.</p>
<p>Is someone working on implementing pytorch operators using risc-v vector intrinsics?</p>
<p>I’m gonna start a project that porting pytorch operators to rvv platform. As far as I can see, most part of the project would be implementing  oprators and fusion oprators by rvv intrinsics.</p>
<p>Could you please give me some advices on my project?</p>
Torch.profiler failed to profile in Megatron-LM

NiuMa-1234: <p>I use torch.profiler to profile the training time of each function in megatron-lm . But it errors: RuntimeError: Can’t disable Kineto profiler when it’s not running.<br>
Here’s the code I use it:</p>
<pre><code class="lang-auto">    torch.cuda.synchronize() 
    with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
        with record_function("model_fwd"):
            output_tensor = model(tokens, position_ids, attention_mask,
                    labels=labels)
    print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))
    torch.cuda.synchronize()
</code></pre>
<p>And here’s the error:</p>
<pre><code class="lang-auto">[rank12]:     with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
[rank12]:   File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 748, in __exit__
[rank12]:     self.stop()
[rank12]:   File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 764, in stop
[rank12]:     self._transit_action(self.current_action, None)
[rank12]:   File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 793, in _transit_action
[rank12]:     action()
[rank12]:   File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 212, in stop_trace
[rank12]:     self.profiler.__exit__(None, None, None)
[rank12]:   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py", line 359, in __exit__
[rank12]:     self.kineto_results = _disable_profiler()
[rank12]: RuntimeError: Can't disable Kineto profiler when it's not running
</code></pre>
TIMM-based ViT masked-image modeling numerically unstable

chocolate_fireball: <p>I am training vision transformers using the masked-image modeling paradigm of MAE (<a href="https://arxiv.org/abs/2111.06377" rel="noopener nofollow ugc">https://arxiv.org/abs/2111.06377</a>), and building off of the paper’s codebase too, but with ViT-tiny models run on the CIFAR-10. The transformers built with TIMM components, are pre-norm, trained with learning rate warmup, xavier uniform initialization, and in pure 32-bit precision.</p>
<p>I can produce deterministic results by setting determinism, seed, and various PyTorch backends, eg. which attention algorithm is used, or whether I use TensorFloat32. I further find that the linear probes I use to evaluate my pretrained models are robust to changes to these settings—all validation accuracies typically fall within a 0.1% interval.</p>
<p>However, when varying just these backends for pretraining, I find enormous differences—linear probes return validation accuracies within a roughly 2% interval. This is significant on the scale of differences between results reported in published papers. Pretraining losses are also different, with relative differences between two runs more than <code>1e-7</code>. But this is after many iterations of training, so the issue is in optimization.</p>
<p>I’ve observed that switching from the transformer-default AdamW to SGD dispels this issue, at the cost of learning far worse representations. So I wonder if the reparameterization in AdamW is actually too sensitive to numerical differences coming from different backends that give different gradients? You can somewhat get around this by increasing <code>eps</code> in AdamW, but making it large enough to fix the problem essentially means barely reparameterizing, and indeed performance degrades enormously.</p>
<p>Is this normal? Documented anywhere? If not, any suggestions for tracking down the actual issue? I would massively appreciate help here.</p>
<p>(I can post a working example, but I will hold off since it’d be a fair amount of code and this might just be a known fact.)</p>

Nburnett: <p>I have a similar repeatability issue with TIMM vit_base_patch16_384 or tinyViT when training on 16 bit through pytorch lightning.</p>
When training with FSDP, optimizer_0 and pytorch_model_fsdp_0 directories are created. How do I extract model.bin and config.json files from them?

ysg5025: <p>When training with FSDP, optimizer_0 and pytorch_model_fsdp_0 directories are created. How do I extract model.bin and config.json files from them?</p>
<p>The file structure is as follows.</p>
<pre><code class="lang-auto">
├── pytorch_model_fsdp_0/
│ ├── __0_0.distcp
│ └── __1_0.distcp
   └── .metadata
│
├── optimizer_0/
│ ├── __0_0.distcp
│ └── __1_0.distcp
   └── .metadata
├──  scheduler.pt
├──  trainer_state.json</code></pre>
Why is the output of the linear layer different with the same input

JM_C: <pre><code class="lang-auto">import torch
torch.manual_seed(42)
torch.cuda.manual_seed(42)
torch.cuda.manual_seed_all(42)
a=torch.randn(1,5)
linear=torch.nn.Linear(5,2)
linear(a)
</code></pre>
<p>the output is</p>
<pre><code class="lang-auto">tensor([[-0.4137,  0.3302]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>but if i run it in CUDA environment</p>
<pre><code class="lang-auto">linear.eval()
linear=linear.cuda()
a=a.cuda()
for i in range(5):
    print(linear(a))
</code></pre>
<p>the result is</p>
<pre><code class="lang-auto">tensor([[-0.2320,  2.0413]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-0.4641,  2.0749]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-0.6961,  2.1084]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-0.9282,  2.1419]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
tensor([[-1.1602,  2.1754]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>it happends only when the batch_size is 1, and run in CUDA environment.<br>
And I found that the accumulated value is exactly equal to</p>
<pre><code class="lang-auto">torch.mm(a,linear.weight.data.T)
</code></pre>
<pre><code class="lang-auto">tensor([[-0.2320,  0.0335]], device='cuda:0')
</code></pre>

ptrblck: <p>I cannot reproduce the issue in <code>2.6.0.dev20241004+cu124</code> on a 3090. I don’t know which device or release you are using, but if it’s an older PyTorch version, could you update to the latest stable or nightly release?</p>
How do I run ddp on Windows?

pardazesh_2050: <p>hello<br>
I have 2 systems (Windows).<br>
And each system has 1 gpu.<br>
i use pycharm.<br>
I want to run a code on pycharm and the program uses 2 gpus.<br>
Two systems are connected to each other through LAN.</p>
<p>Code and dataset should only be in one system.<br>
How can I do it??</p>
Build torch from source for jetson

wangyitao42: <p>When I type</p>
<pre data-code-wrap="bash"><code class="lang-bash">git clone --recursive https://github.com/pytorch/pytorch.git
git submodule sync
git submodule update --init --recursive

# Set Build Environment 
export USE_NCCL=0
export USE_DISTRIBUTED=0                  
export USE_QNNPACK=0
export USE_PYTORCH_QNNPACK=0
export TORCH_CUDA_ARCH_LIST="8.7"

export PYTORCH_BUILD_VERSION="2.3.0"
export PYTORCH_BUILD_NUMBER=1

# source &lt;conda env&gt;
python setup.py bdist_wheel
</code></pre>
<p>Then I got</p>
<pre data-code-wrap="bash"><code class="lang-bash">Building wheel torch-2.3.0
-- Building version 2.3.0
Could not find any of CMakeLists.txt, Makefile, setup.py, LICENSE, LICENSE.md, LICENSE.txt in /home/orin/tools/pytorch/third_party/ios-cmake
Did you run 'git submodule update --init --recursive'?
</code></pre>
<p>Every time there is new third-party comes that, what should I do?</p>

ptrblck: <p>Did you run the suggested command which will recursively initialize all submodules?</p>
What is the difference of .flatten() and .view(-1) in PyTorch?

ipid: <p>Both  <code>.flatten()</code>  and  <code>.view(-1)</code>  flattens a tensor in PyTorch. What’s the difference?</p>
<ol>
<li>Does  <code>.flatten()</code>  copy data of the tensor?</li>
<li>Is  <code>.view(-1)</code>  faster?</li>
<li>Is there any situation that  <code>.flatten()</code>  doesn’t work?</li>
</ol>
<p>I’ve tried to read PyTorch’s docs but it doesn’t answer these questions.</p>

Sunshine352: <p><code>view</code> ( *<em>shape</em> ) → Tensor<br>
<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view</a><br>
Returns a new tensor with the same data as the  <code>self</code>  tensor but of a different  <code>shape</code> .</p>
<p><code>torch.</code> <code>flatten</code> ( <em>input</em> ,  <em>start_dim=0</em> ,  <em>end_dim=-1</em> ) → Tensor<br>
<a href="https://pytorch.org/docs/stable/torch.html#torch.flatten" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.flatten</a><br>
Flattens a contiguous range of dims in a tensor.</p>

ipid: <p>Well I read those descriptions on the docs but they didn’t answer my questions.</p>

gMatas: <ol>
<li>
<p>No, <code>torch.flatten()</code> function does not copy any data, and actually it behaves more like a wrapper around the <code>view()</code> function. Simple way to prove it without having any explicit mention of it in the docs is by running the following lines of code:</p>
<pre><code class="lang-python"># Create (2, 3, 4) shape data tensor filled with 0.
a = torch.zeros(2, 3, 4)

# Flatten 2nd and 3rd dimensions of the original data 
# tensor using `view` and `flatten` methods.
b = a.view(2, 12)
c = torch.flatten(a, start_dim=1)

# Change a distinct value in each flattened tensor object.
b[0, 2] = 1
c[0, 4] = 2

# Compare tensors objects data to each other to look for 
# any mismatches.
print("Tensors A and B data match?", all(a.view(-1) == b.view(-1)))
print("Tensors A and C data match?", all(a.view(-1) == c.view(-1)))
print("Tensors B and C data match?", all(b.view(-1) == c.view(-1)))
</code></pre>
<p>Output:</p>
<pre><code class="lang-markdown">Tensors A and B data match? True
Tensors A and C data match? True
Tensors B and C data match? True
</code></pre>
</li>
<li>
<p>Yes, but the difference is negligible in practice. The overhead that <code>flatten()</code> function introduces is only from its internal simple computation of the tensor’s output shape and the actual call to the <code>view()</code> method or similar. This difference is in less than 1μs.</p>
</li>
<li>
<p>Not any that I would know about.</p>
</li>
</ol>

YaoXiao-CS: <p>I mabye found your question on Stack Overflow, and <a href="https://stackoverflow.com/questions/49643225/whats-the-difference-between-reshape-and-view-in-pytorch/49644300#49644300" rel="noopener nofollow ugc">this answer</a> is useful.</p>
Difference between torch.transpose and torch.movedim

wasabi_linguist: <p>Hi,<br>
I was wondering what the difference is between torch.transpose and torch.movedim as we can use both of them to move dimensions and they return the same values. For example where, <code>A = torch.arange(0,2*3).float().view(2,3)</code> both <code>A.transpose(0,1)</code> and <code>A.movedim(0,1)</code> returns the same value.</p>

jrog: <p>I think the main difference between <code>torch.transpose</code> and <code>torch.movedim</code> is that you can use <code>torch.movedim</code> to move multiple dimensions to new positions at the same time, where <code>transpose </code> can swap only two dimensions directly.</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(2, 3, 4, 5)
# Moves dim 0 to position 1, then dim 2 to pos 3
# Resulting in shape [3, 2, 5, 4] 
x.movedim([0, 2], [1, 3]).shape
</code></pre>
<p>To achieve the same result using torch.transpose, you would need to write:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(2, 3, 4, 5)
x.transpose(0, 1).transpose(2, 3).shape
</code></pre>
<p>More examples in pytorch documentation:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.transpose.html" rel="noopener nofollow ugc">transpose</a><br>
<a href="https://pytorch.org/docs/stable/generated/torch.movedim.html#torch.movedim" rel="noopener nofollow ugc">movedim</a></p>
Understanding stride for the output of nonzero

FSchoen: <p>Hello everyone!</p>
<p>After hours of debugging I noticed that <code>.stride()</code> behaves (in my opinion) strangely when applied to the output of a <code>.nonzero()</code> operation. Here is a minimal working example of what I am encountering:</p>
<pre><code class="lang-auto">import torch


def test_stride():
    print()
    example = torch.tensor([[[False, True],
                             [False, False]]])
    nonzero = torch.nonzero(example).contiguous()

    print(f"Nonzero output: {nonzero}")                     # Nonzero output: tensor([[0, 0, 1]])
    print(f"Nonzero stride: {nonzero.stride()}")            # Nonzero stride: (1, 1)

    equivalent = torch.tensor([[0, 0, 1]]).contiguous()
    print(f"Equivalent output: {equivalent}")               # Equivalent output: tensor([[0, 0, 1]])
    print(f"Equivalent stride: {equivalent.stride()}")      # Equivalent stride: (3, 1)
</code></pre>
<p>I would assume that in both cases the stride of the tensor is <code>(3, 1)</code>, but no matter what I do to the output tensor of the <code>.nonzero()</code> operation the stride stays <code>(1, 1)</code>. This is only the case as long as it contains a single element; as soon as two or more elements are returned the output is as expected.</p>
<p>Is this a bug or am I overlooking something? If the latter is the case please tell me how to resolve my issue. Thanks!</p>

Eduardo_Lawson: <p>This isn’t really a bug, but rather a quirk of how how PyTorch optimizes memory layout for single-element results from <code>nonzero()</code>.<br>
The key difference is that:</p>
<ul>
<li>For single elements, PyTorch uses a compact stride of (1, 1)</li>
<li>For multiple elements, it uses the expected stride of (3, 1)<br>
This different stride pattern doesn’t make the operations wrong, but it can be surprising when you’re explicitly working with strides.<br>
To solve this issue, if you need consistent stride behavior you can convert the single element case to match the multi-element case. I would try:</li>
</ul>
<pre><code class="lang-auto">nonzero = torch.nonzero(example)
if nonzero.shape[0] == 1:
    nonzero = nonzero.clone().view(1, -1)  # This will give you (3, 1) stride
</code></pre>

FSchoen: <p>Thanks for the reply! Although this still seems kind of unintuitive for me, your solution (or in practice one inspired by it) works perfectly fine, thank you!</p>
How to use the gradient of an intermediate variable in the computation graph of a later variable

MartensCedric: <pre><code class="lang-auto">point_2 = torch.tensor([0.2, 0.8], device=device, requires_grad=True)
p = torch.cat((point_2, torch.tensor([0], device=device)), 0)

x_verts = torch.tensor([0.0, 1.0, 0.0], device=device, requires_grad=True)
y_verts = torch.tensor([0.0, 0.0, 1.0], device=device, requires_grad=True)
z_verts = torch.tensor([0.1, -0.1, 0.2], device=device, requires_grad=True)

v1_2d = torch.cat((torch.index_select(x_verts, 0, torch.tensor([0])), torch.index_select(y_verts, 0, torch.tensor([0])), torch.tensor([0])))
v2_2d = torch.cat((torch.index_select(x_verts, 0, torch.tensor([1])), torch.index_select(y_verts, 0, torch.tensor([1])), torch.tensor([0])))
v3_2d = torch.cat((torch.index_select(x_verts, 0, torch.tensor([2])), torch.index_select(y_verts, 0, torch.tensor([2])), torch.tensor([0])))


area_3 = torch.cross(v2_2d - v1_2d, v3_2d - v1_2d)
area = torch.index_select(area_3, 0, torch.tensor([2]))

alpha_3 = 0.5 * torch.cross(v2_2d - p, v3_2d - p) / area
beta_3 = 0.5 * torch.cross(v3_2d - p, v1_2d - p) / area
gamma_3 = 0.5 * torch.cross(v1_2d - p, v2_2d - p) / area

alpha = torch.index_select(alpha_3, 0, torch.tensor([2]))
beta = torch.index_select(beta_3, 0, torch.tensor([2]))
gamma = torch.index_select(gamma_3, 0, torch.tensor([2]))

z = alpha * torch.index_select(z_verts, 0, torch.tensor([0])) + beta * torch.index_select(z_verts, 0, torch.tensor([1])) + gamma * torch.index_select(z_verts, 0, torch.tensor([2]))

z.backward()


grad_norm = torch.norm(point_2.grad ) # &lt;= disconnection

f = torch.tanh(10.0 * (grad_norm - 2.0))
f.backward() # &lt;= error

print(x_verts.grad)
print(y_verts.grad)
print(z_verts.grad)
</code></pre>
<p>I have this code which fails because I use a variable’s <code>.grad</code> value as input to another variable. How can I fix this?</p>

AlphaBetaGamma96: <p>You can write a custom autograd function to define the gradient of your <code>f</code> term directly, docs <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc">here</a></p>
Per-sample gradients w.r.t the output of a layer

fy-meng: <p>Hi guys,</p>
<p>I’m trying to run per-sample <a href="https://captum.ai/api/concept.html" rel="noopener nofollow ugc">TCAV</a>, which requires computing the gradients w.r.t. the output of a layer (e.g. <code>resnet18.layer4</code>). I could set batch size to 1, but it seems that the script would become extremely inefficient and slow.</p>
<p>I’ve also looked at <a href="https://dev-discuss.pytorch.org/t/an-efficient-way-to-compute-per-sample-gradients/695" rel="noopener nofollow ugc"><code>torch.nn.utils._per_sample_grad</code></a> and <code>autograd_hacks</code>, but from my understanding they only store per-sample gradients for each layer with weights, not layer outputs, so they won’t work for <code>resnet18.layer4</code>.</p>
<p>Is there an efficient way to compute such gradients?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/fy-meng">@fy-meng</a>,</p>
<p>Have a look at the <code>torch.func</code> namespace, you can efficiently compute per-sample gradients via <code>torch.func.vmap</code>.</p>
<p>You can find an example I wrote for calculating the per-sample Jacobian over a batch of data <a href="https://discuss.pytorch.org/t/computing-batch-jacobian-efficiently/80771/12">here</a>.</p>