akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What base image that torch lib builds on for linux package

Rhett-Ying: <p>I am wondering how torch lib is built, especially which OS is used for build package for linux platform to make it most widely compatible with users’ linux distribution.</p>
<p>This is also about what the minimum versions of <code>glibc</code>, <code>gcc</code> and other basic dependencies are.</p>
<p>Build upon docker image such as <code>ubi8</code>, <code>ubuntu 20.04</code> to support various linux distributions?</p>

ptrblck: <p>The current CPU and CUDA binaries should use CentOS7 as the base as seen <a href="https://github.com/pytorch/builder/blob/ef4a28a4313c52c6765669c914b4ea4c756d040d/manywheel/build_docker.sh#L20-L52">here</a>.</p>
Forward function running forever because of custom function

Mihai_M: <p>Hi all,</p>
<p>I am running a custom function in the forward function. My forward function looks like the following:</p>
<pre><code>     def forward(self, x: torch.Tensor):

      x = self.layer[0](x)
      v_out= custom_function_call
      x = self.layer[1](v_out)
      x = self.layer[2](x)
      x = self.layer[3](x)
      return x
</code></pre>
<p>The custom function calculates values and returns these in a tensor of the correct size, which is then passed to the second layer in the net.</p>
<p>My issue is that it looks like this function makes the forward function run forever.</p>
<p>Any pointers?</p>

ptrblck: <aside class="quote no-group" data-username="Mihai_M" data-post="1" data-topic="200489">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mihai_m/48/63041_2.png" class="avatar"> Mihai_M:</div>
<blockquote>
<p>My issue is that it looks like this function makes the forward function run forever.</p>
<p>Any pointers?</p>
</blockquote>
</aside>
<p>It’s unclear what <code>custom_function_call</code> is and you might want to share more details about this method so we could check why it’s hanging or takes a long time.</p>
Compile function that uses constant tensor

pimdh: <p>Hi!</p>
<p>Imagine a setup like:</p>
<pre data-code-wrap="python"><code class="lang-python">def construct(device, dtype) -&gt; Tensor:
    return ... # some complicated code to construct a tensor with given device and dtype

def f(x):
    constant_tensor = construct(x.device, x.dtype)
    return x + constant_tensor
</code></pre>
<p>Now, I want to compile <code>f</code>, but I do not want to compile the <code>construct</code> function. Instead, I’d like the tensor <code>constant_tensor</code> to be a constant in the compiled graph. How do I best realize this?</p>
<p>Also, preferably, if my main module uses <code>f</code> multiple times, I’d prefer the constant to be computed/stored on the graph only once. Without compilation, I’d add a <code>functools.lru_cache</code> around <code>construct</code> to memoize the result, however, when I try to compile <code>f</code> with the cache, it tries to compile the caching function, which fails.</p>
<p>Thanks!</p>

pimdh: <p>I think I’ve found a solution. We can make a <code>lru_cache</code>-like memoizer that appears like a dictionary to the compiler. It doesn’t trace the construction function, but instead thinks that all keys are simply present when compiling the function.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch


class Memoizer(dict):
    def __init__(self, fn):
        super().__init__()
        self.fn = fn

    def __missing__(self, item):
        tensor = self.fn(*item)
        self[item] = tensor
        return tensor

    def __call__(self, *args):
        return self[args]


@Memoizer
def constant_constructor(dtype, device):
    print("construct contant")
    return torch.ones(1, dtype=dtype, device=device)


def f(x):
    cnst = constant_constructor(x.dtype, x.device)
    return x + cnst


f_compiled = torch.compile(f, fullgraph=True)
print(f_compiled(torch.ones(1)))  # -&gt; construct constant
print(f_compiled(torch.ones(1)))
print(f_compiled(torch.ones(1).cuda()))  # -&gt; construct constant
print(f_compiled(torch.ones(1).cuda()))
</code></pre>
Pytorch transformer norm_first=False leads to diverging model

Daniel_Maier: <p>I have a small transformer model (~11M parameter) that I train on some data. The model achieved very good accuracies on test data (~99%). After a commit, the model would only reach about 3% or diverge completely, giving NaN for loss (caused by setting weights to NaN I guess). After searching for the cause for the drastic change in behaviour I realized that this behaviour is entirely dependent on the norm_first parameter. Setting it to False (on both Encoder and DecoderLayer) leads to the divergence during training.<br>
It seems very odd to me that that alone would make or brake a model. I also use masks of bool type like this:</p>
<pre><code class="lang-auto">causal_mask = torch.triu(torch.ones(tgt.shape[1], tgt.shape[1], dtype=torch.bool), diagonal=1).to(src.device)
</code></pre>
<p>which gives warnings in eval() mode:</p>
<pre><code class="lang-auto">UserWarning: Converting mask without torch.bool dtype to bool; this will negatively affect performance
</code></pre>
<p>Don’t know if this might be related. Is this behaviour “normal” for not pretrained trasnformer or might there be something wrong with my setup?</p>
BertForMultipleChoice works without captum, breaks with it?

rbelew: <p>Hi, I’m trying to do first experiments with captum, with a pretrained BertForMultipleChoice model over this <a href="https://huggingface.co/datasets/casehold/casehold" rel="noopener nofollow ugc">published Caseholder dataset</a>, and following this <a href="https://github.com/pytorch/captum/blob/master/tutorials/Bert_SQUAD_Interpret.ipynb" rel="noopener nofollow ugc">BERT SQUAD Captum tutorial</a>.  I’ve posted <a href="https://colab.research.google.com/drive/1Om_am_LFxHQEl47B80e5LyYBEhVTqFcX" rel="noopener nofollow ugc">my notebook on Colab</a> if you’re kind enough to look!</p>
<p>My first issue seems to be identifying what “ground truth” means. As this is a multiple choice task, I figured the correct answer would be ground truth?  But this fails trying to get the indices of the GT tokens: <code>ground_truth_end_ind = indices.index(ground_truth_tokens[-1])</code></p>
<pre><code>ValueError: 7607 is not in list
</code></pre>
<p>Question#1: What should ground truth be for this dataset?</p>
<p>To get a bit farther I just reused the same text for ground truth. But this fails when it tries to make a prediction:</p>
<pre><code>start_scores, end_scores = predict(input_ids, \
                               token_type_ids=token_type_ids, \
                               position_ids=position_ids, \
                               attention_mask=attention_mask)
</code></pre>
<p>I traced this error to <code>modeling_bert.BertForMultipleChoice.forward()</code> (cf. lines 1662-1690).</p>
<p>In previous experiments without captum the shape of <code>input_ids</code> comes in as [16,5,128], <code>input_ids.size(-1)</code> = 128, <code>num_choices</code> = 5, and <code>input_ids</code> shape is changed to [80,128].  Then <code>pooled_output.shape=[80, 128]</code>, <code>logits.shape=[80,1]</code>, and <code>reshaped_logits</code> is computed correctly, with shape=[16,5].</p>
<p>But using captum, <code>input_ids</code> shape is unchanged <code>num_choices</code> = 260. Then <code>pooled_output.shape=[1,768]</code>, <code>logits.shape=[1,1]</code>, and the attempt to compute <code>reshaped_logits</code> fails with error:</p>
<pre><code>RuntimeError: shape '[-1, 260]' is invalid for input of size 1
</code></pre>
<p>Question#2: Why is <code>BertForMultipleChoice.forward()</code> behaving differently with captum?</p>
<p>Thanks for any help.  captum looks hugely useful, can’t wait to make use of it!</p>
Torch.jit.script is not working with fasterrcnn

ricardo_assaad: <p>Hello Everyone,</p>
<p>I have been trying to figure out an issue with torchscript in c++<br>
I get the following error:</p>
<blockquote>
<p>forward() Expected a value of type ‘List[Tensor]’ for argument ‘images’ but instead found type ‘Tensor’.<br>
Position: 1<br>
Declaration: forward(<strong>torch</strong>.torchvision.models.detection.faster_rcnn.FasterRCNN self, Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span> images, Dict(str, Tensor)<span class="chcklst-box fa fa-square-o fa-fw"></span>? targets=None) → ((Dict(str, Tensor), Dict(str, Tensor)<span class="chcklst-box fa fa-square-o fa-fw"></span>))</p>
</blockquote>
<p>I tried annotating the images variable in the forward method with List[torch.Tensor] but the error persists</p>
<p>I even annotated the forward method in torchvision.models.detection.generalized_rcnn.py</p>
<p>the code is built off of this tutorial:<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>I am using torch.jit.script(model)</p>
<p>when I debug the scripted model, the forward method code starts with this</p>
<pre><code class="lang-auto">'def forward(self,\n    images: List[Tensor],\n    targets: Optional[List[Dict[str, Tensor]]]=None) -&gt; Tuple[Dict[str, Tensor], List[Dict[str, Tensor]]]:\n
</code></pre>
<p>I would really appreciate the help</p>
How to combine a PyTorch network and a non-differentiable simulator

Carlo_Alessi: <p>I would like to combine a neural network <code>Net</code> with a non-differentiable fixed simulator <code>Sim</code>  as follows:</p>
<p><strong>a</strong> → <code>Net</code> → <code>Sim</code> → <strong>c_pred</strong></p>
<ol>
<li><code>Net</code> takes as input vectors <strong>a</strong> and predicts vectors <strong>b</strong>,</li>
<li><code>Sim</code> (implemented in Python/numpy) takes as input <strong>b</strong> and, after integrating PDEs, produces vectors <strong>c_pred</strong>,</li>
<li>My ground truth are data (<strong>a</strong>, <strong>c</strong>)</li>
<li>I would like to tune the <code>Net</code> to minimize the loss L(<strong>c</strong>, <strong>c_pred</strong>)</li>
</ol>
<p>So far I wrap the <code>Sim</code> in a <code>th.autograd.Function</code> but the learning is not great because I can’t define a good gradient of the <code>Sim</code> in the custom <code>backward()</code>.</p>
<p>How can I solve this?</p>
<p>Kind regards,<br>
Carlo</p>

ptrblck: <aside class="quote no-group" data-username="Carlo_Alessi" data-post="1" data-topic="200597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carlo_alessi/48/69144_2.png" class="avatar"> Carlo_Alessi:</div>
<blockquote>
<p>So far I wrap the <code>Sim</code> in a <code>th.autograd.Function</code> but the learning is not great because I can’t define a good gradient of the <code>Sim</code> in the custom <code>backward()</code>.</p>
<p>How can I solve this?</p>
</blockquote>
</aside>
<p>You approach is valid and you would need to define a “proper” backward method in your custom module.</p>
Which is faster: Loading data in parallel or running multiple training processes in parallel but have data loaded sequentially?

Roy_Gabriel: <p>I currently run an active learning algorithm. Let us say the size of the data is 30,000 samples. I start with 300. I train a model on these 300 images (7 epochs with early stopping let’s say). Then i add more samples (300 + 700 = 1000 samples). I retrain the model. I keep doing this for 15 Active Learning iterations. Total data used will be 300 + 700*(15-1) = 10,100 samples (roughly 1/3 of the original data). This is what I call an experiment.</p>
<p>Now, I need to run the above experiment 10 times to obtain a mean and std of the results. That being said, I am running into long run times. I used to use DataLoader with num_workers &gt; 0 and pin_memory = True wait till an entire experiment is done then run this process 10 times total. Recently, I tried using multiprocessing to run say 3 experiments at a time but it does not allow me to run DataLoader with num_workers &gt; 0. I also had to decrease batch size to around 16 or 8 to avoid GPU memory overload.</p>
<p>That being said, does anyone know what is the optimal way to go about this? Currently I have one GPU RTX 2080 but soon I will have two GPUs RTX TITAN (and maybe even better ones think L series or RTX 4090s). My priority right now is speed.</p>
Indexing in multiple dimensions according to another tensor

i4ata: <p>Consider the following tensors:</p>
<pre><code class="lang-auto">n = 2
a = torch.rand(3,7,7,n,5)
b = torch.rand(3,7,7,n)
</code></pre>
<p>Is it possible to select the elements of <code>a</code>, where the corresponding element of <code>b</code> is the largest without reshaping <code>a</code> and <code>b</code> beforehand? The new tensor must have the shape <code>(3,7,7,5)</code>. That can be easily achieved with <code>torch.where</code> if <code>n = 2</code> but how can it be done for <code>n != 2</code>? I imagine something like <code>torch.index_select</code>, though it only works for 1-dimensional indices.</p>

i4ata: <p>Apparently, It can be achieved as follows:</p>
<pre><code class="lang-auto">b = b.argmax(-1, keepdims=True).unsqueeze(-1).expand(-1,-1,-1,-1,5))
torch.gather(a, 3, b)
</code></pre>
The quantized-weight's key is missing

RonaldYuren: <p>Hi, I am trying using my “static_quantize_weights.pt” to do QAT(or you can call finetuning)；<br>
But it come across the problem that quantized-weight doesn’t have the same key compare with the original model.</p>
<p>the ERROR message is like below:<br>
##########<br>
Traceback (most recent call last):<br>
File “Static_quantize5.py”, line 200, in<br>
train(hyp, opt = opt,device = torch.device(‘cpu’), tb_writer = tb_writer)<br>
File “/home/gene/YuRen/yolov7/train.py”, line 102, in train<br>
state_dict = ckpt[‘model’].float().state_dict() # to FP32<br>
KeyError: ‘model’<br>
############<br>
How could I do next?<br>
Note:  <strong>Pytorch 2.1.0</strong>，<strong>FX-graph-Mode Post-training-static-quantization with YOLOv7</strong><br>
Please help, Thank you!!!</p>

RonaldYuren: <p>Here are the key from original and from static-quantized-weights:</p>
<p><strong>Original keys:</strong><br>
epoch<br>
best_fitness<br>
training_results<br>
model<br>
ema<br>
updates<br>
optimizer<br>
wandb_id</p>
<p><strong>static-quantized-weight 's keys:</strong>    (Too many, Just a part of keys)<br>
model_0_conv_input_scale_0<br>
model_0_conv_input_zero_point_0<br>
model_39_scale_0<br>
model_39_zero_point_0<br>
model_49_scale_0<br>
model_49_zero_point_0<br>
_tensor_constant0<br>
_tensor_constant1<br>
_tensor_constant2<br>
_tensor_constant3<br>
_tensor_constant4<br>
model_77_scale_4<br>
model_77_zero_point_4<br>
_tensor_constant5<br>
_tensor_constant6<br>
model_77_scale_10<br>
model_77_zero_point_10<br>
_tensor_constant7<br>
_tensor_constant8<br>
model_77_scale_16<br>
model_77_zero_point_16<br>
model.0.conv.weight<br>
model.0.conv.bias<br>
model.0.conv.scale<br>
model.0.conv.zero_point<br>
model.0.act.scale<br>
model.0.act.zero_point<br>
model.1.conv.weight<br>
model.1.conv.bias<br>
model.1.conv.scale<br>
model.1.conv.zero_point<br>
model.1.act.scale<br>
model.1.act.zero_point<br>
model.2.conv.weight<br>
model.2.conv.bias<br>
model.2.conv.scale<br>
model.2.conv.zero_point<br>
model.2.act.scale<br>
model.2.act.zero_point<br>
model.3.conv.weight<br>
model.3.conv.bias<br>
model.3.conv.scale<br>
model.3.conv.zero_point<br>
model.3.act.scale<br>
model.3.act.zero_point<br>
model.4.conv.weight<br>
model.4.conv.bias<br>
model.4.conv.scale<br>
model.4.conv.zero_point<br>
model.4.act.scale<br>
model.4.act.zero_point<br>
model.5.conv.weight<br>
model.5.conv.bias<br>
model.5.conv.scale<br>
model.5.conv.zero_point</p>
Questions about how to import stub files(.pyi files) in .py files?

warming423: <p>In torch._C directory, I found many .pyi files. And in some .py files,  I saw “import torch._C” , I try to import  it, but I met error “ModuleNotFoundError: No module named 'torch._C._profiler”(eg.). I am new to this source code. Could anyone explain the question to me? Thank you!</p>
Pytorch speedup

kxkxkxkx: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38469926946b1ad0cd3a4ae2546c4487bed1a0d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/38469926946b1ad0cd3a4ae2546c4487bed1a0d9" title="Screenshot 2024-04-10 at 12.47.58 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38469926946b1ad0cd3a4ae2546c4487bed1a0d9_2_690x327.png" alt="Screenshot 2024-04-10 at 12.47.58 AM" data-base62-sha1="81PZEqle8GAYGwlSzkrdR5SuWi5" width="690" height="327" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38469926946b1ad0cd3a4ae2546c4487bed1a0d9_2_690x327.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38469926946b1ad0cd3a4ae2546c4487bed1a0d9_2_1035x490.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/8/38469926946b1ad0cd3a4ae2546c4487bed1a0d9.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-04-10 at 12.47.58 AM</span><span class="informations">1164×552 50.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Why doesn't the latest version of pyTorch support ONNX opset version: 20 ？？？

wangzhenlin123: <pre><code class="lang-auto">dummy_input = torch.randn(1, 1, 28, 28)
torch.onnx.export(model, dummy_input, "stn_classifier.onnx", verbose=True, opset_version=20)
</code></pre>
<p>vesion：<br>
torch             2.2.2+cu118<br>
torchsummary      1.5.1<br>
torchvision       0.17.2+cu118</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bded43aa76d3a8e23526e6e85d2b7efab1437ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/5bded43aa76d3a8e23526e6e85d2b7efab1437ef" title="49382f88de11d27ea59712420c98466"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5bded43aa76d3a8e23526e6e85d2b7efab1437ef_2_384x499.png" alt="49382f88de11d27ea59712420c98466" data-base62-sha1="d6IRfamUahQuaLSpcHnarYJdOiz" width="384" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5bded43aa76d3a8e23526e6e85d2b7efab1437ef_2_384x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5bded43aa76d3a8e23526e6e85d2b7efab1437ef_2_576x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bded43aa76d3a8e23526e6e85d2b7efab1437ef.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">49382f88de11d27ea59712420c98466</span><span class="informations">766×995 34.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

wangzhenlin123: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26a222566c16688de495c3ca40c4cd49c64996ea.png" data-download-href="https://discuss.pytorch.org/uploads/default/26a222566c16688de495c3ca40c4cd49c64996ea" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26a222566c16688de495c3ca40c4cd49c64996ea.png" alt="image" data-base62-sha1="5vLvUjLztvR89faOcJ0mHbZzL4u" width="690" height="193" data-dominant-color="1C1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">967×271 7.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
RuntimeError: CUDA error: out of memory; Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions

DrizzleShh: <h1><a name="trouble-in-training-yolov5-1" class="anchor" href="#trouble-in-training-yolov5-1"></a>Trouble in training Yolov5</h1>
<p>It worked well when the version of pytorch is 1.7, but when I upgrade pytorch to 2.0, alwanys shows ‘CUDA out of memory’. My GPUs are 3090.<br>
And I have added</p>
<blockquote>
<p>export CUDA_LAUNCH_BLOCKING=1<br>
export TORCH_USE_CUDA_DSA=1</p>
</blockquote>
<p>as the Error suggestions.<br>
Now it shows</p>
<blockquote>
<p>RuntimeError: CUDA error: out of memory;  Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
</blockquote>
<p>What should I do to solve this problem?</p>
<blockquote>
<p>torch.cuda.is_available = True</p>
</blockquote>

eqy: <p>Could you post a script (e.g., with random/dummy data) that reproduces the issue/high memory usage that worked in 1.7 but crashes in 2.0?</p>

DrizzleShh: <p>I’m sorry for bothering you. The issue disappeared on the second day, so I forgot my question. It may because of the unkilled processes, and coincidently I upgraded pytorch, so I  ascribe the error to pytorch…</p>

Bhavay_Malhotra: <p>Heyy, I’m not getting the CUDA:Out of memory error but the later half i.e. Compile with ‘TORCH_USE…’. I don’t have any active processes too. Can you tell which PyTorch version have you upgraded too?</p>

rahulswa: <p>Any update on this. I am facing this issues when running a parellel programming code on a server.<br>
Tried with Pytorch 2.0 and Cuda 11.8, Also tried with Pytorch 1.13 and cuda 11.7. but no luck any thoughts?</p>
How to pre-allocate a tensor and the its gradient?

nahso: <p>I need to communicate with other processes to gather data, so I must pre-register the memory location to the network card before RDMA send/recv to gain a better performance. I only use CPU, and implementing a custom CPU allocator seems to be a good choice.</p>
<p>I noticed that it had been discussed before(<a href="https://discuss.pytorch.org/t/control-tensor-memory-allocation/110085" class="inline-onebox">Control Tensor Memory Allocation?</a>), but I wonder whether the CPU allocator has a memory pool(and where is the code).</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">#include &lt;iostream&gt;
#include &lt;torch/all.h&gt;

int main() {
    auto option = torch::TensorOptions().requires_grad(true).dtype(torch::kFloat32);
    torch::Tensor a = torch::tensor({1, 2, 3}, option);
    torch::Tensor b = torch::tensor({4, 5, 6}, option);
    torch::Tensor c = a * b;
    torch::Tensor grad = torch::autograd::grad({c}, {a}, {torch::ones_like(c)})[0];

    return 0;
}
</code></pre>
<p>I traced this demo above until <code>posix_memalign</code> was called, I didn’t find any memory pool or cached malloc.</p>
<p>The next question is how to pre-allocate the gradient of a tensor. Or is it possible to pre-allocate it?</p>
Conda install pytorch error

dongfengxijian: <p>When I used the official command “conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia” to install pytorch, I encountered the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75e39cdacb97b6cc8b4497605c8771c3ca992396.png" data-download-href="https://discuss.pytorch.org/uploads/default/75e39cdacb97b6cc8b4497605c8771c3ca992396" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75e39cdacb97b6cc8b4497605c8771c3ca992396.png" alt="image" data-base62-sha1="gOTwmGUbXeCCIBnc9ypE9jneDpc" width="690" height="401" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1345×783 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I would like to ask if there is any way to solve this problem?</p>

ptrblck: <p>Are you using the <code>conda-forge</code> channel? If so, try to use the <code>main</code> channel only as we’ve seen smilar GLIBC incompatibilities caused by libs in <code>conda-forge</code>.</p>

dongfengxijian: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="200535">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>conda-forge</p>
</blockquote>
</aside>
<p>Thank you for your relplying. I tired to use “conda install torchaudio --conda-forge” to install torchaudio in another virtual environment before this error happen. May it break my conda environment? When I use conmmand “conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia”, I did not add any command and my .condarc file is like as follow:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b84013cf4eff7bb4a948c1d9c7372801bc38a9db.png" alt="image" data-base62-sha1="qhX8F5iAkOc1E5s3MVAW3cyPELh" width="661" height="299"></p>

dongfengxijian: <p>When I woke up the next day and re-entered the command “conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia”, it didn’t report an error anymore. Thank you for your guidance.</p>
Visualize activations

cobie8: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - are there any updates to the framework that would allow me to visualize activations? i’ve searched far and wide for weeks, using the same suggested methods on the forums but haven’t been successful. any help would be appreciated.</p>
<p>what i’ve done so far:</p>
<ol>
<li>pull conv2d layers out of a custom-designed detectron2 network</li>
<li>of those, there are a few activation layers. i’d like to visualize those mostly for debugging purposes.</li>
</ol>
<p>thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You could use forward hooks to grab the desired activation maps and visualize them as described <a href="https://discuss.pytorch.org/t/visualize-feature-map/29597/2">here</a>. Alternatively, you could also check other repositories, but I’m unsure how compatible these are to the current PyTorch release (e.g. <a href="https://github.com/utkuozbulak/pytorch-cnn-visualizations">this one</a>).</p>
<p>I would also generally not recommend tagging specific users in questions, as it could demotivate others to post valid answers.</p>

cobie8: <p>i see. thanks.</p>
<p>apologies - newb here and will be mindful next time.</p>
Reuse of nn.Sequential()

msv: <p>If I define my net like this:</p>
<pre><code class="lang-auto">class Net(): 
   def __init__
      self.block = nn.Sequential(...)
   def forward
      x0 = self.block(inputs[0])
      x1 = self.block(inputs[1])
</code></pre>
<p>Will the learnables of self.block be shared among input 0 and 1, or will each call to self.block in the forward function automatically upscale the number of learnables?</p>

ptrblck: <p><code>self.block</code> is reused in your approach and so will be its trainable parameters. PyTorch’s Autograd creates a computation graph capturing both operations, won’t create new parameters, but will just reuse them.</p>
Basic Extreme Learning Machine pytorch implementation

derby: <p>Hi all, I wanted to share this little github gist i made that is a basic implementation of the ELM, “extreme learning machine”. a quick explainer is that its a single hidden layer FFNN with randomized and locked/frozen hidden neurons. the input weights/biases are not computed using backprop; it uses least squares solutions and psuedo-inverse of the hidden layer outputs multiplied by target outputs analytically. there are many papers on the ELM and will describe it much better than me.</p>
<p>in this demonstration we classify the MNIST dataset, and our results are seemingly quite good! here are the results of a few runs (data preprocessing takes much longer than the actually training):</p>
<p>1000 Hidden Neurons:<br>
Training Time: 2.3480 seconds<br>
Prediction Time: 0.0872 seconds<br>
Test Accuracy: 94.12%</p>
<p>5000 Hidden Neurons:<br>
Training Time: 39.7465 seconds<br>
Prediction Time: 0.3457 seconds<br>
Test Accuracy: 97.28%</p>
<p>10000 Hidden Neurons:<br>
Training Time: 213.8608 seconds<br>
Prediction Time: 0.6887 seconds<br>
Test Accuracy: 97.66%</p>
<p>maybe too good to be true? idk, see for yourself:</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/derbydefi/616b1d5b986610f7cb5120ccaac85915">
  <header class="source">

      <a href="https://gist.github.com/derbydefi/616b1d5b986610f7cb5120ccaac85915" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/derbydefi/616b1d5b986610f7cb5120ccaac85915" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/derbydefi/616b1d5b986610f7cb5120ccaac85915</a></h4>

  <h5>elm.py</h5>
  <pre><code class="Python">import time
import torch
import torchvision
import torchvision.transforms as transforms

class ELM(torch.nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(ELM, self).__init__()
        self.hidden_size = hidden_size
        self.input_weights = torch.nn.Parameter(torch.randn(input_size, hidden_size) * 0.1, requires_grad=False)</code></pre>
   This file has been truncated. <a href="https://gist.github.com/derbydefi/616b1d5b986610f7cb5120ccaac85915" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Padding mask in attention

bogdan_ka: <p>Here is a simple example of computing attention scores (rather weights before multiplying the q,k product by values.) We have two sequenes, one of which is padded with 0. Attention mask will be dimension 10X10. And when applied to scores tensor it works as expected by changing all values on upper diagonal to 0 for both cases in this batch of 2.</p>
<p>The padding mask will be dimension 2X10, or rather 2X1X10 after unsqueeze(1) so it can be applied to each case in the batch. If we apply this mask to the scores before the attention mask, the last 5 columns of the score tensor of the first case in the batch change to -inf. Score tensor for the second case remains unchanged as expected.</p>
<p>But shouldn’t the last 5 columns AND the last 5 rows in the first tensor score change to -inf? Since the score matrix is essentially a similarity matrix between each element in the sequence, wouldn’t both padded rows and columns zero out? Image of the first case score tensor below.</p>
<pre><code class="lang-auto">x=torch.tensor([[1,2,3,4,5,0,0,0,0,0],[1,2,3,4,5,6,7,8,9,10]])

embed=torch.nn.Embedding(11,12)

q=embed(x)

k=q

# attention mask

atten_mask=np.triu(np.ones((1, 10, 10)), k=1).astype('uint8')

atten_mask = (torch.from_numpy(atten_mask) == 0)

# padding mask

pad_mask=(x!=0)

scores = torch.matmul(q, k.transpose(-2, -1)) / 12 ** 0.5

# masked scores, applying pad mask only 

pad_mask = pad_mask.unsqueeze(1)

scores.masked_fill_(pad_mask == 0, -np.inf)

#scores.masked_fill_(atten_mask == 0, -np.inf)

F.softmax(scores, dim=-1)[0]
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a8eab5cb5160a96e3966b7fad5ce0acb0a4f4b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a8eab5cb5160a96e3966b7fad5ce0acb0a4f4b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a8eab5cb5160a96e3966b7fad5ce0acb0a4f4b1.png" alt="image" data-base62-sha1="cV6DGu3Y6qCza8Yf6WqmUQ8IpvH" width="690" height="377" data-dominant-color="2B2A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">911×499 13.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

bogdan_ka: <p>Just read more on this, apparently padding mask only needs to be applied to the key side (columns), not the query side (rows). Apparently this is not a big deal even though a padded item on the query side is attending to previous non-padded items. This is because the padded output is not used in any way. TIL</p>

AdithyaVenkateshMoha: <p>can you share us the materials / resources on where you read about this</p>
Customized loss function lost grad_fn

leoseak2: <p>I tried to build my own loss function that uses the torch.sort (it is a sequential ranking problem). Here is the code:</p>
<blockquote>
<p>def loss_fn(output, target):<br>
sorted, indices =torch.sort(output,descending=True)<br>
loss = torch.mean((output - target)**2)<br>
loss2 = torch.mean((indices - target)**2)<br>
return loss2</p>
</blockquote>
<p>When I return the loss, there’s no error. However, when I use loss2, I get the runtime error: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.</p>
<p>This error cannot be fixed by setting</p>
<blockquote>
<p>loss2.requires_grad=True</p>
</blockquote>
<p>When I look into the tensor, I think the only difference between loss and loss2 is that loss has “grad_fn= MeanBackward0”. How could I add this to loss2 so that I can use my own loss function?</p>

KFrank: <p>Hi Leo!</p>
<aside class="quote no-group quote-modified" data-username="leoseak2" data-post="1" data-topic="200462" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leoseak2/48/69076_2.png" class="avatar"> leoseak2:</div>
<blockquote>
<blockquote>
<pre><code>    sorted, indices =torch.sort(output,descending=True)
</code></pre>
<p>…</p>
</blockquote>
<p>when I use loss2, I get the runtime error: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.</p>
</blockquote>
</aside>
<p>The core problem is that <code>indices</code> consists of discrete integers (in particular,<br>
<code>long</code>s), so it is not (usefully) differentiable.</p>
<p>This means that what you are trying to do probably doesn’t make sense.<br>
But if it were to make sense somehow, you would have to find a usefully<br>
differentiable proxy for <code>indices</code> that somehow captured something relevant<br>
to <code>output</code>’s sort order (but was still differentiable) and compute your loss<br>
function from that.</p>
<p>Best.</p>
<p>K. Frank</p>
Using torch.logical_and turns requires_grad flag to False

haj: <p>As a toy code</p>
<pre><code class="lang-auto">tseq = torch.linspace(0,1,100).requires_grad_(True)
print(tseq.requires_grad)

result = torch.logical_and(0.2 &lt; tseq, tseq &lt; 0.7)
print(result.requires_grad)
</code></pre>
<p>It seems that <code>torch.logical_and</code> automatically turns <code>requires_grad</code> to <code>False</code>. Is this function not supported by autograd?</p>

KFrank: <p>Hi Hajin!</p>
<aside class="quote no-group quote-modified" data-username="haj" data-post="1" data-topic="200426" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/haj/48/63446_2.png" class="avatar"> haj:</div>
<blockquote>
<pre><code class="lang-auto">result = torch.logical_and(0.2 &lt; tseq, tseq &lt; 0.7)
</code></pre>
<p>Is this function not supported by autograd?</p>
</blockquote>
</aside>
<p>This is correct.  <code>bool</code>s are discrete, so it doesn’t make sense to try to<br>
differentiate them and autograd doesn’t try.</p>
<p>(More specifically, the tensor <code>0.2 &lt; tseq</code> itself has <code>dtype</code> <code>bool</code> and<br>
pytorch has already set its <code>requires_grad</code> property to <code>False</code>, even<br>
though it was computed from a tensor that had <code>requires_grad = True</code>.<br>
So the problem has already occurred before the <code>logical_and()</code> function.)</p>
<p>Best.</p>
<p>K. Frank</p>
Can some body help me, I can’t load my custom model

ratez: <p>Hi guys, I have a problem when I load my model:</p>
<p>import torch<br>
import cv2</p>
<p>path = ‘C:/…/best.pt’<br>
model = torch.hub.load(‘ultralytics/yolov5’, ‘custom’, path , force_reload=True)</p>
<p>img = cv2.imread(‘C:/…/123111.jpg’)<br>
result = model(img)</p>
<p>results.print()<br>
results.show()</p>
<p>problem:</p>
<pre><code class="lang-auto">cannot instantiate 'PosixPath' on your system. Cache may be out of date, try `force_reload=True` or see https://docs.ultralytics.com/yolov5/tutorials/pytorch_hub_model_loading for help.
  File "C:\Users\.........................\ultralytics_yolov5_master\hubconf.py", line 50, in _create
    model = DetectMultiBackend(path, device=device, fuse=autoshape)  # detection model
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\.................................\hub\ultralytics_yolov5_master\models\common.py", line 467, in __init__
    model = attempt_load(weights if isinstance(weights, list) else w, device=device, inplace=True, fuse=fuse)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.
.
.
.
.
        
Exception: cannot instantiate 'PosixPath' on your system. Cache may be out of date, try `force_reload=True` or see https://docs.ultralytics.com/yolov5/tutorials/pytorch_hub_model_loading for help.
</code></pre>
How to achieve Torch-MLIR compatibility with Dynamo backend?

trirpi: <p>Consider the following simple module that only does a matrix multiplication and a torch Dynamo backend called <code>toy_backend</code>.</p>
<pre><code class="lang-auto">class TestModule(nn.Module):
    def forward(a, w):
        return torch.mm(a, w)

def toy_backend(gm, inputs):
    return gm.forward

c = torch.compile(TestModule(), backend=toy_backend)
</code></pre>
<p>Note that <code>gm.forward</code> will return a tuple even though the original <code>nn.Module</code> returns a single value (see [1]).<br>
This is not a problem in the current example. However, if instead of simply returning <code>gm.forward</code> we want to invoke the Torch-MLIR <code>compile</code> function on <code>gm</code> this does not work.<br>
This is the case because Torch-MLIR <a href="https://github.com/llvm/torch-mlir/issues/1471#issuecomment-1273913856" rel="noopener nofollow ugc">does not support single element tuple returns</a>.</p>
<p>Is it possible to change the GraphModule to return a single element (not a tuple)?</p>
<p>[1] Note the output of <code>gm.print_readable()</code> clearly shows a tuple being returned and performing gm.forward also returns a tuple:</p>
<pre><code class="lang-auto">def forward(self, L_a_ : torch.Tensor, L_w_ : torch.Tensor):
    l_a_ = L_a_
    l_w_ = L_w_

    mm = torch.mm(l_a_, l_w_);  l_a_ = l_w_ = None
    return (mm,)
</code></pre>

trirpi: <p>Not sure if this is a good way to do it, but the code below works. It simply deletes the output node and adds a new one after which it recompiles.</p>
<pre><code class="lang-auto">before_output_node, output_node = list(gm.graph.nodes)[-2:]
gm.graph.erase_node(output_node)
gm.graph.output(before_output_node)
gm.recompile()
</code></pre>
Parallel build of c++ extension using setuptools

ThibaultL: <p>How to build a c++ extension in parallel using setuptools?</p>
<p>I followed the tutorial here : <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>.<br>
I have a quite similar <code>setup.py</code> (see below) with several c++ source files for one python module.<br>
I build the extension using the command: <code>pip install -e .</code>.<br>
But the build is sequential and every source files are rebuilt, even if only one file changes… which is very time consuming.<br>
How to perform <strong>incremental parallel</strong> build, in the spirit of <code>make -j16</code>?</p>
<p>I am on Ubuntu 22.04 using PyTorch 2.0.1.<br>
I checked online but could not find any solution yet.<br>
I only saw that it is possible to build several c++ extensions in parallel, but not one unique extension.<br>
I saw a <code>parallel</code> option in the class <code>build_ext</code> of setuptools, but I am not sure how to set it when using the <code>BuildExtension</code> of PyTorch.<br>
Sorry in advance if this not a PyTorch issue but a setuptools one</p>
<p>Here is my <code>setup.py</code> from a minimal example:</p>
<pre><code class="lang-auto">import torch
from setuptools import find_packages, setup
from torch.__config__ import parallel_info
from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension
from pathlib import Path

def get_extensions():
    all_sources = [
        'csrc/src.cpp', 
        'csrc/a.cpp', 
        'csrc/b.cpp', 
        'csrc/c.cpp']
    define_macros = [('WITH_PYTHON', None)]
    undef_macros = []
    include_dirs = []

    extra_compile_args = {'cxx': ['-O3']}
    extra_link_args = ['-s']

    info = parallel_info()
    if ('backend: OpenMP' in info and 'OpenMP not found' not in info):
        extra_compile_args['cxx'] += ['-DAT_PARALLEL_OPENMP']
        extra_compile_args['cxx'] += ['-fopenmp']
    else:
        print('Compiling without OpenMP...')

    extension = CppExtension(
        'torch_test._C',
        all_sources,
        include_dirs=include_dirs,
        define_macros=define_macros,
        undef_macros=undef_macros,
        extra_compile_args=extra_compile_args,
        extra_link_args=extra_link_args,
    )
    return [extension]

setup(
    name='torch_test',
    version='none',
    description=('none'),
    author='none',
    author_email='none',
    url='none',
    download_url='none',
    keywords=[],
    python_requires='&gt;=3.8',
    install_requires=[],
    extras_require={'test': []},
    ext_modules=get_extensions(),
    cmdclass={
        'build_ext':
        BuildExtension.with_options(no_python_abi_suffix=True, use_ninja=False)
    },
    packages=find_packages(),
    include_package_data=True,
)
</code></pre>

ThibaultL: <p>I solved my issue by using Ninja.</p>
GradCam on cifar10 obtains feature maps 1x1

Giovanni_Cena: <p>Hi,</p>
<p>I’m implementing the GradCam algorithm on several architectures, mainly Resnets. The main issue is that the feature maps are very small in the last block, precisely 1x1.</p>
<p>In particular, giving a batch of 64x3x32x32 (CIFAR10) to a Resnet18,<br>
the feature maps after the layer4.conv2 are [1, 512, 1, 1]</p>
<p>Therefore the cams are of size ([64, 1, 1, 1]), which are very small (and not informative at all!)</p>
<p>Is there a simple way to improve this situation? I’m considering interpolating CIFAR10 to 128x128 before training, to have at least more informative CAMs 4x4.</p>
<p>Do you have any advice?</p>
<p>I really appreciate any help you can provide.<br>
G</p>
Background as a Class

Adnan1: <p>In the TorchVision Object Detection Finetuning Tutorial, num_classes is set as 2, as person and background. Is it general thing to set number of classes as real number of classes plus 1? Or this example is true for image segmentation?<br>
My project is related with object detection not image segmentation and I have four number of classes in my dataset. So should I add background as class or not?</p>
Onnx model exported by pytorch 1.12 with wrong output shage

hitbuyi: <p>As shown below ,the left one is wrong when converted to tensor_RT model, the right is good, when I compare those two model ,the output’s shape of left is not specified. Is  this a normal or wrong phenomenon?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/381e17abe7aad8aef115303b5c97a311a050f8e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/381e17abe7aad8aef115303b5c97a311a050f8e0" title="Screenshot from 2024-04-09 21-43-16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381e17abe7aad8aef115303b5c97a311a050f8e0_2_690x282.png" alt="Screenshot from 2024-04-09 21-43-16" data-base62-sha1="80rd5GQK50Ql3CC1p7gG6d4MVm8" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381e17abe7aad8aef115303b5c97a311a050f8e0_2_690x282.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381e17abe7aad8aef115303b5c97a311a050f8e0_2_1035x423.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381e17abe7aad8aef115303b5c97a311a050f8e0_2_1380x564.png 2x" data-dominant-color="EFEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-04-09 21-43-16</span><span class="informations">1851×758 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is it possible to define a custom back-end wrapped with aot_autograd which does not use fake tensors?

cl_d: <p>Is it possible to define a custom back-end wrapped with <code>aot_autograd</code> which does not use fake tensors? Solely considering forward compilation, a custom back-end can be registered as follows:</p>
<pre><code class="lang-auto">def my_forward_compiler_fn(gm, tensor_inputs):
  pass

register_backend(name="my_backend", compiler_fn=aot_autograd(fw_compiler=my_forward_compiler))
</code></pre>
<p>In my_forward_compiler, <code>tensor_inputs</code> are of type <code>FakeTensor</code>. A human-readable table of the GraphModule can be printed using <code>gm.graph.print_tabular()</code>, however, there appears to be no way to access the underlying data of the arg tensors.</p>
2 phase training of autoencoders (e.g. USAD, TranAD)

spacemeerkat: <p>I’m trying to replicate the work of autoencoders which train in two phases. A good example is USAD which constructs loss values as:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44114baa0a1a2d32b20531d91b466453cdab9d12.png" alt="image" data-base62-sha1="9I9wszZ9El4zfdy7kYgbHbKzJYK" width="465" height="105"></p>
<p>Or perhaps more usefully:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/675c1dd975ea1830041bfd81b66b3cf9ce66b9b3.png" alt="image" data-base62-sha1="eKmxnslaV5W2RFThwvFcllktcHx" width="411" height="242"></p>
<p>My perhaps naive PyTorch question is:</p>
<p>For a network like this, do you pass the losses backwards to update the weights in one go (i.e. sum the above losses into one value), or do you need to do something special to apply the updates to the separate decoders?</p>
<p>My thinking is that, simply summing these two loss values before passing them backward will cancel out the second term in both loss calculations. But perhaps I am missing the finer details of how the constructed gradient tree doesn’t work like that?</p>
<p>Any help on the matter would be greatly appreciated, thanks everyone!</p>

spacemeerkat: <p>Ah… so looking at the USAD model on github, it appears that you may need to set up separate optimisers and then pass the loss values back individually for the weight updates: <a href="https://github.com/manigalati/usad/blob/master/usad.py#L91" class="inline-onebox" rel="noopener nofollow ugc">usad/usad.py at master · manigalati/usad · GitHub</a></p>