akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Place minmaxobserver on activation layer

username: wangtiance
cooked: <p>When I apply default qconfig to my model, it automatically puts minmaxobserver on conv and batchnorm layers, but not activation layers. But I need to get the min/max values of those layers too. Is there a simple way to do that?</p>
<pre><code class="lang-auto">import torch.ao.quantization as Q
testmodel = nn.Sequential(
    nn.Conv2d(10,10,3),
    nn.BatchNorm2d(10),
    nn.PReLU(10),
)
testmodel.qconfig = Q.default_qconfig
Q.prepare(testmodel, inplace=True)
print(testmodel) # no observer on prelu
</code></pre>

username: 111357
cooked: <p>What’s the <code>activation layers</code> refer to? <code>nn.PRelu</code>?</p>

username: wangtiance
cooked: <p>Yes. If it was relu or sigmoid I could infer the value range from the batchnorm output, but that’s not the case for prelu</p>

username: wangtiance
cooked: <p>I found that some activation modules (including nn.Hardtanh, nn.Hardswish) get an observer, but others don’t. I’m not sure where the difference lies.</p>

username: wangtiance
cooked: <p>It seems to has to do with the function<br>
torch.ao.quantization.quantization_mappings.get_default_qconfig_propagation_list(),<br>
which lists all quantizable modules. I can override it by specifying the allow_list in torch.ao.quantization.prepare()</p>
Removing last layer from yolov8 pretrained model

username: Ajith
cooked: <p>I have a pretrained model and i want to remove the last layer of it and convert to tflite model . This is the yaml file <a href="https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/v8/yolov8.yaml" class="inline-onebox" rel="noopener nofollow ugc">ultralytics/yolov8.yaml at main · ultralytics/ultralytics · GitHub</a> . i tried by using</p>
<pre><code class="lang-auto">model = YOLO("path to pretrained model")
model = torch.nn.Sequential(*list(model1.model.children())[:-1])
model1.export(format="tflite")
</code></pre>
<p>but it seems that that is not correct method . Is there any way by changing in yolov8.yaml file and removing the last layer and then converting to tflite.</p>
Please help Training Loss and Test Loss is NAN

username: Ashish_Verma
cooked: <p>Hello Team,</p>
<p>I am trying to solve a regression problem of predicting house using pytorch linear model<br>
=<a href="https://www.kaggle.com/code/averma111/house-price-pytorch-v2" class="inline-onebox" rel="noopener nofollow ugc">House-Price-Pytorch-V2 | Kaggle</a></p>
<p>The tensors look okay to me for test and train split but while training the model loss for test and train is coming as NAN.</p>
<p>Let me know how to correct this error.</p>

username: eqy
cooked: <p>Is the loss immediately NaN upon the first iteration (before any weight updates)? I would inspect the intermediate outputs of the model and see where the first NaN originates. You may want to use a utility function like <a href="https://pytorch.org/docs/stable/generated/torch.isfinite.html" class="inline-onebox" rel="noopener nofollow ugc">torch.isfinite — PyTorch 2.0 documentation</a> to help with this.</p>
<p>If the loss doesn’t start out as NaN but becomes NaN after some iterations, I would check the learning rate schedule of your optimizer and see if decreasing the learning rate helps.</p>

username: Ashish_Verma
cooked: <p>The model is starting with NaN from the beginning</p>
How to understand the parameter reduction set to 'none' in the F.cross_entropy function in pytorch?

username: zhengyang_xu
cooked: <p>I would like to use heat maps for pixel classification tasks. Currently, I have a classification heat map with predicted results, with dimensions (batch_num, C, H, W), as well as a ground truth heat map with dimensions (batch_num, C, H, W). At the same time, I want each pixel to have different levels of attention in different channels, so I have generated an energy matrix with dimensions (batch_num, C, H, W). In my experiment, I expect to put this energy matrix into F.cross_entropy(classification, class_heat_maps, reduction=‘none’) to get a result with dimensions (batch_num, C, H, W), which I will use for element-wise multiplication with my energy matrix.</p>
<p>This is my code:</p>
<blockquote>
<p>classification_loss=total_class_energy*F.cross_entropy(classification,class_heat_maps,reduction=‘none’)<br>
classification_loss=torch.sum(classification_loss,dim=1)<br>
classification_loss=torch.mean(classification_loss)</p>
</blockquote>
<p>Could you explain why the dimensions generated by my F.cross_entropy(classification, class_heat_maps, reduction=‘none’) function in the experiment are (batch_num, H, W)? Do I need to customize a cross entropy function to achieve my goal?</p>
<p>I would be very grateful for your help.</p>

username: ptrblck
cooked: <p><code>F.cross_entropy</code> is used for a multi-class classification or segmentation use case where each sample or pixel is assigned to a single class only.<br>
The output, if <code>reduction='none'</code> is used, will be the loss value for each sample or pixel location and thus no channel or class dimension is used. Note that this loss is not an elementwise loss since <code>dim1</code> represents the class dimension in the model output where each “channel” represents the logits for this corresponding class.</p>

username: zhengyang_xu
cooked: <p>This is very helpful, thank you very much.</p>
Laplace.sample() returns inf in half precision

username: Alethia
cooked: <p>Hi,</p>
<p>I’m training a network in fp16 precision, where it has to sample from a laplace distribution.<br>
I find that in fp16 the <code>laplace.sample()</code> method possibly returns <code>torch.inf</code>, but in fp32 this will not happen.<br>
To reproduce the code:</p>
<pre><code class="lang-python">import torch
import torch.distributions as dist
def test(r):
    count = 0
    for i in range(1000):
        if r.sample().isinf().any():
            count += 1
    return count


ra = dist.laplace.Laplace(torch.randn(100).cuda().half(),
                    torch.randn(100).cuda().exp().half())
print(test(ra)) # this returns non-zero value
</code></pre>
<p>Is it possible to solve this problem?<br>
I know I can manually cast to fp32 for this operation, but I wonder if there exists any clean method.</p>

username: ptrblck
cooked: <p>We generally do not recommend using <code>float16</code> manually but our mixed-precsion utils. instead which should cast to the needed wider <code>dtype</code> automatically.</p>
MTCNN- PyTorch processing time in still images and videos

username: axnet
cooked: <p>I’m working to implement the pre-trained model of MTCNN-PyTorch available here <a href="https://github.com/timesler/facenet-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - timesler/facenet-pytorch: Pretrained Pytorch face detection (MTCNN) and facial recognition (InceptionResnet) models</a> in NVIDIA jetson device. however although the code runs using CUDA, testing it in still images outputs processing time around 3-4 seconds per image. i tried to test in videos the processing time varied between 1-5FPS. Is there any explanation to this variation of time like why still images take longer than videos, and why videos can vary between 1 - 5FPS although all are in the same size and quality.</p>
LibTorch symbol lookup error in CPU

username: MrRace
cooked: <p>I try LibTorch in CPU.<br>
version 1: use cxx11 ABI</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=/tools/libtorch-cxx11-abi-shared-with-deps-1.13.0+cpu ..
cmake --build . --config Release
</code></pre>
<p>version 2: use Pre-cxx11 ABI</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=/tools/libtorch-shared-with-deps-1.13.0+cpu ..
cmake --build . --config Release
</code></pre>
<p>Both can build successfully, but when to run the release executable program come same error:</p>
<pre><code class="lang-auto">./test-model: symbol lookup error: ./test-model: undefined symbol: _ZN2at4_ops4ones4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEENS6_INS2_6DeviceEEENS6_IbEE
</code></pre>
<p>However when switch the <code>DCMAKE_PREFIX_PATH</code> to Pytorch install path, for example <code>/opt/conda/lib/python3.8/site-packages/torch/share/cmake</code>, the release executable program can run successfully.<br>
Therefore it seems the LibTorch download from official can not be directly use in C++？ Why ?<br>
My test C++ code snippet is simple below：</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include &lt;numeric&gt;

int main(int argc, const char* argv[]) {
    if (argc != 4) {
        std::cerr &lt;&lt; "usage: test-model &lt;path-to-exported-script-module&gt;\n";
        return -1;
    }
</code></pre>
<p>Above example What should do if I want to use the official LibTorch lib ? Maybe I use wrong version for my CPU? By the way, my OS is “Ubuntu VERSION=20.04.4 LTS (Focal Fossa)” and CPU is AMD EPYC 7713.</p>

username: Tim_Hanson
cooked: <p>For those who may have hit the same error: it seems to be caused by the executable using different libraries (e.g. system libraries) than those used during linking.</p>
<p>Run ldd on the executable to see what it’s using for runtime, and make sure those are the same as passed to g++ or ld.</p>
<p>Note also you can grep the libraries for missing symbols, including that super long “_ZN2at…” one in your error message, to make sure they exist.</p>
XLA debug flags

username: Jerome_Ku
cooked: <p>Hi,</p>
<p>Is it possible to pass <a href="https://github.com/openxla/xla/blob/main/xla/debug_options_flags.cc" rel="noopener nofollow ugc">XLA debug flags</a> to torch-xla?</p>
<p>E.g., when using jax, setting <code>XLA_FLAGS=--xla_force_host_platform_device_count=8</code> will mimic an 8 device / core backend (useful for debugging in CPU-only platforms) such that <code>jax.devices()</code> will return a list of 8 devices.</p>
<p>An equivalent call to <code>torch_xla.xla_model.get_xla_supported_devices()</code> would have similar functionality.</p>

username: JackCaoG
cooked: <p>Hi Jerome,</p>
<p>It is possible to do that through env var. One example is the flag we passed to xla directly <a href="https://github.com/pytorch/xla/blob/master/torch_xla/__init__.py#L48" class="inline-onebox" rel="noopener nofollow ugc">xla/__init__.py at master · pytorch/xla · GitHub</a>.</p>
<p>BTW we checked the github issue more often so next time you have issues feel free to open issues on <a href="https://github.com/pytorch/xla/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/xla · GitHub</a>.</p>
How to get a 'cpu' state dict

username: timbmg
cooked: <p>Assuming my model is on a gpu already, is there a way to get a state_dict of a model with cpu tensors without moving the model first to cpu and then back to gpu again?<br>
Something like:</p>
<pre><code class="lang-auto">state_dict =  model.state_dict()
state_dict = state_dict.cpu()
</code></pre>

username: SimonW
cooked: <pre><code class="lang-auto">for k, v in state_dict.items():
  state_dict[k] = v.cpu()
</code></pre>

username: francois-rozet
cooked: <p>There is also a one-liner to create a <code>cpu</code> copy of the <code>state_dict</code> :</p>
<pre><code class="lang-python">{k: v.cpu() for k, v in model.state_dict()}
</code></pre>

username: RobS
cooked: <p>Small correction: to iterate through, you need the <code>items()</code> from the dict, not the dict itself</p>
<pre><code class="lang-auto">{k: v.cpu() for k, v in model.state_dict().items()}
</code></pre>
How to save memory while having a big Jacobian matirx

username: miraboreasu
cooked: <p>Hello,</p>
<p>I am doing some computation using pytorch, there is a Jacobian matrix, J of a vector of equations, r, and I am using jacfwd to calculate it, then there will be a linear solving</p>
<pre><code class="lang-auto">        r      = get_residual(unknown)
        J      = jacfwd(get_residual)(unknown)
        update       = torch.linalg.solve(J, r)
</code></pre>
<p>I actually don’t need J to be explicit, how to use like COO or CSR matrix to save memory and speedup this process?</p>
<p>Thanks!</p>

username: miraboreasu
cooked: <p>I don’t need the full matrix from jacfwd, so is there way to save it to a tmp and directlly make it COO or CSR?</p>
Sigmoid function problem for NLP tasks

username: Gorgen
cooked: <p>my task is multimodal emotion recognition. audio features is (batch, seq, feature_dimension). if use torch. Sigmoid(audio), it works for feature dimension.</p>
<p>my question is if audio feature is (seq, batch, feature_dimension), batch_first is false, how to use torch. Sigmoid function?</p>
<p>many thanks<br>
best wishes</p>

username: JuanFMontesinos
cooked: <p>Sigmoid is an elementwise operator.</p>
torch.cuda.OutOfMemoryError: CUDA out of memory.

username: kuohao233
cooked: <p>I got this error when I was doing an image classification task, The problem persists even if I lower the batch to 64</p>
<pre><code class="lang-auto">net.train()
        # train and update
        optimizer = torch.optim.Adam(net.parameters(), lr = self.lr)
        privacy_engine = PrivacyEngine()
        net, optimizer, self.ldr_train = privacy_engine.make_private_with_epsilon(
            module=net,
            optimizer=optimizer,
            data_loader=self.ldr_train,
            target_epsilon=2, target_delta=1e-5,
            epochs=50,
            max_grad_norm=1.0,
        )
        epoch_acc = []
        epoch_loss = []
        for iter in range(self.local_ep):
            batch_acc = []
            batch_loss = []

            for batch_idx, (images, labels) in enumerate(self.ldr_train):
                images, labels = images.to(self.device), labels.to(self.device)
                net.zero_grad()
                # ---------forward prop-------------
                fx = net(images)

                # calculate loss
                loss = self.loss_func(fx, labels)
                # calculate accuracy
                acc = calculate_accuracy(fx, labels)

                # --------backward prop--------------
                loss.backward()
                optimizer.step()

                batch_loss.append(loss.item())
                batch_acc.append(acc.item())

</code></pre>

username: ptrblck
cooked: <p>Depending on your model, use case, and device you might need to lower the batch size further or try to reduce the memory usage e.g. via mixed-precision training or checkpointing.</p>

username: kuohao233
cooked: <p><code>loss.backward()</code></p>
<p>The memory overflow is caused at this location and the problem still exists even if the batch is reduced to a very low level</p>

username: kuohao233
cooked: <p>By introducing BatchMemoryManager, the memory overflow problem has been solved, but another problem is that the accuracy rate has dropped to single digits or even 0 and is not getting better</p>
I'm trying to rewrite the CUDA cache memory allocator

username: raining_day513
cooked: <p>Hi, I’m trying to temporarily disable the pre-allocation of CUDA GPU memory to investigate the CUDA GPU memory usage of some experiments with my model. To be clear,</p>
<ol>
<li>I want to make sure that GPU is allocated as late as possible, so I can read the correct and exact memory usage of each layer of my model. (I understood that <code>cudaMalloc</code> is an expensive operation, but that’s fine in my case!)</li>
<li>I want to release the GPU memory as soon as possible. From my current understanding, this has been done decently in the current code base. I just simply mentioned it here.</li>
</ol>
<p>From my current understanding, I will need to investigate <code>cudaMalloc</code>, and the pre-allocation is controlled by the cache memory allocator as mentioned by my title.</p>
<p>Am I on the right track this way trying to understand the source of <code>CUDAMallocAsyncAllocator</code>? It will help me a lot if someone did this before and could share it with me. Or you might point out what’s the problem with this track and point me to another recommended one. Thanks!</p>

username: ptrblck
cooked: <p>You can disable the caching allocator via <code>export PYTORCH_NO_CUDA_MEMORY_CACHING=1</code> which will then use <code>cudaMalloc/Free</code> calls without reusing any memory.</p>

username: raining_day513
cooked: <p>Hi, ptrblck, thanks for your advice! I will try it for sure!</p>
<p>Before I found your answer I was thinking about using both:</p>
<ul>
<li><code>torch.cuda.max_memory_reserved</code></li>
<li><code>torch.cuda.reset_peak_memory_stats</code></li>
</ul>
<p>in the <code>register_forward_hook</code> of nested <code>nn.Module</code> to measure the CUDA GPU memory of each layer of my model as if the CUDA caching mechanism of PyTorch is not enabled. (In detail: I was thinking about subtracting the value returned by the first API from the GPU memory usage including caches.) Do you think this a good idea? Thanks for your kindness <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  (and I probably used the wrong tag, let me fix it now!)</p>

username: ptrblck
cooked: <p>Instead of checking the <code>reserved</code> memory (which includes the cache) and subtracting the reported memory by <code>nvidia-smi</code> I would just use <code>torch.cuda.memory_allocated</code> which will return the used and allocated memory only.</p>

username: ChineseBest
cooked: <p>I agree, memory_allocated will be a better idea. I think raining_day513 is in the right track.</p>
U Net won't train. Any ideas?

username: deveshdatwani
cooked: <p>I know I am being a bit lazy here by asking this question.</p>
<p>I am trying to train a UNet model on Kvasir medical image dataset. My dice loss doesn’t decrease.</p>
<p>I included a basic block to check if the weights of the layers were being updated.</p>
<p>It tunrs out that only my final hidden layer is being updated. Which means the training might have vanishing gradient problem.</p>
<p>Ps- I am trying to train the network without BatchNorm layers.</p>
Can't run inference on FP16 trained model

username: tilhi
cooked: <p>I am using a transformer model that utilizes MultiScaleDeformableAttention. The training of the model using mixed precision was easy by just utilizing the trick described <a href="https://discuss.pytorch.org/t/how-can-i-write-the-cuda-code-to-support-fp16-calculation/107181/3">here</a>. However, inference was not as straight forward. I tried autocasting using <code>torch.autocast</code> but MSDA module gave ‘expected half but got float’. Then when forcing the input values to fp16 using .half() I got ‘expected long but got half’ with one of the inputs (attention_weights). Then when changing that using .long() I got ‘expected half but got long’.</p>
<p>So my question is, is it even possible to run fp16 inference with MSDA module-based model? Input is fp16 and the weights are also fp16.</p>

username: ptrblck
cooked: <p>Could you describe your use case in more detail as it seems you have written a custom kernel using raw <code>float16</code> data instead of the native mixed-precision utils.?</p>

username: tilhi
cooked: <p><code>torch.autocast</code> does not work automatically when using MSDA (Getting an error like ‘half not implemented for MSDA’ or something along those lines), but the training was performed successfully using <code>torch.autocast</code> after the <code>FLOATING_TYPES_AND_HALF</code>-change to that specific module. However, even inside the autocast there is still complaint about the inputs being float instead of half. This happens in the forward of MSDeformAttn-function:</p>
<pre><code class="lang-auto">output = MSDeformAttnFunction.apply(
            value, input_spatial_shapes, input_level_start_index, sampling_locations, attention_weights, self.im2col_step)
</code></pre>
<p>How are there still floats inside that function and why changing them by force to be half gives still errors (expected long got half for attention_weights, expected half got long when not changing it).</p>

username: ptrblck
cooked: <p>Since you are using custom <code>autograd.Function</code>s, check <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#autocast-and-custom-autograd-functions">this example</a> which explains how to add the <code>@custom_fwd</code> and <code>@custom_bwd</code> context managers.</p>

username: tilhi
cooked: <p>I have experimented with the custom context managers, however I did not use them during training and there were no errors. However, now adding them during inference to avoid errors I get less than desirable results with my model.</p>
<p>Is there any way to run the model in a way that the context would be identical to the one during training?</p>
Vgg16 as the perceptual loss function on 1 channel data

username: Harsimar_Singh
cooked: <p>Hi experts, I have one channel data tensor of raw MRI slice i.e. [16,1,320,320] and I want to use Vgg16 as the perceptual loss function. So If I am not wrong, we push the output and the ground truth inside the vgg and compare their activations after relu using a loss function like MSE, so basically comparison the loss between the activation maps right. So vgg, as a pretrained network, works on the 3 channel images but my input tensor is 1 channel. Should I just copy the 1 channel into 3 channels? I need some guidance in this. Thanks!</p>

username: gkalliatakis
cooked: <p><a class="mention" href="/u/harsimar_singh">@Harsimar_Singh</a> Is there an update on this? I’m trying to do something similar.</p>
Saving FSDP model via full_optim_state_dict

username: ipoletaev
cooked: <p>Hi,</p>
<p>Trying to find out why FSDP model does not want to save its optimizer state from within all processes in parallel:</p>
<pre><code class="lang-auto">from torch.distributed import fsdp
...
model = ... # torch.nn.Module
model = fsdp.FSDP(model)
optimizer = torch.optim.adam.AdamW(model)
...

state_dict_config = fsdp.FullStateDictConfig(
            offload_to_cpu=True, rank0_only=True)
state_dict_type = fsdp.StateDictType.FULL_STATE_DICT
with fsdp.FullyShardedDataParallel.state_dict_type(
            model,
            state_dict_type=state_dict_type,
            state_dict_config=state_dict_config):
    model_state = model.state_dict()

optimizer_state = fsdp.FullyShardedDataParallel.full_optim_state_dict(
                model=model, optim=optimizer, rank0_only=True)
</code></pre>
<p>fails with</p>
<pre><code class="lang-auto">File ".venv/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1331, in full_optim_state_dict
    return FullyShardedDataParallel._optim_state_dict_impl(
  File ".venv/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1154, in _optim_state_dict_impl
    return _optim_state_dict(
  File ".venv/lib/python3.8/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1455, in _optim_state_dict
    _gather_orig_param_state(
  File ".venv/lib/python3.8/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1697, in _gather_orig_param_state
    value = value[: flat_param._numels[param_idx]].reshape(
RuntimeError: shape '[50304, 768]' is invalid for input of size 8
</code></pre>
<p>Meaning that the model state is collected, but not the optimizer’s one. Which is somewhat strange given that it follows examples in <a href="https://github.com/pytorch/pytorch/blob/e68e84ef8a3b28780b22e7440bcf5486a05a73d5/test/distributed/fsdp/test_fsdp_optim_state.py#L573" rel="noopener nofollow ugc">pytorch tests</a> or here with <a href="https://github.com/mosaicml/composer/blob/b52f6c8e44441242a8245c00c4d674bbc1573f71/composer/core/state.py#L773" rel="noopener nofollow ugc">mosaic</a>.</p>
<p>Any clue what this could be if the torch version is <code>2.0.0</code>? Thank you very much in advance!</p>

username: agu
cooked: <p>It might be best for you to file a Github issue with a way to repro for us to best help out.</p>
Seeking Advice on Total Grad Norm Values

username: smartbarbarian
cooked: <p>Hello everyone,</p>
<p>I’m currently working on a deep learning project and have a question about total grad norms( total_grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) ). I’ve noticed that the typical total grad norm values are around 0.2 or less than 1.0, even in larger models. This seems to be the reason why people often set clip_grad_norm values to 0.5 or 1.0 during training.</p>
<p>However, in my own model, I’ve observed grad norm values in the hundreds during training, and surprisingly, the model still performs well. Intuitively, it would make sense that the grad norm value should increase with more parameters. So, I’m curious as to why it is usually smaller than 1.0 in most cases.</p>
<p>I would greatly appreciate any insights, advice, or explanations that could help me better understand these grad norm values and their implications on the training process. Thank you in advance for your help!</p>
Error with view, no_grad and inplace modify?

username: Goldname
cooked: <p>I am trying to do this:</p>
<pre><code class="lang-auto">with torch.no_grad():
  for param in net.parameters():
    for j in param.flatten():
        print("current j", j)
        j += 1
</code></pre>
<p>However I get this error:</p>
<blockquote>
<p>RuntimeError: A view was created in no_grad mode and its base or another view of its base has been modified inplace with grad mode enabled. Given that this use case is ambiguous and error-prone, it is forbidden. You can clarify your code by moving both the view and the inplace either both inside the no_grad block (if you don’t want the inplace to be tracked) or both outside (if you want the inplace to be tracked).</p>
</blockquote>
<p>Can someone explain to what has grad mode enabled? The whole block is under no_grad(). Also, the first iterations runs well. I get this output:</p>
<blockquote>
<p>current j tensor(0.0919, requires_grad=True)<br>
current j</p>
</blockquote>
<p>However in the second iteration of the loop I get the error message above. Why is this happening only in the second iteration?</p>

username: ptrblck
cooked: <p>I’m able to reproduce the issue using your posted code and don’t know why the error is raised since all operations are already in the <code>no_grad</code> guard.<br>
However, it also seems that the <code>print</code> statement is causing the issue as removing it gets rid of the error:</p>
<pre><code class="lang-python">net = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU(),
)

with torch.no_grad():
  for param in net.parameters():
    for j in param.flatten():
        #print("current j", j)
        j += 1
</code></pre>
<p><a class="mention" href="/u/alband">@albanD</a> do you know if this is a new regression? Let me know if you want me to create an issue to track it.</p>

username: albanD
cooked: <p>Ho wow this is super fishy… Yes we should definitely open an issue with this small repro!</p>

username: ptrblck
cooked: <p>Created <a href="https://github.com/pytorch/pytorch/issues/99968">#99968</a> to track it.</p>
RuntimeError: Error when trying to collate the data into batches with fa_collate, at least two tensors in the batch are not the same size. Please include a transform in after_item that ensures all data of type Tensor is the same size

username: Niraja
cooked: <p>I am working on different size of image dataset. While preprocessing I got this error,<br>
RuntimeError: Error when trying to collate the data into batches with fa_collate, at least two tensors in the batch are not the same size.</p>
<p>Mismatch found on axis 0 of the batch and is of type <code>Tensor</code>:<br>
Item at index 0 has shape: torch.Size([1, 114, 114])<br>
Item at index 1 has shape: torch.Size([3, 161, 162])</p>
<p>Please include a transform in <code>after_item</code> that ensures all data of type Tensor is the same size<br>
Please tell me how to solve this error as I am first time see this type error. Thank you in advance.</p>
<pre><code>'''
class MTLDataset(Dataset):
def __init__(self,df, transform=None):
    self.path = list(df.path)
    self.age = list(df.age)
    self.gender = list(df.gender)
    #self.ethnicity = list(df.ethnicity)
    self.transform = transform

def __len__(self): 
    return len(self.path)

def __getitem__(self,idx):
    #image
    path = self.path[idx]
    image = read_image(path)
    image = convert_image_dtype(image,dtype = torch.float32)
    if self.transform:
        image = self.transform(image)
    #age,gender,ethnicity
    age = self.age[idx]
    gender = self.gender[idx]
    #ethnicity = self.ethnicity[idx]
    
    return image, age, gender
'''
'''
def generate_dataloader(df, transformation = None):
  ds = MTLDataset(
  df,
  transformation
)

return DataLoader(
  ds,
  batch_size = 100,
  num_workers = 4,
)
'''
'''
transformation = transforms.Compose([
transforms.Resize((100, 100)),
transforms.ToTensor(),
# transforms.Normalize(mean=[0.485, 0.456, 0.406],
#                      std=[0.229, 0.224, 0.225])
])
'''
'''
train_dataloader = generate_dataloader(train_df)
test_dataloader = generate_dataloader(test_df)
'''
'''
train_features, train_age, train_gender = next(iter(train_dataloader))
print(f"Feature batch shape: {train_features.size()}")
print(f"Feature: {train_features[0]}")
'''
</code></pre>
<p>This is my code. I tried to solve this error but I Couldn’t.</p>

username: ptrblck
cooked: <p>You could either resize the images to the same shape thus allowing the <code>DataLoader</code> to stack them to a single batch or you could create a custom <code>collate_fn</code> which would return e.g. a <code>list</code> containing image tensors in different shapes. In the latter case you would need to make sure your model is able to work with this data structure as usually also a single input batch would be expected.</p>
<p>The error message also seems to come from a higher-level API unless I’m not familiar with the <code>after_item</code> call.</p>
Input features in reward function

username: NHarris
cooked: <p>Can the values used as an environments input also be used to calculate the reward or would that somehow lessen the agents performance or “learning”?</p>

username: J_Johnson
cooked: <p>That depends on how directly correlated those values are to the desired outcome. The more correlated, the more beneficial.</p>
<p>For example, in Lunar Lander, the euclidean distance from the landing spot can be directly calculated from the x, y coordinates. And so part of the desired solution is reducing that distance.</p>
<p>However, if you reduce it too quickly, the velocity will be too high and it will crash. So the other part of the solution is maintaining a safe velocity and decelerating when approaching the landing zone.</p>
<p>So one could construct a reward function with some combination of the above points.</p>
<p>However, if you simply give the inverse of the euclidean distance as a reward(i.e. as the distance increases, the reward decreases and v.v.), you will have the issue that the reward may be too high at very close distances. So you either need to clip the reward OR make it 0 when the delta of the distance from the previous frame is unchanged/increased(i.e. the rocket didn’t get closer) and 1 when that distance decreased(the rocket got closer).</p>
"Padded" add in pytorch?

username: zisisnotzis
cooked: <p>How to add a smaller tensor to a bigger tensor out of place, just like padded with zero?</p>
<p>This can be done either by <code>pad+add</code> or by <code>split+add+cat</code>, but both of them wastes time copying data around.</p>
<p>Another possible method is <code>torch.Tensor.put</code>, but I don’t really have or want to generate a index tensor. This would result in wasting time indexing around.</p>
<p>The third method I can think of, is to manually create output tensor, and do <code>torch.add(out=xx)</code> stuff. This theoretically does not waste any time, but might it affect auto-optimization / autograd? This is also not really elegant.</p>
<p>Any better ideas?</p>
Loading ImageNet data from zip files

username: hamedB
cooked: <p>Hi everyone,</p>
<p>I am going to download and store ImageNet training set on the server in my lab. The examples on the internet require to extract all sub folders in the data root in advance. However, due to a memory limitation on the amount of data samples, I can not extract sub folders from ImageNet directory in my server. My question is that whether it is possible to implement custom data loader that only extracts a sub folder in case it wants to load a sample from that zip directory. if so, I would really appreciate any help, explanation, guidance or sharing any code snippet.  My question is opposite to the solution where all data are extracted in advance.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You could check e.g. <a href="https://stackoverflow.com/questions/5710867/downloading-and-unzipping-a-zip-file-without-writing-to-disk">this approach</a> to unzip the desired folder(s) during the actual loading, but I would also expect to see a large performance hit.<br>
In any case you could experiment with it in a custom <code>Dataset</code> and see how well this approach of re-extracting the same folder would work.</p>
Model forward changes the input?

username: aviadp
cooked: <p>Hello, I train model with pytorch and the traning working good.<br>
I try to get the summary from this model and also predict to get result, but i get something strange.<br>
In the train i use batch size that equal to 1, and the image size is 224,224,8.<br>
and the tensor shape is ([1, 8, 244,244]) but in the summary i get torch.Size([2, 8, 224, 224]).</p>
<p>I try to understand what the problem and i don’t success to understand, someone can help?</p>

username: ptrblck
cooked: <p>Could you post the model definition here so that we could try to reproduce the unexpected shape?</p>
Dynamo cache limit and too many graph breaks

username: aifartist
cooked: <p>Previously I had gotten torch.compile inductor max-autotune to work with good results.  When I recently tried again to produce my results I kept hitting the cache size limit of 64.  Is this related to too many “graph breaks”?</p>
<p>I found a way to debug graph breaks by having it dump details about them.  One of them that I get, in large numbers, is in a case where it complains about “torch.float16” being returned from a function.  NOTE: This isn’t a value of this type but the “type” itself.</p>
<p>What might it take to get _dynamo/variables/builder.py:wrap_fx_proxy_cls() to be able to handle a simple constant which is just a type?</p>
<p>Also, I believe there seems to be a lot of graph breaks due to “call_function”.  I don’t understand thing enough probably to ask the right question other than to ask if it could be supported.</p>

username: marksaroufim
cooked: <p>Check out <code>torch._dynamo.explain()</code> to diagnose graph breaks <a href="https://pytorch.org/docs/master/compile/troubleshooting.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.0 Troubleshooting — PyTorch master documentation</a></p>
<p>If you have a repro I can take a look</p>
Treat a custom nn.Module class as a monolithic operator

username: Shaunak_Sachin_Joshi
cooked: <p>Hi I have a following class extending the nn.Module for performing quantization.</p>
<pre><code class="lang-auto">class LTQ(nn.Module):
    def __init__(self, num_bits):
        super(LTQ, self).__init__()
        init_range = 2.0
        self.n_val = 2**num_bits - 1
        self.interval = init_range / self.n_val
        self.start = nn.Parameter(torch.Tensor([0.0]), requires_grad=True)
        self.a = nn.Parameter(
            torch.Tensor([self.interval] * self.n_val), requires_grad=True
        )
        self.scale1 = nn.Parameter(torch.Tensor([1.0]), requires_grad=True)
        self.scale2 = nn.Parameter(torch.Tensor([1.0]), requires_grad=True)

        self.two = nn.Parameter(torch.Tensor([2.0]), requires_grad=False)
        self.zero = nn.Parameter(torch.Tensor([0.0]), requires_grad=False)
        self.eps = nn.Parameter(torch.Tensor([1e-3]), requires_grad=False)

    def forward(self, x):

        x = x * self.scale1

        x_forward = x
        x_backward = x
        step_right = self.zero + 0.0

        a_pos = torch.where(self.a &gt; self.eps, self.a, self.eps)

        for i in range(self.n_val):
            step_right += self.interval
            if i == 0:
                &lt;some code&gt;
            else:
                &lt;some code&gt;
</code></pre>
<p>Here for my use case I would like this entire <code>LTQ(nn.Module)</code> to be treated as a monolithic operator like <code>nn.Conv2d</code> when I run <code>torch._C._jit_pass_inline()</code> on the output of <code>torch.jit.trace</code>. In this case the _jit_pass_inline actually traces every line inside the forward function and maps the operators in each line to the respective <code>aten</code> operators. Can someone please help me if I want the trace to treat this class as an operator?? <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/alband">@albanD</a> your help would be greatly appreciated.</p>

username: ptrblck
cooked: <p>I would recommend not to tag specific users as it might discourage others to post a valid answer.</p>
<p>If I understand your question correctly, you are concerned about tracing showing all internal operations instead of one “wrapper”? If so, I wouldn’t know the answer unfortunately.</p>
"manually" creating a channels-last tensor

username: Axel-Jacobsen
cooked: <p>Howdy! I am relatively fresh to channels last memory format. I’ve spent time reading around, and I don’t think my question has been answered yet.</p>
<p>I am doing some object detection: (a batch of) grayscale image(s) in, to a (batch of) tensor(s) of shape <code>(prediction dim, H, W)</code> out, where <code>H</code> and <code>W</code> specify the “prediction grid” shape, and <code>prediction dim</code> has bounding box and class information.</p>
<p>With results from a batch of images, I naturally have to do a lot of indexing along the prediction grid (i.e. I want to know the result at H = 12, W = 15). Right now, I index the result tensor like this: <code>res[:, 12, 15]</code>. However, I think this is probably slower than if I had the <code>prediction dim</code> last, like <code>res[12,15,:]</code>. So in a sense, I want channels last.</p>
<p>I get the result tensor from a conv net. Right now, if setting <code>memory_format</code> to <code>channels_last</code> also set the shape of the tensor, I could then gladly index the result in the “channels last” format like above. However, <code>channels_last</code> simply sets the stride.</p>
<p>Is it possible to manually set the shape as I desire (e.g. a permute of the input image into channels last) and override the designation of the memory format to channels last? If I do a permute and then <code>to(channels_last)</code>, the stride changes, and I don’t quite get what I want.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I don’t know what exactly you want to do with the result tensor but wouldn’t a simple <code>permute(...).contiguous()</code> and indexing work? The <code>channels_last</code> memory format is used by some layers, such as conv layers, for performance reason but in case you are only concerned about indexing specific values a “manual” permutation could work.</p>

username: Axel-Jacobsen
cooked: <p>Ah OK, thank you. I suppose I don’t quite understand the semantics of how Pytorch handles <code>channels_last</code> memory format. You’re right, the <code>permute(...).contiguous()</code> works. Thanks!</p>
MPS on m2 max working poorly

username: elandy
cooked: <p>I have a macbook pro m2 max and attempted to run my first training loop on device = ‘mps’.</p>
<p>I’m running a simple matrix factorization model for a collaborative filtering problem R = U*V.t, where U and V share a latent factor dimension.  I’ve found that my kernel dies every time I try and run the training loop except on the most trivial models (latent factor dim = 1) and batch sizes (n = 1).  Even with these trivial dimensions, the model seems to run quite slowly, though it does progress through the epochs.</p>
<p>The training loop runs as expected on the CPU.</p>
<p>I’m somewhat new to pytorch and training on GPUs, so I may be doing something wrong.</p>
<p>Training loop code below:</p>
<pre><code class="lang-auto">for epoch in range(EPOCH_NUM):
    model.train()
    train_loss = 0
    train_mae = 0
    for user_item_idx, rating in train_loader:
        user, item = user_item_idx[:, 0].to(device), user_item_idx[:, 1].to(device)
        rating = rating.to(device)
        optimizer.zero_grad()
        prediction = model(user, item)
        batch_mse = loss(prediction, rating)
        batch_loss = batch_mse
        batch_loss.backward()
        optimizer.step()
        train_loss += batch_mse.item()
        train_mae += torch.abs(prediction - rating).mean().item()
    train_loss /= len(train_loader)
    train_mae /= len(train_loader)


    model.eval()
    val_loss = 0
    val_mae = 0
    for user_item_idx, rating in val_loader:
        user, item = user_item_idx[:, 0].to(device), user_item_idx[:, 1].to(device)
        rating = rating.to(device)
        prediction = model(user, item)
        batch_loss = loss(prediction, rating)
        val_loss += batch_loss.item()
        val_mae += torch.abs(prediction - rating).mean().item()
    val_loss /= len(val_loader)
    val_mae /= len(val_loader)

    scheduler.step(val_loss)

    print(f"Epoch {epoch + 1} | Train loss: {train_loss:.4f} | Train MAE: {train_mae:.4f} | Val loss: {val_loss:.4f} | Val MAE: {val_mae:.4f}")

    prev_val_loss = [np.inf]
    # early stopping
    if epoch &gt; 0 and val_loss &gt; (min(prev_val_loss) * 1.05):
        break
    prev_val_loss.append(val_loss)
</code></pre>

username: marksaroufim
cooked: <p>What error are you getting exactly? Make sure to monitor the macOS activity monitor it might give you a hint as to what’s going</p>
Repytorch replace the batch norm layer in resnet50 with group norm layer

username: nikiguo93
cooked: <p>Hi, can anybody tell me how to replace the batch norm layer in resnet50 with group norm layer using pytorch?</p>

username: eqy
cooked: <p>To start I would take a look at the existing reference implementations in <code>torchvision</code> <a href="https://pytorch.org/vision/main/_modules/torchvision/models/resnet.html#resnet50" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.resnet — Torchvision main documentation</a><br>
and see if you can simply modify the handling of the <code>norm_layer</code> argument to handle <code>GroupNorm</code> (since it requires specifying the number of groups in addition to the number of channels)l</p>
RuntimeError: x86 is not a valid value for quantized engine

username: sbht_sbht
cooked: <p>Trying to quantize my model using this documentation<a href="https://pytorch.org/docs/stable/quantization.html#backend-hardware-support" rel="noopener nofollow ugc"> Backend/Hardware Support</a> and get this  error - “<code>RuntimeError: x86 is not a valid value for quantized engine</code>”.</p>
<p>I followed this instruction:</p>
<pre><code class="lang-auto"># set the qconfig for PTQ
# Note: the old 'fbgemm' is still available but 'x86' is the recommended default on x86 CPUs
qconfig = torch.ao.quantization.get_default_qconfig('x86')
# or, set the qconfig for QAT
qconfig = torch.ao.quantization.get_default_qat_qconfig('x86')
# set the qengine to control weight packing
torch.backends.quantized.engine = 'x86'
</code></pre>
<p>Using this model:</p>
<pre><code class="lang-auto">from torch.quantization import QuantStub, DeQuantStub

class QuantizedSiameseNetwork(nn.Module):
    """
        Siamese network for image similarity estimation.
        The network is composed of two identical networks, one for each input.
    """
    def __init__(self, embedding_size=2, use_quant=False):
        super().__init__()
        self.backbone = Backbone(embedding_size)
        self.use_quant = use_quant
        if self.use_quant:
            self.quant_m = QuantStub()
            self.dequant_m = DeQuantStub()
</code></pre>
<p>And this code is how I tried to implement it:</p>
<pre><code class="lang-auto">model_fp32 = QuantizedSiameseNetwork(2, True).to('cpu')
backend = "x86"
model_fp32.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend

model_static_quantized = torch.quantization.prepare(model_fp32, inplace=False)
</code></pre>

username: sbht_sbht
cooked: <aside class="quote no-group" data-username="sbht_sbht" data-post="1" data-topic="178419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sbht_sbht/48/46999_2.png" class="avatar"> sbht_sbht:</div>
<blockquote>
<p><code>fbgemm</code></p>
</blockquote>
</aside>
<p>I find the answer. In my version of PyCharm I should use <strong>fbgemm</strong> instead of <strong>x86</strong></p>
How to sort the last dimension independently of a 3D tensor?

username: weicheng.frank.lee
cooked: <p>Hi, I want to sort a 3D tensor’s last dimension. For example, a 2 * 3 * 4 tensor is as follows.<br>
a=tensor([[[1, 0, 2, 4],<br>
[3, 2, 2, 4],<br>
[0, 0, 2, 2]],<br>
[[1, 4, 0, 2],<br>
[1, 2, 4, 3],<br>
[2, 0, 4, 2]]])<br>
I want to sort the last dimension, resulting in a tensor<br>
b=tensor([[[0, 1, 2, 4],<br>
[2, 2, 3, 4],<br>
[0, 0, 2, 2]],<br>
[[0, 1, 2, 4],<br>
[1, 2, 3, 4],<br>
[0, 2, 2, 4]]])<br>
I think I can write a double for-loop to sort the last dimension independently, but the first dimension times the second dimension would actually be as large as 2048 * 12. I’m afraid it would be too slow. Is there any smarter approach to do this?</p>

username: suraj.pt
cooked: <p>This should work:<br>
<code>torch.sort(a, dim=2, stable=True)</code></p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.sort.html#torch.sort" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/generated/torch.sort.html#torch.sort</a></p>

username: weicheng.frank.lee
cooked: <p>Thank you. It works. Will it be faster than for-loops? Will setting stable sort even improve the speed?</p>

username: suraj.pt
cooked: <p>It should be faster than a double loop in python</p>
TORCH_CHECK exceptions are not printed on Android

username: Alexander_Ivkin
cooked: <p>How do I enable C++ exceptions printing in the Logcat? I only get my to crash with no exception being caught in Kotlin code or print to the logs. I believe the exception is raised from <strong>detail::torchCheckFail</strong><br>
So far, I tried to build my version of Pytorch Mobile to add some debug output, and it worked until I had to use <code>torchvision_ops</code>, which I have no idea how to build from the course code.</p>
How to configure FSDP to allow training a 13B model with 2 A100 80GB GPUs

username: 11178
cooked: <p>Hi,</p>
<p>I am trying to finetune a 13B model and fit the training process into 2 A100 80GB GPUs with FSDP, but I am constantly met with OOM errors. I am using CPU offloading, but it does not seem to address the OOM problem. Any solutions? Would FSDP enable training a very large model on limited GPU resources?</p>
<p>Thanks!</p>

username: mileseverett
cooked: <p>Also interested in this.</p>