akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataloader Data Queue is Empty Exception

Ruben_Band: <p>I am having trouble with understanding the underlying error message:</p>
<pre><code class="lang-auto">Empty                                     Traceback (most recent call last)
File c:\Users\Ruben\miniconda3\envs\CustomEnvironment\lib\site-packages\torch\utils\data\dataloader.py:1163, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
   1162 try:
-&gt; 1163     data = self._data_queue.get(timeout=timeout)
   1164     return (True, data)

File c:\Users\Ruben\miniconda3\envs\CustomEnvironment\lib\queue.py:179, in Queue.get(self, block, timeout)
    178 if remaining &lt;= 0.0:
--&gt; 179     raise Empty
    180 self.not_empty.wait(remaining)

Empty: 

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
File c:\Users\Ruben\Documents\MyProject\Manager.py:121, in Manager.train(self, from_checkpoint)
    119 def train(self, from_checkpoint: str=None):
    120     self.assertCheckpointFile(from_checkpoint)
--&gt; 121     self.trainer.fit(self.model, self.train_dl, self.val_dl, ckpt_path=from_checkpoint)

File c:\Users\Ruben\miniconda3\envs\CustomEnvironment\lib\site-packages\torch\utils\data\dataloader.py:1176, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
   1174 if len(failed_workers) &gt; 0:
   1175     pids_str = ', '.join(str(w.pid) for w in failed_workers)
-&gt; 1176     raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
   1177 if isinstance(e, queue.Empty):
   1178     return (False, None)
RuntimeError: DataLoader worker (pid(s) 20136) exited unexpectedly
</code></pre>
<p>As I’m understanding, the error tells me that one of the workers of my dataloader failed due to the Empty queue exception. How is it possible that my data queue is empty? Why is this a problem? When I’m iterating over my dataloader myself to check if the dataset is correct, everything seems to be fine:</p>
<pre><code class="lang-auto">dataloader = manager.train_dl
for i, v in enumerate(dataloader):
    print(i, v)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">0 DataBatch(x=[1636, 8], edge_index=[2, 10032], y=[32, 3], batch=[1636], ptr=[33])
1 DataBatch(x=[1596, 8], edge_index=[2, 9764], y=[32, 3], batch=[1596], ptr=[33])
2 DataBatch(x=[1597, 8], edge_index=[2, 9721], y=[32, 3], batch=[1597], ptr=[33])
...
779 DataBatch(x=[1586, 8], edge_index=[2, 9660], y=[32, 3], batch=[1586], ptr=[33])
780 DataBatch(x=[1521, 8], edge_index=[2, 9221], y=[32, 3], batch=[1521], ptr=[33])
781 DataBatch(x=[797, 8], edge_index=[2, 4873], y=[16, 3], batch=[797], ptr=[17])
</code></pre>
<p>To be clear: Iterating over the Dataloader from Pytorch works fine. Giving the iterator to the Pytorch Lightning Trainer fit method does not seem to work. Can someone help me understand how to dataloader can be empty?</p>
Cuda Error: RunTimeError: Unrecognized CachingAllocator option: 0

Tim123: <p>Hi,<br>
I have problems with cuda. I have the following versions installed:<br>
Torch version: 2.3.0+cu121<br>
Nvidia information: Driver: 538.62; CUDA Version: 12.1</p>
<p>When I run the example</p>
<p>import torch<br>
print(torch.cuda.is_available())<br>
x = torch.tensor([1.0, 2.0]).cuda()<br>
print(x)</p>
<p>there is no problem at all. But when I run my Code:<br>
if(torch.cuda.is_available()):<br>
model=Model_GRU(input_dim, output_dim,n_hidden,n_layers,<br>
dropout,isBidirectional,dense_layers).cuda()</p>
<p>I geht the following Error Message:</p>
<blockquote>
<p>RunTimeError: Unrecognized CachingAllocator option: 0</p>
</blockquote>
<p>Does anyone know this Error Message and can help?</p>
<p>Thanks a lot<br>
Tim</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the error?</p>
Specifying shape of normal sampling with partially empty keras tensor

lcs-crr: <p>Hi,<br>
I’ve been migrating my code from keras 2 using tf to keras 3 based on torch.<br>
For part of my model I need to sample a tensor of shape Z_mean, where Z_mean is a keras tensor of shape (None, 1024, 12).<br>
When I exectute my code (and the minimal working code below) I get the following error:<br>
<em>Traceback (most recent call last):</em></p>
<ul>
<li>File “/lhome/USER/.cache/JetBrains/RemoteDev/dist/8af836b5b190f_pycharm-professional-2024.1.1/plugins/python/helpers-pro/pydevd_asyncio/pydevd_asyncio_utils.py”, line 117, in _exec_async_code*</li>
<li>result = func()*</li>
<li>File “&lt;‘input’&gt;”, line 1, in *</li>
<li>File “/lhome/USER/PycharmProjects/PROJECT_NAME/venv/lib/python3.10/site-packages/keras/src/random/random.py”, line 27, in normal*</li>
<li>return backend.random.normal(*</li>
<li>File “/lhome/USER/PycharmProjects/PROJECT_NAME/venv/lib/python3.10/site-packages/keras/src/backend/torch/random.py”, line 37, in normal*</li>
<li>return torch.normal(*<br>
<em>TypeError: normal(): argument ‘size’ must be tuple of ints, not tuple</em></li>
</ul>
<p>Minimal working code:</p>
<pre><code class="lang-auto">Z_mean = keras.Input(shape=(1024, 12), batch_size=None)
eps = keras.random.normal(shape=ko.shape(Z_mean))
</code></pre>
<p>Sampling a partially None-shaped tensor worked fine in Keras 2, why am I doing wrong?</p>

ptrblck: <p>Since your question is Keras-specific, you might want to cross-post it into their discussion board.</p>
Difficult to understanding about the filters and kernels?

Guna_Sekhar_Venkata: <p>Hi, Im newbiee to deep learning. I have came across the pytorch documentation about nn.functional.conv2d. I have a doubt that suppose my image is 10x10x3. If i have taken the kernel size as 3x3 with stride as 2 means the following operatons will happens on my image as follows:<br>
A=10x10x1 * 3x3x1 (stride=2) # Feature Map 1<br>
B=10x10x2 * 3x3x2 (stride=2) <span class="hashtag-raw">#Feature</span> Map 2<br>
C=10x10x3 * 3x3x3 (stride=2) <span class="hashtag-raw">#Feature</span> Map 3</p>
<p>So the final feature map is addtion of A+B+C right. So here we are using the 3 channeled kernel because you image has three channels right. But now how can i intutively think about the filters like the above explanation…! I mean in the vgg16 architecture first block contains some thing like:</p>
<pre><code class="lang-auto">x = layers.Conv2D(
        64, (3, 3), activation="relu", padding="same", name="block1_conv1"
    )(img_input)
</code></pre>
<p>Im interested to know about what is that 64 and how can i intutively think like above for better understanding?</p>
<p>Thanks in advance</p>

Aniruth_Sundararaja1: <p>A filter is window of numbers which slides over the image and captures local features.<br>
Each filter has a different set of weights , and the stride is like by how much pixels the window moves at a time.</p>
<p>For VGG Architecture , here 64 means , the number of different filters to apply on the image , which results in 64 different feature maps.<br>
(3 , 3) is the kernel size , i.e the kernel will be a 3*3 window.</p>
<p>For intuition , think of it like , each of the 64 filters , can be used to express some characteristic pattern of the feature map , and the output is that particular feature representation.</p>
<p>Hope this clears your Doubt</p>
Anyone know how to get pytorch for pi zero 2 w

raf789: <p>Does anyone know how to get Pytorch for pi zero 2 W running Kali Linux? I tried to git-clone it before but when I tried to use it it didn’t work. I realised that Python 3.11.8 may not be supported only in Nightly versions. Just wondering if anyone knows how to download it.</p>

QFuser: <aside class="quote no-group" data-username="raf789" data-post="1" data-topic="203361">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raf789/48/70250_2.png" class="avatar"> raf789:</div>
<blockquote>
<p>pi zero 2 W</p>
</blockquote>
</aside>
<p>The Python version shouldn’t be a problem  but the amount of memory on this system seems  too little alone the frameworks are in the GB range. Besides this main branch is always a bit experimental on github I wished they made release branches (maybe there are there any haven’t found them) . The rest look at the build instructions.</p>
Rewriting the batch files into powershell

QFuser: <p>Okay I have also another topic I rewrite on the side the batchfiles which are part of pytroch helper files in powershell because:<br>
1.Microsoft want to drop the support for batchfiles(since MS-DOS) at some point in time at least they said so.<br>
2. Duplicate sections inside not sure what going on<br>
3.I can check if accepted files and environment variables are there and valid.</p>
<p>My problems is I can reproduce what those lines do :</p>
<blockquote>
<p>echo @“%TMP_DIR_WIN%\bin\randomtemp.exe” “%TMP_DIR_WIN%\bin\sccache.exe” “%CUDA_PATH%\bin\nvcc.exe” %%* &gt; “%TMP_DIR%/bin/nvcc.bat”<br>
cat %TMP_DIR%/bin/nvcc.bat<br>
set CUDA_NVCC_EXECUTABLE=%TMP_DIR%/bin/nvcc.bat<br>
for /F “usebackq delims=” %%n in (<code>cygpath -m "%CUDA_PATH%\bin\nvcc.exe"</code>) do set CMAKE_CUDA_COMPILER=%%n<br>
set CMAKE_CUDA_COMPILER_LAUNCHER=%TMP_DIR%/bin/randomtemp.exe;%TMP_DIR%\bin\sccache.exe</p>
</blockquote>
<p>I know its for the Visual Studio support but I have only a rough idea what it put out without it the Ninja build system is used but I want to reproduce what the original script does. Can someone show me how the nvcc.bat looks like when this work ?</p>
Different behavior of x * h and x.mul_(h)

Jiachen: <p>In my code, I have one operation that I need multiple x (an intermediate activation) with a scaler parameter h, so in my forward pass, I have some code like this:<br>
def forward(x):<br>
x = x * h</p>
<p>Alternatively, I tried with<br>
def forward(x):<br>
x.mul_(h)</p>
<p>So when I tested for one interation, I found both of the produce correct output and correct gradient. But if i train with them, I found the first one always better than the same one. Is there any difference between the two operations? (I used fp16, but I found the same behavior for fp32)</p>

Aniruth_Sundararaja1: <p>There is a small difference in how both of the methods work.</p>
<p>x*h does out-place multiplication , i,e it creates a new tensor which is a product of x and h.<br>
x.mul(h) is in place ,i,e it replaces the original x variable with the product.</p>
<p>So , i think x.mul(h) , as it overwrites x , it is removing the gradient history of the model .<br>
So , gradient calculation is not perfect in the second method</p>
How to get intermediate result, make some modifications to it, and then feed it back into the next layer?

gregorymjenson: <p>Hello,</p>
<p>I was using TensorFlow and Keras which has a nice and simple option to get the output of the intermediate layer and then I can feed the same data back into the next layer</p>
<p>Is there something similar available in PyTorch without having to retrain the model?</p>
<p>Kind regards,<br>
Greg</p>

Aniruth_Sundararaja1: <p>hi <a class="mention" href="/u/gregorymjenson">@gregorymjenson</a></p>
<p>Check out hooks in pytorch docs</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html</a></p>
What is the best and right way to do forward and backprop during model training and evaluation

Sajin_P: <p>1 ) model.train()<br>
output = model(cat_feat_data, num_feat_data)<br>
loss = criterion( output, target ) # Reshape for regression<br>
epoch_loss += loss.item()<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<ol start="2">
<li>
<p>model.train()<br>
optimizer.zero_grad()<br>
output = model(cat_feat_data, num_feat_data)<br>
loss = criterion( output, target ) # Reshape for regression<br>
epoch_loss += loss.item()<br>
loss.backward()<br>
optimizer.step()</p>
</li>
<li>
<p>model.train()<br>
output = model(cat_feat_data, num_feat_data)<br>
loss = criterion( output, target ) # Reshape for regression<br>
epoch_loss += loss.item()<br>
loss.backward()<br>
optimizer.step()<br>
optimizer.zero_grad(set_to_none=True)</p>
</li>
</ol>

Aniruth_Sundararaja1: <p><a class="mention" href="/u/sajin_p">@Sajin_P</a></p>
<p>The best way to do would be like this<br>
model.train()<br>
optimizer.zero_grad()<br>
output = model(cat_feat_data, num_feat_data)<br>
loss = criterion( output, target ) # Reshape for regression<br>
loss.backward()<br>
optimizer.step()</p>
<p>epoch_loss += loss.item()</p>
<p>During evaluation , we dont do back propagation , we only do forward propagation</p>
Argmax function in Multi-Class Classification

sumona: <p>Hi, can anyone review my multiclass classification GCN model below? I have a dataset of 4 classes with a total of 1800 samples. My model gives raw logits. I am not able to converge the model as validation loss stays above 1 even after 1000 epochs. I doubt if I am using the argmax function correctly as I have targets or labels in integer form e.g. 0, 1, 2, 3 NOT one-hot encoded.</p>
<pre><code class="lang-auto">loss_fn = torch.nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-3)
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.9, patience=10, verbose=True)

def train(data_loader, is_validation=False):
    model.train() if not is_validation else model.eval()
    correct = 0
    total = 0
    loss_total = 0

    for batch in data_loader:
        optimizer.zero_grad()
        pred, embedding = model(batch.x.float(), batch.edge_index.long(), batch.batch.long())  
        target = batch.y.long()

        target = target.view(-1) 

        loss = loss_fn(pred, target)

        if not is_validation:
            loss.backward()
            optimizer.step()

        predicted = torch.argmax(pred, dim=1)

        total += target.size(0)
        correct += (predicted == target).sum().item()
        loss_total += loss.item()

    accuracy = correct / total
    return loss_total / len(data_loader), accuracy

def validate(data_loader):
    return train(data_loader, is_validation=True)

print("Starting training...")
train_losses = []
train_accuracies = []
val_losses = []
val_accuracies = []

for epoch in range(1001):
    train_loss, train_accuracy = train(train_loader)
    val_loss, val_accuracy = validate(val_loader)

    train_losses.append(train_loss)
    train_accuracies.append(train_accuracy)
    val_losses.append(val_loss)
    val_accuracies.append(val_accuracy)

     if val_accuracy &gt; best_val_accuracy:
        best_val_accuracy = val_accuracy
        best_epoch = epoch
        #Saving the model weights
       torch.save(model.state_dict(), 'best_model_weights.pth')

    if epoch % 100 == 0:
        print(f"Epoch: {epoch}, Train Loss: {train_loss}, Train Accuracy: {train_accuracy}, Validation Loss: {val_loss}, Validation Accuracy: {val_accuracy}")

    scheduler.step(val_loss)
</code></pre>
<p>If you have any advice to improve my model. Please let me know. Thank you!!!</p>

ptrblck: <p>The usage of <code>predicted = torch.argmax(pred, dim=1)</code> looks correct assuming <code>pred</code> is the model output in the shape <code>[batch_size, nb_classes, *]</code>.</p>

sumona: <p>Yes, the model works as it does not throw any error. Thank you for the clarification, maybe I don’t have enough data for the learning.</p>
PyTorch Dropout Behavior - Same Seed Different Dropout Percentage

animikhaich: <p>After setting manual seed in PyTorch, if we apply dropouts in two scenarios on the same layers:</p>
<ul>
<li>Scenario 1: 10% dropout on linear layer 6 with seed 2024</li>
<li>Scenario 2: 20% dropout on the same linear layer 6 with the same seed 2024</li>
</ul>
<p>Will this always mean (deterministically) that in Scenario 2, the exact same neurons get dropped out as in Scenario 1 (for the 10%) + some additional neurons (for the remaining 10%)?</p>
<p>Or, does Scenario 2 randomly drop out neurons that may or may not be a part of the original 10% from Scenario 1?</p>

ptrblck: <p>You should assume the second behavior, i.e. no random numbers will be shared between these calls sampling different masks. You <em>could</em> see the same values but there would be no guarantee as e.g. the vectorization might differ between these calls.</p>

animikhaich: <p>Thank you. Is there any documentation that shows this? Alternatively, is there any method to experimentally verify this?</p>

KFrank: <p>Hi Animik!</p>
<p>It is plausible that your 20% dropout drops all of the same elements as your 10%<br>
dropout and then some.</p>
<aside class="quote no-group" data-username="animikhaich" data-post="3" data-topic="203328" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/animikhaich/48/53875_2.png" class="avatar"> animikhaich:</div>
<blockquote>
<p>Is there any documentation that shows this?</p>
</blockquote>
</aside>
<p>I’m not aware of any documentation that addresses this detail.</p>
<aside class="quote no-group">
<blockquote>
<p>is there any method to experimentally verify this?</p>
</blockquote>
</aside>
<p>Yes.  It is straightforward to test, and it appears to work as you suggest:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.3.0
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.ones (1000000, 1)
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt; d10 = torch.nn.Dropout1d (p = 0.10) (t)
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt; d20 = torch.nn.Dropout1d (p = 0.20) (t)
&gt;&gt;&gt;
&gt;&gt;&gt; (d20[d10 == 0.0] != 0.0).sum()   # number of elements that were dropped out in d10, but not in d20
tensor(0)
</code></pre>
<p>However, you shouldn’t rely on this behavior.  It could change based on the size<br>
of your tensor, on gpu vs. cpu, etc., and could well change in future versions.</p>
<p>Best.</p>
<p>K. Frank</p>
How to sort 2d tensors by row-wise pairs

vlacerda: <p>I am trying to sort a tensor first column-wise, preserve this sorting, and sort the second column accordingly. The tensors are always 2d, and they contain only torch.int64 elements.</p>
<p>The intuitive way to think about it is that each row represents a sample, and I want to sort these samples in ascending order, first by the first column and then by the second.</p>
<p>For example, if I have a tensor:</p>
<pre><code class="lang-auto">a = torch.tensor([ [3,0], [2,5], [2,4], [3,9] ])
</code></pre>
<p>I would like the output to be:</p>
<pre><code class="lang-auto">torch.tensor([ [2,4], [2,5], [3,0], [3,9] ])
</code></pre>
<p>I suspect the “stable” argument of the torch.sort function can help me achieve this, but I’m not sure I understand the documentation.</p>

KFrank: <p>Hi Victor!</p>
<aside class="quote no-group" data-username="vlacerda" data-post="1" data-topic="203301" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vlacerda/48/60337_2.png" class="avatar"> vlacerda:</div>
<blockquote>
<pre><code class="lang-auto">a = torch.tensor([ [3,0], [2,5], [2,4], [3,9] ])
</code></pre>
<p>I would like the output to be:</p>
<pre><code class="lang-auto">torch.tensor([ [2,4], [2,5], [3,0], [3,9] ])
</code></pre>
<p>I suspect the “stable” argument of the torch.sort function can help me achieve this</p>
</blockquote>
</aside>
<p>Yes, <code>stable</code> will be part of the approach.</p>
<p>Because you want to reorder the rows themselves, rather than the elements within<br>
a given column, you will need to index into <code>a</code> with the sorted-order indices given by<br>
<code>argsort()</code>.</p>
<p>Then, using a standard approach for sorting by a “primary” and “secondary” key,<br>
first sort by the secondary key, and then perform a stable sort with the primary key.</p>
<p>Thus:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.3.0'
&gt;&gt;&gt; a = torch.tensor([ [3,0], [2,5], [2,4], [3,9] ])
&gt;&gt;&gt; b = a[a[:, 1].argsort (dim = 0, stable = False)]
&gt;&gt;&gt; b[b[:, 0].argsort (dim = 0, stable = True)]
tensor([[2, 4],
        [2, 5],
        [3, 0],
        [3, 9]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Memory leak in Drqv2

zichunxx: <p>Hi!</p>
<p>I’m a rookie at reinforcement learning and trying to reproduce the result of <a href="https://github.com/facebookresearch/drqv2" rel="noopener nofollow ugc">Drqv2</a>.</p>
<p>However, I met the memory leak problem when training <code>drqv2</code> with 2,000,000+ training frames, which is categorized as a medium or hard task in <code>drqv2</code>.</p>
<p>After eliminating some unnecessary factors (i.e., environment and wrapper), I found that it’s the way that <code>drqv2</code> saves and loads data that causes the memory leak. <code>drqv2</code> writes replay buffer to the disk and loads it into <code>IterableDataset</code>, then samples a batch through <code>DataLoader</code>. I’m not sure if such a frequent operation by multiple workers causes the memory leak.</p>
<p>Has anyone encountered the same problem? I hope this will initiate some discussions and get some help. I’d be glad to collaborate positively.</p>
How does torch.linalg.solve work under the hood?

michaeleisel: <p>I’m trying to replicate torch.linalg.solve in a project where I’m not able to depend on libtorch, and so I’m trying to figure out exactly what it does (and the many layers of abstraction in the C++ code have been hard to navigate). Specifically, I’m interested to know what it’s doing for the CPU and/or cuSOLVER cases. Is it just calling <code>*getrf</code> to get the LU factorization of A and <code>*getrs</code> to solve? When I profile the call, I see a lot of time spent in triangular_solve_cublas (in the cuSOLVER case), which calls directly into <code>*trsm</code>, which seems unrelated. Does anyone have any guesses?</p>
Get intermediate operations for BatchNorm

sabiha_afroz: <p>I am using Fx graph to visualize the operation steps. For  “x = self.bn1(x)” operation, the Fx graph shows that this operation consists of multiple operations (below snippet). How to get those intermediate operations for BatchNorm. I tried to debug the code but failed.</p>
<pre><code class="lang-auto"># File: miniconda3/envs/pytorch/lib/python3.11/site-packages/torchvision/models/resnet.py:269, code: x = self.bn1(x)
        add: "i64[]" = torch.ops.aten.add.Tensor(primals_164, 1)
        var_mean = torch.ops.aten.var_mean.correction(convolution, [0, 2, 3], correction = 0, keepdim = True)
        getitem: "f32[1, 64, 1, 1]" = var_mean[0]
        getitem_1: "f32[1, 64, 1, 1]" = var_mean[1];  var_mean = None
        add_1: "f32[1, 64, 1, 1]" = torch.ops.aten.add.Tensor(getitem, 1e-05)
        rsqrt: "f32[1, 64, 1, 1]" = torch.ops.aten.rsqrt.default(add_1);  add_1 = None
        sub: "f32[1, 64, 112, 112]" = torch.ops.aten.sub.Tensor(convolution, getitem_1)
        mul: "f32[1, 64, 112, 112]" = torch.ops.aten.mul.Tensor(sub, rsqrt);  sub = None
        squeeze: "f32[64]" = torch.ops.aten.squeeze.dims(getitem_1, [0, 2, 3]);  getitem_1 = None
        squeeze_1: "f32[64]" = torch.ops.aten.squeeze.dims(rsqrt, [0, 2, 3]);  rsqrt = None
        mul_1: "f32[64]" = torch.ops.aten.mul.Tensor(squeeze, 0.1)
        mul_2: "f32[64]" = torch.ops.aten.mul.Tensor(primals_162, 0.9)
        add_2: "f32[64]" = torch.ops.aten.add.Tensor(mul_1, mul_2);  mul_1 = mul_2 = None
        squeeze_2: "f32[64]" = torch.ops.aten.squeeze.dims(getitem, [0, 2, 3]);  getitem = None
        mul_3: "f32[64]" = torch.ops.aten.mul.Tensor(squeeze_2, 1.0000797257434426);  squeeze_2 = None
        mul_4: "f32[64]" = torch.ops.aten.mul.Tensor(mul_3, 0.1);  mul_3 = None
        mul_5: "f32[64]" = torch.ops.aten.mul.Tensor(primals_163, 0.9)
        add_3: "f32[64]" = torch.ops.aten.add.Tensor(mul_4, mul_5);  mul_4 = mul_5 = None
        unsqueeze: "f32[64, 1]" = torch.ops.aten.unsqueeze.default(primals_2, -1)
        unsqueeze_1: "f32[64, 1, 1]" = torch.ops.aten.unsqueeze.default(unsqueeze, -1);  unsqueeze = None
        mul_6: "f32[1, 64, 112, 112]" = torch.ops.aten.mul.Tensor(mul, unsqueeze_1);  mul = unsqueeze_1 = None
        unsqueeze_2: "f32[64, 1]" = torch.ops.aten.unsqueeze.default(primals_3, -1);  primals_3 = None
        unsqueeze_3: "f32[64, 1, 1]" = torch.ops.aten.unsqueeze.default(unsqueeze_2, -1);  unsqueeze_2 = None
        add_4: "f32[1, 64, 112, 112]" = torch.ops.aten.add.Tensor(mul_6, unsqueeze_3);  mul_6 = unsqueeze_3 = None
</code></pre>
Accuracy Drop During Calibration (before Conversion) in Post Training Quantization

bcxiao: <p>I am new to quantization and am following this <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">official tutorial</a>, I’ve noticed that the accuracy after calibration before converting (using the official tutorial code below) is lower than the accuracy of the original <code>float_model</code> . Here is the relevant part of the code:</p>
<pre><code class="lang-auto">myModel.qconfig = torch.ao.quantization.default_qconfig
print(myModel.qconfig)
torch.ao.quantization.prepare(myModel, inplace=True)

# Calibrate first
print('Post Training Quantization Prepare: Inserting Observers')
print('\n Inverted Residual Block:After observer insertion \n\n', myModel.features[1].conv)

# Calibrate with the training set
top1, top5 = evaluate(myModel, criterion, data_loader, neval_batches=num_calibration_batches)
print('Post Training Quantization: Calibration done')
</code></pre>
<p>The <code>top1</code> and <code>top5</code> returned from calibration are worse than original float32 model’s <code>top1</code> and <code>top5</code> on the same dataset (the line of code below)</p>
<pre><code class="lang-auto">top1, top5 = evaluate(float_model, criterion, data_loader, neval_batches= num_calibration_batches)
</code></pre>
<p>From my understanding, calibration should only be gathering statistics like the range of activations and shouldn’t alter the network’s computations at this stage. I thought any computational changes would only occur after calling <code>torch.ao.quantization.convert</code> . Could someone clarify if a decrease in accuracy during calibration is expected?</p>
<p>Thank you for your insights!</p>

jcaip: <p>Hi <a class="mention" href="/u/bcxiao">@bcxiao</a>,</p>
<p>Are you comparing the top1 and top5 accuracy to that produced by this line in the tutorial?</p>
<blockquote>
<p>top1, top5 = evaluate(float_model, criterion, data_loader_test, neval_batches=num_eval_batches)</p>
</blockquote>
<p>It is expected for the result you gave to be different, not because of differences to the model, but because num_eval_batches != num_calibration_batches, since we don’t need to calibrate on all of the evaluation data. This is because you are now only doing evaluation on a subset of the data, so it is not a fair comparison of these two metrics.</p>
<p>I would expect them to be the same, or at least very similar, if you changed num_calibration_batches to be num_eval_batches though. Can you try making that change and seeing what you get?</p>
How do I hook into the backward computional graph?

NightMachinery: <p>Suppose I have two intermediate nodes <code>o_1 = f(a, ...)</code> and <code>o_2 = g(a, ...)</code>. Both <code>o_1</code> and <code>o_2</code> contribute to the final loss. I want to get the gradient flowing to <code>a</code> from <code>o_1</code> only, and not from <code>o_2</code>. PyTorch:</p>
<p>Suppose I have two intermediate nodes <code>o_1 = f(a, ...)</code> and <code>o_2 = g(a, ...)</code>. Both <code>o_1</code> and <code>o_2</code> contribute to the final loss. I want to get the gradient flowing to <code>a</code> from <code>o_1</code> only, and not from <code>o_2</code>. This is the part of the gradient of the loss that flows to <code>a</code> through the node <code>o_1</code>.</p>
<p>To reiterate what I want for more clarity:<br>
When computing gradients, a single node A can have multiple paths to the final gradient target (the loss). The upstream gradients reach A and are accumulated by summation in A to form its final gradient value. I want to hook into this accumulation process and store each upstream gradient that comes. I.e., I want to store the upstream gradient from the different edges flowing from A separately, instead of having the summed.</p>
<p>The only way I can think of is to detach the gradient of <code>o_2</code>, but I actually need to get the gradients of a lot of these edges in the computational graph, and detaching <code>o_2</code> will make the downstream edges incorrect.</p>
<p>The only way I can think of is to detach the gradient of <code>o_2</code>, but I actually need to get the gradients of a lot of these edges in the computational graph, and detaching <code>o_2</code> will make the downstream edges incorrect.</p>

ptrblck: <p>Using <code>torch.autograd.grad</code> should work as it would allow you to compute the gradients w.r.t. a specific variable:</p>
<pre data-code-wrap="python"><code class="lang-python">f = nn.Linear(1, 1)
g = nn.Linear(1, 1)

a = torch.randn(1, 1, requires_grad=True)

o_1 = f(a)
o_2 = g(a)

g_1 = torch.autograd.grad(outputs=o_1, inputs=a, retain_graph=True)
g_2 = torch.autograd.grad(outputs=o_2, inputs=a)

o_1 = f(a)
o_2 = g(a)
(o_1 + o_2).backward()
print(a.grad)
# tensor([[0.6055]])
print(g_1[0] + g_2[0])
# tensor([[0.6055]])
</code></pre>

NightMachinery: <p>Thanks! I actually thought of a solution that works:</p>
<pre><code class="lang-auto">import torch
from icecream import ic


v = torch.tensor([1., 1., 1.], requires_grad=True)

v_dummy1 = v.view(*v.shape)
v_dummy1.retain_grad()

out = v_dummy1 + v**2
out.backward(torch.tensor([1., 2., 3.]))
ic(v_dummy1.grad, v.grad)
</code></pre>
<pre><code class="lang-auto">v_dummy1.grad: tensor([1., 2., 3.]), v.grad: tensor([3., 6., 9.])
</code></pre>
<p>I don’t want to use <code>autograd.grad</code>, as it mixes the backward computation into the forward definition of the model. I also suspect it’s less efficient when called multiple times.</p>
Mixing C++ part and Python inside PyTorch

Woosung: <p>Hello,</p>
<p>It might be more of a general question about programming, not specifically about PyTorch.</p>
<p>I noticed some functionalities, like CUDA-related parts, are coded with C++ on c10.<br>
But since most of the parts in PyTorch are Python-based, how can they interact with each other?</p>
<p>More specifically, I want to understand how PyTorch manages the memory in the GPU.<br>
I want to find out how much memory they try to allocate for a single convolutional layer or something like that.<br>
So, I looked into the code CUDACachingAllocator.cpp in c10, but I could not figure out which part of the code calls such functions.<br>
Moreover, I don’t know which function I should search for.<br>
Do you guys have any tips on this?</p>
<p>thx</p>

ptrblck: <aside class="quote no-group" data-username="Woosung" data-post="1" data-topic="203273">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/bb73d2/48.png" class="avatar"> Woosung:</div>
<blockquote>
<p>But since most of the parts in PyTorch are Python-based, how can they interact with each other?</p>
</blockquote>
</aside>
<p><a href="https://github.com/pybind/pybind11"><code>pybind11</code></a> is used to allow Python to C++ calls.</p>
<aside class="quote no-group" data-username="Woosung" data-post="1" data-topic="203273">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/bb73d2/48.png" class="avatar"> Woosung:</div>
<blockquote>
<p>More specifically, I want to understand how PyTorch manages the memory in the GPU.</p>
</blockquote>
</aside>
<p>PyTorch uses a caching allocator by default and <code>CUDACachingAllocator.cpp</code> is the right file to read more about its usage. Search for <a href="https://github.com/pytorch/pytorch/blob/1b29c16e5e143eb49a902d384b3d8a47dbe62a31/c10/cuda/CUDACachingAllocator.cpp#L1009"><code>malloc</code></a> to see how the memory pools are used.</p>
Training via FSDP - loss calculation

rondoll: <p>I’m using this tutorial: <a href="https://pytorch.org/tutorials/intermediate/FSDP_adavnced_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Advanced Model Training with Fully Sharded Data Parallel (FSDP) — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>
<p>Looking at the train function</p>
<pre><code class="lang-auto">def train(args, model, rank, world_size, train_loader, optimizer, epoch, sampler=None):
    model.train()
    local_rank = int(os.environ['LOCAL_RANK'])
    fsdp_loss = torch.zeros(2).to(local_rank)

    if sampler:
        sampler.set_epoch(epoch)
    if rank==0:
        inner_pbar = tqdm.tqdm(
            range(len(train_loader)), colour="blue", desc="r0 Training Epoch"
        )
    for batch in train_loader:
        for key in batch.keys():
            batch[key] = batch[key].to(local_rank)
        optimizer.zero_grad()
        output = model(input_ids=batch["source_ids"],attention_mask=batch["source_mask"],labels=batch["target_ids"] )
        loss = output["loss"]
        loss.backward()
        optimizer.step()
        fsdp_loss[0] += loss.item()
        fsdp_loss[1] += len(batch)
        if rank==0:
            inner_pbar.update(1)

    dist.all_reduce(fsdp_loss, op=dist.ReduceOp.SUM)
    train_accuracy = fsdp_loss[0] / fsdp_loss[1]


    if rank == 0:
        inner_pbar.close()
        print(
                f"Train Epoch: \t{epoch}, Loss: \t{train_accuracy:.4f}"
            )
    return train_accuracy
</code></pre>
<p>The loss is calculated via <code>fsdp_loss[0] / fsdp_loss[1]</code> where <code>fsdp_loss[0]</code> is the summed loss and <code>fsdp_loss[1]</code> is the summed <code>len(batch)</code>, however a couple of things don’t make sense to me</p>
<ol>
<li>
<p>In the example the tutorial gives, batch is a dictionary of length 4, so <code>fsdp_loss[1] += len(batch)</code> would just increment by 4 every time. Is that a mistake?</p>
</li>
<li>
<p>In the example the tutorial gives, the model is a hf transformer’s t5. Looking at the code, loss is calculated via <code>torch.nn.CrossEntropyLoss</code> defined here: <a href="https://github.com/huggingface/transformers/blob/v4.41.1/src/transformers/models/t5/modeling_t5.py#L1772" class="inline-onebox" rel="noopener nofollow ugc">transformers/src/transformers/models/t5/modeling_t5.py at v4.41.1 · huggingface/transformers · GitHub</a><br>
And since no reduction was specified, ‘mean’ is chosen by default, which averages the loss across the batch anyway, right?</p>
</li>
</ol>
<p>So instead of <code>fsdp_loss[1] += len(batch)</code> should it be <code>fsdp_loss[1] += 1</code>?</p>
How to compute/plot magnitude of gradients of each layer

DeepLearner17: <p>Hello,</p>
<p>I would like to know if there is a straightforward way (less memory consumption) to compute the magnitude of gradients of each layer at every epoch and plot them with tensorboard ?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/deeplearner17">@DeepLearner17</a>,</p>
<p>You can compute the norm efficiently via a view like so,</p>
<pre><code class="lang-auto">norms = {}
for name, param in model.named_parameters():
  norm[name] = param.grad.view(-1).abs().pow(2).sum(-1).sqrt()
</code></pre>
<p>Then you can pass these to <code>Tensorboard</code> like normal.</p>
<p>EDIT: Added <code>.grad</code> attribute to compute Euclidean norm of gradient.</p>

DeepLearner17: <p>Thank you <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>.<br>
It’s not the weights of the layers but the gradients associated to each weight.</p>
<p><code>norms[name] = param.grad.abs().view(-1).pow(2).sum(-1).sqrt()</code></p>