akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model Agnostic Approach to create a 2nd output head?

sad_robot: <p>I’ve run into a strange problem where I need to take an arbitrary existing pytorch model and add a 2nd output head of the same kind as the original output head then have the model.forward() produce outputs from both output heads (e.g. head1_output, head2_output = model.forward(X)).<br>
Alternatively it would also work if there was a way to increase the output channel/feature dimension of an existing arbitrary model.</p>
<p>Model Agnostic Constraints:</p>
<ul>
<li>We cannot assume the outputs have a certain size which is consistent across inputs.</li>
<li>We also can’t assume particular model architecture (e.g. transformer vs CNN, vs fully-connected model).</li>
</ul>
<p>Any ideas?</p>

sad_robot: <p>Hey thanks, the tricky part is that this might not work <code>features = self.original_model(x)</code>. The original model probably evaluates the entire model back-bone and head and produces the output of the original head rather than a hidden representation.</p>
<p>So if we clone the original head it might not be compatible with the model’s outputs (e.g. Conv layer that has 64 input channels but 1 output channel cannot be re-applied to its own output).</p>

ptrblck: <p>Ignore the previous response as it’s a chatbot generated one and usually quite unhelpful.</p>
Is torch2.4.1 compatible with CUDA12.5

Pied-Piper1: <p>I locally installed my <strong>CUDA Toolkit12.5</strong>. And everything went well (I can successfully call <code>torch.cuda.is_available()</code> and returns <code>ture</code>) before I install torchvision with cmd <code>conda install torchvision -c pytorch</code> . After installing torchvision, I even couldn’d import torch with error <code>OSError: [WinError 127] Error loading "D:\anaconda3\envs\pytorch2.4.1\Lib\site-packages\torch\lib\torch_cuda.dll" or one of its dependencies.</code>  And <code>torch              2.4.1+cu124</code>,<code>torchvision        0.19.1+cu124</code> are results of<code>pip list | findstr torch</code>. How can I do to tackle with this issue?</p>

system: 

system: 

ptrblck: <p>Check which dependency is missing and/or try to reinstall <code>torchvision</code>.<br>
I would not recommend creating other fake accounts to bump your topic.</p>
Computing averages

KukumavMozolo: <p>Hi there,<br>
if have a data matrix x of dim n x d, where n is the batch dimension and d is the dimensionality of my data and a matrix m in d x m.<br>
I want to use m to compute averages over certain dimensions: e.g. x @ m  =&gt; in n x m.<br>
Therefor m[i,j] = 1.0/i_z for j in [idx_i_1,idx_i_2,…,idx_i_z], 0 else<br>
As one can see the internal dimension of m is actually d+1, where the parameter 1.0/i_z repeats i_z times in dimension i and the matrix is 0 everywhere else.<br>
Is there a clever way to produce such a matrix using views?</p>

ptrblck: <p>Could you post a (slow) reference code using nested loops, please?</p>

KukumavMozolo: <p>Gladly:</p>
<pre data-code-wrap="python"><code class="lang-python">    #nested loops
    indices = [[0,1],[2]]
    x = torch.tensor([[1,2,3],[4,5,6],[7,8,9]])
    res = torch.zeros((3,2))
    for i in range(3):
        for j in range(2):
            res[i,j] = torch.sum(x[i,indices[j]]) / len(indices[j])
    print(res)

    #with matrix as previously described
    avg_mat = torch.zeros((2,3))
    counter = 0
    for i, idx in enumerate(indices):
        avg_mat[i,idx] = 1.0/len(idx)
    print(x.to(torch.float) @ avg_mat.T)
    print(avg_mat.T)
</code></pre>
<p>—&gt;</p>
<pre><code class="lang-auto">tensor([[1.5000, 3.0000],
        [4.5000, 6.0000],
        [7.5000, 9.0000]])
tensor([[1.5000, 3.0000],
        [4.5000, 6.0000],
        [7.5000, 9.0000]])
tensor([[0.5000, 0.0000],
        [0.5000, 0.0000],
        [0.0000, 1.0000]])
</code></pre>

KukumavMozolo: <p>I found out about torch.scatter, maybe thats a good solution to my troubles…</p>

ptrblck: <p>Yes, <code>.scatter_reduce_</code> should work, but you would need to modify your index and check the count:</p>
<pre data-code-wrap="python"><code class="lang-python">idx = torch.tensor(sum([[i]*len(a) for i, a in enumerate(indices)], []))
_, count = idx.unique(return_counts=True)
idx = idx.unsqueeze(0).expand(3, 3)

out = torch.zeros(idx.size(0), idx.max()+1, dtype=x.dtype)
out.scatter_reduce_(1, idx, x, reduce="sum")
out = out / count

print(out)
# tensor([[1.5000, 3.0000],
#         [4.5000, 6.0000],
#         [7.5000, 9.0000]])
</code></pre>

KukumavMozolo: <p>thank you very much!</p>
[CPU] Train network using float 16?

Aknw_Fen: <p>I’m considering ways to improve speed and memory footprint of models and data.</p>
<p>Reading SO, this forum and docs, it’s still unclear to me whether it’s possible to use either float16 or bfloat16 and whether I can set this somewhere as a top level parameter.</p>
<p>What I have found is the setting:</p>
<pre data-code-wrap="python"><code class="lang-python">dtype = torch.float16
torch.set_default_dtype(dtype)
</code></pre>
<p>but this returns errors due to un-supported ops. (i’m using it with a standard neural network.)</p>
<p>Could anyone offer some hints?</p>
<p>Running <code>lscpu | grep dtype</code> shows apparently support for f16 (not bf16.)</p>

ptrblck: <p>CPU workloads should support <code>bfloat16</code> in <code>autocast</code> as described in the <a href="https://pytorch.org/docs/stable/amp.html#automatic-mixed-precision-package-torch-amp">docs</a>:</p>
<blockquote>
<p>As shown in the CPU example section of <a href="https://pytorch.org/docs/stable/amp.html#torch.autocast"><code>torch.autocast</code></a>, “automatic mixed precision training/inference” on CPU with datatype of <code>torch.bfloat16</code> only uses <a href="https://pytorch.org/docs/stable/amp.html#torch.autocast"><code>torch.autocast</code></a>.</p>
</blockquote>
<p>I don’t know what the status of <code>float16</code> support on CPU is and if it’s planned.</p>
What is the use of tensor.share_memory_()?

dcusmeb: <p>I have two codes</p>
<pre><code class="lang-auto">import torch
import multiprocessing

def worker(tensor):
    tensor += 1
    print(f"Worker tensor: {tensor}")

if __name__ == '__main__':
    tensor = torch.zeros(3)
    p = multiprocessing.Process(target=worker, args=(tensor,))
    p.start()
    p.join()
    print(f"Main tensor: {tensor}")
</code></pre>
<pre><code class="lang-auto">import torch
import multiprocessing

def worker(tensor):
    tensor += 1
    print(f"Worker tensor: {tensor}")

if __name__ == '__main__':
    tensor = torch.zeros(3)
    tensor.share_memory_()
    p = multiprocessing.Process(target=worker, args=(tensor,))
    p.start()
    p.join()
    print(f"Main tensor: {tensor}")
</code></pre>
<p>Only difference between two is the use of tensor.share_memory_(). My output is same in both cases</p>
<pre><code class="lang-auto">Worker tensor: tensor([1., 1., 1.])
Main tensor: tensor([1., 1., 1.])
</code></pre>
<p>Can anyone explain if tensors are shared directly, why do we even need shared memory?</p>
Problem with fork-like multiprocess Dataloader on Ubuntu

neyronon: <p>A week ago after tons of work on code base I noticed a segmentation fault problem right before starting the main loop of training process.<br>
I found that training interrupts on fetching a batch from train dataloader. Interenstingly that some timm models like ese_vovnet and mobilenetv3_large_075 doesn’t work absolutely but some models like mobilenet_large_100 sometime work especially if I use EMA but not every time. If the first epoch can start, then the training process lasts til the end (many tens epochs) without any problems every time. I assume that there occurs a deadlock during processes creating.<br>
If I place a code <code>multiprocessing.set_start_method('spawn')</code> in <code>__main__</code> all problems dissapear except one: training time rises three times becauses time that is taken to create every worker increases and I can see from profiling that <code>_io.BufferedWriter.write()</code> takes much time. In total a time to enable multiprocessing (<code>_io.BufferedWriter.write(), select.pool.pool(), _thread.lock.acquire()</code>) three-four times exceeds a time taken by whole training stuff (mainly <code>imread(), torch.conv2d(), run_backward()</code>).</p>
<p>So I would like to use ‘fork’ method but I dont know what to start from. I don’t use homemade multiprocessing in my code. Only framework Dataloaders and some functions like pandarallel use multiprocessing.<br>
I have some experience in C++ threads but I’ve never worked with multiprocessing in Python. How to tackle such kind of problem?</p>
Understanding channel dim in audio processing

axion66: <p>Hi!</p>
<p>I’m working on generative audio model, and I want to what’s the most common way of selecting channel dimension. I saw some people after doing torch.stft or spectrogram, add additional dimension to have (batch,1,freq_bins,length) and work on additional channel. But I felt like it’s better to use freq_bins as channel dimension for more Transformer-like style. (batch, length,freq_bins) → (batch,length,latent channel)<br>
Any thought on this?</p>
<p>Thank you,</p>

shivammehta007: <p>Hello,<br>
It is definitely recommended to use the freq_bin dimensions. All you have to do is change Conv2D operations to Conv1D operations and you should be good to go! It is more elegant and fast while reducing the number of parameters compared to 2d operations on the additional dimension.</p>
<p>hope this helps!</p>
Tensor parallelism simple Embedding Example

caglar_demir: <p>Hi,<br>
I was doing through <a href="https://pytorch.org/docs/stable/distributed.tensor.parallel.html" rel="noopener nofollow ugc">the examples of tensor parallel</a>. After spending few hours, I still don’t get my head around.<br>
Let’s assume that we have the following class  and we have 2 GPUs.<br>
How can we use ColwiseParallel tensor paralelisim to store the first half of the entity_embeddings and of the relation_embeddings in the first GPU and other halfs into second GPU.</p>
<pre data-code-wrap="python"><code class="lang-python">
class DistMult(nn.Module):
    def __init__(self):
        super().__init__(args)
        self.entity_embeddings = torch.nn.Embedding(135, 32)
        self.relation_embeddings = torch.nn.Embedding(46, 32)
    def forward(self,h,r,t):
    h=self.entity_embeddings(h)
    r=self.relation_embeddings(r)
    t=self.entity_embeddings(t)
    return ((h * r) * t).sum(dim=1)
</code></pre>
<p>Constantly, the following RuntimeError occurs<br>
RuntimeError: Function EmbeddingBackward0 returned an invalid gradient at index 0 - got [135, 32] but expected shape compatible with [46, 32]</p>

tianyu: <p>Hi. Can you provide the end-to-end code so that we can help? In your code I didn’t see TP being applied.</p>
<p>Btw, you may find the tutorial helpful<br>
<a href="https://pytorch.org/tutorials/intermediate/TP_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/TP_tutorial.html</a></p>

caglar_demir: <p>Hi <a class="mention" href="/u/tianyu">@tianyu</a>,</p>
<p>Thank you for your response. Thank you for the link. I have went through it already.<br>
Yet, I couldn’t really make use of it</p>
<p>Here is an end-to-end code example.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
class DistMult(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.entity_embeddings = torch.nn.Embedding(135, 32)
        self.relation_embeddings = torch.nn.Embedding(46, 32)
    def forward(self,h,r,t):

        h=self.entity_embeddings(h)
        r=self.relation_embeddings(r)
        t=self.entity_embeddings(t)
        return ((h * r) * t).sum(dim=1)

model=DistMult()
optim=torch.optim.Adam(model.parameters(), lr=1e-3)
triples=torch.LongTensor([[0,0,0],
                          [1,1,1],
                          [2,2,2]])
for i in range(100):
    yhat=model.forward(triples[:,0],triples[:,1],triples[:,2])
    loss=torch.nn.functional.binary_cross_entropy_with_logits(yhat,torch.ones_like(yhat))
    loss.backward()
    print(loss)
    optim.step()
    optim.zero_grad()
</code></pre>
<p>The goal is to apply columnwise TP so that<br>
the first 16 columns of entity_embeddings and relation_embeddings are kept on the first GPU and the last 16 columns of entity_embeddings and relation_embeddings are on the second GPU.</p>

tianyu: <p>In your code, I didn’t see TP being applied. You need to call <code>parallelize_module</code> explicitly with a plan (e.g. <code>"tok_embeddings": RowwiseParallel()</code>) to parallelize the module before usage.</p>
Output different with and without FSDP

Abhiraj_Kanse: <p>Hi i tried running a small model with and without FSDP to check if i get the same output when identical input is provided. Below is my code for the same , basically the assert fails when running the code.</p>
<blockquote>
<p>import os<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>import torch.distributed as dist<br>
import torch.multiprocessing as mp<br>
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP</p>
<p>def setup(rank, world_size):<br>
os.environ[‘MASTER_ADDR’] = ‘localhost’<br>
os.environ[‘MASTER_PORT’] = ‘12355’</p>
<pre><code># initialize the process group
dist.init_process_group("nccl", rank=rank, world_size=world_size)
</code></pre>
<p>def cleanup():<br>
dist.destroy_process_group()</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1, 32, 3, 1)<br>
self.conv2 = nn.Conv2d(32, 64, 3, 1)<br>
self.dropout1 = nn.Dropout(0.25)<br>
self.dropout2 = nn.Dropout(0.5)<br>
self.fc1 = nn.Linear(9216, 128)<br>
self.fc2 = nn.Linear(128, 10)</p>
<pre><code>def forward(self, x):

    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = F.max_pool2d(x, 2)
    x = self.dropout1(x)
    x = torch.flatten(x, 1)
    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout2(x)
    x = self.fc2(x)
    output = F.log_softmax(x, dim=1)
    return output
</code></pre>
<p>def fsdp_main(rank, world_size):</p>
<pre><code>setup(rank, world_size)

model = Net()
model.eval()
inp = torch.rand(4,1,28,28)
out_og = model(inp)

fsdp_model = FSDP(model, device_id=rank)
fsdp_model.eval()
inp_cuda = inp.to(rank)
out = fsdp_model(inp_cuda)

assert torch.max(torch.abs(out_og - out.cpu())) &lt; 1e-6, f"parallelized and original model have different outputs max diff {torch.max(torch.abs(out_og - out.cpu()))}"

print("ALL Done")
cleanup()
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
WORLD_SIZE = torch.cuda.device_count()<br>
mp.spawn(fsdp_main,<br>
args=(WORLD_SIZE,),<br>
nprocs=WORLD_SIZE,<br>
join=True)</p>
</blockquote>
<p>I was not expecting this behavior and expected the outputs to be same. I don’t understand why the output is different here?</p>

tianyu: <p>You have to guarantee the initialization of model parameters and inputs to be the same, e.g. using the same random seeds on all ranks, to achieve identical outcome.</p>
Computing mIoU during validation

jawi289p: <p>I am working on a binary segmentation task and have implemented the following training and validation loop. I need help with two points:</p>
<ol>
<li>How can I compute the IoU for each class after every epoch and print the Class 1 IoU, Class 2 IoU, and the overall mIoU score?</li>
<li>Is it better to save the model based on the best mIoU score or the lowest validation loss?</li>
</ol>
<p>Any guidance would be greatly appreciated. Thanks!</p>
<p>Here is my code:</p>
<pre><code class="lang-auto"># Initialize lists to store loss values
train_losses = []
val_losses = []

# Training and validation loop
for epoch in range(n_eps):
    model.train()
    train_loss = 0.0

    # Training loop
    for images, masks in tqdm(train_loader):
        images, masks = images.to(device), masks.to(device)
        
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, masks)
        loss.backward()
        optimizer.step()
        
        train_loss += loss.item()

    avg_train_loss = train_loss / len(train_loader)
    train_losses.append(avg_train_loss)
    print(f"Epoch [{epoch+1}/{n_eps}], Train Loss: {avg_train_loss:.4f}")

    model.eval()
    val_loss = 0.0

    # Validation loop
    with torch.no_grad():
        for images, masks in val_loader:
            images, masks = images.to(device), masks.to(device)
            outputs = model(images)
            val_loss += criterion(outputs, masks).item()

    avg_val_loss = val_loss / len(val_loader)
    val_losses.append(avg_val_loss)
    print(f"Epoch [{epoch+1}/{n_eps}], Val Loss: {avg_val_loss:.4f}")
</code></pre>

Eduardo_Lawson: <p>I like to use <a href="https://lightning.ai/docs/torchmetrics/stable/segmentation/mean_iou.html" rel="noopener nofollow ugc">torchmetrics miou</a>. You just need to pass your target and the models prediction, I think this function its a good choice for your problem.</p>
<p>Answering the second question, it depends of what you are looking for. If you are just training and want to use the model to test i suggest to take the highest miou, but if you are training for using it as a pretrained model maybe taking the loss will be better.</p>

jawi289p: <p><a class="mention" href="/u/eduardo_lawson">@Eduardo_Lawson</a> thank you for response. I did try with torchmetrics but the problem is that it is not giving individual class scores for the binary segmentation.</p>
DDP leads to Out of Memory error

enterthevoidf22: <p>I’m encountering an issue while running a distributed PyTorch script for model evaluation and metrics reporting in pycharm debugger. The script uses DistributedDataParallel (DDP) for multi-GPU evaluation, and I suspect the issue lies in distributed initialization, rank assignments, or dataset loading.</p>
<p>The command I use to run the script is:</p>
<pre><code class="lang-auto">/home/user/envs/my_project_env/bin/python -m torch.distributed.run \
    --nproc_per_node=8 \
    --master_port=12346 \
    /specific/a/home/users/username/.pycharm_helpers/pydev/pydevd.py \
    --multiprocess \
    --qt-support=auto \
    --client localhost \
    --port 34779 \
    --file /home/user/projects/my_project/eval/evaluate.py \
    --model_path /home/user/models/checkpoints/2024_11_21_model \
    --data_path /home/user/data/dataset \
    --batch_size 4

</code></pre>
<p>here is the broad overview of my code:</p>
<pre><code class="lang-auto">import argparse
import torch
import os
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from expir.utils import disable_torch_init
from expir.model.builder import load_custom_model
from expir.utils.collators import DataCollatorForSupervisedDataset
from expir.dataset import CustomDataset
from expir.eval import evaluate_dataset

def evaluate_model(args):
    if 'LOCAL_RANK' in os.environ:
        args.local_rank = int(os.environ['LOCAL_RANK'])
    else:
        args.local_rank = -1

    if args.local_rank != -1:
        torch.cuda.set_device(args.local_rank)
        device = torch.device(f'cuda:{args.local_rank}')
    else:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    if args.local_rank != -1:
        dist.init_process_group(backend='nccl')
        args.distributed = True
    else:
        args.distributed = False

    model_path = os.path.expanduser(args.model_path)
    model = load_custom_model(model_path).to(device)
    if args.distributed:
        model = DDP(model, device_ids=[args.local_rank], output_device=args.local_rank)

    collator = DataCollatorForSupervisedDataset()
    dataset = CustomDataset(args.data_path)
    evaluate_dataset(model, dataset, collator, device, args)

    if args.distributed:
        dist.destroy_process_group()

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_path", type=str, required=True, help="Path to the pre-trained model.")
    parser.add_argument("--data_path", type=str, required=True, help="Path to the dataset directory.")
    parser.add_argument("--batch_size", type=int, default=4, help="Batch size for evaluation.")
    parser.add_argument("--local_rank", type=int, default=-1, help="Local rank for distributed training.")
    args = parser.parse_args()

    evaluate_model(args)

</code></pre>
<p>the strange behaviour that i see is that befor calling DDP i already see the model on all gpus. i’m not able to make sense of the example</p>
Rand perm vectorized version

whoab: <p>Let’s say I want to create an (3, 2, 100) tensor where the last dimension is a randomized 100D permutation tensors.</p>
<p>torch.randperm only creates a SINGLE permutation tensor. In order to make a (3, 2, 100) tensor, I would have to create a for loop over dimension 3 and 2. How do I do this in an efficient vectorized way, instead of a very slow for loop?</p>

KFrank: <p>Hi Whoab!</p>
<aside class="quote no-group" data-username="whoab" data-post="1" data-topic="118508" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/4bbf92/48.png" class="avatar"> whoab:</div>
<blockquote>
<p>How do I do this in an efficient vectorized way, instead of a very slow for loop?</p>
</blockquote>
</aside>
<p>First, as a practical matter, looping over a small number of items (in your<br>
example, <code>3 * 2</code>) won’t be very expensive.  If the size of the permutation<br>
(<code>100</code> in your example) is relatively large (which probably means rather<br>
larger than the <code>100</code> in your example), the cost of the permutation will<br>
dwarf the cost of the loop.</p>
<p>So, as a practical matter, I wouldn’t worry about it (unless you can show<br>
with actual timings that it matters).</p>
<p>Having said that, here is a “vectorized” method that introduces a<br>
different source of computational inefficiency.</p>
<p>Note the <code>randperm()</code> <em>should</em> have a cost of O (n), where n is the<br>
length of the permutation.  My sorting approach (below) has a cost<br>
of O (n log (n)) (because of the <code>sort()</code>).  So for a big loop, but a<br>
smallish <code>n</code>, it should run faster, but for large enough <code>n</code>, it will run<br>
more slowly.</p>
<p>Note that using <code>randint()</code> has a (very low) chance of generating<br>
duplicates that would (very slightly) bias the distribution of the<br>
random permutations.  <code>multinomial()</code> (with its default of<br>
<code>replacement = False</code>) will not generate duplicates, but is probably<br>
more expensive.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; dim1 = 2
&gt;&gt;&gt; dim2 = 2
&gt;&gt;&gt; nPerm = 5
&gt;&gt;&gt; torch.randint (torch.iinfo (torch.int64).max, (dim1, dim2, nPerm)).argsort()
tensor([[[0, 2, 1, 3, 4],
         [0, 3, 1, 2, 4]],

        [[1, 4, 0, 3, 2],
         [0, 3, 1, 4, 2]]])
&gt;&gt;&gt; torch.multinomial (torch.ones (dim1 * dim2 * nPerm), dim1 * dim2 * nPerm).reshape ((dim1, dim2, nPerm)).argsort()
tensor([[[0, 3, 1, 2, 4],
         [0, 4, 1, 2, 3]],

        [[4, 1, 2, 3, 0],
         [2, 4, 0, 3, 1]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Space4444: <p>Cool, it works faster than torch.randperm placed inside torch.vmap</p>
DDP device hanging before running torch.dist.all_reduce()

jacksettles: <p>I am training a (relatively) small transformer language model that is about 1.15B params. I am running this training job on my university HPC which uses SLURM. I have tried to adapt my code to the DDP tutorials provided on YouTube here: <a href="https://www.youtube.com/watch?v=-K3bZYHYHEA&amp;list=PL_lsbAsL_o2CSuhUhJIiW0IkdT5C2wGWj" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=-K3bZYHYHEA&amp;list=PL_lsbAsL_o2CSuhUhJIiW0IkdT5C2wGWj</a></p>
<p>My steup:<br>
Python 3.10.4<br>
PyTorch/2.1.2-foss-2022a-CUDA-12.1.1<br>
CUDA/12.1.1<br>
4 Nvidia A100 GPUs w/ 80 GB</p>
<p>Currently, my script runs the eval() function at the start of training to go over the validation dataset and get baseline perplexity metrics. Then I run the first epoch. Then I go back to eval() to see how well the model improved after the first epoch. I noticed that Rank 3 finishes eval first, so it hits the torch.dist.all_reduce() line first. I tried placing dist.barrier() in front of it, but that does not seem to have done anything. I checked the data sizes for each rank. They all process the same number of batches, and each batch has a batch size of 16. The sequence length varies from batch to batch because I did not pad all of them to be the same length (there is a specific reason for this), but all sequences within a batch have the same length. This made me think that Rank 3 had less data to process in terms of number of total tokens, but it doesn’t - it has the second fewest. There is another device that has about 2,000 tokens less that doesn’t even finish running eval before rank 3 does. They all process roughly the same number of tokens per batch at about 15-16 tokens per batch.</p>
<p>Any idea why the job keeps hanging?</p>
Why does PyTorch + Ray Tune save the data twice?

Aknw_Fen: <p>In the <a href="https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html" rel="noopener nofollow ugc">beginners tutorial there is the code</a> (comments are mine.)</p>
<pre data-code-wrap="pytorch"><code class="lang-pytorch">        # makes a tmp dir in /tmp/adfgsdf in Unix.
        with tempfile.TemporaryDirectory() as checkpoint_dir:
            data_path = Path(checkpoint_dir) / "data.pkl"
            # tmp dir and contents are removed after
            # upper context exits.
            with data_path.open("wb") as fp:
                pickle.dump(checkpoint_data, fp)
            
            # loads the data again, and saves it persistent?
            checkpoint = Checkpoint.from_directory(checkpoint_dir)
            train.report(
                {"loss": val_loss / val_steps, "accuracy": correct / total},
                checkpoint=checkpoint,
            )
</code></pre>
<p>Why does PyTorch/RayTune save the data twice, one with <code>dump</code> then with <code>report</code> ?</p>
<hr>
<p><strong>Reply from ChatGPT</strong></p>
<p>The key part from ChatGPT answer (If i paste the above description) is:</p>
<blockquote>
<p>When <code>Checkpoint.from_directory(checkpoint_dir)</code> is called, Ray Tune encapsulates all the contents of <code>checkpoint_dir</code> (including the serialized <code>data.pkl</code> file) into a <strong>checkpoint object</strong>. This abstraction allows Ray Tune to integrate with its distributed training and experimentation framework seamlessly.</p>
</blockquote>
<blockquote>
<p>If you only use <code>train.report</code> to save checkpoints, Ray Tune might not have sufficient information to reconstruct or manage the experiment properly, especially in complex training pipelines.</p>
</blockquote>
<p>I’m not very convinced by this explanation, but it may be correct. I find it hard to see why there isn’t a way to pass the checkpoint data directly to <code>report(...)</code></p>
[Question] How to Parallelizing the Loading of Serialized Tensor Files?

HyperHyper: <p>Hello PyTorch community,</p>
<p>I am exploring ways to speed up the loading process for a serialized tensor file in PyTorch.</p>
<p>Here’s my scenario:<br>
I have a single large tensor that I previously saved using <code>torch.save()</code> into a file on local disk, say <code>tensor.pt</code>. Now, I want to optimize the time it takes to load this tensor back using <code>torch.load(tensor.pt)</code>. As I understand, <code>torch.load</code> uses Python’s unpickling mechanism, which is also single-threaded. I’m wondering if there are ways to speed up this loading process.</p>
<p>Some ideas I’ve considered:</p>
<ol>
<li><strong>Splitting the tensor files</strong>: Save the one large tensor into multiple files, each containing a chunk of the tensor. Then, use multithreading to load these files in parallel and reconstruct the original tensor.</li>
<li><strong>Alternative formats</strong>: Serialize the tensor into formats like Pandas DataFrame or NumPy ndarray, and explore multithreaded deserialization options. After loading, perform a zero-copy conversion to a PyTorch tensor if possible.</li>
</ol>
<p>So far, I haven’t found an elegant solution to this problem. I’m open to suggestions beyond <code>.pt</code> files, as my goal is more general—optimizing the loading of serialized binary tensor files. The <code>.pt</code> file is just the simplest example as a tensor file.</p>
<p>Additional context:</p>
<ul>
<li>My primary operation involves performing mathematical computations on the tensor once it’s loaded. I don’t think <code>Dataset</code> or <code>DataLoader</code> fits this use case, as this is not about shuffling, sampling or batching—it’s just about efficiently loading one large tensor.</li>
</ul>
<p>Any hints, advice, or alternative approaches would be greatly appreciated!</p>
How to compute the Validation loss

ali_fahad: <p>Hi there<br>
we use the train and validation loss to determine the model performance in terms of whether it is overfitting ,underfitting ,or well generalization<br>
so do we use the teacher forcing in validation also with no parameter update since it is validation<br>
as if we use the autoregressive the validation loss start increasing from start of first epoch<br>
can you kindly explain this<br>
also if we use the teacher forcing in validation there is still the question that why we should use it since the validation should mimic the test scenario?</p>

ptrblck: <p>I don’t think using teacher forcing during the validation run is valid since you are interested in computing a proxy of the test accuracy which won’t use teacher forcing.</p>

ali_fahad: <p>if we switches to autoregressive technique  then how can we compute the loss<br>
means do we force the model to give same token probabilities equal to the respective ground truth length<br>
to compute the validation loss?</p>
Accessing weights and biases without detaching them

dmeister: <p>How can I access weights/biases without detaching them? I access them through state_dict, but it seems to be detached from the graph, getting the <em>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</em>. Here is a minimal example code to illustrate what I mean:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

X = torch.randn(100, 1)
Y = 2 * X + 1 + 0.1 * torch.randn(100, 1)

model = nn.Linear(1, 1)

optimizer = optim.Adam(model.parameters(), lr=0.1) 
num_epochs = 100

for epoch in range(num_epochs):
    optimizer.zero_grad()
    
    loss = model.state_dict()['weight']
    loss.backward()
    optimizer.step()
      
    if (epoch + 1) % 10 == 0:
        print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}")
</code></pre>

dmeister: <p>I have figure it out. Instead of <code>model.state_dict</code>, I can use simply <code>model.weight</code> or <code>model.bias</code>.</p>
How to customise LSTM in python

Omar_Abozaid: <p>How to customise LSTM cell and integrate that to work with auto grad. Suppose part of the weights in the LSTM cells are to be described by some formulas and to be updated given that formula? For example in this snippet I try a cubic / quadratic approximation of the input weights but when I print, I see that weights never change. Here is a minimal example of what I try to do</p>
<pre><code class="lang-auto">class CubicLSTM(nn.Module):
    def __init__(self, input_size=1, hidden_size=64, num_layers=1, output_size=1):
        super(CubicLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size
        self.num_layers = num_layers

        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True) 

        # Initialize 'a' parameters with 6.5e-5 and 'b' parameters with 0
        self.a = nn.Parameter(torch.tensor(2))  
        self.fc1 = nn.Linear(hidden_size, output_size)
        
    def _update_lstm_weights(self):
        """Update the LSTM input weights (weight_ih_l0) based on cubic-quadratic forms."""
        z = torch.arange(self.hidden_size) 
       W = self.a * z**3
        
        with torch.no_grad():
            self.lstm.weight_ih_l0.copy_(W)
        # self.lstm.weight_ih_l0.copy_(W) #uncomment to avoid no grad

    def forward(self, x):
        self._update_lstm_weights()
        out, (hn, cn) = self.lstm(x)
        out = self.fc1(hn[-1]) 
        return torch.tanh(out)
</code></pre>
<p>I also tried to add those weights directly in the optimiser but I see they are never updated.</p>

ptrblck: <p>Your code does not show where <code>self.a</code> is used so I guess you are resetting the lstm weight to the same value. Did you check which values the weight has?</p>
"Index put" causing runtime errors in torch.func.hessian, but not in the forward pass of my model

carneirj: <p>I am copying a physics-informed neural network (PINN) architecture found in the following paper by Andres Beltran-Pulido et al: <a href="https://ieeexplore.ieee.org/abstract/document/9788008" class="inline-onebox" rel="noopener nofollow ugc">Physics-Informed Neural Networks for Solving Parametric Magnetostatic Problems | IEEE Journals &amp; Magazine | IEEE Xplore</a></p>
<p>I have implemented the Encoding layer as described, and the layer works as expected in the forward pass.</p>
<pre><code class="lang-auto">class Encoding(nn.Module):
    
    def __init__(self, m=3):
        super().__init__()
        self.m = m

    def forward(self, data, norm_params):

        x, y = data[0], data[1]
        Lx, Ly = norm_params[-2], norm_params[-1]
        phi_x = data.new_ones(size=(self.m*2+1,))
        phi_y = data.new_ones(size=(self.m*2+1,))

        j = torch.arange(1, self.m+1)

        phi_x[2*j-1] = torch.cos(2*torch.pi*x*j/Lx)
        phi_x[2*j] =  torch.sin(2*torch.pi*x*j/Lx)
        phi_y[2*j-1] = torch.cos(2*torch.pi*y*j/Ly)
        phi_y[2*j] =  torch.sin(2*torch.pi*y*j/Ly)

        phi_vec = torch.outer(phi_y, phi_x).view(-1) #equivalent to vec(phix*phiy^T) as it stacks columns of phi_vec
        return phi_vec
</code></pre>
<p>Because this is a PINN for solving a PDE, I need the Hessian of the output wrt the input for each input individually. So, for a batch size of 32, I would want a 32x2x2 Hessian tensor, where the <code>h[i,:,:]</code> slice is the 2x2 Hessian for the <code>data[i,:]</code>. The forward pass of the model is called using <code>vmap</code> so that I could <code>vmap</code> the Hessian call as well. The following call works properly:</p>
<pre><code class="lang-auto">layer = torch.func.vmap(Encoding())
data = torch.rand(32, 2)
params = torch.rand(32, 4)
layer(data, params).shape
</code></pre>
<p>However, trying to get the Hessian of this layer using <code>h = torch.func.hessian(model)(data, params)</code> fails at the line <code>phi_x[2*j-1] = ...</code> with following error:</p>
<p><code>RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and Double for the source.</code></p>
<p>I have double checked the dtypes of <code>x</code>, <code>Lx</code>, <code>j</code>, <code>phi_x</code>, <code>phi_x[2*j-1]</code>, and <code>torch.cos(2*torch.pi*x*j/Lx)</code>, and they are all <code>torch.float32</code>. I have also explicitly declared everything with the <code>.float()</code> method, on all of the variables in this snippet. It still throws this error when I run it.</p>
<p>Does the error message not refer to the dtypes as I understand it? Why does it fail in the Hessian call (based on the error log it is in the <code>jacfwd</code> call so it doesn’t even get to the <code>jacrev</code> call) but not the forward pass? Any insight into what is happening under the hood would be much appreciated.</p>
Bfloat16 dtype runtime error in amp auto_cast enable mode

dongdongtong: <p>Recently I am running into the amp issue when using bfloat16 dtype in pytorch amp auto_cast mode. I am trying to train the SAM2 locally, but I find it is hard to fit it in bfloat16.</p>
<p>I searched in the pytorch issues that I found one issue may related to my scenario, that is <a href="https://github.com/pytorch/pytorch/issues/132715" rel="noopener nofollow ugc">bfloat16 auto_cast runtime error</a>. Despite that the issue had been closed, the code displayed in the issue still cause an error in my environment (i.e., pytorch 2.4.0, 3090 GPU, cuda 12.1).</p>
<p>However, I cannot handle this issue, and I have checked the PR which aims to solve this issue. The issue should have been gone but I still encounter the same error.</p>
<p>So could anybody save me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

dongdongtong: <p>Hi, everyone, after carefully checking the date and time of the issue, I found the issue is opened in August, which is newer than the release date of pytorch 2.4.0.</p>
<p>So I tried to upgrade my pytorch to the latest pytorch 2.5.0, then the issue is solved.</p>
Memory increasing after hugging face generate method

ima882112: <p>Hello! I wanted to make an inference with codegemma model from huggingface, but when I use model.generate(**inputs) method GPU memory cost increases from 39 GB to 49 GB in peak usage with torch profiler no matter max_token_len number. I understand that we need to save activations of model on inference and context of like 4096 input tokens but I can’t believe that it can increase inference memory usage on 10 GB. Can someone explain me how could it be? Thank you in advance.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/5912afc1d8481a4525526edf1be1739b6684177e.png" data-download-href="https://discuss.pytorch.org/uploads/default/5912afc1d8481a4525526edf1be1739b6684177e" title="Снимок экрана 2024-11-23 в 05.20.02"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5912afc1d8481a4525526edf1be1739b6684177e_2_690x337.png" alt="Снимок экрана 2024-11-23 в 05.20.02" data-base62-sha1="cHYwTI6qE5RcANsv73d35TQ8WnA" width="690" height="337" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5912afc1d8481a4525526edf1be1739b6684177e_2_690x337.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5912afc1d8481a4525526edf1be1739b6684177e_2_1035x505.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/9/5912afc1d8481a4525526edf1be1739b6684177e.png 2x" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Снимок экрана 2024-11-23 в 05.20.02</span><span class="informations">1348×660 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Request to Join PyTorch Slack Channel

FFFrog: <p>Hello, I am the leader of a team from China. Currently, our team is working on optimizing the registration mechanism of PrivateUse1 to facilitate the access to other new backends.</p>
<p>At present, our team has submitted more than 20 PRs. For the convenience of subsequent development, we want to apply to join the Pytorch Slack Channel. we all submitted “PyTorch Slack Invitation” half a month ago, but there is no reply yet.</p>
<p>The currently submitted PR link can be referred to here: <a href="https://github.com/pytorch/pytorch/issues/98406#issuecomment-1506466347" rel="noopener nofollow ugc">PRs</a></p>
<p>If possible, please add the following 6 domains to PyTorch Slack, thank a lot.</p>
<p><a href="mailto:youansheng@pku.edu.cn">youansheng@pku.edu.cn</a></p>
<p>Kind regards <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20"></p>

FFFrog: <p>And other addresses are as follow:</p>
<p><a>18207133434@163.com</a><br>
<a>519182487@qq.com</a></p>

FFFrog: <p><a>shaoyf42@outlook.com</a><br>
<a>medivh-x@outlook.com</a></p>

FFFrog: <p><a>yanpq@zju.edu.cn</a><br>
<a>ljw1101.vip@gmail.com</a></p>

Sunil_Singh: <p>Can you please add me on slack<br>
<a href="mailto:sunilsingh2019@gmail.com">sunilsingh2019@gmail.com</a></p>
How to Parallelize Image Loading for Multiple 4K Images in __getitem__?

MalumaDev: <p>Hello,</p>
<p>I’m working on a dataset class in PyTorch where I need to load several 4K images in the <code>__getitem__</code> method. Each time, I load <strong>7 images</strong>, including a ground truth image (GT). Currently, I’m using the following approach:</p>
<pre><code class="lang-auto">def __getitem__(self, index):
    imgs = self.files[index]
    imgs["files"].append(imgs["GT"])

    # Synchronously loading images one by one
    imgs = [Image.open(img) for img in imgs["files"]]

    # Transform images to tensors
    images = torch.stack(self.to_tensor(self.dataset_config.tfms(imgs)))
</code></pre>
<p>However, loading these images, especially at <strong>4K resolution</strong>, is quite slow and is significantly bottlenecking my training pipeline.</p>
<p>Additionally, the <code>to_tensor</code> function I’m using for transformations also seems to be slow, which further impacts performance.</p>
<pre><code class="lang-auto">self.to_tensor: Callable[..., torch.Tensor] = TF.Compose(
            [
                TF.ToImage(),
                TF.ToDtype(torch.float32, scale=True),
            ]
        )
</code></pre>
<p>Transformations I’m applying:</p>
<ul>
<li><strong>RandomShortestSize</strong></li>
<li><strong>RandomCrop</strong></li>
<li><strong>RandomHorizontalFlip</strong></li>
<li><strong>RandomVerticalFlip</strong></li>
</ul>
<p>I’ve considered using <code>asyncio</code> to fetch images in parallel, but seems to be slower (?).</p>
<h3><a name="p-459857-my-question-1" class="anchor" href="#p-459857-my-question-1"></a>My question:</h3>
<ul>
<li>How can I efficiently <strong>parallelize</strong> the loading of <strong>multiple images</strong> (7 images per sample) in PyTorch’s <code>__getitem__</code>?</li>
<li>I’m open to using threading, multiprocessing, or any other solution that can speed up.</li>
<li>Also the function to_tensor seems to be slow</li>
</ul>
<p>Any suggestions would be greatly appreciated!</p>
<p>Thanks!</p>

T0BIAS: <p>Im guessing you have your imgs in a .jpg or.png format (if not just skip this)<br>
Than you could use PIL to load the imgs and transforms.Compose ([*args]) to turn them to tensors and apply any tranformation you need. (This is basically what you already seem to do but I put this here as your to_tensor method might be different)</p>
<p>For faster / parallel loading you can use torchs dataLoader with pin_memory=True and num_workers=whatever works.<br>
Note that the memory required is very large especially with your resolution.<br>
E.g dataloader  =DataLoader (your_custom_dataset,batch_size =7,num_workers= x,pin_memory=True,prefetch_factor=y,persistent_workers=True)</p>
<p>Also if possible you could resize the images to a smaller res but given your emphasis on the img size Im guessing that isnt an option.</p>
<p>Maybe you could also experiment with turning the data to tensors beforehand and saving the tensor dataset. This would ensure you’d only turn every img to tensors once.</p>
<p>Edit: I’m guessing you set device to ‘cuda’ already</p>
Executorch - Encoder/Decoder models?

cptspacemanspiff: <p>Not sure if this is the right place, but I am trying to understand what people would recommend for implementing an encoder-decoder architecture in executorch. Would you implement them as 2 separate models and call the encoder once in the c++ runner, then pass the results as an input to the decoder (followed by the same type of decoder iteration that decoder only models do?)</p>
<p>Or is there a better way?</p>
<p>Thanks</p>

ptrblck: <p>Would the alternative be to wrap both modules into a single parent module and execute them together? If so, I would not expect to see a difference between these approaches and would claim it comes down to your coding style and if you want to use the intermediates in another way.</p>
Gather data from multiple processes in one gpu

Huang_Jiawei: <p>I started 4 processes on 2 GPUs (2 processes in 1 gpu). However, when I try to use distributed.all_gather_object(…) to collect the data (as follow)</p>
<pre><code class="lang-auto">torch.distributed.all_gather_object(all_process_list, [data])
</code></pre>
<p>I will receive the following error:</p>
<pre><code class="lang-auto">...
    work = default_pg.allgather([tensor_list], [tensor])
    torch.distributedwork = default_pg.allgather([tensor_list], [tensor]).
DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.18.1
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
Duplicate GPU detected : rank 2 and rank 0 both on CUDA device 46000
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.18.1
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
Duplicate GPU detected : rank 1 and rank 3 both on CUDA device 49000
    all_gather(object_size_list, local_size, group=group)
...
</code></pre>
<p>It seems that <code>torch.distributed.all_gather_object</code> does not support to collect data from the processes on the same device. Does anyone know if there is anyway to fix it?</p>

tianyu: <p>cc: <a class="mention" href="/u/wconstab">@wconstab</a> would you please help?</p>

kwen2501: <p>NCCL does not support having multiple ranks on the same GPU.<br>
You can use Gloo (CPU communication library) to gather the objects in CPU memory.</p>
<p>Method 1: create a separate ProcessGroup with Gloo support</p>
<pre><code class="lang-auto">cpu_pg = dist.new_group(backend="gloo")
dist.all_gather_object(..., group=cpu_pg)  # specify separate group
</code></pre>
<p>Method 2: create a ProcessGroup with both NCCL and Gloo support</p>
<pre><code class="lang-auto">dist.init_process_group("cuda:nccl,cpu:gloo")
dist.all_gather_object(...)  # no need to specify separate group
</code></pre>
DDP (multigpu) on multivariate time series dataset

lorsonb: <p>Hi everyone. I am doing some time series training on multiple GPUs using DDP, where my accuracy (mse) decreases with increasing number of GPUs. My guess is this has something to do with the way data is being split across the GPUs. Given the importance of order in time series datasets, i.e., time moving sequential from row 0 to row “len of dataset”, I am considering splitting the dataset via columns/features. Is this possible with Pytorch’s distributed sampler or DDP? Also, am I right with my assumption that the distributed sampler splits the data by rows? Thanks.</p>

fegin: <p>DDP doesn’t control what data feed into the model and is only responsible for reducing the gradients. So I would say this issue is orthogonal to DDP.</p>

lorsonb: <p>Thanks fegin. I just read in the torch.nn.parallel.distributed.py file that the module does not control how data is split across the GPUs. The user, i.e., me, will have to handle that. My guess is that it is possible. I’ll look further into it. If you have any ideas, you can share if you don’t mind.</p>
<p>So what I basically want to do is to let each GPU handle a set of features (all rows/timesteps for those features), instead of what is currently being done, where each GPU handles batches of all features up to a given timestep.</p>