akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Shuffling Concatenated Datasets

PopHSKE: <p>Hello.</p>
<p>I am training a recommender model using DDP and currently have two datasets, one that returns a positive interaction (customer id and item id) and the other returns a negative example ( customer id and a random item id from a list of candidates for that user).</p>
<p>The data looks like</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6007098a9dee70bc47a771f7ee54ca9440c15d7.png" alt="image" data-base62-sha1="pY3QcsJFT3HiepkdMl1YemHWt7h" width="443" height="301"></p>
<p>The first dataset takes in the user and item IDs and outputs these with a label of 1</p>
<pre><code class="lang-auto">class IntDataset(Dataset):
    def __init__(self, user_item_inters):
        self.inters = torch.LongTensor(user_item_inters) #  user and item
        
    def __len__(self):
        return len(self.inters)  # number interactions

    def __getitem__(self, idx):
        inter = self.inters[idx]

        return inter[0], inter[1], torch.DoubleTensor(np.array([1.0])).squeeze(-1)  

</code></pre>
<p>The second one takes in the same customers as the first one but also the list of candidate items. It outputs a customer id, a chosen item and the label (0)</p>
<pre><code class="lang-auto">class NegDataset(Dataset):
    def __init__(self, custs, negs, len_negs):
        
        self.custs = torch.tensor(custs) #  user and negs
        self.negs = torch.tensor(negs)
        self.len_negs = len_negs
        
    def __len__(self):
        return len(self.custs)  # number users
    

    def __getitem__(self, idx):
        
        user = self.custs[idx] # userID
        neg = self.negs[idx]
        
        
        neg_item = neg[np.random.randint(low =0, high = self.len_negs, size =1)]

        return user, torch.tensor(neg_item).squeeze(-1),  torch.tensor(np.array([0.0])).squeeze(-1)
</code></pre>
<p>Then I concatenate them:</p>
<pre><code class="lang-auto">neg_dataset = NegDataset(custs = pdf_ints['cust'].values, negs = pdf_ints['wght_new_smpl'], len_negs = args.len_hard_samples)
  
int_dataset = IntDataset(user_item_inters = pdf_ints[['cust', 'item']].values)
    
dataset = torch.utils.data.ConcatDataset([int_dataset,neg_dataset])
</code></pre>
<p>If I load all the data in memory and shuffle the concat dataset, this seems to work. However, when i cant load into memory, I am not sure how to proceed. I can use two iterable datasets, reading from disk but these can’t be shuffled AFAIK, the concat dataset will be stacked with all the positives on top.</p>
Performance problem with column major matrix multiplication

xdaniel9: <p>Here is the code</p>
<pre><code class="lang-auto">w_np = np.random.randn(4096).astype(np.float32)
R_np = np.random.randn(4096, 64).astype(np.float32)
R_np_c = R_np.copy(order='C')
R_np_f = R_np.copy(order='F')

w_torch = torch.from_numpy(w_np)
R_torch_c = torch.from_numpy(R_np_c)
R_torch_f = torch.from_numpy(R_np_f)

%timeit w_np@R_np_c
# 15.9 µs ± 76.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

%timeit w_np@R_np_f
# 4.64 µs ± 23.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

%timeit w_torch@R_torch_c
# 26.4 µs ± 20.5 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

%timeit w_torch@R_torch_f
# 28 µs ± 19.7 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
</code></pre>
<p>it is clear that numpy  version speedup with fortran order as desired while torch not, is there any problem with my code? how can i achieve the same performance gain using torch</p>
Model loading destroy my jupyter kernel

Samurai_Spirit: <p>Hi all, I am training a model, and every n iterations I save it to disk with the code.</p>
<pre><code class="lang-auto">torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict()
            }, os.path.join(model_dir, f'model_{epoch}_{i}.pth'))
</code></pre>
<p>Everything works fine, the model is trained.But when I try to load the model from disk, jupyter notebook crashes with error “Disposing session as kernel process died ExitCode: 3221225477, Reason: 0.00s - Debugger warning: It seems that frozen modules are being used, which may…”. According to my observations, this error occurs when memory overflows occur. How can i optimize model saving/loading?</p>
<p>Edit1: It looks like loading exactly the latest model breaks everything. I’ve tried before with a save every 6000 batches, and I only had one saved model file on which the error occurred. Now I tried saving the model every 80 batches (for the sake of experimentation), the most recent file did not load, but the one before that loaded fine. Apparently it’s not a memory shortage error, but a serialization error?</p>

mattmaitland1: <p>I am running into the same issue. Did you ever find a solution?</p>
A big picture to show the relationship among Compile, Dynamo, Inductor, AOTAutograd, AOTInductor, torch. export and related modules since PT2

111357: <p>Is there a big picture to show the relationship among <code>torch. Compile</code>, <code>dynamo</code>, <code>Inductor</code>, <code>AOTAutograd</code>, <code>AOTInductor</code>, <code>torch. export</code> and other modules since PT2.</p>
Is there a git repository where AMD generates their PyTorch WHL version from?

mzimmerm: <p>AMD is placing their Pytorch WHL files which correspond to ROCm here:</p>
<p><a href="https://download.pytorch.org/whl/nightly/rocm6.0" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/rocm6.0</a></p>
<p>Does anyone know where I can find the repository the AMD WHLs are build from?</p>
<p>My motivation is: I am trying to build PyTorch from master (also tried v2.2.1 tag) with <strong>USE_ROCM=1 USE_CUDA=0</strong>, and all the builds are failing with messages related to caffe2. Instead of asking here, I should first try to build from AMD source, as they may be making some changes that will clarify my issues.</p>
<p>Thanks</p>

mzimmerm: <p>Ah, I am sorry to have jumped. Was not able to find it last night, but did now by an unrelated search.</p>
<p>The AMD Pytorch repo seems to be here:</p>
<p><a href="https://github.com/ROCm/pytorch" rel="noopener nofollow ugc">https://github.com/ROCm/pytorch</a></p>
Focal loss for regression

Nason: <p>I have a regression problem with a training set which can be considered unbalanced. I therefore want to create a weighted loss function which values the loss contributions of hard and easy examples differently, with hard examples having a larger contribution.</p>
<p>I know this is possible type of weighted loss is possible as its implemented when using Focal loss.</p>
<p>My question is if its possible to convert focal loss for regression based problems using L1 loss and a linear output layer?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.cuda.amp as amp


##

class FocalLoss(nn.Module):

    def __init__(self,
                 alpha=0.25,
                 gamma=2,
                 reduction='mean',):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma
        self.reduction = reduction
        self.crit = nn.BCEWithLogitsLoss(reduction='none')

    def forward(self, logits, label):
        '''
        Usage is same as nn.BCEWithLogits:
            &gt;&gt;&gt; criteria = FocalLoss()
            &gt;&gt;&gt; logits = torch.randn(8, 19, 384, 384)
            &gt;&gt;&gt; lbs = torch.randint(0, 2, (8, 19, 384, 384)).float()
            &gt;&gt;&gt; loss = criteria(logits, lbs)
        '''
        probs = torch.sigmoid(logits)
        coeff = torch.abs(label - probs).pow(self.gamma).neg()
        log_probs = torch.where(logits &gt;= 0,
                F.softplus(logits, -1, 50),
                logits - F.softplus(logits, 1, 50))
        log_1_probs = torch.where(logits &gt;= 0,
                -logits + F.softplus(logits, -1, 50),
                -F.softplus(logits, 1, 50))
        loss = label * self.alpha * log_probs + (1. - label) * (1. - self.alpha) * log_1_probs
        loss = loss * coeff

        if self.reduction == 'mean':
            loss = loss.mean()
        if self.reduction == 'sum':
            loss = loss.sum()
        return loss
</code></pre>

shmaronshmevans: <p>Did this work for you? I am trying to do something similar and am curious how your results were?</p>
Libtorch previous release version download access denied

santisy: <p>Hi,</p>
<p>I am trying to download prebuilt libtorch release <a href="https://download.pytorch.org/libtorch/libtorch-cxx11-abi-shared-with-deps-2.0.1%2Bcu117.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/libtorch-cxx11-abi-shared-with-deps-2.0.1%2Bcu117.zip</a> but it shows access denied. Is the link not correct? How can I donwload previous built libtorch release.</p>
<p>Best</p>
<p>Dingdong</p>
Cannnot create the MFCC of a tensor that is already on a gpu

Greenscreen23: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I recently noticed that it is not possible to create the MFCC of a tensor that is already on a gpu. This is because two tensors are created during initialization, which cannot be moved to another device using the arguments of the MFCC class.</p>
<p>The tensors are the filterbanks for transforming the spectogram to mel scale and the dct matrix used by the mfcc.</p>
<p>Example code that crashes:</p>
<pre data-code-wrap="py"><code class="lang-py">import torch
from torchaudio.transforms import MFCC

device = torch.device('cuda')
mfcc = MFCC(melkwargs={ "wkwargs": { "device": device } })

data = torch.rand([400, 400]).to(device)

print(mfcc(data))
</code></pre>
<p>To be able to compute the MFCC of a tensor that is on a gpu, one has to add these two lines after the creation of the mfcc object, which is not good, as one has to access torchaudio internals.</p>
<pre data-code-wrap="py"><code class="lang-py">mfcc.MelSpectrogram.mel_scale.fb = mfcc.MelSpectrogram.mel_scale.fb.to(device)
mfcc.dct_mat = mfcc.dct_mat.to(device)
</code></pre>
<p>It would be great if this was possible without accessing the internals. I’d be happy to open a PR on the torchaudio repo, but I read that additions to the code should be discussed before creating a PR.</p>
<p>The way I would solve this issue in my PR would be to create a parameter “device” for the MFCC class, a parameter “mel_scale_kwargs” for the MelSpectrogram class and a parameter “device” for the MelScale class. Passing a device argument would transfer the dct_mat to that device, and pass this device to the spectogram window function (“melkwargs” &gt; “wkwargs” &gt; “device”) and the mel scale (“melkwargs” &gt; “mel_scale_kwargs” &gt; “device”), which transfer their respective tensors to the device.</p>
<p>I’m happy about any thoughts or feedback.</p>

ptrblck: <p>You could create an issue in the <code>audio</code> repository explaining the issue and your proposal to fix it.</p>
<p>CC <a class="mention" href="/u/moto">@moto</a> for visibility.</p>

Greenscreen23: <p>I’ll do that, thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I wanted to do that, but moved it to here because of this hint in the issue template:</p>
<blockquote>
<p><strong>PLEASE NOTE THAT THE TORCHAUDIO REPOSITORY IS NO LONGER ACTIVELY MONITORED.</strong> You may not get a response. For open discussions, visit <a href="https://discuss.pytorch.org/">https://discuss.pytorch.org/</a>.</p>
</blockquote>

moto: <p>Replied on the github.</p>
ModuleValidator.fix() causes gradients to be None

clr: <p>After I replacing BatchNorm by GroupNorm with ModuleValidator.fix(), and the gradients of parameters become None when I get them from model.named_parameters(). The grad_sample of parameters are also None. So the gradient is not flowing to the replaced GroupNorm weights when running backward pass.</p>

ptrblck: <p>Could you explain what <code>ModuleValidator.fix()</code> does as it seems to be a custom method?</p>

clr: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="199113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>ModuleValidator.fix()</code></p>
</blockquote>
</aside>
<p>It’s a method from Opacus <a href="https://opacus.ai/tutorials/guide_to_module_validator" class="inline-onebox" rel="noopener nofollow ugc">Opacus · Train PyTorch models with Differential Privacy</a></p>
Combination of two independent models

rynczakd: <p>Hello,</p>
<p>I want to combine two independent models as follows:</p>
<p>I have two independent models, which are stored in different folders - Model A and Model B. I want to attach selected blocks of Model A to Model B, so that they work as a single model. Both models have been pre-trained before and I have files with weights for both models.<br>
Please find a visualization of this problem in the image below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8bf6fdd9c17cb4b5206ab3df91d09fa9730f37c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8bf6fdd9c17cb4b5206ab3df91d09fa9730f37c" title="combined_models_questions"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8bf6fdd9c17cb4b5206ab3df91d09fa9730f37c.png" alt="combined_models_questions" data-base62-sha1="zuwyZVYiu0tZlh11dwDUI5zDVy4" width="690" height="496" data-dominant-color="EFF2F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">combined_models_questions</span><span class="informations">1032×743 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My idea was to create a Model A object in the Model B constructor, and then extract the individual Model A blocks so that they could be used in the Model B <code>forward</code> function. Due to the complexity of the models, it was not so easy and I was not able to connect the models in this way.</p>
<p>Another solution that comes to mind is to modify the source code of Model A so as to keep only the necessary blocks of the model and then use only their outputs, but I’m not sure if that would work.</p>
<p>What is the most correct and the most PyTorch way to connect models in such a way?<br>
How would you combine the models if you were me?</p>
<p>Thanks!</p>
Protobuf versions for pytorch-2.1.2

climbfuji: <p>In the spack package manager developed by LLNL we need to know the dependencies and their versions for pytorch. Previous versions of pytorch had problems with external protobuf v4, wondering if this is still the case in pytorch-2.1.2?</p>
<p>Issue <a href="https://github.com/pytorch/pytorch/issues/78362" class="inline-onebox" rel="noopener nofollow ugc">Numerous builds/tests are broken by protobuf update · Issue #78362 · pytorch/pytorch · GitHub</a> describes the problems and was closed as “mitigated” almost a year ago, but it’s unclear to me if latest versions of pytorch work with protobuf v4. Being limited to v3 is problematic, because many other Python packages require newer versions of protobuf.</p>
<p>If the issue preventing to update to protobuf v4 has not been fixed yet, are there plans (and a timeline) for this to happen?</p>
<p>Thanks!</p>
Loading model weights via timm from huggingface not possible. `_pickle.UnpicklingError: invalid load key, '<'.`

jS5t3r: <p>I want to load this model:</p>
<pre data-code-wrap="python"><code class="lang-python">import timm
model = timm.create_model("hf_hub:edadaltocg/resnet18_cifar100", num_classes=100, pretrained=True)
</code></pre>
<p>Error</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for ResNet:
        size mismatch for conv1.weight: copying a param with shape torch.Size([64, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 3, 7, 7]).
</code></pre>
<p>Then, I changed the code to:</p>
<pre data-code-wrap="python"><code class="lang-python">        model = timm.create_model("hf_hub:edadaltocg/resnet18_cifar100", num_classes=100, pretrained=False)
        # override model
        model.conv1 = nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        model.maxpool = nn.Identity()

        model.load_state_dict(
        torch.hub.load_state_dict_from_url(
            "https://huggingface.co/edadaltocg/resnet18_cifar100/resolve/main/pytorch_model.bin",
            map_location="cpu", 
            file_name="resnet18_cifar100.pth",
            )
        )
</code></pre>
<p>Is this correct? It tells me then</p>
<pre><code class="lang-auto">torch/serialization.py", line 1246, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '&lt;'.
</code></pre>
<p>My package versions:</p>
<pre><code class="lang-auto">timm                      0.9.12
torch                     2.1.1
torcheval                 0.0.7
</code></pre>

jS5t3r: <p>Solved.</p>
<p>I just removed <code>file_name="resnet18_cifar100.pth",</code>.</p>
Typeerror - RandomIoUCrop() requires input sample to contain tensor or PIL images and bounding boxes. Sample can also contain masks

ranjaniocl: <p>Hi,<br>
I am getting started on torch vision to train and evaluate object detection models. I am getting into issues and need some help.<br>
I am setting up basic steps to evaluate a pre-trained model on coco 2017 dataset. Here is my colab notebook.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/13nY17EEUx7IDsJCOmY9XEY9FjFX3E1pA?usp=drive_link">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/d00b91438f03c9783753d1b62bfddb21/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/13nY17EEUx7IDsJCOmY9XEY9FjFX3E1pA?usp=drive_link" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/13nY17EEUx7IDsJCOmY9XEY9FjFX3E1pA?usp=drive_link" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am using this example  from PyTorch to set up ‘data loader’ for coco dataset.<br>
<a href="https://pytorch.org/vision/main/auto_examples/transforms/plot_transforms_e2e.html#sphx-glr-auto-examples-transforms-plot-transforms-e2e-py" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/auto_examples/transforms/plot_transforms_e2e.html#sphx-glr-auto-examples-transforms-plot-transforms-e2e-py</a></p>
<p>After the data loader steps, I added steps to train / evaluate using ‘engine.py’.</p>
<p>Any help to resolve this issue will be appreciated. Or any suggestion on alternate ways.</p>
<p>Thanks,</p>
<p>Amit</p>
Can I use Dataparallel if my loss is calculated on the whole batch?

Timka: <p>Let’s say i have 20 batch and 10 image per batch. (I have some loaded tensors that has to be seen for every image) Would I be able to split my Data into 5-5 image per batch if the loss i am using should be calculated on the results of the 10 image together? Can I use Distributed Dataparalell?</p>
What's the difference between pass the model and the model.parameters()?

raegan_yang: <p>I have a model(A) which needs to be trained based on its similarity to another one(B).</p>
<p>Solution1: I pass B to A’s loss function, in this function, I calculate B.parameters().</p>
<p>Solution2: I pass B.parameters() to A’s loss function.</p>
<p>I run experiments for both methods but the results are very different. I can’t see why… solution 1&amp;2 look the same to me. Did I do something to B in-place when I pass it/its parameters? Why they function differently?</p>
Run comfy, but error, fail to install torch I think

Zhiyuai: <p>when I run comfy, it show some error. please help.</p>
<p>I try to install in the following two ways, but still the same error.</p>
<p><strong>conda install pytorch::pytorch torchvision torchaudio -c pytorch</strong></p>
<p><strong><a href="https://developer.apple.com/metal/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Accelerated PyTorch training on Mac - Metal - Apple Developer</a></strong></p>
<p>can anyone help?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/574f9010b9448389562528df33342028d0e96b1a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/574f9010b9448389562528df33342028d0e96b1a" title="截屏2024-03-20 14.19.18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/574f9010b9448389562528df33342028d0e96b1a_2_512x500.jpeg" alt="截屏2024-03-20 14.19.18" data-base62-sha1="cso0915zf8W8nYXXu8uyiLaA1B0" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/574f9010b9448389562528df33342028d0e96b1a_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/574f9010b9448389562528df33342028d0e96b1a_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/574f9010b9448389562528df33342028d0e96b1a_2_1024x1000.jpeg 2x" data-dominant-color="DDE5EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2024-03-20 14.19.18</span><span class="informations">2062×2010 768 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Zhiyuai: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/efef622bc26e267650e2734c695b591cf823b911.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/efef622bc26e267650e2734c695b591cf823b911" title="截屏2024-03-20 14.19.27"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efef622bc26e267650e2734c695b591cf823b911_2_690x433.jpeg" alt="截屏2024-03-20 14.19.27" data-base62-sha1="yeyZvVSjMxcsnBNwL6cLRFUqKkN" width="690" height="433" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efef622bc26e267650e2734c695b591cf823b911_2_690x433.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efef622bc26e267650e2734c695b591cf823b911_2_1035x649.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efef622bc26e267650e2734c695b591cf823b911_2_1380x866.jpeg 2x" data-dominant-color="575454"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2024-03-20 14.19.27</span><span class="informations">3520×2212 1010 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>screen of comfy</p>
How to use mixed precision training models in libtorch

w1005444804: <p>Thanks to anyone for providing clues</p>

eqy: <p>An ancient snippet from the author of AMP that might be relevant… <a href="https://github.com/pytorch/pytorch/issues/46649#issuecomment-715471342" class="inline-onebox" rel="noopener nofollow ugc">autocast get OOM with c++ api · Issue #46649 · pytorch/pytorch · GitHub</a></p>

w1005444804: <p>thanks any way, after at::autocast::set_enabled(true);The model accuracy has decreased。</p>
`python setup.py install` vs. `pip install .`

collinmccarthy: <p>When building from source I would prefer to use <code>pip install .</code> or <code>pip install -e</code>, instead of <code>python setup.py install</code> or <code>python setup.py develop</code>, so I can uninstall the package more easily.</p>
<p>Is there any reason I should NOT do this? Seems like a simple question but I can’t find the answer anywhere.</p>
<p>Thanks.</p>

collinmccarthy: <p>Looks like I can still <code>pip uninstall torch</code> even if using <code>python setup.py</code>, but I’m still curious, is there a reason not to use pip explicitly during the install?</p>

ptrblck: <p>If I’m not mistaken, the <code>setup.py</code> approach is considered legacy as mentioned <a href="https://pip.pypa.io/en/stable/reference/build-system/setup-py/">here</a> so you might be find building via <code>pip</code>. However, I haven’t carefully read through the mentioned PEPs.</p>
Is there a risk of log conflict when sharing a same log directory by using torchrun with multiple nodes?

ayl: <p>Consider the scenario where we have two machines, each equipped with 8 GPUs, all of which can access the same folder through a distributed file system. I’m curious whether conflicts might arise when these two nodes attempt to save logs into the same directory. Take for instance the following <code>torchrun</code> commands executed on these nodes, both utilizing the same <code>log_dir</code> :</p>
<pre><code class="lang-auto"># node #0
torchrun --rdzv-id aaaa --master_addr bbbb --master_port cccc --nnodes 2 --node_rank 0 --nproc_per_node 8 --log_dir ./logs --redirects 3 --tee 3 train.py
# node #1
torchrun --rdzv-id aaaa --master_addr bbbb --master_port cccc --nnodes 2 --node_rank 1 --nproc_per_node 8 --log_dir ./logs --redirects 3 --tee 3 train.py
</code></pre>
<p>The question that arises is whether these two nodes might concurrently attempt to write to the same log file within the specified <code>log_dir</code> .</p>
How to find which ops are not supported by torch.compile

bigtree: <p>Hey, I am new with toch.compile and tried this feature on llama2.<br>
What I found is it doesn’t work:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/403a1ef078d5eac3408e10133f232259aec59685.png" data-download-href="https://discuss.pytorch.org/uploads/default/403a1ef078d5eac3408e10133f232259aec59685" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/403a1ef078d5eac3408e10133f232259aec59685.png" alt="image" data-base62-sha1="9ab53aCv6QbuYI6iKBXRJrTEDyt" width="690" height="256" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1249×465 19.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After I tried the " TORCH_LOGS=“+dynamo” TORCHDYNAMO_VERBOSE=1", I still can’t find which ops or layers are not supported. Could some one help me about how to find which layer of ops the torch.compile() doesn’t support?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2e26cd48a24fb9b765c324e19d2c6e62b2d0278.png" data-download-href="https://discuss.pytorch.org/uploads/default/e2e26cd48a24fb9b765c324e19d2c6e62b2d0278" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2e26cd48a24fb9b765c324e19d2c6e62b2d0278.png" alt="image" data-base62-sha1="wn71mbjaIthoSvRNdTxqy2ynSZO" width="690" height="237" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1169×403 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Exact Correspondence between nightly build and the main branch?

youkaichao1: <p>Earlier today I was told that I can find the exact commit used for nightly at <a href="https://hud.pytorch.org/hud/pytorch/pytorch/nightly" class="inline-onebox" rel="noopener nofollow ugc">PyTorch CI HUD &lt;!-- --&gt;</a> . This is great, however, it seems difficult to figure out the commits in nightly branch and the main branch.</p>
<p>For example, if I find <code>torch-2.3.0.dev20231224</code> does not work, but <code>torch-2.3.0.dev20231223</code> works, and I want to find out all the commits in the main branch to further investigate which commit causes the change, how can I find the correspondence?</p>

youkaichao1: <p>I find that 2023-12-22 nightly <a href="https://github.com/pytorch/pytorch/commit/b6723b1317679b7d4ea6b93797457666cfd41eba#diff-d7302d133bb5e0890fc94de9aeea4d9d442555a3b40772c9db10edb5cf36a35c" class="inline-onebox" rel="noopener nofollow ugc">2023-12-22 nightly release (b47aa69685aa6e561d3ba0edaff84b7833b98a82) · pytorch/pytorch@b6723b1 · GitHub</a> contains a PR in 2023-12-20 . Quite strange though.</p>
<p>I used to think that nightly is just a build against a commit in main <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> So I thought 2023-12-22 nightly only contains PR from 2023-12-21 .</p>

ptrblck: <p>Nightly branches might branch from <a href="https://github.com/pytorch/pytorch/commits/viable/strict"><code>viable/strict</code></a> instead of <code>main</code> to avoid building with known regressions, but <a class="mention" href="/u/malfet">@malfet</a> can correct me if I’m wrong.</p>
How to distinguish datasets spawned by the dataloader moudule?

chener: <p>If both the dataset and dataloader moudules are used as the document, the spawned datasets are the same. But for my situation, in dataset getitem function, I used a model to preprocess the data. If I don’t specify the cuda index, all the models used in dataset for preprocessing will occupy the same gpu. So, how can I get something like index in each dataset.<strong>init</strong> function to assign different gpus to the models in the dataset.</p>

ptrblck: <p>You could try to use the worker id and use it to move the model to the corresponding device.</p>

chener: <p>Hi, how can I get the worker id, any links suggested?</p>

ptrblck: <p>This code should work inside the <code>Dataset.__getitem__</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">    def __getitem__(self, index):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info:
            worker_id = worker_info.id
            print('worker_id {} calling with index {}'.format(worker_id, index))
</code></pre>
Fine-grained control over gradient computation

Faberix: <p>Hi,<br>
I am doing contrastive learning in a setting similar to CLIP (i.e. I have n correct pairs of samples in a batch and the task is to find which of the n^2 possible pairs are correct). This is done by computing cosine similarities of the embeddings and then applying cross entropy loss. Since I suspect that a significant part of the information is available in only one of the two modalities, I want to add an additional link between the modalities after a part that is the same for all pairs and before the loss is computed. Unfortunately, this means that more computation needs to be done for all the n^2 pairs rather than just the n individual samples.<br>
In the inference setting, this is doable if I compute the cosine similarities in sub-batches. However, when training, autograd stores the information for all n^2 pairs that is needed for gradient computation. This severely limits the batch size, which should actually be as large as possible for contrastive learning.<br>
I could still do larger batches if I could compute the gradients for all pairs where one of the samples is the same as in a given correct pair at a time and aggregate the gradients. I know that it is possible to compute per sample gradients (I saw <a href="https://pytorch.org/tutorials/intermediate/per_sample_grads.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/per_sample_grads.html</a>), but that would be inefficient because all the gradients for the common part of the model would need to be calculated for each correct pair.<br>
So my question is: Can I compute the gradients per pair in the part with links between the modalities, aggregate it and then compute the gradients for the rest of the model and feed them to the optimizer?</p>
<p>It would be ideal if I could do this without affecting the way the model needs to be handled for training, because I use Pytorch Lightning and want to keep using it.</p>

TheOrbitHeart: <p>Hey,<br>
The process you’re describing sounds awfully similar to a technique called GradCache (<a href="https://arxiv.org/pdf/2101.06983.pdf" rel="noopener nofollow ugc">Paper</a>, <a href="https://github.com/luyug/GradCache" rel="noopener nofollow ugc">Implementation</a>). They basically first calculate the representation gradients with respect to the loss function (without calculating the model gradients) then later accumulate the model gradients for smaller sub-batches that can fit into memory while utilising the cached gradients. finally they optimize. I think they do support pytorch lightning but I’m not sure.</p>
About the gradient of intermediate variables w.r.t. input

Jinghong_Zhang: <p>Is there a good way to compute the following: suppose <code>x</code> is a tensor that requires gradient, and I have two functions f(x) and g(x), we define h(x) = g(x)/f(x). When I call <code>h.backward()</code> (I need to find the gradient \partial h/ \partial x, how can I also extract \partial f/ \partial x efficiently since f is also part of the computation graph?</p>
<p>Help would be much appreciated!</p>
Imaginary Component Error when calculating FID

kwhew: <p>So this bit of <a href="https://colab.research.google.com/drive/1eifd4oix1q5eJlhSG0ONsu3T6m33BaEM?usp=sharing" rel="noopener nofollow ugc">code</a> has been working fine for me the last time I ran it… but has a recent update to torch caused the FID score to blow up ? I am a bit suspicious of this warning:<br>
<code>UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors.</code></p>
<p>So whenever I am running the training process, I get:</p>
<pre><code class="lang-auto">&lt;ipython-input-8-6bc3f7d49a34&gt; in calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps)
     32         if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):
     33             m = np.max(np.abs(covmean.imag))
---&gt; 34             raise ValueError('Imaginary component {}'.format(m))
     35         covmean = covmean.real
     36

ValueError: Imaginary component 1.172166351981351e+117
</code></pre>
<p>This is just an example provided by my class (which I haven’t changed) but I am not sure how to go about debugging this especially since I have limited GPU usage on colab. Any ideas? Thanks!</p>

kwhew: <p>I have tried changing the code as per the user warning from:<br>
<code>z = Variable(Tensor(np.random.normal(0, 1, (imgs.shape[0], opt.latent_dim)</code>)))<br>
to:<br>
<code>z = Variable(torch.tensor(data=np.random.normal(0, 1, (imgs.shape[0], opt.latent_dim)),dtype=torch.float, device='cuda')) if cuda else torch.FloatTensor</code></p>
<p>Still seems to blow up the FID value and throws the imaginary value error</p>
How to access the data of a Tensor

sroyyors: <p>Hi,<br>
I wanted to directly access the data matrix of the Tensor to store in a primary type, e.g.a double array. I was doing this to write the Tensor to a binary file. Following this post <a href="https://discuss.pytorch.org/t/convert-tensor-into-an-array/56721" class="inline-onebox">Convert tensor into an array</a> I tried</p>
<pre><code class="lang-auto"> torch::Tensor preds = torch::zeros({ 4,8 });
        preds.index_put_({0,1},2);
        preds.index_put_({0,3},4);
        preds.index_put_({1,7},5);
        preds.index_put_({2,5},-5);
        preds.index_put_({3,6},-3.96);
        preds.index_put_({2,6},-1e-3);
       double* mydata=preds.data_ptr&lt;double&gt;();
</code></pre>
<p>But I get sigabort every time I do this. Any ideas how to address this?<br>
thanks!</p>

ptrblck: <p>You could try to initialize or transform the tensor to <code>double</code> as the default <code>dtype</code> would be <code>float</code> (i.e. <code>float32</code>).</p>

sroyyors: <p>Thank you that worked! I did it like</p>
<pre><code class="lang-auto">torch::Tensor preds = torch::zeros({ 4,8 }).to(torch::kDouble);
</code></pre>
<p>However, I noted that although this is a 2d tensor, the data_ptr will only return a double* pointer. So I always have to do this:</p>
<pre><code class="lang-auto">double* mydata1d=preds.data_ptr&lt;double&gt;();
</code></pre>
<p>which I would read as a 1D array. I wonder if it is possible to read it in as a 2D array.</p>
<p>That said, I realized I can also just write the Tensor directly to a binary file like</p>
<pre><code class="lang-auto">mytensorfile.write(reinterpret_cast&lt;char*&gt;(&amp;preds),4*8*sizeof(torch::kDouble));
</code></pre>
Skip incompatible - lcuda when searching for/usr/lib/i386 Linux gnu/libcuda. so

dkz: <pre><code class="lang-auto">/usr/bin/ld: 当搜索用于 /usr/lib/i386-linux-gnu/libcuda.so 时跳过不兼容的 -lcuda
/usr/bin/ld: 当搜索用于 /usr/lib/i386-linux-gnu/libcuda.so 时跳过不兼容的 -lcuda
/usr/bin/ld: 找不到 -lcuda
collect2: error: ld returned 1 exit status
</code></pre>
<p>When I use torch.compile, this error occurs. How should I solve this problem？ Thanks<br>
OS is ubuntu.</p>

ptrblck: <p>Could you create an issue on GitHub describing your use case, please?</p>
A few questions about QConfig in quantization

minhhotboy9x: <p>Hello, I’m a beginner in quantization. When we want to quantize a model, we must specify a qconfig for a model to choose scales and zero-points. For example:</p>
<pre><code class="lang-auto">QConfig(activation=torch.quantization.default_observer,
     weight=torch.quantization.default_observer)
</code></pre>
<p>I think that the <code>weight</code> param of <code>QConfig</code> is the observer of the weight tensors. But what about <code>activation</code>? Does <code>activation observer</code> watch the output values of a layer or the input? And it is to choose the scales and zero-points for the input of layers, right?</p>

HDCharles: <p>for static quantization, the activation are quantized at the output. The input is assumed to be quantized.</p>
<p>for dynamic quantization, the output is not quantized while the input is so the qconfig gets applied to the input.</p>

minhhotboy9x: <p>Thank you. I am confused that the term <code>activation</code> here is the output of an arbitrary layer (conv2d, linear, …) right? Or it must be the output of an activation function?</p>

HDCharles: <p>generally we use the term activation to mean any flow from one layer to another, known only at runtime. Whereas weights are known before runtime.</p>
Transformer "tgt" Question

thebeginning_isnigh: <p>Hi, I’m trying to do an experiment with a Transformer LLM in Pytorch.</p>
<p>The idea is to pass a sequence of tokens (each token is a 200 dim vector) into the transformer. The max token sequence length would 500, and the batch size would be 20.</p>
<p>According to the docs, the shape of the input is:<br>
[seq_length (500), batch_size (20), feature_dims (200)].</p>
<p>The thing I’m trying (part of speech softmax classification) requires the decoder to output a 2 dimensional vector for each output, either [1, 0] or [0, 1]. Given this:</p>
<p>According to the docs, the shape of the output should be:<br>
[seq_length (250), batch_size (20), feature_dims (2 in this case)]</p>
<p>But according to the docs, The output of the decoder’s feature length <strong>MUST</strong> match the input’s (200) - which is fine, because I can add a fully connected layer to get that down to 2.</p>
<p>BUT the decoder seems to also require a “tgt” variable for what the decoder’s targets should be. So training can’t happen any other way than the decoder trying to aim for the target vector?</p>
<p>Am I misunderstanding this? Why is the decoder so strict about for what &amp; how it must be implemented?</p>
<p>Is there any way around this?</p>
How to suppress this warning?

gradwolf: <p>UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. warnings.warn('Was asked to gather along dimension 0, but all ’</p>

SimonW: <p>You are probably using DataParallel but returning a scalar in the network. You should return a batched output.</p>

pritamqu: <p>i faced the same issue, and you’re right, i am using data parallel, but could you please elaborate how to tackle this?</p>

ronlek: <p>Hey Simon. Could you please share how to returned a batched output? Here’s my <code>finetune</code> function</p>
<pre><code class="lang-auto"># Step 4: Fine-tuning
def fine_tune(model, train_loader, optimizer, num_epochs, save_dir):
    model.train()
    for epoch in range(num_epochs):
        progress_bar = tqdm(enumerate(train_loader), total=len(train_loader), desc=f"Epoch {epoch+1}/{num_epochs}", unit="batch")
        for batch_idx, batch in progress_bar:
            optimizer.zero_grad()
            input_ids = batch['input_ids'].to(device)
            attention_mask = batch['attention_mask'].to(device)
            labels = batch['label'].to(device)
            outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
            loss = outputs.loss
            loss.backward()
            optimizer.step()

            # Update progress bar
            progress_bar.set_postfix({'loss': loss.item()}, refresh=True)
        # Save model weights
        save_path = os.path.join(save_dir, f"model_epoch_{epoch+1}.pt")
        torch.save(model.state_dict(), save_path)
</code></pre>
<p>I’m aware I need to do <code>loss.sum().backwards()</code> to get the sum of all losses across all machines.</p>
Poisson-beta Distribution

Carzit: <p>Recently I’m developing a neural network that uses probability distributions to fit chemical reaction processes. Biologically, the Poisson-beta distribution is what we are interested in. But its original formula is probably an integral that cannot be calculated. Note that an approximation can be made using Gaussian-Jacobian quadrature. In fact, <code>scipy.special.roots_jacobi</code> can be used in scipy to calculate Gauss-Jacobi orthogonal sampling points and weights. I’m curious if we can implement a Poisson-beta Distribution class for torch.distributions based on this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>