akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x73984 and 4096x1)

bash: <p>Hi please i need help i have a discriminator architecture of</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()

        self.main = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, stride=2, padding=1),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.25),
            nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1),
            nn.ZeroPad2d((0, 1, 0, 1)),
            nn.BatchNorm2d(32, momentum=0.82),
            nn.LeakyReLU(0.25),
            nn.Dropout(0.25),
            nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1),
            nn.BatchNorm2d(64, momentum=0.82),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.25),
            nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1),
            nn.BatchNorm2d(128, momentum=0.82),
            nn.LeakyReLU(0.25),
            nn.Dropout(0.25),
            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(256, momentum=0.8),
            nn.LeakyReLU(0.25),
            nn.Dropout(0.25),
            nn.Flatten(),
            nn.Linear(256 * 4 * 4, 1),  
            #nn.Linear(32, 73984),  
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.main(x)
</code></pre>
<p>the nn.linear output is</p>
<pre><code class="lang-auto"> Linear(in_features=4096, out_features=1, bias=True)
</code></pre>
<p>i am trying to change the nn.linear to in_features=32, out_features=73984 from my architecture please help</p>

smth: <p>change this line</p>
<pre><code class="lang-auto">nn.Linear(256 * 4 * 4, 1),  
</code></pre>
<p>to:</p>
<pre><code class="lang-auto">nn.Linear(256 * 4 * 4, 32),  
</code></pre>
<p>And then uncomment the line <code>#nn.Linear(32, 73984),  </code></p>
<p>That’s it, you’ll be all set</p>

bash: <p>yes i have done that but this is the error</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x278784 and 32x278784)

</code></pre>

smth: <p>you must’ve not done what I mentioned. that error message would come if you put the numbers in the wrong order.</p>

bash: <p>yes i understand what you are saying but it will give the same error</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x73984 and 4096x3)
</code></pre>
<pre><code class="lang-auto">(16): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (17): BatchNorm2d(256, eps=1e-05, momentum=0.8, affine=True, track_running_stats=True)
    (18): LeakyReLU(negative_slope=0.25)
    (19): Dropout(p=0.25, inplace=False)
    (20): Flatten(start_dim=1, end_dim=-1)
    (21): Linear(in_features=4096, out_features=3, bias=True)
    (22): Linear(in_features=32, out_features=73984, bias=True)
    (23): Sigmoid()
  )
)
</code></pre>
Global pruning uses increasing amounts of memory

costelljoe: <p>I’m performing pruning on a simple model but am running into GPU out-of-memory errors. At each consecutive pruning iteration more memory is used, rather than maintaining a constant amount (see outputs below). Pruning using the “l1_unstructured” function works fine and doesn’t use increasing memory, while the “global_unstructured” function has this issue.</p>
<p>Am I doing something wrong here or this a bug?</p>
<p>Here’s simple code to reproduce the error:</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.prune as prune

class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = torch.nn.Linear(2000, 2000)
    
model = Net()
model.to(device="cuda:0")

iter = 0
for _ in range(1000):

    # Doesn't have any memory issue:
    # prune.l1_unstructured(model.fc, name="weight", amount=0.01)

    # Rapidly uses memory:
    prune.global_unstructured(
        [(model.fc, "weight")],
        pruning_method=prune.L1Unstructured,
        amount=0.01,  # fails with 0, 0.01, or 2
    )

    iter += 1
    if iter % 100 == 0:
        print(f"\nIter {iter}: allocated mem (gb) = {torch.cuda.memory_allocated(0) / 1e9}")
</code></pre>
<p>And the output, where it’s using multiple gb of gpu memory after only a couple hundred iterations:</p>
<pre data-code-wrap="Text"><code class="lang-Text">Iter 100: allocated mem (gb) = 1.6775424

Iter 200: allocated mem (gb) = 3.303190528

Iter 300: allocated mem (gb) = 4.929615872

Iter 400: allocated mem (gb) = 6.555264
</code></pre>
<p>This issue occurs for the following “amount” values: 0, 0.01, 2. All of these values are fine when using the “l1_unstructured” function.</p>
<p>I’m using PyTorch 2.1.2+cu121.</p>

costelljoe: <p>After some investigation I found that global pruning uses a <code>torch.nn.utils.prune.PruningContainer</code> which stores the entire history of the pruning mask during iterative pruning. It probably wasn’t designed for iterative pruning using 100s-1000s of steps, like what I’m trying to do.</p>
<p>Ideally there’d be an option to not store the entire history. For now, calling <code>prune.remove()</code> every ~10-100 iterations is a workaround to delete all the masks and free memory. Calling this after every iteration appears to add some delay so I’m only doing it when needed.</p>
Logging information from Dataset?

desert_torch: <p>I’d like to log various information about each dataset “record” consumed during the training loop. This includes the idx that was passed from the DataLoader, plus various detailed information such as the exact augmentations that were applied, how long it took to produce the record, etc.</p>
<p>My current solution is to return this information from the Dataset by combining it with the label as a 2-tuple, and then I unpack that in the training loop to separate the label from this extra information. So instead of the dataset returning img, label, it returns img, (label, extra_information).</p>
<p>See last line in block of code below:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, csv_with_image_label_pairs):
        self.csv_with_image_label_pairs = csv_with_image_label_pairs

        df = pd.read_csv(self.csv_with_image_label_pairs)

    def __len__(self):
        return len(self.df)

    def produce_img(img_filepath):
        # lots of complex code
        pass

    def __getitem__(self, idx):
        record = self.df.iloc[idx]

        img = self.produce_image(record["img_filepath"])
        label = record["label"]
        extra_information = (idx, augmentation_params, other_stuff)

        return img, (label, extra_information)
</code></pre>
<p>Is this a good way of doing this? Any other approaches to consider? I’m an “advanced beginner” with Python and a beginner with Pytorch, btw.</p>
<p>Thanks!</p>
Distributed Data and Model

Mordehay_M: <p>I am attempting to train a model with a size of only 5 million parameters. However, when I try to train the model with a batch size of 1, I encounter the “CUDA out of memory” error. I suspect that this issue arises due to the LSTM layers producing a significant number of intermediate tensors.</p>
<p>Upon this realization, I understand that data parallelism may not be a solution in this case, as even with a batch size of one, the problem persists. What steps should I take to address this issue?</p>
<p>I addressed the issue by manually splitting the model on different devices, but I am sure it’s not efficient.</p>
<p>Thanks.</p>

H-Huang: <p>5 million parameters seems like a pretty small model to OOM out on.</p>
<p>Do you have more details on your system and memory available?</p>
<p>In the case of OOMs you are correct that data parallelism may not be the solution, but there are workarounds. FSDP (<a href="https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/" class="inline-onebox" rel="noopener nofollow ugc">Introducing PyTorch Fully Sharded Data Parallel (FSDP) API | PyTorch</a>) can shard the model across multiple ranks which will save memory, but is still under the data parallelism umbrella. In the case where you manually split the model, that is a form of model parallelism. We have support for pipeline parallelism which performs automatic model splitting for you (<a href="https://github.com/pytorch/PiPPy" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/PiPPy: Pipeline Parallelism for PyTorch</a>)</p>

Mordehay_M: <p>I will check your links.<br>
Here you can find more details about my system and memory<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9b4ec6954138be9704bf8a6483bba60ea7cbccb.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9b4ec6954138be9704bf8a6483bba60ea7cbccb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9b4ec6954138be9704bf8a6483bba60ea7cbccb.png" alt="image" data-base62-sha1="odijNj49arfYRIO8qK8cNxqTJFx" width="488" height="500" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">821×841 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Multiple Instance Learning - Implementation of dataset object to load data bag-wise

gotlieb: <p>Hi everyone! I am kind of new in deep learning and pytorch. That’s why I’m having some issues and thought to address them here… <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>I need to create a dataset object for loading the data afterwards as a set of labeled bags (Multiple instance learning). Each bag will contain 10 images. And to label the bags (directories), I created a CSV file with two columns - one holding the directory name and the other some certain value as a label (regression task). How can I code the dataset object, such that when I provide it to the DataLoader, my data will be loaded bag-wise?</p>
<p>I could not find any piece of code or any link which could help me further. Any help would be much appreciated!</p>

ptrblck: <p>Could you explain the use case of creating the bags a bit more, please?<br>
If I understand it correctly, you would like to sample from specific indices to create these bags?<br>
Would you like to return a data tensor in <code>[batch_size, 10, ...]</code> or would 10 already be the batch size?</p>

gotlieb: <p>Thank you for your answer <a class="mention" href="/u/ptrblck">@ptrblck</a>! Sorry for not explaining it more concisely. Actually, my batch size should be 10, meaning that it will have 10 bags per batch. And the bags will contain a specific number of images, let’s say 15. Each bag is labelled by a number, from a CSV file.</p>
<p>Is this according to you more reasonable? Or you think sampling from loaded images to create bags makes more sense…</p>

HARIHK: <p>Hi, I am doing the same project with medical data.</p>
<p>The images are super large with a resolution of 5000*5000, to solve this I made patches of it and saved in folders. Then created a bag for each folder images</p>
<p>my bag has more than 187 images which I couldn’t in the gpu memory. Is there any other way to solve it ?</p>

ptrblck: <p>I’m not fully understand what “bad” means in this use case.<br>
If your data still doesn’t fit into the GPU memory, you might need to reduce the patch sizes further.</p>

HARIHK: <p>I tried reducing the patch size as well to 100*100, but still can’t able to fit in the GPU memory!!</p>

Bibhabasu_Mohapatra: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> actually bag means set of images. Like in HISTOPATHOLOGY one works with very large images called WSIs.<br>
In pathology WSI is given a label. It’s now has become standard in histology to make bags of images for each WSI and give them a weak label. It’s weakly supervised.</p>
<p>What’s the major issues is image model work with (b,c,h,w).  The MIL thing works with (1,num-patches,c,h,w) the major disadvantage is they work on 1-batch size.</p>
<p>People are here in this might want to work with multiple batch size as far as I know there are not definate ways around it but grad accumulation can be one of the methods to address multibatch training</p>
Deep learning model accuracy change when number of training GPUs changes

Nick1: <p>Hello everyone,</p>
<p>I’m new to PyTorch and deep learning, and I have a question regarding an action recognition model (supervised) using Graph Convolutional Network (GCN). In two experiments, where I trained the model and tested it with all parameters and the dataset constant, I observed a significant accuracy difference. Specifically, with 2 GPUs, I got a top-1 accuracy of 89.06, and with 4 GPUs, I obtained a top-1 accuracy of 89.96.</p>
<p>I want to highlight that I initialize the weights of the model using a random seed to ensure result reproducibility. Despite my efforts to identify the cause of this accuracy difference, I haven’t been successful. Has anyone else encountered a similar issue or have any insights into what might be happening?</p>
<p>I’m conducting these experiments on:</p>
<ul>
<li>GPU: Tesla P100-PCIE-16GB</li>
<li>CUDA version: 12.2</li>
<li>torch version: 1.7.1+cu110</li>
<li>Utilizing DataParallel with torch.nn.DataParallel</li>
</ul>
<p>Any experiences or ideas would be greatly appreciated. Thanks!</p>
Advice on Transformer Models for EDU Segmentation and Topic/Sentiment Analysis in Hugging Face

qrm_dev: <p>Hello Pytorch Community,</p>
<p>I am reaching out to seek your expertise in developing a PyTorch-based architecture for a complex NLP task. Specifically, my project involves processing textual comments, where the primary objectives are twofold:</p>
<ol>
<li><strong>Segmentation into Elementary Discourse Units (EDUs):</strong> Each comment needs to be divided into its constituent EDUs. I have access to ground truth data that defines these EDU boundaries, which should aid in training and validating the segmentation model.</li>
<li><strong>Classification of Topics and Sentiments:</strong> Once segmented, each EDU requires analysis for two aspects - its topical content and the sentiment it expresses. Again, I have ground truth data available for both topics and sentiments associated with each EDU.</li>
</ol>
<p>Given these requirements, I am looking for advice on designing an effective PyTorch architecture that can handle both segmentation and classification tasks efficiently. I am particularly interested in any insights on:</p>
<ul>
<li>Suitable model architectures or pre-trained models that can be adapted for this task.</li>
<li>Strategies for multi-task learning, if applicable, to handle both segmentation and classification in a unified framework.</li>
<li>Data preprocessing and feature engineering techniques that are effective for EDU-based analysis.</li>
</ul>
<p>Any examples, reference implementations, or pointers to relevant literature would be greatly appreciated. Additionally, I am open to collaboration or further discussion if anyone is particularly interested in this area of NLP.</p>
<p>Thank you in advance for your time and assistance!</p>
How to adjust training hyper-params to train a multi-gpu project on less-cards or even a single-card environment?

JohnHerry: <p>Hi, all<br>
I have a torch DDP project which can be well trained on a 8-card GPU machine, while will sometimes go fail on 4-card machine [ Here fail means the training loss is downing as normal, but the validation result is not good ], or if I train it on a single-card env, it always go to fail. The batchsize can not increase any more, so I can not train it on single-card env  with larger batch size,  So which hyper parameters I can adjust to make it train good on single-card or less card [eg ,2-cards, 3-cards   ]  environment? will a larger learning-rate and warmup steps help?</p>
Is it able to share memory pool of pytorch among different processes?

zizhao.mo: <p>Hi there,</p>
<p>I would like to ask that if it is possible that multiple processes on a GPU device can share the same GPU memory address? (like IPC in CPU memory) Is it implemented in PyTorch memory pool? If no, is it possible to do so via modifying the memory pool?</p>
<p>Thanks.</p>
How can Pytorch share memory among several processes?

jabberwoo: <p>According to <a href="https://stackoverflow.com/a/3044626/7238787" rel="nofollow noopener">this</a>, ‘processes have separate memory’. But Pytorch can somehow share memory among several processes, according to this <a href="https://pytorch.org/docs/stable/multiprocessing.html" rel="nofollow noopener">link</a>: <strong>‘Once the tensor/storage is moved to shared_memory (see share_memory_()), it will be possible to send it to other processes without making any copies.’</strong> Why is it possible to share memory among separate memory? Doesn’t it sound like a paradox?</p>

pietern: <p>It uses <em>shared memory</em>. Multiple processes can map the same shared memory segment into their own private memory space. The same segment may have a different address in each process, but maps to the same underlying physical memory. Also see <a href="https://en.wikipedia.org/wiki/Shared_memory" rel="nofollow noopener">https://en.wikipedia.org/wiki/Shared_memory</a>.</p>

zizhao.mo: <p>Hi pietern,</p>
<p>Does the shared memory you mentioned locate in the CPU? If it is located on the GPU, is it possible to share the GPU memory across jobs?</p>
Can we use pre allocate output tensor for forward execution when inference model?

Chunmyong_Park: <p>Is that possible to use forward execution with preallocated output tensors?</p>
<p>For example, can we use forward function like below?:</p>
<pre><code class="lang-auto">model = torch.load(…)
input = torch.Tensor(…)
output = torch.Tensor(…)
model.forward(input, output=[output])
</code></pre>
<p>When inferencing model, I need to save output into cuda ipc shared memory based tensor. But I didn’t found the way executing model with pre allocated shared memory based tensor. so, now I just use memcpy from model’s out to shared memory based tensor, but it seems not efficient.</p>
Help with mc3_18 model

Vinicius_Carvalho: <p>Hi folks, I’m new to ML and pytorch, so please apologies in advance for some very beginner quesstions.</p>
<p>I’m trying to extract features from a video using this model, but I’m bit confused on how to use it.<br>
Reading the docs, it seems the model accepts an input as (B, T, C, H, W), so this is what I’ve done to capture frames using opencv and convert them on the 5D tensor:</p>
<pre><code class="lang-auto">def preprocess_video_frames(video_path, frame_size=(224, 224)):
    frames = []
    capture = cv2.VideoCapture(video_path)
    while True:
        ret, frame = capture.read()
        if not ret:
            break

        # Resize each frame to the desired frame_size
        frame = cv2.resize(frame, frame_size)
        
        #ipdb.set_trace()
        # Normalize pixel values to the range [0, 1]
        frame = frame / 255.0
        #frame = np.transpose(frame, (2, 0, 1))
        
        # Append the preprocessed frame to the list
        frames.append(frame)

    capture.release()
    return frames

def load_and_preprocess_video(video_path):

    frame_list = preprocess_video_frames(video_path)
    video_frames = np.stack(frame_list) # Convert list into a 4D array
    
    video_tensor = torch.tensor(video_frames, dtype=torch.float32).permute(0, 3, 1, 2)
    

    
    # Extract features using the pre-trained I3D model
    with torch.no_grad():
        features = i3d_model(video_tensor)
    return features
</code></pre>
<p>I get <code>Given groups=1, weight of size [64, 3, 3, 7, 7], expected input[1, 303, 3, 224, 224] to have 3 channels, but got 303 channels instead</code></p>
<p>This is where I’m a bit confused, I thought that this tensor would have (BxTxCxWxH) and hence the 3 channels dimension being correct.</p>
<p>Could someone shed a light on what I’m doing wrong?</p>
<p>Thank you</p>

ptrblck: <p><code>nn.Conv3d</code> expects inputs as <code>[N, C, D, H, W]</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#torch.nn.Conv3d">docs</a>.</p>

Vinicius_Carvalho: <p>Thanks I was looking at this page: <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.video.mc3_18.html" class="inline-onebox" rel="noopener nofollow ugc">mc3_18 — Torchvision main documentation</a><br>
I’m afraid I do not understand well enough how to properly do this, my example seems to be way off, I’ll do some more digging on how to use the model for feature extraction.</p>
<p>Thanks</p>

ptrblck: <p>From the link:</p>
<blockquote>
<p>The inference transforms are available at <code>MC3_18_Weights.KINETICS400_V1.transforms</code> and perform the following preprocessing operations: Accepts batched <code>(B, T, C, H, W)</code> and single <code>(T, C, H, W)</code> video frame <code>torch.Tensor</code> objects.</p>
</blockquote>
<p>In this case I might be wrong and the model seems to expect a different format than e.g. native <code>nn.Conv3d</code> layers. Could you post a minimal and executable code snippet reproducing the issue using random tensors?</p>
Torch.cuda.device_count() behavior on multiple nodes

Xiongfei_Wei: <p>Hi team.</p>
<p>I want to confirm the expected behavior of <code>torch.cuda.device_count()</code> on a multi-node environment. Should it return the device count on a single node or the total device count across the nodes?</p>
<p>Let’s say I have 2 GPU VMs and each of which has 4 GPU devices. If I run <code>torch.cuda.device_count()</code> in a multi-node way (such as using torchrun), I observed that the command on each terminal returns 4. That means the <code>torch.cuda.device_count()</code> returns the device count on a single node instead of across the node. Is it expected?</p>
<p>Thanks!</p>

ptrblck: <p>Yes, that’s expected as the Python process wouldn’t have knowledge about other nodes.</p>

Xiongfei_Wei: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for confirming!</p>
Difference between multi label and single label CrossEntropyLoss

Giovanni_Puccetti: <p>Hi, I have question about <code>CrossEntropyLoss</code></p>
<p>If I understand correctly, this two should give the same output:</p>
<pre><code class="lang-auto">loss_fct = CrossEntropyLoss(reduction="none")
t = torch.randn(2, 10, 5)
l = torch.randint(0, 5, (2, 10))

loss_fct(t.transpose(-2, -1), l)
loss_fct(t.view(-1, t.size(-1)), l.view(-1)).view(t.size(0), -1)
</code></pre>
<p>However, while with this simple example they do, in some real loss calculations I have noticed they don’t.</p>
<p>Are these rounding errors of some kind or is there another possible explanation?</p>

ptrblck: <p>The second approach will fail with:</p>
<pre data-code-wrap="python"><code class="lang-python">ValueError: Expected input batch_size (20) to match target batch_size (2).
</code></pre>
<p>since <code>t</code> will be flattened to <code>[20, 5]</code> thus increasing the batch size while <code>l</code> still has the shape <code>[2, 10]</code>.</p>

Giovanni_Puccetti: <p>Sorry I missed a <code>.view(-1)</code> now it should work.</p>

ptrblck: <p>Yes, in this case it should yield the same result and the backend should also flatten the tensors in the same way as seen <a href="https://github.com/pytorch/pytorch/blob/24c39bb5e50848a5418bf79ebec51e02ffc4304d/aten/src/ATen/native/LossNLL.cpp#L723-L731">here</a>.</p>
Resnet18 fx_qat to onnx

aibear: <pre><code class="lang-auto">  from torch.ao.quantization import (
    get_default_qat_qconfig_mapping,
    QConfigMapping,
  )
  
  import copy
  import torch
  import torch.nn as nn
  import torch.optim as optim
  import torch.ao.quantization.quantize_fx as quantize_fx
  
  from resnet import resnet18
  from utils import prepare_dataloader, model_timeit
  from tqdm import tqdm
  
  
  def train(args):
      
      device = torch.device("cuda:%s" % args.device if torch.cuda.is_available() else "cpu")
  
      model = resnet18(num_classes=10)
      
      qconfig_mapping = get_default_qat_qconfig_mapping("fbgemm")
      
      model.train()
      example_inputs = torch.randn((1, 3, 224, 224))
  
      model = quantize_fx.prepare_qat_fx(model, qconfig_mapping, example_inputs)
      
      train_dataloader, test_dataloader = prepare_dataloader(batch_size=args.batch_size)
  
      loss_fc = nn.CrossEntropyLoss()
      optimizer = optim.Adam(model.parameters(), lr=1e-2)
      
      model.to(device)
      
      for epoch in range(args.epoch):
          epoch_loss = 0.0
          for idx, (image, label) in enumerate(tqdm(train_dataloader, total=len(train_dataloader))):
              image, label = image.to(device), label.to(device)
              output = model(image)
              loss = loss_fc(output, label)
              
              optimizer.zero_grad()
              loss.backward()
              optimizer.step()
              
              epoch_loss += loss.item()
          
          print(f"Epoch: {epoch}, Epoch Loss: {epoch_loss}")
      
      model = quantize_fx.convert_to_reference_fx(model)
</code></pre>
<p>what should do next to export this model to run tensorrt or onnx?</p>

HDCharles: <p>we don’t really support export to onnx, you’d be better off following a guide or asking onnx when you run into an issue.</p>
Type information for arguments of ATen operations

Abel_Nieto: <p>The following code</p>
<pre><code class="lang-auto">def test_add(x):
    return torch.add(x, 1)


module = export(test_add, (torch.rand(4, 3),))
print(module)

</code></pre>
<p>generates an <code>aten.add.Tensor</code> operation, instead of an <code>aten.add.Scalar</code></p>
<pre><code class="lang-auto">ExportedProgram:
    class GraphModule(torch.nn.Module):
        def forward(self, arg0_1: "f32[4, 3]"):
            add: "f32[4, 3]" = torch.ops.aten.add.Tensor(arg0_1, 1);  arg0_1 = None
            return (add,)
            
Graph signature: ExportGraphSignature(input_specs=[InputSpec(kind=&lt;InputKind.USER_INPUT: 1&gt;, arg=TensorArgument(name='arg0_1'), target=None)], output_specs=[OutputSpec(kind=&lt;OutputKind.USER_OUTPUT: 1&gt;, arg=TensorArgument(name='add'), target=None)])
Range constraints: {}
Equality constraints: []

</code></pre>
<p>This in spite of the fact that <code>aten.add.Tensor</code> appears to take two tensors, as per its documentation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/2899fc50013e60d4bb509234b78a72f793575719.png" data-download-href="https://discuss.pytorch.org/uploads/default/2899fc50013e60d4bb509234b78a72f793575719" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/2899fc50013e60d4bb509234b78a72f793575719_2_690x167.png" alt="image" data-base62-sha1="5Nb0HXmVLv73EnlT4Bwp64lI0zL" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/2899fc50013e60d4bb509234b78a72f793575719_2_690x167.png, https://discuss.pytorch.org/uploads/default/original/3X/2/8/2899fc50013e60d4bb509234b78a72f793575719.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/8/2899fc50013e60d4bb509234b78a72f793575719.png 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">972×236 14.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Two questions:</p>
<ol>
<li>Under what conditions does <code>torch.export</code> geneate an <code>aten.add.Scalar</code>?</li>
<li>More generally, if the type information in the IR documentation <a href="https://pytorch.org/docs/main/torch.compiler_ir.html" class="inline-onebox" rel="noopener nofollow ugc">IRs — PyTorch main documentation</a> doesn’t accurately capture the types that can appear in ATen operations, is there another source that does?</li>
</ol>
How to modify the forward pass of a loaded torch model with forward and backward hooks?

lflx: <p>I have a pretrained architecture which I load from <code>torch.hub.load()</code>. I know the forward pass is not as straightforward as a simple pass through all the layers but do not have access to the forward pass. Let’s say that this model looks like this.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.A = nn.Linear(128, 128)
        self.B = nn.Linear(128, 128)
        self.C = nn.Linear(128, 128)
        self.D = nn.Linear(128, 128)
    
    def forward(self, x):
        x = F.relu(self.A(x)) + x
        x = F.relu(self.B(x))
        x = F.relu(self.C(x))
        x = self.D(x)
        return x
</code></pre>
<p>I would like to have a network E, which would run in parallel to B: takes as input the output of A, and the output of B and E are added and are the input of C. If I could have access to the forward pass, it would look like this.</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = F.relu(self.A(x)) + x

        x1 = F.relu(self.B(x))
        x2 = F.relu(self.E(x))

        x = F.relu(self.C(x1+x2))
        x = self.D(x)
        return x
</code></pre>
<p>I would like to do this with hooks (both forward and backward), without modifying the Model class defined, but am not sure how to do it in a way so that the computation graph is similar to the forward pass shown above.</p>
UnboundLocalError: local variable referenced before assignment from import torch.optim._multi_tensor

Jiawei_Fu: <p>Hi all,</p>
<p>I am facing the error</p>
<pre><code class="lang-auto">UnboundLocalError: local variable 'torch' referenced before assignment
</code></pre>
<p>from running the following snippet</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class ASmallModel(nn.Module):
    def __init__(self, *args, **kwargs) -&gt; None:
        super().__init__(*args, **kwargs)

        print(f"from small model: {torch}")

        import torch.optim._multi_tensor


def main():
    mym = ASmallModel()


if __name__ == "__main__":
    main()
</code></pre>
<p>The solution is that moving the <code>import torch.optim._multi_tensor</code> to the start of the script. However, I am just curious how it causes the error when I do the import after <code>print(f"from small model: {torch}")</code></p>
<p>Thanks a lot!</p>
<p>Best,<br>
Jiawei</p>

Jiawei_Fu: <p>My torch version is</p>
<pre><code class="lang-auto">torch                    2.0.1
torchvision              0.15.2
</code></pre>
Using seperate encoder & decoder for transformer

Stingpie_52: <p>Hello, I’m messing around with transformers right now, and I’m trying to modify the encoded representation with a modified LSTM (the goal is to continue text in a specific style). I’ve found an example on how to use T.nn.TransformerEncoder, but no examples on how to properly use T.nn.TransformerDecoder. How am I supposed to use it? I’ve read about how decoders work in general, but I can’t find anything about the specific pytorch implementation. How should I use it for training vs inference? do I manually have to put the output of the transformer into the tgt during inference, or is that done automatically? What does tgt_is_causal do?</p>
<p>I’ve included a snippet of my code if that’s useful at all.</p>
<pre><code class="lang-auto">    def forward(self, x, mhx, tgt= None):
        
        
        embedded_seq = self.embedding(x) * math.sqrt(self.emb_dim)
        embedded_seq = self.pos_encoder(embedded_seq)
        if src_mask is None:
            """Generate a square causal mask for the sequence. The masked positions are filled with float('-inf').
            Unmasked positions are filled with float(0.0).
            """
            src_mask = nn.Transformer.generate_square_subsequent_mask(len(embedded_seq)).to(device)
        encoded_seq = self.transformer_encoder(embedded_seq, src_mask)
        
        
        
        # Take the encoded sequence, repeat once over the time axis, and stick that into the DNC. (to give the DNC time to analyze &amp; plan)
        processed_seq, (chx, mhx, rv) = self.vector_machine(T.cat( (encoded_seq, encoded_seq),1), (None, mhx, None), reset_experience=True, pass_through_memory=True)

        #split the processed sequence into two parts, taking the second half and adding it to the encoded sequence as a skip layer.
        processed_seq = T.chunk(processed_seq,2,dim=1)[1] + encoded_seq
        
        #TODO: put decoder here. 
        
        return decoded_seq, (chx, mhx, rv)
</code></pre>
<p><strong>Is there any example for proper use of the pytorch transformer decoder?</strong></p>
Estimate gradient of model output w.r.t its parameters

Salman_Abbasi: <p>Hi<br>
I have a model dnn_Linear. I would like to take derivative output(v2)of this model(dnn_Linear) w.r.t to its parameters.</p>
<p>ss = torch.autograd.grad(v2,dnn_Linear.parameters(), grad_outputs=None, retain_graph=None, create_graph=False, only_inputs=True, allow_unused=False)</p>
<p>I get the following error: grad can be implicitly created only for scalar outputs.</p>
<p>Also, it would be helpful how to estimate the gradient of model output w.r.t its parameters.</p>
<p>Best<br>
Sal</p>

ptrblck: <p>You would have to reduce the output or pass a gradient in the same shape as the output to solve the issue:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
model = nn.Linear(10, 10)
x = torch.randn(1, 10)

out = model(x)

# fails
torch.autograd.grad(out, model.parameters())
# RuntimeError: grad can be implicitly created only for scalar outputs

# works
torch.autograd.grad(out.mean(), model.parameters())

# also works
out = model(x)
torch.autograd.grad(out, model.parameters(), grad_outputs=torch.ones_like(out))
</code></pre>

Salman_Abbasi: <p>Thank you. It does not show errror with the proposed solution.<br>
.<br>
Sal</p>
Detached weights still being updated

gathierry: <p>Hi, I’m building a model that will freeze some layers after several training steps. But the weights which should be frozen are still updated.<br>
Here’s a piece of code to reproduce the issue. The printed <code>self.conv1.weight</code> keeps changing even though it’s already detached</p>
<pre><code class="lang-auto">import torch
from torch import nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 8, 3, padding=1)
        self.conv2 = nn.Conv2d(8, 8, 3, padding=1)
        self.pool = nn.AdaptiveAvgPool2d(1)
        self.linear = nn.Linear(8, 2)
        self.loss = nn.CrossEntropyLoss(reduction="mean")
        
    def forward(self, x, y, step):
        if step &lt; 5:
            self.conv1.train()
            self.conv2.eval()
        else:
            self.conv1.eval()
            self.conv2.train()

        if step &lt; 5:
            x = self.conv1(x)
        else:
            with torch.no_grad():
                x = self.conv1(x).detach()
            x = self.conv2(x)
        x = self.pool(x)[:, :, 0, 0]
        x = self.linear(x)
        print(step, self.conv1.weight.mean())
        loss = self.loss(x, y)
        return loss
          
model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4)

for i in range(10):
    input_data = torch.rand(size=[2, 3, 16, 16])
    target_data = (torch.rand(size=[2]) * 2).long()
    loss = model(input_data, target_data, i)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>

srishti-git1110: <p>Hi, your code doesn’t do what you require.<br>
Set <code>requires_grad=True</code> for the parameters that you want frozen.</p>

Iyad_alshami: <p>I am not an expert,  but this might be helpful in debugging the problem.<br>
when using just:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
</code></pre>
<p>you get what you expected, so maybe there is something happening when using<br>
<code>momentum, weight_decay</code></p>

ptrblck: <p>You are right and <code>weight_decay</code> should still update the parameters even with a zero gradient.<br>
<a class="mention" href="/u/gathierry">@gathierry</a> you could disable this behavior by using <code>optimizer.zero_grad(set_ton_none=True)</code> which will delete the <code>.grad</code> attribute and thus skip the parameter update. In newer PyTorch version <code>set_to_none</code> is <code>True</code> by default.</p>

gathierry: <p>Thanks, <code>set_ton_none=True</code> can solve my problem. But I guess the change is because of momentum?<br>
I tried <code>weight_decay=0</code>, but the weights are still updated. For <code>momentum=0</code>, the weight turns out to be consistent.</p>

Y-T-G: <p>Thanks. Spent a lot of time trying to figure out why they were still getting updated.</p>
Object detection inference results are not consistent with training results

gleow: <p>Hi all,</p>
<p>I have implemented a pytorch retinanet object detection model based on the yhenon github implementation - <a href="https://github.com/yhenon/pytorch-retinanet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - yhenon/pytorch-retinanet: Pytorch implementation of RetinaNet object detection.</a>.</p>
<p>I tried to perform inference on a specific image (image A) that is also in the training dataset. I was expecting to get identical, if not similar, results (bounding box locations, confidence scores, etc) for the same image. However, it is not the case at all. I get drastically different inference results.</p>
<p>Is this behavior expected?</p>
Tracing provenance of nodes in the backward pass

0x6b64: <p>Hi,</p>
<p><code>aot_module</code> can be used to get handle on the traced forward &amp;&amp; backwards passes by autograd. For example,</p>
<pre><code class="lang-auto">def print_aten_ops(gm, example_inputs):
  from functorch.compile import aot_module
  def trace_printer(gm, _):
    return gm
  return aot_module(gm, fw_compiler=trace_printer, bw_compiler=trace_printer)
model_compiled = torch.compile(model, fullgraph=True, backend=print_aten_ops)
</code></pre>
<p>I’m trying trace the generation of nodes in the backward pass, so I can attribute the node(s) in the forward pass which caused the creation of the said node in the backward graph. I see there is facility to add hooks to the modules/tensors, but that doesn’t necessarily let me get the effect I need.</p>
<p>Ideally, the nodes in the returned fx graph has been “annotated” with some node identified from the forward pass computation.</p>
<p>Any pointers will be very helpful!</p>
Inconsistent LSTM batched vs. unbatched error during inference

shu_bekk: <p>Hi</p>
<p>I’m trying to debug some inconsistencies I get for batched vs unbatched inference with LSTM.<br>
In short, I get different outputs depending on whether I compute the output in one batch or pass elements one-by-one to the LSTM. These differences are somewhat small, but my ultimate application relies on these LSTMs in succession, so the errors accumulate.</p>
<p>Additionally, I notice that this error even changes with completely unrelated changes to the code! For example, if I add some model initialization, the batched-vs-unbatched error can change significantly.</p>
<p>I use a specific set of pretrained weights, you can find them <a href="https://drive.google.com/file/d/1GqoA23fGp97HwxiIXzdiun8WZWQJgdyP/view?usp=sharing" rel="noopener nofollow ugc">here</a>. You can reproduce the issues with the following code snippet:</p>
<pre><code class="lang-auto">import torch
import random
import numpy as np


class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = torch.nn.LSTM(32,
                                  64,
                                  1,
                                  batch_first=True,
                                  bidirectional=True)
        self.linear = torch.nn.Linear(128, 32)
    
    def forward(self, x):
        lstm_out, _ = self.lstm(x)
        linear_out = self.linear(lstm_out)
        return linear_out

if __name__ == "__main__":
    torch.use_deterministic_algorithms(True)
    torch.manual_seed(0)
    random.seed(0)
    np.random.seed(0)

    # UNCOMMENT THIS, AND ERROR BECOMES 7.0512e-05
    # my_model = MyModel()

    # UNCOMMENT THIS AS WELL, AND ERROR BECOMES 0.0009
    # my_model = MyModel()

    my_model = MyModel()
    my_model.load_state_dict(torch.load('my_state_dict.pt'))
    my_model.eval()

    X = torch.randn(2, 288, 32)
    with torch.no_grad():
        Y_batched = my_model(X)
        
        Y_individiual = torch.zeros_like(X)
        for i in range(Y_individiual.shape[0]):
            Y_individiual[i:i+1] = my_model(X[i:i+1])

    # ERROR IS 0.0001
    print(torch.abs(Y_batched - Y_individiual).max())
</code></pre>
<p>I know that there is some tolerance with floating points and such, but I don’t see why it would affect the outputs in other batches, and I certainly don’t understand why changing a line of unrelated code would change any behavior.</p>
<p>Any help would be appreciated. Thanks!</p>

shu_bekk: <p>Hello,</p>
<p>I am still looking for help on this issue.<br>
Any ideas? Thanks.</p>

shu_bekk: <p>After some debugging, I think the error is sufficiently small and I had some problems in another part of the code.<br>
As for the inconsistent error when uncommenting “unrelated” lines, I think these lines were changing the random state so that when I initialized the tensor X, it was different every time.</p>
Displaying and Saving the Best Trial During Hyperparameter Search with Optuna in Parallel GPU Environment

YuhskeHujisaki: <h3><a name="issue-1" class="anchor" href="#issue-1"></a>Issue</h3>
<p>I am conducting hyperparameter optimization using Optuna integrated into the <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py" rel="noopener nofollow ugc">PyTorch Imagenet example</a>, while parallelizing across GPUs. However, I am unsure how to display and save the results of the best trial (optimal trial).</p>
<h4><a name="details-2" class="anchor" href="#details-2"></a>Details</h4>
<p>I am using the <code>study.optimize</code> method from Optuna as shown in the code below. <code>mp.spawn</code> is utilized to distribute the workload across each GPU. The issue arises in capturing and saving the results of the best trial while maintaining GPU parallelization. How can this be achieved?</p>
<pre data-code-wrap="python"><code class="lang-python">study.optimize(
    lambda trial: mp.spawn(
        objective, args=(ngpus_per_node, args, trial), nprocs=ngpus_per_node
    ),
    n_trials=100,
)

def objective(gpu, ngpus_per_node, args, trial):
    return main_worker(gpu, ngpus_per_node, args, trial)
</code></pre>
<h4><a name="question-3" class="anchor" href="#question-3"></a>Question</h4>
<p>What is the recommended approach to display and save the best trial while performing hyperparameter search with GPU parallelization using Optuna? Is there a specific method or practice within Optuna or PyTorch that facilitates this?</p>
Dynamically Loaded Dataset Using DDP

bigtuna: <p>Hello. I am training a language model using DDP on multiple compute nodes. The training data I’m using is a large set of millions text sequences and it doesn’t fit in my memory so I will have to load batches on the fly. Since each sequence may have different lengths, I want to concatenate short sequences using an eos token to avoid wasted computation on padding tokens. Also, I need to implement the dynamic loading in such a way the all my nodes are getting different data, ideally it would be shuffled as well. Is there a good way of doing this?</p>
<p>My current thinking is take total number of nodes and assign each node a (manageably sized) chunk of the total data available, then locally create a different dataset from that chunk and train with that. Would this approach work? Also, if one of the chunks is shorter than the others causing that node to finish early, will that interfere with the ddp training?</p>
How can I extend TripletMarginLoss to a quadruple?

mttbrv: <p><code>TripletMarginLoss</code> measures the relative similarity between three embeddings: <em>a</em>, <em>p</em> and <em>n</em> (i.e. anchor, positive example and negative example, respectively) and it penalizes a higher similarity between the anchor and the negative example as follows:</p>
<p>L(a,p,n) = max{ d(a​,p​) − d(a​,n​) + margin, 0 }</p>
<p>My goal is to extend this loss function to a quadruple <em>a</em>, <em>p</em>, <em>n1</em> and <em>n2</em>. In other words, I want to penalize instances where either d(a​,n1​) and/or d(a​,n2) are smaller than d(a​,p​), where d is a distance measure such as the Cosine Similarity.<br>
In short, I want <em>a</em> and <em>p</em> to be the most similar items in each quadruple, where <em>p</em> is the output of the model I’m training. Intuitively, I believe this could be achieved as follows:</p>
<p>L(a,p,n1,n2) = max{ d(a​,p​) − d(a​,n1​) + margin, 0 } + max{ d(a​,p​) − d(a​,n2) + margin, 0 }</p>
<p>My question is, how can I adapt the existing <code>TripletMarginLoss</code> to this case? Is it enough to compute <code>TripletMarginLoss</code> twice, once with <em>n1</em> and once with <em>n2</em>, and then sum the results as follows?</p>
<pre data-code-wrap="python"><code class="lang-python">triplet_loss_1 = nn.TripletMarginLoss(margin=1.0, p=2, eps=1e-7)
triplet_loss_2 = nn.TripletMarginLoss(margin=1.0, p=2, eps=1e-7)

anchor = torch.randn(100, 128, requires_grad=True)
positive = torch.randn(100, 128, requires_grad=True)
negative_1 = torch.randn(100, 128, requires_grad=True)
negative_2 = torch.randn(100, 128, requires_grad=True)

output_1 = triplet_loss_1(anchor, positive, negative_1)
output_2 = triplet_loss_2(anchor, positive, negative_2)
total_loss = output_1 + output_2

total_loss.backward()
</code></pre>
Fine tuned efficientnet doesn't train

pizzaball: <p>Hello! Can someone explain me why this model can’t train?</p>
<pre><code class="lang-auto">class EfficientNetClassifier(pl.LightningModule):
    def __init__(self, num_classes, transfer=True, freeze="last"):
        super().__init__()

        self.efficientNet = models.efficientnet_b4(pretrained=transfer)
        linear_size = 1792
        self.efficientNet.classifier[1] = nn.Sequential(
            nn.Linear(linear_size, 300),
            nn.BatchNorm1d(300),
            nn.LeakyReLU(),
            nn.Linear(300, 50)
        )
        
</code></pre>
<pre><code class="lang-auto">class LightningClassifier(pl.LightningModule):
    def __init__(self, lr_rate=1e-5, freeze="last"):
        super().__init__()

        self.model = EfficientNetClassifier(50, True, freeze)

        self.lr_rate = lr_rate

    def forward(self, x):
        return self.model(x)

    def cross_entropy_loss(self, logits, labels):

        return F.nll_loss(logits, labels)

    def training_step(self, train_batch, batch_idx):
        x, y = train_batch
        logits = self.forward(x)
        loss = self.cross_entropy_loss(logits, y)

        logs = {"train_loss": loss}

        acc = torch.sum(logits.argmax(axis=1) == y) / y.shape[0]
        self.log("train_acc", acc, on_step=True, prog_bar=True)
        self.log("train_loss", loss, on_step=True, prog_bar=True)

        return {"loss": loss, "log": logs}

    def validation_step(self, val_batch, batch_idx):
        x, y = val_batch
        logits = self.forward(x)
        loss = self.cross_entropy_loss(logits, y)
        acc = torch.sum(logits.argmax(axis=1) == y) / y.shape[0]

        self.log("val_loss", loss, on_step=True, on_epoch=True, prog_bar=True)
        self.log("val_acc", acc, on_step=True, on_epoch=True, prog_bar=True)

        return {"val_loss": loss, "val_acc": acc}
</code></pre>
<p>I chcked everything in code few times but didn’t find mistake. I thought that is because of <code>dropout</code> in efficientnet model, but as I know pytorch lightning include <code>model.train()</code> and <code>model.eval()</code> in <code>training_step()</code>, <code>validation_step()</code> and turns off <code>dropout</code> on evaluation</p>