akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Getting Runtime Error when testing quantized model

Zowie_Tay: <p>i am getting the runtime error when i try to test evaluate my quantized model, how do i go about fixing this? i have no clue where to even begin</p>
<pre><code class="lang-auto">     81 for X, y in test_dataloader:
     82     X, y = X.to(self.device), y.to(self.device)
---&gt; 83     pred = self.model(X)
     84     test_loss += self.loss_fn(pred, y).item()
     85     correct += (pred.argmax(1) == y).type(torch.float).sum().item()

RuntimeError: getCudnnDataTypeFromScalarType() not supported for QUInt8
</code></pre>
<pre><code class="lang-auto">    def calibrate(model, data_loader):
        model.eval()
        with torch.no_grad():
            for image, target in data_loader:
                image = image.to(self.device)
                model(image)

    model_fp32 = copy.deepcopy(self.model)
    qconfig_mapping = get_default_qconfig_mapping("qnnpack")
    model_fp32.eval()
    example_inputs = (next(iter(data_loader))[0])
    prepared_model = prepare_fx(model_fp32, qconfig_mapping, example_inputs)
    calibrate(prepared_model, data_loader_test)
    self.model = convert_fx(prepared_model)
</code></pre>
<pre><code class="lang-auto">    def test(self, test_dataloader:DataLoader)-&gt;list:
        size = len(test_dataloader.dataset)
        num_batches = len(test_dataloader)
        self.model.eval()
        test_loss, correct = 0, 0
        with torch.no_grad():
            for X, y in test_dataloader:
                X, y = X.to(self.device), y.to(self.device)
                pred = self.model(X)
                test_loss += self.loss_fn(pred, y).item()
                correct += (pred.argmax(1) == y).type(torch.float).sum().item()
        test_loss /= num_batches
        correct /= size
        print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")
        return [correct*100, test_loss]
</code></pre>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.dropout1 = nn.Dropout(0.25)
        self.dropout2 = nn.Dropout(0.5)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output
</code></pre>

Vasiliy_Kuznetsov: <p>int8 quantization is not supported natively on GPUs in PyTorch yet, you can move the model to CPU to get around this.</p>
Projecting embeddings on a hypershpere of radius 1

K_awanish: <p>Hi all, I have been trying to run a triplet loss based neural network and have been facing difficulties in converging it to the global minima. I recently came across “FaceNet: A Unified Embedding for Face Recognition and Clustering” that mentions that the embeddings should lie on a hypershpere of radius 1 in the n dimensional space and I can’t seem to find a way to do this in my code. does pytorch have in-built functions like this ?</p>

AbdulsalamBande: <p>I think you can use <code>nn.functional.normalize()</code>  to normalize the embeddings so that they lie on a hypersphere of radius. I think the below would work</p>
<pre><code class="lang-auto">import torch.nn.functional as F
normalized_embeddings = F.normalize(embeddings, p=2, dim=1)
</code></pre>

K_awanish: <p>that’s perfect. Thanks <a class="mention" href="/u/abdulsalambande">@AbdulsalamBande</a></p>
What is the difference between VAD and Speaker Segmentation?

laro: <p>I’m not sure I can distinguish and understand the difference between:</p>
<ul>
<li>
<code>VAD</code> (Voice Activity Detection) and</li>
<li><code>Speaker Segmentation</code></li>
</ul>
<p>I understand that:</p>
<ul>
<li>VAD - split audio to <strong>segments</strong> of speech or not speech</li>
<li>Speaker Segmentation - split audio to <strong>segments</strong> of not speech and <strong>different speakers</strong>
</li>
</ul>
<p>for example:</p>
<pre><code class="lang-auto">VAD                  = [not speech, speech,  not speech,         speech,      not speech] 

Speaker Segmentation = [not speech, speech , not speech,  speech A, speech B, not speech] 
</code></pre>
<p>Am I right ?</p>
<p>Is my example correct ?</p>
Calculating the psudoinverse quickly and accurately

serafej: <p>I am interested in calculating the psudoinverse, X^p = X^T (X \cdot X^T)^{-1}, for a relatively large matrix X (ideally up to a size of (10000, 60000)). Unfortunately, I am finding this to either be (a) too slow or (b) imprecise for standard implementations. The following test script highlights the problem for a much smaller matrix.</p>
<pre><code class="lang-auto">import torch as torch
import time

def method_1(X):  # manual method
    XXt = X @ X.t()
    return XXt @ torch.inverse(XXt)

def method_2(X):  # pinv method
    return X @ torch.linalg.pinv(X)

def method_3(X):  # solve method
    return X @ torch.linalg.solve(X @ X.t(), X).t()

options = {'manual method': method_1,
           'pinv method': method_2,
           'solve method': method_3}

torch.manual_seed(0)
X = torch.randn((5000, 7500))
identity = torch.eye(5000)

for device in ['cpu', 'cuda']:
  print(f'Device: {device}')
  X = X.to(device); identity = identity.to(device)
  for name, method in options.items():
      start = time.time()
      soln = method(X)
      end = time.time()
      correctness = torch.allclose(soln, identity, rtol=0, atol=1e-5)
      tot_time = end - start 
      print(f'Method: {name}, Correct: {correctness}, Time: {tot_time}' )
</code></pre>
<p>Which outputs the following using Google Colab (Tesla T4 for the GPU)</p>
<pre><code class="lang-auto">Device: cpu
Method: manual method, Correct: True, Time: 6.604369878768921
Method: pinv method, Correct: True, Time: 37.75372791290283
Method: solve method, Correct: True, Time: 9.509742498397827
Device: cuda
Method: manual method, Correct: False, Time: 0.31755805015563965
Method: pinv method, Correct: False, Time: 19.725148677825928
Method: solve method, Correct: False, Time: 0.3164680004119873
</code></pre>
<p>Is there a way in which I could calculate this accurately and in a reasonable time for matrix X of size (10000, 60000)?</p>
Difficulty matching scikit-learn LASSO results with L1 Regularization or AdamW

jludwig: <p>Hello everyone, I’m trying to replicate some basic linear regression results from sci-kit learn’s LASSO implementation into pyTorch and finding that the solution quality is nowhere near as good. I’ve tried this two separate ways, first by explicitly adding a penalty to the loss function, similar to what is described here: <a href="https://stackoverflow.com/questions/44641976/pytorch-how-to-add-l1-regularizer-to-activations" class="inline-onebox" rel="noopener nofollow ugc">python - Pytorch: how to add L1 regularizer to activations? - Stack Overflow</a></p>
<p>The second way I’ve tried is using AdamW and effectively treating lambda (weight decay) the same way it’s used in LASSO (creating curves of train and cross-validation error as a function of lambda).</p>
<p>In both cases, I get sub-optimal results vs. sci-kit learn (i.e. the validation and test set objective functions are higher). Also, I get a substantial level of noise in the training, even with a lower learning rate (1.0e-4) and a fairly high weight decay:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f03171e853aec6713c23acf0f3012f98e1a79e4.png" alt="LASSO_example" data-base62-sha1="4qlqcfrA7LQShbwHgiXtnl8ozRy" width="588" height="442"></p>
<p>The obvious solution is to use sci-kit learn, but I’m trying both dense and LSTM models on the same datasets which show very similar issues, I thought it easiest to debug the linear model first. Thoughts?</p>
Do gradient descent once an epoch cooperating with dataloader

wan_cai: <p>hi, I am new to pytorch and I am trying to do pseudo-label on an image segmentation task. I am struggling to implement loss.backward() in the training process. From the original paper, if I am correct, the loss will be accumulated for all mini-batch in an epoch and do back propagate once an epoch. I am not sure how to implement this.</p>
<p>Original paper is on: <a href="https://www.researchgate.net/publication/280581078_Pseudo-Label_The_Simple_and_Efficient_Semi-Supervised_Learning_Method_for_Deep_Neural_Networks" rel="noopener nofollow ugc">pseudo-label paper</a></p>
About FSDP work problems

junetou: <p>I have a issue about FSDP:<br>
I have two devices and 8 gpu on each devices. Name is device1 and device2. I use pytorch 2.0 and fsdp train model.<br>
I train a 3.5B gpt2 model in a device1 or device2. It work.<br>
I train a 3.5B gpt2 model in device1 and device2, it work.<br>
I train a 4B gpt2 model in a device1 or device2. It not work.<br>
I train a 4B gpt2 model in device1 and device2, It still not work.Output:torch.cuda.OutOfMemoryError: CUDA out of memory.<br>
I want to train a 4B model in device1 and device2 use fsdp.How should I solve this problem? thx</p>
<p>This is config files about fsdp:<br>
fsdp_config:<br>
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP<br>
fsdp_backward_prefetch_policy: NO_PREFETCH<br>
fsdp_offload_params: true<br>
fsdp_sharding_strategy: 1<br>
fsdp_state_dict_type: FULL_STATE_DICT<br>
fsdp_transformer_layer_cls_to_wrap: GPT2Block</p>
<p>This is device1 output:<br>
ubuntu-SYS-4028GR-TR:30007:30101 [0] NCCL INFO comm 0x7b44ec0 rank 8 nranks 16 cudaDev 0 busId 4000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30008:30100 [1] NCCL INFO comm 0x844ccf0 rank 9 nranks 16 cudaDev 1 busId 5000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30011:30102 [4] NCCL INFO comm 0x6b29170 rank 12 nranks 16 cudaDev 4 busId 82000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30012:30095 [5] NCCL INFO comm 0x72e6670 rank 13 nranks 16 cudaDev 5 busId 85000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30013:30097 [6] NCCL INFO comm 0x67e6340 rank 14 nranks 16 cudaDev 6 busId 86000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30009:30099 [2] NCCL INFO comm 0x83e3070 rank 10 nranks 16 cudaDev 2 busId 8000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30010:30098 [3] NCCL INFO comm 0x7dce1b0 rank 11 nranks 16 cudaDev 3 busId 9000 - Init COMPLETE<br>
ubuntu-SYS-4028GR-TR:30015:30096 [7] NCCL INFO comm 0x8d140c0 rank 15 nranks 16 cudaDev 7 busId 8a000 - Init COMPLETE</p>
<p>This is device2 output:<br>
ubuntu1-NF5588M4S:7039:7128 [0] NCCL INFO comm 0x7dd2bb0 rank 0 nranks 16 cudaDev 0 busId 4000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7041:7135 [2] NCCL INFO comm 0x82337c0 rank 2 nranks 16 cudaDev 2 busId 7000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7043:7133 [4] NCCL INFO comm 0x73f0d40 rank 4 nranks 16 cudaDev 4 busId c000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7045:7132 [6] NCCL INFO comm 0x7e93c70 rank 6 nranks 16 cudaDev 6 busId e000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7047:7130 [7] NCCL INFO comm 0x8532370 rank 7 nranks 16 cudaDev 7 busId f000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7042:7131 [3] NCCL INFO comm 0x80c3180 rank 3 nranks 16 cudaDev 3 busId 8000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7044:7129 [5] NCCL INFO comm 0x8403930 rank 5 nranks 16 cudaDev 5 busId d000 - Init COMPLETE<br>
ubuntu1-NF5588M4S:7040:7134 [1] NCCL INFO comm 0x8da3d00 rank 1 nranks 16 cudaDev 1 busId 6000 - Init COMPLETE</p>
____TypeError: forward() takes 2 positional arguments but 3 were given____

nathan345345: <pre><code class="lang-auto">class EasyQAEarlyFusionNetwork(nn.Module):

    def __init__(self, hyperparms=None):

        super(EasyQAEarlyFusionNetwork, self).__init__()        
        self.dropout = nn.Dropout(0.3)
        self.vision_projection = nn.Linear(2048, 768) 
        self.text_projection = nn.Linear(768, 768) # here
        self.fc1 = nn.Linear(1536, 256) 
        self.bn1 = nn.BatchNorm1d(256)
        self.classifier = nn.Linear(256, 17) 
        W = torch.Tensor(768, 768)
        self.W = nn.Parameter(W)
        self.relu_f = nn.ReLU()
        # initialize weight matrices
        nn.init.kaiming_uniform_(self.W, a=math.sqrt(5))

        self.attention = nn.Linear(768, 768)
        
    def forward(self, image_emb, text_emb, fact_emb):

        x1 = image_emb   
        x1 = torch.nn.functional.normalize(x1, p=2, dim=1)
        Xv = self.relu_f(self.vision_projection(x1))
        
        x2 = text_emb
        x2 = torch.nn.functional.normalize(x2, p=2, dim=1)
        Xt = self.relu_f(self.text_projection(x2))

        x3 = fact_emb
        x3 = torch.nn.functional.normalize(x3, p=2, dim=1)
        Xf = self.relu_f(self.text_projection(x3))

        attention_scores = self.attention(Xt, Xf)
        attention_weights = F.softmax(attention_scores.squeeze(), dim=1).unsqueeze(2)

        # Apply attention weights to fact embeddings
        attended_facts = torch.sum(attention_weights * Xf.unsqueeze(0), dim=1)

        Xvt = Xv * Xt
        Xvt = self.relu_f(torch.mm(Xvt, self.W.t()))

        Xvtf = torch.cat([Xvt,attended_facts], dim=1)

        Xvtf = self.fc1(Xvtf)
        Xvtf = self.bn1(Xvtf)
        Xvtf = self.dropout(Xvtf)
        Xvtf = self.classifier(Xvtf)

        return Xvtf
</code></pre>

ptrblck: <p>Could you post a code snippet showing how you are calling into the model as well as the full stacktrace, please?</p>

AbdulsalamBande: <p>It seems you are getting a TypeError because the <code>self.attention</code> layer is expecting only a single input, but you are passing two inputs (Xt and Xf) to it in this line:</p>
<pre><code class="lang-auto">attention_scores = self.attention(Xt, Xf)

</code></pre>
<p>The  <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">Linear</a> layer only takes a single input</p>

nathan345345: <p>Fixed, thanks for the help!</p>
Masking inputs of variable length in SB3-contrib's MaskablePPO source code

zezimabig: <p>Hi Pytorch community,</p>
<p>I have recently posted a question, on SB3-contrib’s repository, titled " <em>What modifications do I need to mask the inputs similar to how MaskablePPO masks the outputs?</em>".</p>
<p>Unfortunately, it seems that no one has an answer to this particular question of mine. Since, SB3 and SB3-contrib are codes implemented using Pytorch, I thought that posting the question here would be of relevance and effectiveness!</p>
<p><strong>Here is the question that I posted [Avoid Replicates]:</strong><br>
<a href="https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/issues/168" rel="noopener nofollow ugc">What modifications do I need to mask the inputs similar to how MaskablePPO masks the outputs?</a></p>
<p>Any help is greatly appreciated with regards to this issue. If I must say, I am looking for suggestions for points of modification in the source code, or if anyone is able to guide me in this endeavour, I will be really thankful for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Hai everyone Can anyone give me some suggestions!

Teddy_Salas: <p>Hai everyone , i know pytorch, in pytorch i know linear regression,binary classification, multi classification,computer vision and custom datasets, is this enough to get a job as a pytorch developer?</p>

ptrblck: <p>I’m unsure if you are looking for a job to <strong>develop</strong> PyTorch directly or to <strong>use</strong> PyTorch in an ML-related job. Assuming the latter I would see PyTorch more as a tool to get the job done and would recommend checking the actual job requirements which might list specific ML skills.</p>

Teddy_Salas: <p>Thank you ptrblck, i am asking what kind ML jobs i will get if i only know pytorch</p>
How can I wait to finish training model B after model A is completed before applying loss.backward() for each epoch

tnguyen87: <p>Hello,</p>
<p>Let’s say that I have 2 models A &amp; B to train on one task, and I train both models at the same time.</p>
<p>For each epoch (or each dataloader batch), if model A completes training faster than model B, how can I write a statement so that model A can wait for model B and a loss is updated (loss.backward()) based on both models instead of either A or B. Is it necessary to write a wait statement, like sleep(), in the training loop? Thanks!</p>
<p>Best,<br>
Tommy</p>

AbdulsalamBande: <p>In PyTorch, you can achieve this by using the built-in <code>torch.utils.data.DataLoader</code> to handle your data and training both models in the same loop. You don’t need to use a wait statement or <code>sleep()</code> in this case. Instead, you can compute the loss for both models in the same loop and update the gradients based on the combined loss. Here’s an example of how to do this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

# Load your dataset and create DataLoader
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = DataLoader(trainset, batch_size=64, shuffle=True)

# Define your models A and B
model_A = ...
model_B = ...

# Define your loss function and optimizers
criterion = nn.CrossEntropyLoss()
optimizer_A = optim.Adam(model_A.parameters(), lr=0.001)
optimizer_B = optim.Adam(model_B.parameters(), lr=0.001)

# Train the models
num_epochs = 10
for epoch in range(num_epochs):
    for i, (inputs, targets) in enumerate(trainloader):
        # Zero the parameter gradients
        optimizer_A.zero_grad()
        optimizer_B.zero_grad()

        # Forward pass for both models
        outputs_A = model_A(inputs)
        outputs_B = model_B(inputs)

        # Calculate the loss for both models
        loss_A = criterion(outputs_A, targets)
        loss_B = criterion(outputs_B, targets)

        # Combine the losses
        combined_loss = loss_A + loss_B

        # Backward pass and optimization
        combined_loss.backward()
        optimizer_A.step()
        optimizer_B.step()

        # Print the losses
        print(f'Epoch [{epoch + 1}/{num_epochs}], Step [{i + 1}/{len(trainloader)}], Loss A: {loss_A.item():.4f}, Loss B: {loss_B.item():.4f}')

</code></pre>

tnguyen87: <p>Thank you so much <a class="mention" href="/u/abdulsalambande">@AbdulsalamBande</a></p>
What is difference torch.cuda.memory_allocated() vs. max_memory_allocated

JuyiLin: <p>I want to know how my GPU is used for my gradients and optimizer state;<br>
which function should I use?<br>
And I wonder where I should place this function. Is there any good demo?</p>

JuyiLin: <p>Now I know <code>memory allocated</code> is the current state.<br>
But I still don’t know when I should use <code>max_memory_allocated</code>.</p>

ptrblck: <p><code>max_memory_allocated</code> will show the peak memory usage for allocated memory since the beginning of the script execution or since you have explicitly reset the peak via <code>reset_peak_memory_stats()</code>.<br>
This can be useful to check e.g. how high the peak usage is during the backward pass, as it might be tricky to add <code>memory_allocated()</code> calls into the backward pass directly e.g. via hooks.</p>
How to use at::_upsample_nearest_exact2d?

3togo: <p>auto size = img_tensor.sizes();<br>
int w = size[0], h = size[1];<br>
c10::ArrayRef&lt;int64_t&gt; hw_t({h, w});<br>
c10::ArrayRef scale_t({384. / h, 384. / w});<br>
img_tensor = at::_upsample_nearest_exact2d(img_tensor, hw_t, scale_t);</p>
<p>below is the error dump<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  Must specify exactly one of output_size and scale_factors<br>
Exception raised from compute_output_size at …/aten/src/ATen/native/UpSample.cpp:18 (most recent call first):</p>
<p>if I change the code to<br>
int w = size[0], h = size[1];<br>
c10::ArrayRef&lt;int64_t&gt; hw_t({h, w});<br>
c10::ArrayRef scale_t({384. / h, 384. / w});<br>
img_tensor = at::_upsample_nearest_exact2d(img_tensor, c10::nullopt, scale_t);</p>
<p>I got this error<br>
what():  Expected static_cast&lt;int64_t&gt;(scale_factors-&gt;size()) == spatial_dimensions to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
<p>how to fix it?</p>

AbdulsalamBande: <p>The issue you’re encountering is due to the mismatch between the number of spatial dimensions and the size of the <code>scale_factors</code> array. Your input image tensor seems to have 2 spatial dimensions (height and width), so the <code>scale_factors</code> array should have 2 elements as well.<br>
Here’s how you can fix it:</p>
<ol>
<li>First, make sure that you’re correctly extracting the height and width dimensions from the tensor shape. You should ensure that the first dimension of your tensor is the height and the second is the width.</li>
</ol>
<pre><code class="lang-auto">auto size = img_tensor.sizes();
int h = size[0], w = size[1];

</code></pre>
<ol start="2">
<li>Then, create the <code>scale_factors</code> array with the correct number of elements:</li>
</ol>
<pre><code class="lang-auto">c10::ArrayRef&lt;double&gt; scale_t({384.0 / h, 384.0 / w});

</code></pre>
<p>Note that I changed <code>scale_t</code> to <code>c10::ArrayRef&lt;double&gt;</code> instead of <code>c10::ArrayRef</code>.</p>
<ol start="3">
<li>Finally, call the <code>_upsample_nearest_exact2d</code> function with the <code>output_size</code> set to <code>c10::nullopt</code> and the correct <code>scale_factors</code>:</li>
</ol>
<pre><code class="lang-auto">img_tensor = at::_upsample_nearest_exact2d(img_tensor, c10::nullopt, scale_t);

</code></pre>
<p>Now, the function should execute without throwing any exceptions, and your image tensor should be correctly resized using nearest-neighbor upsampling.</p>
DataLoader not callable

bornexmachina: <p>My custom dataset has the following in the <strong>init</strong>()</p>
<pre><code class="lang-auto">...
self.train_loader = torch.utils.data.DataLoader(
    self.train_set,
    batch_size=batch_size, shuffle=shuffle, batch_sampler=batch_sampler,
)
</code></pre>
<p>now I assume, that when in my modeling process I gonna use this DataLoader, I should be able to give <code>batch_size</code> to the arguments</p>
<p><code>train_loader = dataset.train_loader(batch_size=self.batch_size)</code></p>
<p>where dataset is my custom Dataset. But I get</p>
<pre><code class="lang-auto">train_loader = dataset.train_loader(batch_size=self.batch_size)
TypeError: 'DataLoader' object is not callable
</code></pre>
<p>Why and how to fix it?</p>

AbdulsalamBande: <p>The <code>TypeError</code> occurs because you are trying to call the <code>train_loader</code> object like a function. Instead, you should create a method within your custom dataset class to return a new DataLoader with the desired batch size. Here’s how you can do it:</p>
<ol>
<li>In your custom dataset class, remove the initialization of <code>self.train_loader</code> from the <code>__init__</code> method.</li>
<li>Add a new method to the custom dataset class that returns a DataLoader with the given batch size:</li>
</ol>
<pre><code class="lang-auto">class MyCustomDataset(torch.utils.data.Dataset):
    def __init__(self, ...):
        # Your dataset initialization code here

    def get_train_loader(self, batch_size, shuffle=True, batch_sampler=None):
        train_loader = torch.utils.data.DataLoader(
            self.train_set,
            batch_size=batch_size, shuffle=shuffle, batch_sampler=batch_sampler,
        )
        return train_loader

</code></pre>
<ol start="3">
<li>Now, in your modeling process, you can get a train_loader with the desired batch size like this:<br>
train_loader = dataset.get_train_loader(batch_size=self.batch_size)</li>
</ol>
<pre><code class="lang-auto">train_loader = dataset.get_train_loader(batch_size=self.batch_size)
</code></pre>
<p>By using this approach, you can create a DataLoader with the specified batch size when needed without getting the <code>TypeError</code></p>
Indexing 3D input matrix using 2D index

anupsingh15: <p>Hey all!</p>
<p>I want to index an input 3D matrix of size, say, (3, 4, 2) using a 2D index matrix of size (3, 4).  Each row of index matrix is a combination of 0s and 1s.</p>
<pre><code class="lang-auto">For example: 
Input matrix (A) = torch.tensor([[[1,2],[3,4],[0,4],[9,2]], [[5,6],[7,8],[8,8],[7,6]], [[9, 10],[11,12],[1,9],[2,2]]])
index matrix (B) = torch.tensor([[1,1,0,0], [0,0,0,1], [1,0,0,0]])
Want: torch.tensor([[2,4,0,9], [5,7,8,6], [10,11,1,2]])
</code></pre>
<p>I am using an inefficient way as:<br>
<code>torch.cat([A[i][torch.arange(B.size(1)).unsqueeze(0), B[i]] for i in range(B.size(0))], dim=0)</code></p>
<p>It would be nice if someone could please suggest a neat way to get the above.</p>

AbdulsalamBande: <p>To index a 3D matrix using a 2D index matrix, you can use the <code>torch.gather</code> function. Here’s how to do it:</p>
<pre><code class="lang-auto">import torch

# Create the input 3D tensor (A)
input_matrix = torch.tensor([[[1, 2], [3, 4], [0, 4], [9, 2]],
                             [[5, 6], [7, 8], [8, 8], [7, 6]],
                             [[9, 10], [11, 12], [1, 9], [2, 2]]])

# Create the index 2D tensor (B)
index_matrix = torch.tensor([[1, 1, 0, 0],
                             [0, 0, 0, 1],
                             [1, 0, 0, 0]])

# Index the input tensor using the index tensor
output_matrix = torch.gather(input_matrix, 2, index_matrix.unsqueeze(2))

# Remove the extra dimension
output_matrix = output_matrix.squeeze(2)

print(output_matrix)

</code></pre>
CPU utilization of main process is very high!

lkp411: <p>Hi there, I am noticing that the CPU utilization of my main process is much much higher than that of any of my dataloader worker processes even though I am doing almost no computation in the main process and merely pushing data from host to device. Moreover, the larger I set the value of num_workers to, the larger this value of main process CPU utilization is. Is this expected behavior and/or can anybody explain why this is happening?</p>

eqy: <p>You might want to check how much CPU load is incurred from the host-to-device copies via a microbenchmark, as this could be expected especially if nonblocking copies are used.</p>

lkp411: <p>I am using non_blocking calls in this case and it’s still very high</p>

lkp411: <p>And it turns out, the CPU utilization for the main process reaches over 100% even when I use just a single background worker in my dataloader, when the only true CPU involvement in the main process is host-device transfer of tensors. For context, every iteration involves 1.1GB of data transfer from host to device at about 10GB/s bandwidth which is still much smaller than the peak theoretical bandwidth for the PCIe GH100.</p>

lkp411: <p><a class="mention" href="/u/eqy">@eqy</a> Any ideas as to why this might be happening?</p>
Intel MKL FATAL ERROR: Cannot load libmkl_core.dylib

Autumn_Thompson: <p>Hi! I am new and just installed pytorch via anaconda. I used the steps listed on the pytorch website and even tried reinstalling following those instructions as well. My installation of pytorch goes smoothly but when I try running the verification code, I get the error listed: Intel MKL FATAL ERROR: Cannot load libmkl_core.dylib. I am using a Mac. Whenever I have researched this topic in the past, the most similar solution I have seen is: <a href="https://github.com/JuliaPy/PyPlot.jl/issues/315" rel="nofollow noopener">https://github.com/JuliaPy/PyPlot.jl/issues/315</a> but I am still a beginner and all the comments on this site seem incredibly advanced ): thank you in advance!</p>

Mikele: <p>It seems youre missing the intel core math libraries from what I gather. Have you already tried installing mkl in the same environment that you installed pytorch in, i.e.:<br>
<code>conda install mkl</code> in your terminal.</p>

Autumn_Thompson: <p>I just tried that and it downloaded, but whenever I restarted anaconda, the same error was given. Here’s the expanded code if it’s helpful:</p>
<p>Adapting from protocol version 5.1 (kernel 79056037-3763-439f-b1f0-27896a9bb6e1) to 5.3 (client).</p>
<p>INTEL MKL ERROR: dlopen(/Users/name/anaconda3/lib/libmkl_core.dylib, 9): image not found.</p>
<p>Intel MKL FATAL ERROR: Cannot load libmkl_core.dylib.</p>

Cami_Williams: <p>Hey Autumn! Have you tried:</p>
<pre><code class="lang-auto">conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service
</code></pre>
<p>I have had similar issues myself trying to run PyTorch locally. If you can try using Google colab or Jupyter notebooks instead for experimentation. You can switch the runtime type’s hardware accelerator to CPU/GPU/TPU which (I have found) sometimes is all it takes. The cloud notebooks make setup a lot easier.</p>

Lorsmo: <p>Thank you Cami! That worked fine for me!</p>

Jeff_Katz: <p>that saved me, thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hieu_Nguyen1: <p>I get this issue too and i fixed it. Thanks so much! <a class="mention" href="/u/autumn_thompson">@Autumn_Thompson</a>, <a class="mention" href="/u/cami_williams">@Cami_Williams</a></p>

edguba: <p>It took me an hour to install and remove as per your instructions.</p>
<p>Unfortunately my problem still persists.</p>
<p>I am using Jupyter.</p>

Abu_Tauheed: <p>This link gives a detailed explanation for the problem in question from multiple aspects along with solutions.</p>
<p><a href="https://docs.anaconda.com/mkl-optimizations/index.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.anaconda.com/mkl-optimizations/index.html</a></p>
<p>Enjoy!</p>
What causes "skipping cudagraphs due to input mutation"?

jack_S: <p>This message <code>torch._inductor.utils: [WARNING] skipping cudagraphs due to input mutation</code> appeared when I tried to train <code>resnet50</code> with <code>CIFAR100</code>. I set a fixed batchsize 500, which can divide 50000. To be honest, I don’t clearly understand what ‘input mutation’ means here. I just thought if the shape of the input tensor is fixed then it’s ok.</p>

ptrblck: <aside class="quote no-group" data-username="jack_S" data-post="1" data-topic="176427">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_s/48/59032_2.png" class="avatar"> jack_S:</div>
<blockquote>
<p>I just thought if the shape of the input tensor is fixed then it’s ok.</p>
</blockquote>
</aside>
<p>I would assume the same and you should check when this warning is raised.<br>
E.g. often the last batch of the <code>DataLoader</code> might be smaller, which could already be seen as a mutation of the computation graph.</p>

jack_S: <p>Thanks, but the shape of my input tensors is fixed. I checked my code and found this warning appeared when I tried to calculate the accuracy of my model (at the first epoch). Is this just because I set <code>with torch.no_grad()</code> ?</p>
Finetuning GPT2 for text to text generation

msabrii: <p>Hi!</p>
<p>I am trying to finetune gpt 2 for a project. I have a dataset of Reddit posts with one comment each and the task is to generate a comment for a given post. I think this is a conditional text generation task but I am not sure how to use gpt2 to do this. I was trying to use the transformers library to do so, but I am not sure how to format the data. There is not a lot online for text to text generation using gpt2. Should the model be given just the post or the post and the comment as input? Not sure how this works, can someone please give me some instruction/advice?</p>
<p>Thanks</p>

J_Johnson: <p>Most text to text generation are trained on next <strong>token</strong> prediction. Along with making use of bos and eos tokens. For instance, you might give the model post + bos + comment + eos and mask after bos.</p>
<p>Please consider taking this free online course on Huggingface on NLP. It might fill in some of the gaps.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/course/chapter1/1">
  <header class="source">

      <a href="https://huggingface.co/course/chapter1/1" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/course.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/course/chapter1/1" target="_blank" rel="noopener nofollow ugc">Introduction - Hugging Face Course</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Automatic differentation and Slicing

spike354: <p>Hello,</p>
<p>lets say my network produces an output x as an 1D-tensor with [1,5,2,10]. If I slice it into two different 1D-tensors like a = x[0:1], b=x[2:3] and put them in a loss function. Will the automatic differentiation algorithm be able to do the differentiation in my network? how should the automatic differentiation algorithm know that these tensors are from my network?</p>

soulitzer: <p>Yes, autograd is still able to construct the graph in this case, as in the case with any viewing and reshaping operation. Autograd is able to do this because slicing is overloaded by pytorch, so any call to slice will have autograd logic interposed to build the backward graph.</p>

spike354: <p>Can I autograd some Tensor automatically, if it can’t reccronstruct the graph?</p>

soulitzer: <p>Not completely sure what you mean by that, but maybe you are looking for <a href="https://pytorch.org/docs/stable/autograd.html?highlight=autograd+function#torch.autograd.Function" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.0 documentation</a>?</p>
Torch 2 installed - Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory

FurkanGozukara: <p>Hello. I have installed torch 2 via this command on runpod io instance</p>
<p>everything installed perfectly fine</p>
<p>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
<p>with torch 1 we were not getting any error but with torch 2 we are getting</p>
<p>how to fix?</p>
<p>it is using unix</p>
<p>Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory</p>
<p>On Windows same prodecure working very well</p>
<p>We are using automatic1111 web ui to use stable diffusion</p>

FurkanGozukara: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any ideas how to solve?</p>

ptrblck: <p>Sorry, for the late reply as I missed the ping.<br>
You are running into <a href="https://discuss.pytorch.org/t/how-to-properly-set-up-an-conda-environment-with-pytorch/176437/2">this issue</a> and could apply the workaround by creating the symbolic link. I’m unsure if this is already fixed in the current nightly, but the next minor/patch release should fix it.</p>

FurkanGozukara: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="176167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>but the next minor/patch release should fix it.</p>
</blockquote>
</aside>
<p>this is very good to here<br>
when can we know that patch will be released? or how can we know when it is released?</p>

ptrblck: <p>It will be announced as usual and will show up in the install matrix.</p>

FurkanGozukara: <p>where can i follow it can you give link</p>
Use index tricks to avoid second memmap file

HelixPiano: <p>Hello everyone,<br>
I have a question about the way pytorch handles indices.<br>
My memmap file is really huge (~70Gb). I want to train my CNN on a certain subset of this memap which contains around half the data points which are not evenly distributed through the data set. I don’t want to create a second memmap file because this would take up much disk space (especially since I already have other memmap files of similar size).<br>
So my idea is to do something like this:</p>
<ol>
<li>Get the length of my memmap subset (e.g. 10k points) and store the corresponding indices (1,4,7,9, etc.) .</li>
<li>Give my data class the subset length as length of the dataset (10k points)</li>
<li>Whenever the dataloader requests certain indices I simply look up the indices from the stored indices of the subset.</li>
</ol>
<p>Does something like this work or does it require a different workaround?</p>

ptrblck: <p>Your workflow describes what <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset"><code>torch.utils.data.Subset</code></a> does and sounds reasonable.</p>
Bigger batch size slows down epoch training time

Yazan_Maarouf: <p>I am training a MobileNetV3 + DeepLabV3 large (taken directly from Pytorch)</p>
<pre><code class="lang-auto">    opt_model = torchvision.models.segmentation.deeplabv3_mobilenet_v3_large(
        progress=True,
        num_classes=1,
    ).cuda()
</code></pre>
<p>I have a custom dataset I am using for body segmentation. When increasing batch size from 32 images to 256, the time it takes to go through one epoch increases. I tested to see which part of the iterations in the epoch took most of the time and it was these 2 :</p>
<pre><code class="lang-auto">            scaler.scale(loss).backward()  # type: ignore
            scaler.step(optimizer)
</code></pre>
<p>I am not sure what’s going on. My assumptions are that with a bigger batch size, the faster it will be to go through 1 epoch.</p>
<h1>
<a name="solution-was-found-1" class="anchor" href="#solution-was-found-1"></a>Solution was found</h1>
<p>The default pytorch model I was using has a bottleneck</p>

ptrblck: <p>Did you create a visual profile to narrow down the bottleneck or how did you isolate these lines?</p>

Yazan_Maarouf: <p>I too the time of each line to execute</p>

ptrblck: <p>I still don’t know how exactly you’ve profiled the code, but be careful with host timers in case you are using the GPU as CUDA operations are executed asynchronously. You would thus need to synchronize the code before starting and stopping your timers.</p>
GATorch: An Energy-Aware PyTorch Extension

dajtmullaj: <p>We deployed <strong>GATorch</strong>! This is a python library seamlessly integrated with PyTorch that enables ML developers to generate an energy consumption report. By attaching your model, the tool automatically tracks the energy consumption of your model’s training and generates graphs and plots to gain in-depth insights into the energy consumption of your model.</p>
<p>Here is an example of the plots you can generate:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccd4dd518ccb6ef5e557c0eba9eb6070cbc2ada4.png" alt="total" data-base62-sha1="te1sM3s7bhoG8Ei9o37XucvTA9K" width="640" height="480"></p>
<p>We integrated tensorboard functionalities for an in depth analysis of your models energy consumption. Through the tensorboard dashboard it is possible to visualize how much energy your model consumes per forward pass or how much energy is needed to decrease your loss by a constant amount.<br>
We hope this tool can be another step towards developing energy aware models!</p>
<p>You can also consult the <a href="https://github.com/GreenAITorch/GATorch" rel="noopener nofollow ugc">Official GATorch Repository </a> and its <a href="https://gatorch.readthedocs.io/en/latest/index.html" rel="noopener nofollow ugc">Manual </a>.</p>
Problems in model init method

dlxdlxdlx: <p>Friends help me! Recently I was running a paper with open source model code, and after some structural changes, I changed the xavier init method to kaiming init method, and the final result of the model improved by 3 points(comparied with xavier init method), is this possible? Or is it possible that there is a problem that causes the final result to be wrong.</p>

thecho7: <p>Initial value is just the initial.<br>
Those methods are proposed to prevent the gradient explosion/vanishing and something.<br>
I think there’s no critical difference between them.</p>
<p>No problem:)</p>

dlxdlxdlx: <p>But why is the final result so different, without changing the other structure of the model.</p>

thecho7: <ul>
<li>Train more epochs</li>
<li>Do many trainings to validate stable result.</li>
<li>Some tasks show the different results depend on the initializer</li>
</ul>
<p>Would help.</p>

dlxdlxdlx: <p>Thanks a lot! I will try this methods</p>
Adaptive, global, extended kalman filters for training feedforward neural networks

ArkadyBogdanov: <p>Hello,</p>
<p>could anybody help me how to implement the extended kalman filter to train a neural network in PyTorch based on this paper <a href="https://doi.org/10.1002/cem.1180080605" rel="noopener nofollow ugc">https://doi.org/10.1002/cem.1180080605</a> ?</p>
<p>I’m a noob in pytorch and am having really hard time coding it.</p>
<p>Thank you</p>
About the version of mmcv corresponding to torch 2.0.0 and cuda118

wx0618: <p>Hello everyone!<br>
I am trying <code>from .iou3d import iou3d_nms_cuda</code> and an error is raised as follows</p>
<pre><code class="lang-auto">iou3d_nms_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: 
_ZNK3c1010TensorImpl36is_contiguous_nondefault_policy_implENS_12MemoryFormatE
</code></pre>
<p>From the <a href="https://github.com/open-mmlab/mmcv/issues/2514" rel="noopener nofollow ugc">site</a> I know that there is no <code>mmcv-full</code> package in my environment. However, when I turn to Docs of <code>mmcv-full</code>, there is no corresponding version to torch2.0.0+cuda118, the environment in my computer.</p>
<p>Then I tried to <code>pip install mmcv-full==1.7.1</code> which is the newest version, but that doesn’t work.</p>
<p>So what can I do to solve this problem? Or should I create a new environment with lower version of torch and cuda?</p>

ptrblck: <p><code>is_contiguous_nondefault_policy</code> was removed in <a href="https://github.com/pytorch/pytorch/commit/2896f81dd4f51062f4bf5338e31d15cc3c599eb7">this PR</a> in May 2022 and thus is not available anymore since <code>torch&gt;=1.12.0</code>. I don’t know if the authors of the <code>mmcv</code> repository are rebuilding their binaries using their latest release, which might be compatible to the current PyTorch version, but based on their <code>README</code> it seems their latest pip wheel supports <code>torch==1.8.0+cu102</code>, so you might need to build <code>mmcv</code> from source.</p>

wx0618: <p>Now I’m going to create a virtual env to satisfy the version requirements.</p>