akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Re-using intermediate features without second forward pass

zelazny: <p>I have a setup with two networks, a <code>classifier</code> and a <code>predictor</code>. The <code>classifier</code> has two outputs, the class predictions and the intermediate features, <code>cls, f = classifier(x)</code>. The <code>predictor</code> simply outputs a scalar based on those intermediate features <code>pred = predictor(f)</code>.</p>
<p>Now, I would like to train both networks on the following objective: <code>obj = CE + MSE</code>, where <code>CE</code> is the cross-entropy of the <code>classifier</code> and <code>MSE</code> is the mean squared error of the <code>predictor</code>. The training is adversarial, i.e. the <code>classifier</code> attempts to both lower the <code>CE</code> and increase the <code>MSE</code>, while the <code>predictor</code> attempts to lower the <code>MSE</code>. The relevant snippet of the training procedure:</p>
<pre data-code-wrap="python"><code class="lang-python">cls, f = classifier(x)

predictor.eval()
with torch.no_grad():
    pred = predictor(f)
classifier_loss = ce_loss(cls, class_target) - mse_loss(pred, prediction_target)
classifier_loss.backward()

predictor.train()
pred = predictor(f)
predictor_loss = mse_loss(pred, prediction_target)
predictor_loss.backward()
</code></pre>
<p>Needless to say, this requires double computation of <code>pred = predictor(f)</code>. GAN-style <code>detach</code>ing does not seem to be applicable here.</p>
<p>Is there any way to perform these two backward passes without recomputing <code>pred = predictor(f)</code> or is it strictly necessary to construct two computation graphs and this part cannot be circumvented?</p>
Pytorch Installation CUDA 11.4

pyt: <p>Hi, I have a computer with ubuntu 20.04, CUDA 11.4 and NVIDIA drivers 470.<br>
I don’t have the permissions to update it. Which version of Pytorch could I install without having to update the drivers and CUDA?</p>
<p>According to this post<a href="https://discuss.pytorch.org/t/which-pytorch-version-2-0-1-support-cuda-11-4/190446">https://discuss.pytorch.org/t/which-pytorch-version-2-0-1-support-cuda-11-4/190446</a> I can’t install versions compiled with CUDA 12.1 but I can with CUDA 11.8. Is this correct and why?</p>
<p>Thanks in advance.</p>

ptrblck: <p>Yes, since the PyTorch binaries ship with their own CUDA 11.8 runtime and would use the NVIDIA driver from your system which should be compatible with 11.8.</p>
Freeze 3D backbone before nn.MultiheadAttention

Byte247: <p>Hello,</p>
<p>I try to train a multimodal system using nn.MultiheadAttention for cross-attention. When I freeze the 3D backbone responsible for one of the branches the model fails to learn any useful relationships. However it does seem to learn if the layers are trainable. Why does this happen? As far as I can see from the source code there is an embedding layer included in nn.MultiheadAttention so it should still be able to train this without having to rely on previous layers in the network?</p>
Is it a good idea to use cross entroy to implement InfoNCE?

Mehran_Ziadloo: <p>Sorry if my question is too obscure. To provide a short description of what InfoNCE is, it’s a loss function used in SimCLR paper which is a contrastive training used in a joint embedding model. This means the output of the model will be a network that outputs an embedding per each input sample. If the model is trained correctly, the embeddings of similar inputs will be close to each other while the embedding of dissimilar inputs will be farther away from each other (usually based on cosine similarity). The job of InfoNCE is to implement this push and pull.</p>
<p>The important part of contrastive training is that each batch should come in pairs. These pairs will be of samples similar to each other while the rest of the pairs in the batch are considered dissimilar to the other pairs. Each sample in the pair will be fed to the network which will generate the embedding for that sample. After normalizing each outputs as part of the loss function, we’ll calculate the matrix multiplication of all the samples within the batch. This will give us a square matrix by the size of number of pairs in the batch and each cell of the this matrix will hold the cosine similarity between the first item of the pair in the column with the second sample of the pair in the row column of that cell. Of course, if the network is trained perfectly, this matrix will be an identity matrix.</p>
<p>Now, my question is, if I have the cosine similarity matrix for the batch and I want to calculate a loss value to push the matrix to identity matrix, do you think the following implementation should work?</p>
<pre data-code-wrap="python"><code class="lang-python">def info_nce_loss(out1, out2):
    # Normalizing the output vectors so their length is one
    out1 = torch.nn.functional.normalize(out1, p=2, dim=-1)
    out2 = torch.nn.functional.normalize(out2, p=2, dim=-1)
    # Calculating similarity scores
    scores = torch.mm(out1, out2.T)
    # Labels is a vector holding the diagonal indices of an identity matrix
    labels = torch.arange(scores.size(0)).to(scores.device)
    return torch.nn.functional.cross_entropy(scores, labels)
</code></pre>
<p>I’ve been using this implementation in my experiment. In some scenarios it works and in some scenarios it does not which makes me wonder whether I should use a different implementation for InfoNCE.</p>
<p>Any suggestion is very much appreciated.</p>
<p>P.S. If anyone’s interested, this is the <a href="https://arxiv.org/abs/2002.05709" rel="noopener nofollow ugc">SimCLR paper</a></p>
Realtime object detection

sahar.mi: <p>Hi everyone. I’m working on a simple object detection project, in this project I read every frame from my camera, and process frames, detect cars and then plot license plates on those frames, so I have a loop for every frame and do the detection with yolov7. this takes time and when I run the program the speed of video gets too slow, how can I make it fast? is there a problem with my architecture? what can I do to run my program real time? I’m working on nvidia RTX3060, I’ll appreciate any suggestion.</p>
Why am I getting segfault when loading an AOT compiled model twice?

rbavery: <p>I’m getting an error when I load an AOT compiled model twice. in between the loads I unzip the model file over the old one that was loaded first. I would expect this to work with models that are fully loaded into memory but is this not how AOT compiled models work? I don’t really know the ins and outs of dynamic shared libraries.</p>
<p>Below is my code that segfaults on the second load after unzipping the model over the first model file.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import zipfile
import os


def load_pt2_model_from_local_zip(zip_file_path: str, device: str):
    """Function to unzip a local ZIP file correctly and load a model.

    Args:
        zip_file_path (str): Local path of the ZIP file containing the model.
        device (str): The device to load the model onto.

    Returns:
        Callable: The loaded model.
    """
    zip_file_name = os.path.basename(zip_file_path)
    extract_dir_base = zip_file_name.rsplit(".", 1)[0]
    save_dir = os.path.join("/opt/workspace", extract_dir_base)

    os.makedirs(save_dir, exist_ok=True)

    with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
        zip_ref.extractall(save_dir)

    model = torch._export.aot_load("/opt/workspace/aot_inductor_gpu_tensor_cores/solar_satlas_sentinel2_model_pt2.so", "cuda") # segfault error is here

    return model


model1 = torch._export.aot_load("/opt/workspace/aot_inductor_gpu_tensor_cores/solar_satlas_sentinel2_model_pt2.so", "cuda")
model2 = load_pt2_model_from_local_zip("/opt/workspace/aot_inductor_gpu_tensor_cores.zip", "cuda") # segfault error is here

print("done") # never reached

</code></pre>

marksaroufim: <p>Hey <a class="mention" href="/u/rbavery">@rbavery</a> I’d post this on github and tag the aot inductor folks - since this is all in private namespaces I doubt there’s much expertise on this issue here</p>
How to make the GCN output to be of binary type

ElenaRz: <p>Hello. I am very new in PyTorch graph Neural Network. I have multiple graphs and I want to do node classification. I divided the graphs to 80% of them for training and 10% for evaluation and 10% for testing. I tried to use GCN to classify nodes. To do that, I implemented the following code:</p>
<p>class GCN(torch.nn.Module):<br>
def <strong>init</strong>(self, in_channels, hidden_channels, out_channels):<br>
super().<strong>init</strong>()<br>
self.conv1 = GCNConv(in_channels, hidden_channels)<br>
self.conv2 = GCNConv(hidden_channels, out_channels)<br>
# Define the linear layer<br>
self.linear = torch.nn.Linear(out_channels, out_channels)</p>
<pre><code>def forward(self, x,edge_index,edge_weight):
    # x, edge_index = data.x, data.edge_index

    x = self.conv1(x, edge_index,edge_weight)
    x = F.relu(x)
    x = F.dropout(x, training=self.training)
    x = self.conv2(x, edge_index, edge_weight)
    # Apply the ReLU activation function
    x = F.relu(x)
    # Apply the linear layer
    x = self.linear(x)
    return F.log_softmax(x, dim=1)
</code></pre>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>in_channels = gInfo.getNodeNumberOfFeatures() # Number of input features per node</p>
<p>hidden_channels = gInfo.getNodeNumberOfFeatures()+4 # Number of output features for the first convolutional layer</p>
<p>max_num_classes = 2 # Since the target output can be either 0 or 1</p>
<h1><a name="initialize-the-gcn-model-with-the-maximum-number-of-classes-1" class="anchor" href="#initialize-the-gcn-model-with-the-maximum-number-of-classes-1"></a>Initialize the GCN model with the maximum number of classes</h1>
<p>model = GCN(in_channels, hidden_channels, max_num_classes).to(device)</p>
<h1><a name="use-binary-cross-entropy-loss-for-scenarios-where-target-has-only-one-class-2" class="anchor" href="#use-binary-cross-entropy-loss-for-scenarios-where-target-has-only-one-class-2"></a>Use Binary Cross Entropy Loss for scenarios where target has only one class</h1>
<p>binary_loss_op = torch.nn.BCEWithLogitsLoss()</p>
<p>criterion = torch.nn.BCELoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=5e-4)</p>
<p>def train():<br>
model.train()</p>
<pre><code>total_loss = 0
for data in train_loader:
    data = data.to(device)
    data.requires_grad_(True)
    optimizer.zero_grad()
    output = model(data.x, data.edge_index, data.edge_weight)
    # Calculate loss
    loss = binary_loss_op(output, data.y.squeeze(dim=1))
    total_loss += loss.item() * data.num_graphs
    loss.backward()
    optimizer.step()
return total_loss / len(train_loader.dataset)
</code></pre>
<p>Now, here for the loss part, the output has 2 dimensions and the data.y is only 0 or 1. I want to know how can I output to become 0 or 1 so to be able to compare with data.y?</p>
Where is the cuLaunchKernel in Pytorch v1.7 static graph mode

zal-orz: <p>I am reading the source code of PyTorch v1.7, but when debugging the C++ source code, I cannot find the specific kernel function, such as when I use static graph mode and set breakpoints on cuLaunchKernel throughout the entire source code, the breakpoints are not triggered. I want to know which file or function specifically schedules operators to run on the GPU</p>
<p>here is my test code:</p>
<pre><code class="lang-auto">import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = torch.nn.Linear(8000, 2000)
        self.fc2 = torch.nn.Linear(2000, 1000)
        self.relu = torch.nn.ReLU()
        self.fc3 = torch.nn.Linear(1000, 100)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.relu(x)
        return self.fc3(x)


model = MyModel().to(device)

input_data = torch.randn(8000, 8000).to(device)

scripted_model = torch.jit.trace(model, input_data)

output = scripted_model(input_data)

print(output)

</code></pre>
Issue of weight and biases in fully connected linear layers and CNN

Vaibhav_Chahar: <p>Hi,<br>
In the past I was working with fully connected linear layers. there I noticed that each batch has different weights and biases but now I am working with CNN there for all the batches weights and biases are same. Can you explain me why is this happening?</p>

ptrblck: <aside class="quote no-group" data-username="Vaibhav_Chahar" data-post="1" data-topic="200078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vaibhav_chahar/48/60910_2.png" class="avatar"> Vaibhav_Chahar:</div>
<blockquote>
<p>there I noticed that each batch has different weights and biases but now I am working with CNN there for all the batches weights and biases are same.</p>
</blockquote>
</aside>
<p>I don’t understand this claim as “batch” usually refers to the input batch and does not provide any weights and biases, so could you explain in more detail what exactly you are referring to?</p>
More efficient norm of gradient computations using vmap

matekrk: <p>Hi, currently I compute the norms of the gradients w.r.t. model parameters of loss function applied individually per example in the following way:</p>
<pre><code class="lang-auto">vmap_loss = torch.vmap(compute_loss_for_single_instance, in_dims=(None, None, 0, 0))
losses = vmap_loss(network, loss_fn, X, y)
norm_gradients = [compute_grad_norm(torch.autograd.grad(loss, network.parameters(), retain_graph=True)).cpu().numpy() for loss in losses]
</code></pre>
<p>where I define the auxiliary methods:</p>
<pre><code class="lang-auto">def compute_loss_for_single_instance(network, loss_function, image, label):
    y_pred = network(image.unsqueeze(0))
    loss = loss_function(y_pred, label.unsqueeze(0))
    return loss

def compute_grad_norm(grads):
    grads = [param_grad.detach().flatten() for param_grad in grads if param_grad is not None]
    norm = torch.cat(grads).norm()
    return norm
</code></pre>
<p>It works, but takes a while.<br>
Is there a way to compute norm_gradients more efficiently?</p>
<p>Many thanks</p>
What should I put in my bbox when there is no object to detect in the image?

4bach: <p>Hi everyone, i have a custom dataset for a binary object detection problem. I use the pytorch model ’ `` <code>fasterrcnn_resnet50_fpn</code>. For this, as it said in the doc i need a target dictionnary containing ‘boxes’ and ‘label’.<br>
There is an error in case my class is 0 (no object to detect), so I have no bbox to put in targets[‘boxes’].<br>
As a temporary solution I set a value of bbox=[0,0,0,0] when there isn’t object in my image.<br>
Is it a correct way to do ?</p>
<p>def my_transform(self,img,bboxes,label):</p>
<pre><code>    if self.transforms is not None and self.train:
        transformed = self.transforms(image=np.array(img),bboxes=bboxes) # bbox can be an empty list 
        img = transformed['image']
        bboxes = transformed['bboxes']
    img = TF.to_tensor(img)
    if self.normalize:
        img = TF.normalize(img,self.mean,self.std)
    label = torch.LongTensor(label)
    bboxes = [l[:4] for l in bboxes] #we dont need name of the object anymore 
    bboxes = torch.FloatTensor(bboxes)
    targets = {'boxes':bboxes,'labels':label}
    return img, targets

def __getitem__(self,index):
    img = Image.open(os.path.join(self.root_images,self.file_names[index]+'.png')).convert('RGB')
    with open(os.path.join(self.root_annotation,self.file_names[index]+'.json'),'r') as f:
        annotation = json.loads(f.read().strip())
    bboxes = []
    label = []
    try:
        for b in annotation['outputs']['object']:
            bbox = [i for i in b['bndbox'].values()] #xmin ymin xmax ymax #VOC Format
            bbox.append(b['name']) # Le label ici est exclusivement drone pour l'instant 
            bboxes.append(bbox)
            label.append(1)
    except:
        raise RuntimeError('Something wrong with the annotation format')
    
    if len(annotation['outputs']['object']) == 0:
        label = [0] # if no object label is 0
        bboxes = [[0,0,0,0]]
    img, targets = self.my_transform(img, bboxes,label)
    return img,targets
</code></pre>
<p>Thank you for your help</p>

gifted_spence: <p>Hi <a class="mention" href="/u/4bach">@4bach</a>, I’m facing a similar problem. I want to train my model with some unannotated images and I don’t know what to do with them. I tried to stick to the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial</a> where a dictionary is created for the target that contains bounding boxes and labels among other things. However, it is not really about <code>empty images</code>.</p>
<p>Have you found a solution? Have you tried setting your bounding boxes to <code>None</code> or something like that? I tried to use <code>bbox = torch.zeros((0, 4), dtype=torch.float32)</code> without success.</p>
Speedup with max_autotune even though all the triton mm kernels are slower

ad8e: <p>I’m comparing these two codepaths:<br>
<code>torch.compile(my_function, options={"max_autotune"})</code></p>
<p><code>torch.compile(my_function)</code></p>
<p>And <code>max_autotune</code> is faster. But in the logs, the triton kernels are all at 50% speed and look like this:</p>
<pre><code class="lang-auto">SingleProcess AUTOTUNE takes 3.5235 seconds
AUTOTUNE mm(16384x12288, 12288x4096)
  mm 2.3869 ms 100.0%
  triton_mm_661 4.4904 ms 53.2%
  triton_mm_662 4.6002 ms 51.9%
  triton_mm_667 4.6453 ms 51.4%
  triton_mm_664 4.7055 ms 50.7%
  triton_mm_663 5.0753 ms 47.0%
  triton_mm_660 5.9063 ms 40.4%
  triton_mm_668 6.2341 ms 38.3%
  triton_mm_670 7.5920 ms 31.4%
  triton_mm_669 12.1617 ms 19.6%
</code></pre>
<p>In my understanding, this means max_autotune should be doing nothing, but it’s actually 1% faster. Why might this be? Is autotune doing something else?</p>
ModuleNotFoundError with torch.load

Fabrice_Auzanneau: <p>Hello<br>
I’m trying to load a model that I saved using:<br>
<code>torch.save(model, filename)</code><br>
The model was saved on a Nvidia A100 GPU, using Python 3.8.10<br>
Then I transferred the <code>pth</code> file to my PC, where I want to open it. On this PC I have Python 3.10.9. I run the following command:<br>
<code>model = torch.load(file).to('cpu')</code><br>
and I get this error message:</p>
<blockquote>
<p>Traceback (most recent call last):</p>
<p>File ~\AppData\Local\anaconda3\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec<br>
exec(code, globals, locals)</p>
<p>File c:\users\fa125436\documents_boulot_liae\projets\refined\codes\bbb sound\test_encdec.py:27<br>
model = torch.load(file).to(‘cpu’)</p>
<p>File ~\AppData\Local\anaconda3\lib\site-packages\torch\serialization.py:1014 in load<br>
return _load(opened_zipfile,</p>
<p>File ~\AppData\Local\anaconda3\lib\site-packages\torch\serialization.py:1422 in _load<br>
result = unpickler.load()</p>
<p>File ~\AppData\Local\anaconda3\lib\site-packages\torch\serialization.py:1415 in find_class<br>
return super().find_class(mod_name, name)</p>
<p>ModuleNotFoundError: No module named ‘EncDec’</p>
</blockquote>
<p>EncDec is the custom model class I define and import using:<br>
<code>from Classes.EncDec import EncDec</code><br>
It is fine, I can declare a model by:</p>
<pre><code class="lang-auto">m = EncDec(512, 256, 100, 1024)

m
Out[7]: 
EncDec(
  (layers): Sequential(
    (0): Linear(in_features=1024, out_features=512, bias=True)
    (1): Tanh()
    (2): Linear(in_features=512, out_features=256, bias=True)
    (3): Tanh()
    (4): Linear(in_features=256, out_features=100, bias=True)
    (5): Tanh()
    (6): Linear(in_features=100, out_features=256, bias=True)
    (7): Tanh()
    (8): Linear(in_features=256, out_features=512, bias=True)
    (9): Tanh()
    (10): Linear(in_features=512, out_features=1024, bias=True)
    (11): Tanh()
  )
)
</code></pre>
<p>I work under the Spyder 5.4.3 IDE, if it can help…</p>
<p>I don’t understand what can create this problem. I don’t have it on the A100.<br>
Thanks for your help.</p>

ptrblck: <p>Saving the entire model is not recommended since you would have to restore the same code structure on the target system and can run into these issues.<br>
Save the <code>state_dict</code> instead and recreate the model object before loading its <code>state_dict</code>. This won’t avoid having to port the code but would relax the dependency on the same file structure.</p>

Fabrice_Auzanneau: <p>Yes, it works much better, thanks !</p>
Convolution speedup (slowdown) with torch.compile

AidanGoldfarb: <p>Hi,</p>
<p>I am comparing the performance of <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">convolutions</a> using both a regular timer and the torch profiler.</p>
<pre><code class="lang-auto">#a variety of (kernel,stride,padding) are used
conv = nn.Conv2d(in_channels,out_channels,kernel,stride=stride,padding=padding,bias=False,device=DEVICE)
convComp = torch.compile(conv)

# Regular timer
st = time.perf_counter_ns()
output = conv(input)
et = time.perf_counter_ns()

st = time.perf_counter_ns()
output = convComp(input)
et = time.perf_counter_ns()

# With profiler
with torch.autograd.profiler.profile(record_shapes=True, profile_memory=True, with_stack=True) as prof:
     output = conv(input)
with torch.autograd.profiler.profile(record_shapes=True, profile_memory=True, with_stack=True) as prof:
     output = convComp(input)
</code></pre>
<p>All experiemts are being run on the GPU, so CUDA or Triton. I have tried with and without cuda.synchronize(), as well as torch.no_grad().</p>
<p>Per the torch profiler, there are massive speedups (up to 8x in some configurations, most around 1.5x). However per the regular timer, all configurations are slower (down to .2x speedup, most around .6x).</p>
<p>I am curious what accounts for this large discrepancy in duration. Any ideas would be greatly appreciated</p>
<p>Thanks!</p>

ptrblck: <p>Profiling CUDA kernels without a synchronization is invalid as the kernels are executed asynchronously and you would profile the host only (including the dispatching, compiling, etc.).</p>

AidanGoldfarb: <p>Good to know. However, I have done the experiment with synchronization, and the results are the same</p>

marksaroufim: <p><code>torch.compile()</code> is a JIT compiler and the first time it’ll compile is when it sees the first input so in your first measurement you are measuring both the compilation time and inference time. Wheras in your second measurement with the profiler only the inference time</p>

AidanGoldfarb: <aside class="quote no-group" data-username="marksaroufim" data-post="4" data-topic="200111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marksaroufim/48/36747_2.png" class="avatar"> marksaroufim:</div>
<blockquote>
<p>irst measurement you are measuring both the compilation time and inference time. Wheras in your second measurement with the profiler only the inference time</p>
</blockquote>
</aside>
<p>I ran each convolution 10 times, and took the median time for each configuration. The two methods of profiling were not run sequentially, I wanted to create a simple presentation for this post.</p>
PyTorch code stuck at torch.nn.Linear() in the __init__() function

Archit_Mukherjee: <p>Hi,<br>
I am trying to train a model which has an MLP layer. But when I run the training code, it is stuck at nn.Linear() initialization and I cannot even terminate the process using Ctrl+C</p>
<p>My PyTorch version is 2.2.2<br>
CUDA version is 12.2.1</p>
<p>I have tried with other CUDA version (11.8), but still it is not working.</p>
<p>The MLP class code is as follows:</p>
<pre><code class="lang-auto">class MLP_5layer(nn.Module):
    def __init__(self, in_features, out_features):
        super(MLP_5layer, self).__init__()
        
        print('-init-mlp5-')
        layer1_out = int(in_features + (out_features-in_features)/5)
        layer2_out = int(in_features + 2*(out_features-in_features)/5)
        layer3_out = int(in_features + 3*(out_features-in_features)/5)
        layer4_out = int(in_features + 4*(out_features-in_features)/5)
        print(layer4_out)
        print("in_features:", in_features)
        print("layer1_out:", layer1_out)
        self.linear1 = torch.nn.Linear(in_features, layer1_out)
        print('-linear1-')
        self.batchnorm1 = torch.nn.BatchNorm1d(layer1_out)
        print('-layer1-')
        self.linear2 = torch.nn.Linear(layer1_out, layer2_out)
        self.batchnorm2 = torch.nn.BatchNorm1d(layer2_out)
        print('-layer2-')
        self.linear3 = torch.nn.Linear(layer2_out, layer3_out)
        self.batchnorm3 = torch.nn.BatchNorm1d(layer3_out)
        print('-layer3-')
        self.linear4 = torch.nn.Linear(layer3_out, layer4_out)
        self.batchnorm4 = torch.nn.BatchNorm1d(layer4_out)
        print('-layer4-')
        self.linear5 = torch.nn.Linear(layer4_out, out_features)
        self.batchnorm5 = torch.nn.BatchNorm1d(out_features)
        print('-layer5-')
        self.leakyrelu = torch.nn.LeakyReLU()
        self.dropout_01 = torch.nn.Dropout(p=0.1)
        self.dropout_05 = torch.nn.Dropout(p=0.5)
        print('-done-mlp5-')

    def forward(self, x):
        x = self.dropout_01(self.leakyrelu(self.batchnorm1(self.linear1(x))))
        x = self.dropout_05(self.leakyrelu(self.batchnorm2(self.linear2(x))))
        x = self.dropout_05(self.leakyrelu(self.batchnorm3(self.linear3(x))))
        x = self.dropout_05(self.leakyrelu(self.batchnorm4(self.linear4(x))))
        x = self.leakyrelu(self.batchnorm5(self.linear5(x)))
        return x
</code></pre>
<p>When I run the main script, I get the following output:</p>
<pre><code class="lang-auto">-init-mlp5-
226867
in_features: 347904
layer1_out: 317644

</code></pre>
<p>The process is stuck here. Please let me know what is the issue.</p>
<p>Thanks!</p>

ptrblck: <p>Your linear layer tries to initialize <code>347904 * 317644 * 4 / 1024**3 = 411.68 GB</code> which is why your code is “stuck”. Reduce the number of features to avoid allocating ~400GB.</p>
Error with static quantization

lzhao4ever: <p>I tried to do static quantization of Restormer model and got the following error:</p>
<p>RuntimeError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend.</p>
<p>Please let me know if you have encounter the similar issue and how you resolve it.</p>
<p>Thanks,</p>

jerryzh168: <p>please checkout <a href="https://pytorch.org/docs/stable/quantization.html#common-errors" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.2 documentation</a></p>

HDCharles: <p>i’ll add, this generally happens when you didn’t put the quant stubs in the correct place and so you used a non quantized arg as input to a quantized op.</p>
I saved the quantized weight and loaded it with the model after torch.ao.quantization.convert(). how do I print the output of each layer of the network?

TD-wzw: <p>After going through torch.ao.quantization.convert(), adding printing in the original network code seems to have no effect.</p>

HDCharles: <p>torch.ao.quantization.convert is eager mode quantization meaning it fully swaps the module to a quantized version, if you put print into the module that gets swapped, it wont do anything once its gone.</p>
<p>we usually do something like:</p>
<pre><code class="lang-auto"># intermediate layer analysis with forward hooks                        
                                                                        
activation = {}                                                         
def get_activation(name):                                               
    def hook(model, input, output):                                     
        activation[name] = {'input': input, 'output': output}           
    return hook                                                         
                                                                        
def add_hooks(m):                                                       
    for k, v in m.named_modules():                                      
        print(k, v)                                                     
        v.register_forward_hook(get_activation(k))                      
                                                                        
m = nn.Sequential(nn.Sequential(nn.Conv2d(1, 1, 1)), nn.Conv2d(1, 1, 1))
add_hooks(m)                                                            
m(torch.randn(1, 1, 1, 1))                                              
print(activation)
</code></pre>

chenster_liu: <p>Absolutely forward_hook is the simplest solution. I’m curious if it’s applicable on the quantized model as well. At least I’ve tried the above code snippet but after the quantization flow apply</p>
<blockquote>
<p>add_hook(model_int8)</p>
</blockquote>
<p><code>activation</code> doesn’t hook anything. Is there any officially recommended way to extract intermediate activations from the quantized model?</p>

HDCharles: <p>when you run the above snipped it doesn’t hook anything? Because i use this regularly to capture intermediate activations for quantized models.</p>

chenster_liu: <p>You are right, the code works.<br>
In my last reply it was not working for me because my q_model is scripted, I think.</p>

HDCharles: <p>yeah hooks are module oriented so scripting which removes modules would break it</p>
Is pytorch simulating the quantization?

abidi_taha_yassine: <p>I’m curious about how PyTorch handles operations involving int8 tensors. Is PyTorch actually doing the computation in int8, or does it convert int8 tensors to floating-point format (fp32) before computation? (which I think this is what’s happening)<br>
If so, I’m not seeing the potential benefits of quantization, unless they’re supposing that there’s specialized hardware capable of directly computing with fixed-point int8. Could someone shed light on this?</p>

HDCharles: <p>yes generally pytorch quantization is actually doing the computation in int8 with int8 hardware operations.</p>
<p>some info here: <a href="https://github.com/google/gemmlowp/blob/master/doc/quantization.md" class="inline-onebox" rel="noopener nofollow ugc">gemmlowp/doc/quantization.md at master · google/gemmlowp · GitHub</a></p>
Ada delta torch implementation

gautamvarmadatla: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ad0def297f112ab1a0071403ed9307219abd728.png" alt="image" data-base62-sha1="aFQPS0dI4xw3kPxcxLqUQl45b28" width="478" height="327"><br>
This is an image from Torch’s documentation…!</p>
<p>Adadelta’s original paper completely removed learning rate and uses RMS( Parameter updates )  instead . But torch’s implementation has an additional learning rate added to it… Any particular reason why this was done?</p>

ptrblck: <p>I guess to give users more flexibility. The learning rate is set to <code>1.0</code> by default so should match the original implementation you are referring to.</p>
Roadmap for torch.ao?

mklasby: <p>Curious about the roadmap for torch.ao, overall design philosphy, and integration with pytorch-labs/ao. Looks like lots of active development going on and would love to contribute and help flesh out the pruning / sparsity support.</p>

111357: <p>It could be useful to you. <a href="https://github.com/pytorch-labs/ao/issues/47" class="inline-onebox" rel="noopener nofollow ugc">[RFC] Plans for torchao · Issue #47 · pytorch-labs/ao · GitHub</a></p>

HDCharles: <p>you’re certainly welcome to contribute <a class="mention" href="/u/jcaip">@jcaip</a> is running the pruning/sparsity work and can perhaps direct you to which tasks would be helpful to get additional contributors for.</p>
Export module (via torch_dynamo) with arbitrary tensors marked as sharded

hsnbrg: <p>Hi, I’m trying to export the following module via <code>torch.export.export</code>.</p>
<pre><code class="lang-auto">class ShardedScaleModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.mesh = Mesh((0, 1, 2), (3, 1), ('x', 'y'))
        self.partition_spec = ('x', None, None, None)
        self.device = xm.xla_device()

    def forward(self, x: torch.Tensor):
        y = 2.0 * x
        xs.mark_sharding(y.to(self.device), self.mesh, self.partition_spec, use_dynamo_custom_op=True)
        z = 3.0 * y
        return z
</code></pre>
<p>However, it seems like <code>torch_dynamo</code> tries to trace through <code>xs.mark_sharding</code> and it consequently fails.<br>
If this is an incorrect way of exporting a module with arbitrary tensors being marked as sharded, what would be the correct way of doing so?</p>
<p>Thanks!</p>
Recurrent Attention Unit in GRU c++

amgb20: <p>Hello everyone,</p>
<p>I would like to make some modifications to the current python torch.GRU by adding an attention model within the cell. For this, I have made some research and it seems that I need to go all the way down to the C++ code where the GRU has been implemented with the mathematical expressions.</p>
<p>Any idea of how I could access this C++ model?</p>
<p>Cheers,</p>
<p>Alex</p>
Evaluate a pretrained Model Using Imagenet

Izan_C_G: <p>Is there any example code that evaluates the entire ImageNet dataset using a pre-trained model like those from Torchvision’s official website (using quantized Resnet50, for example)?</p>

ptrblck: <p>I would assume the <a href="https://github.com/pytorch/examples/blob/2d725b6ab255e05c55e0b08925f06f171aaedc0c/imagenet/main.py#L350"><code>validate</code></a> function was used to create the <code>torchvision</code> results.</p>

Izan_C_G: <p>Yes, I tried, and it works only on fp32 pre-trained models, not with int8 quantized ones. When the program computes “output = model(images)”, it reports a segmentation fault and fails.</p>
<p>This only happens with quantized models, so I guess there is no issue with my cuda/pytorch/torchvision configuration: Cuda 11.2, torch 1.12.1+cu113, torchaudio 0.12.1+cu113, torchsummary  1.5.1, torchvision          0.13.1+cu113. My Cuda is older than the Cuda built with pytorch (11.3), but it is recognized well.</p>
<p>I wonder, if is there any issue with quantized models.</p>
Torch.cuda.is_available() : False

kzxkelvin: <p>Please Help, I am trying to use YOLOv8 in my jetson nano.</p>
<p>torch.cuda.is_available() : False<br>
print(torch.version.cuda) : None</p>
<p>python3 --version = 3.6.9<br>
nvcc --version :<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Sun_Feb_28_22:34:44_PST_2021<br>
Cuda compilation tools, release 10.2, V10.2.300<br>
Build cuda_10.2_r440.TC440_70.29663091_0</p>
<p>pip3 install torch: Successfully installed torch-1.10.2<br>
pip3 install torchvision: Successfully installed torchvision-0.11.3</p>
<pre><code class="lang-auto">pip3 install torch==1.10.0+cu102 torchvision==0.11.0+cu102 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>I uninstalled both torch, torchvision and tried the command above, still failed. Why<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fed74764a7592ae1ab79876abab923611f2a7f1c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fed74764a7592ae1ab79876abab923611f2a7f1c" title="ask2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed74764a7592ae1ab79876abab923611f2a7f1c_2_625x500.jpeg" alt="ask2" data-base62-sha1="AmqvDQBOaJNU4rZPKZJMhVeyZC4" width="625" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed74764a7592ae1ab79876abab923611f2a7f1c_2_625x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed74764a7592ae1ab79876abab923611f2a7f1c_2_937x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed74764a7592ae1ab79876abab923611f2a7f1c_2_1250x1000.jpeg 2x" data-dominant-color="674330"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ask2</span><span class="informations">1280×1024 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Neilstid: <p>Hi,</p>
<p>Can you try to install cuda-toolkit 10.2 from conda:</p>
<pre><code class="lang-auto">conda install conda-forge::cudatoolkit-dev=10.2
</code></pre>
<p>And then reinstall pytorchn since you have a only-cpu version of pytorch (<code>print(torch.version.cuda) : None</code>)</p>

ptrblck: <p>The binaries hosted on <a href="http://PyTorch.org">PyTorch.org</a> don’t support Jetson devices so you would need to download them from <a href="https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/">here</a> or <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>