akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Impact of resuming from a checkpoint vs training/finetuning from the start

raghavm1: <p>I had a doubt about the impact resuming fine-tuning/training from checkpoints might have.</p>
<p>Is there anything that resets like the optimizer state, or perhaps some other parameter in the case of resuming training from a checkpoint instead of a model fine-tuning continuously without interruptions? I wanted to figure out the impact of resuming training vs keeping it going from scratch - is it equivalent or does it impact something in the model updates?</p>

ptrblck: <p>Resuming from a checkpoint should not break the training run and you should see the same model convergence if all states are restored. Besides restoring the <code>state_dict</code>s of the model and optimizer you could also check if you need to seed the code carefully to restore the data loading state as well.</p>

raghavm1: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="209499">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Besides restoring the <code>state_dict</code>s of the model and optimizer you could also check if you need to seed the code carefully to restore the data loading state as well.</p>
</blockquote>
</aside>
<p>Is this related to loading data for continuing the training/fine-tuning task? Does this only apply when data is shuffled? Or am I misinterpreting you here?</p>

ptrblck: <p>Yes, if you are shuffling the data and are not (re-)storing the seed before resuming the training process, the sampled indices would not be the same compared to a full training run without any interruptions. I don’t know, if you would need to restore the shuffling state as well, but wanted to mention it just in case.</p>
I can't import torch. I'm confused

Mahedi: <p>I can’t import torch. I never faced this issue. Below is the error. I have tried with different version of typing_extension also reinstalled Pytorch with cuda=11.8.</p>
<p>OS : Ubuntu 18.4<br>
GPU : RTX2080<br>
Python : 3.8</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/data/studentenv/lib/python3.8/site-packages/torch/<strong>init</strong>.py”, line 1694, in <br>
_C._initExtension(_manager_path())<br>
File “/data/studentenv/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 31, in <br>
from .graphs import (<br>
File “/data/studentenv/lib/python3.8/site-packages/torch/cuda/graphs.py”, line 6, in <br>
from torch.utils import _pytree<br>
File “/data/studentenv/lib/python3.8/site-packages/torch/utils/_pytree.py”, line 51, in <br>
from typing_extensions import deprecated<br>
ImportError: cannot import name ‘deprecated’ from ‘typing_extensions’ (/data/studentenv/lib/python3.8/site-packages/typing_extensions.py)</p>

InnovArul: <p>Googling suggests that the issue is similar to:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/fastapi/fastapi/discussions/9808">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/fastapi/fastapi/discussions/9808" target="_blank" rel="noopener">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/149ba2deea7792f2a5ec095552deeebee52c48aee8f7f4c610f35678f2e92c6c/fastapi/fastapi/discussions/9808" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/fastapi/fastapi/discussions/9808" target="_blank" rel="noopener">cannot import name 'deprecated' from 'typing_extensions' · fastapi/fastapi ·...</a></h3>

  <p>Privileged issue I'm @tiangolo or he asked me directly to create an issue here. Issue Content When I use fastapi in Python 3.10.11. Throw some errors like "cannot import name 'deprecated' from 'typ...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Could you try installing <code>typing_extensions-4.7.1</code>?</p>

Mahedi: <p><a class="mention" href="/u/innovarul">@InnovArul</a> Thank you for your reply. Problem solved. Previously I had typing_extensions 4.4.0 and I tried below that versions. Now I installed 4.11. Everything is working fine.</p>
Unhashable list while training sbert

mantics: <p>Hello,</p>
<p>I am trying to create embeddings first before feeding into the model. Why I need to create the embeddings first is because of max_seq_len, I want to make use of long sequences and I think the best way is to generate embeddings, fine-tune on the embeddings but I have ran into issues with it and tried out things not working yet.</p>
<p>I am using the sentence bert model [query,positive,negative] pairs.</p>
<pre><code class="lang-auto">    model = SentenceTransformer(model_name)
query = qry + ' ' + history + ' ' + tpl.Occ.strip().lower() + ' ' + tpl.Pr.strip().lower() + ' '  + tpl.D.strip().lower() + ' '  + tpl.G.strip().lower() + ' '  + tpl.D.strip().lower() + ' '  + tpl.HomeOwner.strip().lower()
            emb_query = np.array(model.encode(sentences=query, normalize_embeddings=True, show_progress_bar=True))
            emb_positive = np.array(model.encode(sentences=positive, normalize_embeddings=True, show_progress_bar=True))
            emb_hneg = np.array(model.encode(sentences=hard_negative, normalize_embeddings=True, show_progress_bar=True))

train_data['anchor'].append(emb_query)
            train_data['positive'].append(emb_positive)
            train_data['negative'].append(emb_hneg)
    train_dataset = Dataset.from_dict(train_data)

trainer = SentenceTransformerTrainer(
            model=model,
            args=args,
            train_dataset=train_dataset,
            eval_dataset=val_dataset,
            loss=train_loss,
          
            evaluator=dev_evaluator,
        )
        trainer.train()
</code></pre>
<p>Now I get the error that unhashable list. I have tried converting to numpy array and even explored using tuple but by the time I use train_dataset = Dataset.from_dict(train_data), it converts it back to list. [[0.1,0.1], [0.1,0.1]]. If I use raw text, it is fine but I want to maximise my max_seq_len and I don’t have the luxury of using bigger max_se_len beyond 128</p>
Slice_backward op

inging550: <p>When I use with torch.autograd.profiler.profile I saw a op called slice_backward, and there are some questions about slice_backward op</p>
<ol>
<li>where is the source code and what does slice_backward do</li>
<li>what is the difference with the backward function SliceBackward0</li>
<li>where is the source code about SliceBackward0<br>
I am so appreciate that some one can tell me</li>
</ol>

ptrblck: <p>The backward kernel should be defined <a href="https://github.com/pytorch/pytorch/blob/50efbb9f1e7111b4b6d5b8e9a6064ee9783930be/aten/src/ATen/native/TensorShape.cpp#L2566">here</a>. I don’t know how exactly <code>SliceBackward0</code> is generated, but it should be located in <code>torch/csrc/autograd/generated/Functions.cpp</code> (or a related file) in a local source build. <a class="mention" href="/u/alband">@albanD</a> might know details about the internals how the auto-generated method calls into the kernel.</p>

albanD: <p>That is correct!<br>
SliceBackward0 is the autograd “Node” that goes into the “autograd graph”<br>
While at::slice_backward() is the aten op being called by this node to perform the computations it needs to perform.</p>

inging550: <p>I’m understand. THX!</p>
Expert Parallelism and Expert Parallelism + Tensor Parallelism need

Guo_Danding: <p>Hi, I have no idea how to implement ep and ep+tp with current torch api. Is there any examples or tutorials? Is there anyone has tried it?</p>

fegin: <p>If you mean MoE, yes it is possible to implement MoE with TP via DTensor. We are also planning to explore this parallelism combination but it is not ready yet.</p>

lovanto: <p>Hi! If I’m not mistaken expert parallelism is implemented in <a href="https://github.com/databricks/megablocks" class="inline-onebox" rel="noopener nofollow ugc">GitHub - databricks/megablocks</a></p>

Guo_Danding: <p>Thanks. I will check it.</p>

Guo_Danding: <p>Yes, it is what i mean. Pytorch has great api to support TP. But i cannot find any tutorials helpful to implement EP or EP+TP. So i am confused.</p>
Is there a way to build Pytorch with ONNX/protobuf deps

yinghai: <p>Hi folks,<br>
We try to build pytorch without ONNX as its protobuf tends to conflict with other libs. I’ve seen <code>ONNX_ML</code> and <code>INTERN_DISABLE_ONNX</code>. But there are a bunch of references to onnx proto in jit export (<a href="https://github.com/pytorch/pytorch/blob/b608ff3bea7799f76a2293f362f949851ab547b0/torch/csrc/jit/serialization/export.cpp#L19" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/jit/serialization/export.cpp at b608ff3bea7799f76a2293f362f949851ab547b0 · pytorch/pytorch · GitHub</a>).</p>
<p>Can we juse something like</p>
<pre><code class="lang-auto">#ifdef ONNX_ML
</code></pre>
<p>to also make jit to onnx export optional? Thanks!</p>

yinghai: <p>hmm it’s overall the place <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> <a href="https://github.com/pytorch/pytorch/blob/a00faf440888ffb724bad413f329a49e2b6388e7/cmake/TorchConfig.cmake.in#L96" class="inline-onebox" rel="noopener nofollow ugc">pytorch/cmake/TorchConfig.cmake.in at a00faf440888ffb724bad413f329a49e2b6388e7 · pytorch/pytorch · GitHub</a></p>
IMDB_TorchText_Interpret tutorial uses deprecated torchtext

rbelew: <pre><code class="lang-auto">captum 0.7.0
torch 2.3.0
torchtext 0.18.0
spacy 3.7.6
</code></pre>
<p>i’d like to be able to replicate the results in this <a href="https://captum.ai/tutorials/IMDB_TorchText_Interpret" rel="noopener nofollow ugc">captum tutorial</a>, but can’t figure out how to get the vocabulary associated with the pre-trained CNN’s embedding for use by spacy on test sentences?</p>
<p><code>interpret_sentence()</code> makes critical use of the legacy <code>TEXT.vocab.stoi()</code> and <code>Label.vocab.itos()</code> dropped by torchtext version &gt;= 0.9?</p>
<p>this has been a <a href="https://github.com/pytorch/captum/issues/1019#issuecomment-2349958581" rel="noopener nofollow ugc">captum issue</a> for two years?!</p>
Torch.nn.functional.linear is slow when built from source

ashari4: <p>I am building torch from source and seeing slow performance when using torch.nn.functional.linear.</p>
<p>OS: Ubuntu 20.04<br>
commit hash: 4d6314a16e78027832186f5442df888dbabbc159 (9/21/21)</p>
<pre><code class="lang-auto">export USE_CUDA="0"
python3.9 setup.py bdist_wheel
</code></pre>
<pre><code class="lang-auto">import datetime
import torch
import torch.nn.functional as F

t1 = torch.ones((32, 1024, 512))
t2 = torch.ones((50257, 512))
e = datetime.datetime.now()
print(f"Before F.linear: hour: {e.hour}, minute: {e.minute}, second: {e.second}", flush=True)
logits_parallel = F.linear(t1, t2)
e = datetime.datetime.now()    
print(f"After F.linear: hour: {e.hour}, minute: {e.minute}, second: {e.second}", flush=True)
</code></pre>
<p>torch built from source takes 15m 15s while the latest torch from PyPI takes 17 s. Any ideas on what might be the issue?</p>
<ul>
<li>Running htop while running the above script shows that only 1 core is getting exercised</li>
<li>I am not building with MKL</li>
</ul>

ashari4: <p>Building with MKL speeds up F.linear(t1, t2) to what is expected.</p>

Cummins597: <p>Your loss function is programmatically correct except for below:</p>
<pre><code class="lang-auto">    # the number of tokens is the sum of elements in mask
    num_tokens = int(torch.sum(mask).data[0])
</code></pre>
<p>When you do <code>torch.sum</code> it returns a 0-dimensional tensor and hence the warning that it can’t be indexed. To fix this do <code>int(torch.sum(mask).item())</code> as suggested or <code>int(torch.sum(mask))</code> will work too.</p>
<p>Now, are you trying to emulate the CE loss using the custom loss? If yes, then you are missing the <code>log_softmax</code></p>
<p>To fix that add <code>outputs = torch.nn.functional.log_softmax(outputs, dim=1)</code> before statement 4. Note that in case of tutorial that you have attached, <code>log_softmax</code> is already done in the forward call. You can do that too<a href="https://www.omegle.zone/" rel="noopener nofollow ugc">.</a></p>

ashari4: <p>Hi <a class="mention" href="/u/cummins597">@Cummins597</a> - I didn’t specify a loss function in my example. Did you mean to reply to this post?</p>

miandrew: <p>I’m waiting for your reply<a href="https://www.spiritedpuddlejumper.com/how-to-maximize-your-weight-loss-journey-with-supplements/" rel="noopener nofollow ugc">.</a></p>
Computing the trace of the jacobian of the score function

sherlock.h: <p>Hello!</p>
<p>I am interested in the trace of the jacobian of the score function. Is there any efficient way to get it?</p>
<pre data-code-wrap="python"><code class="lang-python">x = ... # (B, N, D)
log_p = dist.log_prob(x) # (B, N)
score_fn = grad(log_p.sum(), x, create_graph=True)[0] # (B, N, D)
</code></pre>
<p>I can get it with for loops, but it’s quite slow.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/sherlock.h">@sherlock.h</a>,</p>
<p>You could have a look at defining a functional approach and using the <code>torch.func</code> namespace to efficiently compute the jacobian of your function. The docs are here: <a href="https://pytorch.org/docs/stable/generated/torch.func.jacrev.html" class="inline-onebox" rel="noopener nofollow ugc">torch.func.jacrev — PyTorch 2.4 documentation</a></p>

sherlock.h: <p>Thanks for the suggestion!</p>
<p>I ended up going with Hutchinson’s trace estimator (expectation with respect to the Rademacher distribution):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/746b465ad95fe606807670b4b0623ba610b0ec94.png" alt="image" data-base62-sha1="gBTdKGo7a4WnLxTlaAC2htAF0ZS" width="192" height="53"></p>
<pre data-code-wrap="python"><code class="lang-python">v = torch.randn(M, B, N, D).sign()
tr = (
    torch.autograd.grad(
        outputs=score_fn,
        inputs=x,
        grad_outputs=v,
        is_grads_batched=True,
    )[0]
    .mul(v)
    .sum(-1)
).mean(-3)
</code></pre>
Distributed Autograd + FSDP?

Jacob_Buckman: <p>The FSDP wrapper seems to use the regular <code>torch.distributed</code> NCCL primitives, but distributed autograd requires RPC primitives instead. Is it possible to use these two in conjunction? Is there any example of this?</p>

fegin: <p>FSDP doesn’t support RPC primitives. Curious in what scenario do you need distributed autograd with FSDP?</p>

Jacob_Buckman: <p>It’s for a custom sharding strategy I am implementing as part of a research project – more or less a ringattention alternative.</p>

fegin: <p>The current approach for implementing ring-attention algorithms is to utilize send/recv and customized forward/backward functions to perform the necessary ring computations. One example is <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/tensor/experimental/_attention.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/tensor/experimental/_attention.py at main · pytorch/pytorch · GitHub</a>.</p>

Jacob_Buckman: <p>Thanks for the reference! I’ll probably end up doing it like this then, implementing manual gradient calculations with NCCL seems like less work than reimplementing FSDP in RPC.</p>
CUDA and PyTorch

Halil_Huseyin_Calisk: <p>Hello, I have a Nvidia Jetson Orin Nano that comes with the Jetpack system and supports CUDA 12.2.1. I want to process images using CUDA and the GPU. If I download torch, torchvision, and torchaudio via pip from the official PyTorch website, which are compatible with CUDA 12.1, will I be able to process images using CUDA and the GPU?</p>

ptrblck: <p>No, since these PyTorch binaries were built for x86. You would this need to download the PyTorch wheels for Jetson from e.g. <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>
Batch size scaling poorly with bigger input

Piotr_Paziewski: <p>Hello, I noticed that inference time is scaling poorly with bigger input size using resnet50 network:</p>
<p>input size: 224<br>
batch size: 1: 6.78 (ms)<br>
batch size: 4: 11.99 (ms)<br>
batch size: 16: 40.49 (ms)</p>
<p>input size: 640<br>
batch size: 1: 20.93 (ms)<br>
batch size: 4: 84.83 (ms)<br>
batch size: 16: 331.74 (ms)</p>
<p>Reproduction:</p>
<p>Code:</p>
<pre><code class="lang-auto">import time
from statistics import mean
import torch
import torchvision.models as models

if __name__ == "__main__":
    if not torch.cuda.is_available():
        print("cuda not available")
        exit()
    device = torch.device("cuda")

    model = models.resnet50()
    model = model.to(device)
    model.eval()

    input_sizes = [224,640]
    batch_sizes = [1, 4, 16]
    
    for input_size in input_sizes:
        print(f"\ninput size: {input_size}")
        for batch_size in batch_sizes:
            # Warm-up 
            inputs = torch.randn(
                batch_size, 3,input_size, input_size).to(device
            )
            with torch.no_grad():
                _ = model(inputs)
            
            measures = []
            for _ in range(100):
                inputs = torch.randn(
                    batch_size, 3,input_size, input_size).to(device
                )

                start_time = time.time()
                with torch.no_grad():
                    _ = model(inputs)
                torch.cuda.synchronize()
                elapsed_time = time.time() - start_time
                
                measures.append(elapsed_time)
            mean_measure = mean(measures)

            print(
                f"  batch size: {batch_size}: " f"{mean_measure*1e3:.2f} (ms)"
            )
</code></pre>
<p>Environment: google colab with T4 GPU. I registered same behavior on L4 GPU with pytorch/pytorch:2.4.1-cuda12.1-cudnn9-devel docker.</p>
<p>Also, I monitored GPU utalization and memory usage on L4 and noticed its using 70% of GPU and ~10% memory with bigest batch with biggest input.</p>
<p>Also, I tried using different networks, even simple conv2d layer and registered same results.</p>
<p>Is that behaviour expected?</p>
Larger batch inference sizes for torch.compile yields untenable warm up times

frdchang: <p>Observations:</p>
<p>For a large batch inference workload, torch.compile takes a very long time to warm up, e.g. quickly goes from minutes to hours with modestly sized batches.</p>
<p>However, the un-compiled model can easily ingest much larger size batches.</p>
<p>Modeling context:</p>
<ul>
<li>torch.inference_mode()</li>
<li>mixed precision context of fp16.</li>
<li>LLM with scaled dot product attention module</li>
<li>large batch inference job using multiple GPU workers</li>
</ul>
<p>So is it correct that:</p>
<ul>
<li>there is significant performance gains to be had with the compiled version</li>
<li>but their is also significant warm up time for larger batch sizes</li>
<li>the warm-up artifacts cannot easily be cached and shared as of today</li>
<li>thus is a trade-off in throughput between compiled vs not-compiling when accounting for the warmup time.</li>
</ul>
<p>Question 1: How does the warmup time scale with batch sizes?</p>
<p>Question 2: Is it correct to think that smaller batch sizes should be used with torch.compile to balance the warmup time penalty and inference throughput?</p>
<p>Thanks,<br>
Fred</p>
How to make eval time prelu inplcae?

Lemar_R: <p>Prelu couldnt be inplaced during training for recording gradients.</p>
<p>But when during eval mode, it seems better to make it inplcae for saving memory. So I wrote a kernal to replace the trained prelu module inherited from augrad class. But It didnt help.</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">__global__ void prelu_forward_cuda_kernel(float* input, float* weight, int64_t batch_size, int64_t channels, int64_t height, int64_t width) {
    int64_t index = blockIdx.x * blockDim.x + threadIdx.x;
    int64_t num_elements = batch_size * channels * height * width;
    
    if (index &lt; num_elements) {
        int c = (index / (height * width)) % channels;
        if (input[index] &lt; 0) {
            input[index] *= weight[c];  // Apply PReLU to negative elements inplace
        }
    }
}
std::vector&lt;torch::Tensor&gt; prelu_backward_cuda(torch::Tensor grad_output, torch::Tensor input, torch::Tensor weight) {
    auto batch_size = input.size(0);
    auto channels = input.size(1);
    auto height = input.size(2);
    auto width = input.size(3);

    auto grad_input = torch::zeros_like(input);
    auto grad_weight = torch::zeros_like(weight);

    int64_t num_elements = batch_size * channels * height * width;
    int threads = 1024;
    int blocks = (num_elements + threads - 1) / threads;

    prelu_backward_cuda_kernel&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(grad_output.data_ptr&lt;float&gt;(), input.data_ptr&lt;float&gt;(), grad_input.data_ptr&lt;float&gt;(), weight.data_ptr&lt;float&gt;(), grad_weight.data_ptr&lt;float&gt;(), batch_size, channels, height, width);

    return {grad_input, grad_weight};
}
</code></pre>
<p>Could anyone help me out with the code?</p>
Get metadata of .pt model without torch.load()

ambitious-octopus: <p>Hey, I need to load a .pt model and get the metadata. I save my models with a function similar to the following:</p>
<pre><code class="lang-auto">torch.save(
{"n_epochs", n_epochs}
model)
</code></pre>
<p>How can i get n_epochs without loading the actual model?<br>
This is because i need to access  the metadata without having the model class.</p>
Did something with the recurrent network models change?

pytorcher: <p>I remember a couple years ago working with GRUs and LSTMs and it was always the case that in the forward function it would return the output and the hidden state and that hidden state should be passed back later on.  Now I’m seeing lots of posts that don’t seem to do that and just ignore the hidden state (<a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="noopener nofollow ugc">LSTM</a>).  Is that handled internally now?  Are LSTMs and GRUs still useful if they are just passed zeros every time?</p>
How to find source code of diag in torch

xkwnzxxjw: <p>Maybe someone can teach me how to search the source code of diag in torch</p>

Soumya_Kundu: <p>Which function exactly?</p>
<p>For torch.diag, it maybe this: <a href="https://github.com/zdevito/ATen/blob/4aa3e1de29ed58457e530f84217e53db0998476c/aten/src/TH/generic/THTensorMath.h#L139" class="inline-onebox" rel="noopener nofollow ugc">ATen/aten/src/TH/generic/THTensorMath.h at 4aa3e1de29ed58457e530f84217e53db0998476c · zdevito/ATen · GitHub</a></p>
Access externally updated tensors on gpu from Torchscript

torchlearner: <p>Hi team,</p>
<p>I have a use-case like this. There is a pipeline which will be updating embeddings on GPU. Consider these standard embeddings of different dimensions. And lets say there is a pipeline written in cuda or otherwise which updates these tensors in GPU. This is live ingestion.</p>
<p>Now I have pytorch model which wants to run certain computation on this data. For example the model might want to do a dot product between say a query embedding it received as input with this data. The cardinality of this data could be high. For example we might be talking of millions of documents on gpu. I needed an efficient way to access this data within the pytorch model on every request. Within the model I do batch operations, so for example query.document_transposed,  where both query and document are vectors.</p>
<p>Also note that the data is live ingested. Which means at any point some documents will be marked dirty and I will need a way to access the document tensors which are not dirty.</p>
<p>What is the best way to go about this ? My environment is Torchscript models on nvidia A100s. And this is an online system, so latency is critical.</p>
<p>Currently I have implemented this by implementing update/insert APIs within the model and defining storage within the model to store the embeddings within the model itself. However I want to externalize this.</p>
<p>Thanks,</p>
Unsupported FX node: <built-in function mod>

Gaulex: <p>Hello!</p>
<p>I am trying to export the <a href="https://github.com/VikParuchuri/surya" rel="noopener nofollow ugc">Surya OCR model</a> to the ONNX format in order to improve its inference latency on CPU with TorchDynamo. The export procedure works as expected with the default export option, but with the <strong>dynamic_shapes=True</strong> option, it does not work anymore:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.onnx.dynamo_export(model, **kwargs) # OK

export_options = torch.onnx.ExportOptions(dynamic_shapes=True)
torch.onnx.dynamo_export(model, **kwargs, export_options=export_options) # ERROR
</code></pre>
<p>The error I have is:</p>
<blockquote>
<p>torch.onnx._internal.diagnostics.infra.context.RuntimeErrorWithDiagnostic: Unsupported FX &gt; nodes: {‘call_function’: [‘built-in function mod’]}.</p>
</blockquote>
<p>I do not understand the error, as the modulus operator is supposed to be supported for the ONNX export, and works as expected on smaller networks that uses modulus.</p>
<p>I cannot provide the full code, as the whole model is gigantic, but I can provide further information if necessary, of course.</p>
<p>Thank you!</p>

Kuei-Da_Liao: <p>I’m wondering if there’s a solution to this?</p>
<p>I’m seeing the same error when trying to export the “BertEncoder” in “transformers.models.bert.modeling_bert.BertEncoder”.</p>
Still Facebook employees only?

Robert_Crowe: <p>The <a href="https://pytorch.org/docs/master/community/governance.html#faq" rel="noopener nofollow ugc">governance FAQ</a> still says that:</p>
<blockquote>
<p><strong>Q: Can I become a committer on the project?</strong> Unfortunately, the current commit process to PyTorch involves an interaction with Facebook infrastructure that <strong>can only be triggered by Facebook employees</strong>. We are however looking at ways to expand the committer base to individuals outside of Facebook and will provide an update when the tooling exists to allow this.</p>
</blockquote>
<p>Is this still true?</p>
Privacy accounting based on DPDataLoader

ShadiRahimian: <p>Dear Opacus community,</p>
<p>I have implemented the DP-SGD algorithm myself, by first clipping the per-sample gradients and noising the batch. I am combining it with the opacus.data_loader.DPDataLoader class to be sure that the sampling for the boosting is being done correctly.</p>
<p>When I use the DPDataLoader, in my training loop, for each epoch I see different batch_sizes being selected (as expected) and the “epoch” continues until the sum of these batches is a number close to the size of the training dataset.<br>
Now my question is if I want to calculate the privacy budget using the RDP as implemented <a href="https://github.com/pytorch/opacus/blob/main/opacus/scripts/compute_dp_sgd_privacy.py" rel="noopener nofollow ugc">here</a>, is the parameter “epoch” the same as the usual epochs of my training loop that in expectation parses through as many data points as the training dataset?</p>
<p>My code looks something like this:</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(torch.from_numpy(d2[:2500, :]).type(torch.LongTensor), batch_size=batch_size, shuffle=True, num_workers=4)

train_dataloader = DPDataLoader.from_data_loader(train_dataloader, distributed=False)

for epoch in range(num_epochs):
	train_acc = 0
	train_loss = 0
	num_samples = 0
	sample_rate = 1 / len(data_loader)
	expected_batch_size = int(len(data_loader.dataset) * sample_rate)
	self.model.train()
	print_interval = 100
	params = {k: v.detach() for k, v in self.model.named_parameters()}
	buffers = {k: v.detach() for k, v in self.model.named_buffers()}
	ft_compute_grad = grad(self.compute_loss, argnums=(0, 1), has_aux=True)
	ft_compute_sample_grad = vmap(ft_compute_grad, in_dims=(None, None, 0))
	for idx, batch in enumerate(tqdm(data_loader)):
		batch_size = len(batch)
		size += batch_size
		if batch_size &gt; 40 :
			continue
		ft_per_sample_grads, loss = ft_compute_sample_grad(params, buffers, batch)
		loss = loss.mean()
		ft_per_sample_grads = ft_per_sample_grads[0]
		for key in ft_per_sample_grads:
			for i in range(batch_size):
				utils.norm_clipper(ft_per_sample_grads[key][i], max_norm=max_norm) #per-sample norm clipping in-place
			ft_per_sample_grads[key] = ft_per_sample_grads[key].sum(0) / expected_batch_size #sum up over batch
			rand = torch.zeros_like(ft_per_sample_grads[key])
			rand.normal_(mean=0, std=noise_multiplier*max_norm/expected_batch_size) #! consider the size of lot in std of Gaussian
			ft_per_sample_grads[key].add_(rand) #add in-place random noise to averaged grads

		for key, p in zip(ft_per_sample_grads, self.model.parameters()):
			p.grad = ft_per_sample_grads[key]
		num_samples += batch_size
		self.optimizer.step()
		self.optimizer.zero_grad()
		train_loss += loss.cpu().data.numpy().item() * batch_size
		if idx % print_interval == 0:
			print(loss.item())
	train_loss /= num_samples
	train_acc /= num_samples
	print(f"total size: {num_samples}")
</code></pre>

HuanyuZhang: <p>Yes. It is the same. Opacus and DP-SGD by default uses Poisson subsampling so it is expected to see variable batch size.</p>
Cant prevent overfitting after 2 epochs

cevvalu: <p>Built a small merge-image captioning model using  pretrained vgg16. But cant prevent overfitting even adding droputs. Validation loss keeps incresing after 2 epochs. So is it possible that this is my model’s maximum performance?</p>

albanD: <p>If you tried to train a vgg16 on a small dataset, it is possible that your model is just too powerful and will always overfit, you might want to use a simpler model.</p>

cevvalu: <p>actually im using an already trained vgg16 which is trained on ImageNet. I use vgg as an encoder . I only train my decoder and after 2 epochs overfits</p>

albanD: <p>I would try to use a simpler decoder still.<br>
Also you can try different regularization techniques (batchnorm, weight decay, etc).<br>
Another thing to check is that your training and validation sets are actually properly correlated. Make you split them with samples from each class in each, etc.</p>

cevvalu: <p>i will check them thank you. I think my dataset is too small i use 6000 train images from Flickr8k.</p>

F_f: <p>Try a model that the output feature vector is smaller like 2048 or make the output of youre vgg16 smaller by maxpooling or avarage pooling<br>
Or make youre dropout rate bigger like 0.6 or 0.7<br>
Or use batchnormalization layer or make youre learning rate smaller or use dynamic learning rate</p>
Query on ExpandedWeights

Pooja_Agarwal1: <p>Hi,</p>
<p>I saw that Opacus supports ExpandedWeights that can potentially improve the latency of per-sample gradient computation over the GradSampleModule (“hooks” approach) (<a href="https://github.com/pytorch/opacus/blob/f2a591aeae627c1ae0486129cf65347fa4d9b3b3/opacus/grad_sample/gsm_exp_weights.py#L46" rel="noopener nofollow ugc">ref</a>).</p>
<p>The README file mentions that there <a href="https://github.com/pytorch/opacus/blob/f2a591aeae627c1ae0486129cf65347fa4d9b3b3/opacus/grad_sample/README.md?plain=1#L13" rel="noopener nofollow ugc">may be unexpected errors</a> when using ExpandedWeights and that it is in beta.</p>
<p>What is the current state of ExpandedWeights, is it out of beta and in stable mode now? If not, is there a timeline for it? Additionally, I saw that it is not supported for RNN, LSTMs and a few other ops, is there a plan to enable ExpandedWeights over these?</p>
<p>Thanks,<br>
Pooja</p>

HuanyuZhang: <p>Hi,<br>
We are still in the beta mode. Right now our priority is on other modes (“hook”, and “functorch”) and there is no recent plan to support new functions for EW.<br>
Thanks,<br>
Huanyu</p>
ㅇdropout deactivating

HYUL: <p>I understand that in PyTorch, dropout is automatically disabled when not in training mode. However, I observed that changing the dropout value during inference affects the results. Are the conditions for disabling dropout only being not in training mode, or is it specifically when <code>eval()</code> is activated?</p>

ptrblck: <aside class="quote no-group" data-username="HYUL" data-post="1" data-topic="209400">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyul/48/72666_2.png" class="avatar"> HYUL:</div>
<blockquote>
<p>Are the conditions for disabling dropout only being not in training mode, or is it specifically when <code>eval()</code> is activated?</p>
</blockquote>
</aside>
<p>“Not in training mode” and calling <code>.eval()</code> is the same as internally <code>.eval()</code> will set the <code>self.training</code> attribute of the module (and all regustered submodules) to <code>False</code>.<br>
Do you have a minimal code snippet showing this behavior?</p>

HYUL: <p>sorry for late reply…</p>
<pre><code class="lang-auto">from model_zoo.RCF import RCF
model = RCF(dropout_prob = 0)
...
model.load_state_dict(torch.load(model_path))
model.to(device)
...
imgs = Variable(imgs.cuda())
        outputs = model(imgs)
</code></pre>
<p>This is all code using model and when I set dropout different makes varying result. I didn’t set anny mode (train, eval) and this case What cause result varying?</p>

ptrblck: <p>If you don’t explicitly call <code>model.eval()</code> the default training mode is active, dropout thus also activated, and it’s expected to see different outputs.</p>