akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
"How to quantize the bias of a convolution in QAT (Quantization Aware Training) mode?

seamoonlight-YBY: <p>My question is similar to the discussions</p><aside class="quote" data-post="1" data-topic="97368">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jonson/48/15855_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-pytorch-simulates-bias-during-quantization-aware-training/97368">How pytorch simulates bias during quantization aware training</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    It seems that pytorch qat doesn’t simulate bias quantization error during qat. And I found that qat.Conv2d only fake-quantize weight and activation. So pytorch’s quantization strategy does not quantize the bias, right?
  </blockquote>
</aside>
<aside class="quote" data-post="1" data-topic="146416">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayankumarbhunia/48/44803_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-bias-quantized-while-doing-pytorch-static-quantization/146416">Is bias quantized while doing pytorch static quantization?</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    Why is bias not quantized upon pytorch static quantization? Or is it not required for deployment?
  </blockquote>
</aside>

<p>From these two discussions, I roughly understand the reason why quantizing bias is not possible in eager mode. I also gained some insights into quantizing bias in FX. However, I would like to confirm: In the latest version of eager mode QAT quantization, is it still not possible to quantize bias? <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>

supriyar: <p>Hi, we haven’t made changes to how we quantize bias in eager model. In the graph mode flows fx and pt2 export based - you can configure this yourself by annotating the graph correctly if your backend supports quantized bias operators.</p>
Dataloader - how to parallelize loading several PNGs per item?

aguaviva: <p>What is the fastest way to load several PNGs per item?</p>
<p>I tried parallelizing the loading by:</p>
<ul>
<li>using asyncio, this doubled the speed but hangs one of our units tests</li>
<li>using <code>multiprocessing</code>, but the Dataloader complaints it is alrady using it and doesn’t like children</li>
</ul>
<p>Thanks!</p>

tom: <p>Except at the beginning of an epoch, the idea is that DataLoader processes pre-prepare batches (in multiprocessing) so you’d hide the latency. That said, if you’re not doing a lot of Python computation (needing the GIL) in it, you could try threads.</p>
<p>Best regards</p>
<p>Thomas</p>
Model doesn't train correctly with custom Dataset with DataLoader(.., num_workers>0)

nbmk: <p>This has been bugging me the last few days. I have narrowed this issue down (I hope) to my custom dataset, that subclasses the <code>torch.utils.data.Dataset</code>.<br>
The model trains fine with num_workers=0 in the dataloader. Different to all the other topics, I can train with num_workers&gt;0, but get weird results: instead of converging to ~70% accuracy and corresponding loss, the model converges to ~20% accuracy, and also weird numbers in validation.</p>
<ol>
<li>My code has a <code>if __name__ == "__main__"</code> safeguard and is simply called with <code>python main.py</code></li>
<li>I only change the <code>num_workers</code> parameter in the training dataloader</li>
<li>I can train fine with other  datasets, say <code>torchvision.datasets.MNIST</code></li>
<li>My custom dataset has the following <code>__getitem__</code> function:</li>
</ol>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        sample = json.loads(self.data[idx])
        x = torch.tensor(sample["x"])
        label = sample["y"]
        return x, label
</code></pre>
<p>which doesn’t look like something could go wrong in the num_workers&gt;0 multiprocessing. Any tips? For now, I can work with <code>num_workers=0</code>, but I am taking a performance hit, I would rather not have.</p>

nbmk: <p>Further investigation shows that I can get both options (<code>num_workers=0</code> and <code>num_workers&gt;0</code>) to behave the same by turning off shuffling in the training dataloader. How exactly is shuffled and yielded with multiple workers? Is it still guaranteed that the each sample in the dataset is only shown once per epoch?</p>

nbmk: <p>Alright, solved it, finally.<br>
It seems that each worker clones everything, meaning in my case also my self-defined collator. In this, I generated my one-hot encodings on the fly, which meant that with multiple workers they don’t necessarily match up, because of randomness in batch index generation.<br>
This was solved by putting the one hot encoding in the <code>__getitem__</code> function of the dataset.<br>
Leaving this up as a reminder for my stupidity.</p>
What is the use of exportedprogram?

sanbuphy: <p>Hello, recently I’ve been researching the latest torch export related things, but I don’t know why we need this exportedprogram. What’s the difference between this and the torchscript obtained using jit and script? I want to run ts using c++ and can use libtorch, so what should I do if I want to run exportedprogram using c++?</p>
<p>by the way , If I want to register some graph optimization operations myself, how can I achieve this with export? or just run it on python ir,  Thank you very much .</p>

marksaroufim: <p>torch.export and torch.jit differ in their graph capture mechanism, jit is on maintenance mode so you should be using export moving forward. in particular aot inductor is the new recommended path for people wanting c++ deployments</p>

sanbuphy: <p>Thanks！  is that anything useful learning materials about  aot inductor?  something like video or code</p>

marksaroufim: <div class="youtube-onebox lazy-video-container" data-video-id="w7d4oWzwZ0c" data-video-title="Lightning Talk: AOTInductor: Ahead-of-Time Compilation for PT2 Exported Models - Bin Bao, Meta" data-video-start-time="771s" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=w7d4oWzwZ0c&amp;t=771s" target="_blank" rel="noopener">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/w7d4oWzwZ0c/maxresdefault.jpg" title="Lightning Talk: AOTInductor: Ahead-of-Time Compilation for PT2 Exported Models - Bin Bao, Meta" width="690" height="388">
  </a>
</div>
FFT backwards error

xianggkl: <p><a href="https://discuss.pytorch.org/t/svt-backward-error-for-low-rank-structure/192397">SVT backward error for Low-rank structure</a><br>
Above is other problem which I meet in the same work I do.<br>
What’s more, for tensor-SVD, we usually transform the original data to the Fourier domain (using FFT or DCT), then do matrix-SVT for each frontal slice, and at last transform back (using inverse-FFT or some inverse function),  to obtain a tensor-SVD-based Low-rank approximation of the original data.</p>
<p>But when I use the autograd and expect to learn the thresholding operator <code>tau</code>, the error occurs: <code>svd_backward:The singular vectors in the complex case are specified up to multiplication by e^{i phi}. The specified loss function depends on this phase term, making it ill-defined.</code></p>
<p>Is the reason of this error that the complex case leads to incorrect gradients?</p>
The Loss can't back propagate to model's parameters with my customized loss function

abc50111: <p>I designed a customized loss:</p>
<pre><code class="lang-auto">class CustomIndicesEdgeAccuracyLoss(torch.nn.Module):
    def __init__(self, num_classes: int, selected_indices: list):
        super(CustomIndicesEdgeAccuracyLoss, self).__init__()
        self.num_classes = num_classes
        self.selected_indices = selected_indices

    def forward(self, input: torch.Tensor, target: torch.Tensor) -&gt; torch.Tensor:
        batch_size, num_classes, feature_size = input.shape
        selected_input = input[::, ::, self.selected_indices]
        selected_target = target[::, self.selected_indices]
        selected_preds = torch.argmax(selected_input, dim=1)
        edge_acc = torch.eq(selected_preds, selected_target).sum()/torch.numel(selected_preds)
        loss = 1 – edge_acc
        loss.requires_grad = True

        return loss
</code></pre>
<p>But the <code>loss</code> won’t back propagate to model’s parameters, in other word, the gradient of model’s parameters are always 0 and the model’s parameters can’t be updated.<br>
What’s possible reasons? How should I revise the codes?</p>
<p>Here is some information of the local variables of <code>forward()</code>:</p>
<pre><code class="lang-auto">input.shape: torch.Size([64, 3, 5])
target.shape:torch.Size([64, 5])
selected_input.shape: torch.Size([64, 3, 2]) 
selected_target.shape:torch.Size([64, 2])
</code></pre>

JuanFMontesinos: <p>Indeed, the gradients are zero because there is no single mathematical operation in your loss.<br>
Actually, the only reason you don’t get an exception is very likely this <code>loss.requires_grad = True</code> cos otherwise I even doubt you have a backprop graph.</p>
<p>Note that indexing (<code>input[::, ::, self.selected_indices]</code>) will just take some elements, namely, gradients for all the non-selected elements will be zero.<br>
Argmax is same thing, only the selected element (the max element) will backpropagate a copy of the upstream gradients.<br>
Finally, torch.eq is neither differentiable operation. I’m not even sure it has a backprop method implemented.<br>
Also, I believe (although I don’t recall it right now) that <code>loss.requires_grad = True</code> may even reset the graph.</p>
Passing fine-tuned resnet encoder into SMP Unet

ronnief1: <p>I’m using the Segmentation Models Pytorch (SMP) library to train a U-net on healthcare images. To initialize the model, you run</p>
<pre><code class="lang-auto">model = smp.Unet(
    encoder_name='resnet50',
    encoder_weights='imagenet',
    classes=len(CLASSES),
    activation=ACTIVATION
)
</code></pre>
<p>However I’d like to pass my own resnet50 (from torchvision.models) that I pre-trained on a specific dataset into this argument. When I do that, I get the following error:</p>
<p>AttributeError: ‘ResNet’ object has no attribute ‘startswith’</p>
<p>Looking in the source code, the encoder_name arg clearly expects a string, so I don’t know how I would pass in my own encoder backbone?</p>
Pytorch export to ONNX

Ars_ML: <p>If I would use torch.autograd.Function like this:</p>
<pre><code class="lang-auto">class CustomLayerFunction(torch.autograd.Function):
    @staticmethod
    def symbolic(g, features):
        return g.op("CustomLib::CustomLayer", features)

    @staticmethod
    def forward(ctx, features):
        return 10 * features;
    
    @staticmethod
    def backward(ctx, grad_output):
        pass
</code></pre>
<p>Pytorch will create CustomLayer node while exporting to ONNX.</p>
<p>Is there another way (without subclassing torch.autograd.Function) to tell Pytorch stop tracing further and insert CustomLayer node?</p>
RNN model embeddings in preprocessing or in model

alexgagnon: <p>Hi,</p>
<p>I’m trying to make a basic RNN using glove for embeddings. Currently I’m thinking I might be able to do this in a preprocessing stage like so:</p>
<pre><code class="lang-auto">embeddings = vocab.GloVe(name='6B', dim=50)

def embed_text(embeddings, x):
    return embeddings.get_vecs_by_tokens(x[0]), x[1]

...
train_datapipe, test_datapipe = IMDB(split=('train', 'test'))
train_datapipe = train_datapipe.map(partial(process_labels, labels))
train_datapipe = train_datapipe.map(partial(tokenize_text, tokenize))
train_datapipe = train_datapipe.map(partial(embed_text, embeddings)) 
train_datapipe = train_datapipe.batch(batch_size=batch_size)
train_datapipe = train_datapipe.rows2columnar(["text", "label"])
train_dataloader = DataLoader(train_datapipe, batch_size=None)
</code></pre>
<p>This seems to make sense to me, however the models I’m seeing online often look like this, with the embeddings as a parameter</p>
<pre><code class="lang-auto">class VanillaRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(VanillaRNN, self).__init__()
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.rnn = nn.RNN(hidden_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        embedded = self.embedding(x)
        output, _ = self.rnn(embedded)
        out = output[:, -1, :]
        return self.fc(out)
</code></pre>
<p>Does the model need to have the embedding param if I’m already doing it in the preprocessing stage? (I’m thinking yes for backpropagation purposes.) If I need it in the model, is there a point in doing it during preprocessing? And finally, it seems like the DataLoader spits out batches in a list format, whereas embedding() requires a single tensor… how can I make these play nice?</p>

vdw: <p>As long as you don’t have the intention to further train the embedding layer, I see no reason why you could consider as part of the data preparation. Given your methode</p>
<pre><code class="lang-auto">    def forward(self, x):
        ...
</code></pre>
<p>The only difference would be how <code>x</code> looks like. With an embedding layer as part of the network, the shape would be <code>(batch_size, seq_len)</code>. Without that layer it would need to be <code>(batch_size, seq_len, embed_size)</code>.</p>
<p>This of course also means that you have to ensure that all sequences in a batch have the same length, for example, by padding all sequences shorter than the longest sequence. If you can ensure this, you should be able to convert the output of the data loader to a tensor of shape <code>(batch_size, seq_len, embed_size)</code>.</p>
<p>Is there any particularly reason you want to do this?</p>
PyTorch version differences

pkadambi10: <p>Hello, I am comparing two PyTorch versions for JetPack 5.1 on a Jetson AGX Orin</p>
<p>torch-2.0.0+nv23.05-cp38 from <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048" rel="noopener nofollow ugc">here</a></p>
<p>torch-2.0.0a0+ec3941ad.nv23.2-cp38 from <a href="https://github.com/dusty-nv/jetson-containers/blob/L4T-R35.2.1/scripts/docker_build_ml.sh" rel="noopener nofollow ugc">here</a></p>
<p>I would like to understand how different these two versions are and possibly find the commits that are different between the two. Would appreciate any suggestions on how to do this.</p>

tom: <p>The 2.0.30a0+ec3941ad is a pre-2.0 version from the git repo of PyTorch from January 12 (the ec… is the commit id) while the torch-2.0.0 identifies PyTorch 2.0(.0) (git tag v2.0.0). Based on the NVidia page you linked, I would get the impression that they’re built directly from the source checkout as described on the pages you linked.</p>
<p>Best regards</p>
<p>Thomas</p>

pkadambi10: <p>Thanks for your reply, this is clear now.</p>
Creating custom tensor mask

PySmoke: <p>I am trying to find the fastest/most efficient way to create a tensor to use for a custom dropout layer.  I have not being able to figure out how to do this other than in a python loop to set the index of the tensor to 0 then multiply with the input.</p>
<p>More specifically, I want to set X number of random elements to 0 within a Y sized “group”. Example: 1d tensor of 10 elements, drop/group =2, groupSize=5: then 2 random indices in the the first half are 0 and 2 random indices from the second half are 0. Below is the code that I have working, but is terribly slow.</p>
<pre><code class="lang-auto">batch,feature = x.size()
scalar_mask = torch.tensor([self.scalar for _ in range(feature)])
for i in range(0,feature,self.group_size):
     masked_count=0
     while masked_count &lt; self.drop_per_group:
          index = random.randint(0, self.group_size-1) 
          if scalar_mask[i+index] !=0:
               scalar_mask[i+index]=0
               masked_count+=1
x = torch.multiply(x,scalar_mask)
</code></pre>
Element wise tensor multiplication and sums

luchungi: <p>Can DataParallel / DistributedDataParallel be used for basic tensor operations and sums without a model being involved?</p>
<p>For example, I have a custom loss function with computation that blows up the size of the output tensors from the model and I want to split these operations across GPUs. It is mostly basic operations such as element wise multiplication, sums and cumsums. I tried using DDP but it requires a module that has parameters which is not the case with the loss function.</p>

ptrblck: <p>No, <code>DistributedDataParallel</code> wasn’t designed to share operations but you could manually move parts of your tensors to another device via the <code>.to()</code> operation.</p>
Convolutional Neural Network Experimentation & Multi-Task Learning Implementation Suggestions

Ddav118: <p>I’m training multiple convolutional neural networks to identify a specific disease from chest x-rays (similar to <a href="https://arxiv.org/abs/1711.05225" rel="noopener nofollow ugc">this paper</a>). Two separate questions:</p>
<p>First, what is the best tool/library/package within the Pytorch ecosystem to systematically run experiments and log model results? Ideally, this tool would allow for training models with different sets of hyperparameters back-to-back (so I can leave it running and get all the results at once). I have tried Pytorch Lightning, but I am looking for a tool that allows more flexibility to modify architectures.</p>
<p>Second, I would like to implement multi-task learning - the second task is to predict the size of the heart. Any suggestions for modifying the architecture to accept this would be greatly appreciated.</p>
Torch.save output is a zip archieve

caner-ercan: <p>I am extracting features from each patches in .pt files and using these files for training in the downstream of my project. At one point I concatenate the feature files using a code similar to following:</p>
<pre><code class="lang-python">features1 = torch.load('File1.pt')
features2 = torch.load('File2.pt')
merged_features = torch.cat([features1, features2], dim=0)
torch.save(merged_features, 'merged_feature.pt')
</code></pre>
<p>However, I receive following error for some files when I try to load:</p>
<p><code>RuntimeError("merged_feature.pt is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))</code></p>
<p>Previously, I run the feature extraction and merging codes again and magically the issue was solved. However, it became persistent and for all the files. I do not know how to solve. From the following location the working and not working files can be seen:<br>
<a href="https://drive.google.com/drive/folders/1V2JLAfF9o2WjLUxwQPwYvxKfCr_bFuu1?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/1V2JLAfF9o2WjLUxwQPwYvxKfCr_bFuu1?usp=sharing</a></p>
<p>I would be glad if anyone can help to solve this issue.</p>
<p>pytorch: 1.3.1<br>
Red Hat 7.9</p>
Dist.init_process_group get stuck after dist.destroy_process_group()

zyhe: <p>Hi, everyone. I found dist.init_process_group() hang after dist.destroy_process_group().</p>
<p>How to solve this problem?</p>
<p>Here is my test code:</p>
<pre><code class="lang-auto">world_size = int(os.environ.get("WORLD_SIZE", "1"))
rank = int(os.environ.get("RANK", "0"))
dist.init_process_group(
    backend="nccl",
    init_method="env://",
    world_size=world_size,
    rank=rank,
)
dist.destroy_process_group()
print("destroy finish")
dist.init_process_group(    ## get stuck here
    backend="nccl",
    init_method="env://",
    world_size=world_size,
    rank=rank,
)
dist.destroy_process_group()
</code></pre>
Does PyTorch 2.1 Support Learnable Post-Training Quantization?

siahuat0727: <p>I’m exploring PyTorch 2.1’s quantization features and have a question: Is learnable post-training quantization (PTQ) supported in this version? Specifically, I’m interested in adjusting quantization parameters, like scales, while keeping other model weights fixed.</p>
<p>Thank you for your assistance and insights.</p>

jerryzh168: <p>yes, it’s possible to support this, you’ll need to write your own quantizer though: <a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2 Export Quantization — PyTorch Tutorials 2.1.1+cu121 documentation</a> and you can use this to specify the LearnableFakeQuant: <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/quantizer/quantizer.py#L59" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/quantizer/quantizer.py#L59</a><br>
<a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/_learnable_fake_quantize.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/_learnable_fake_quantize.py</a></p>
<p>I’m not sure what learnable PTQ means actually</p>
Compiling scriptmodel with tensorRT?

may1: <p>How can I compile scripted module with TensorRT?<br>
I read this pytorch docs <a href="https://pytorch.org/TensorRT/tutorials/use_from_pytorch.html#use-from-pytorch" class="inline-onebox" rel="noopener nofollow ugc">Using Torch-TensorRT Directly From PyTorch — Torch-TensorRT master documentation</a><br>
and followed the instruction.</p>
<p>made the script_model using torch.jit.script<br>
and put the same spec to</p>
<pre><code class="lang-auto">spec = {
    "forward":
        torch_tensorrt.ts.TensorRTCompileSpec({
            "inputs": [torch_tensorrt.Input([1, 3, 300, 300])],
            "enabled_precisions": {torch.float, torch.half},
            "refit": False,
            "debug": False,
            "device": {
                "device_type": torch_tensorrt.DeviceType.GPU,
                "gpu_id": 0,
                "dla_core": 0,
                "allow_gpu_fallback": True
            },
            "capability": torch_tensorrt.EngineCapability.default,
            "num_min_timing_iters": 2,
            "num_avg_timing_iters": 1,
        })
    }
</code></pre>
<pre><code class="lang-auto">trt_model = torch._C._jit_to_backend("tensorrt", script_model, spec) 
</code></pre>
<p>However, it made the input type error.<br>
Input should have received that torch_tensorrt.Input but it received the whole dictionary as input. So I changed the dictionary format into</p>
<pre><code class="lang-auto">    spec = {
        "forward":
            torch_tensorrt.ts.TensorRTCompileSpec(
                inputs = [torch_tensorrt.Input((10,1,19,500))],
                enabled_precisions = {torch.float},
                refit = False,
                debug = False,
                device = {
                    "device_type": torch_tensorrt.DeviceType.GPU,
                    "gpu_id": 0,
                    "dla_core": 0,
                    "allow_gpu_fallback": True
                },
                capability = torch_tensorrt.EngineCapability.default,
                num_min_timing_iters= 2,
                num_avg_timing_iters= 1,
            )
        }
</code></pre>
<p>and now torch_tensorrt.ts.TensorRTCompileSpec function could read input well.<br>
However, still it gives the following error.</p>
<pre><code class="lang-auto">trt_model = torch._C._jit_to_backend("tensorrt", model, spec)
RuntimeError: [Error thrown at /workspace/Torch-TensorRT/py/torch_tensorrt/csrc/tensorrt_backend.cpp:69] Expected core::CheckMethodOperatorSupport(mod, it-&gt;key().toStringRef()) to be true but got false
Method forwardcannot be compiled by Torch-TensorRT
</code></pre>
<p>I am following the tutorial docs and checked that the model was scripted well.<br>
How can I solve this problem?<br>
Also, when I tried to convert the pytorch model with torch_tensorrt.compile() directly.<br>
It made segmentation fault error.</p>

Yue_Wang1: <p>same error here. any workarounds?</p>
How do I add overloads for custom ops defined via torch.Library?`

rvijayc: <p>I am trying to use the <a href="https://pytorch.org/docs/main/library.html" rel="noopener nofollow ugc">torch.library API</a> to create my own torch custom operators. The API seems to be missing an option to allow overloaded operators.</p>
<p>For example, suppose I define a Op like this:</p>
<pre><code class="lang-auto">torch.library.define("my_lib::my_op", "(Tensor x, float y) -&gt; Tensor")
</code></pre>
<p>I want to add an overloaded version that takes two Tensor arguments instead.</p>
<pre><code class="lang-auto">torch.library.???("my_lib::my_op", "(Tensor x, Tensor y) -&gt; Tensor", overload_name="tensor_version")
</code></pre>
<p>How can I do this?</p>

rvijayc: <p>Turned out to be something quite simple. I had to do this, but this wasn’t obvious from <a href="https://pytorch.org/docs/main/library.html" rel="noopener nofollow ugc">torch.library API</a>. The documentation may need to include this.</p>
<pre><code class="lang-auto">torch.library.define("my_lib::my_op.&lt;overload_name&gt;", "(Tensor x, float y) -&gt; Tensor")
</code></pre>
<p>Now, I need to figure out how to add “out” variants for these ops <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Whats the rule of thumb when it comes to module fusion?

Shisho_Sama: <p>Hello everyone, hope you are all having a great time.<br>
I was recently thinking about module fusion and since we have many examples of fused operators in torch I tried a simple example to see how much of an improvement can be expected even in a simple case, I noticed not much really, and I even got worse results.<br>
As an example, I tried</p>
<pre><code class="lang-auto">key = nn.Linear(5,5, bias=False)
query = nn.Linear(5,5, bias=False)
value = nn.Linear(5,5, bias=False)
x = torch.randn(size = (3,2,5))
k,q,v = [m(x) for m in (key, query, value)]

kqv = nn.Linear(5,15, bias=False)
kqv_output = kqv(x)
k2,q2,v2 = torch.split(kqv_output, 5, dim=-1)
</code></pre>
<p>for large numbers, the unfused version works better, for smaller sizes, the second one seems better.<br>
so what is the culprit here? is it the abstraction thats causing the second method to perorm worse? or are there any rules of thumb when it comes to fusion in general or in pytorch?</p>
<p>Thanks a lot in advance</p>
Imbalanced sampling of WeightedRandomSampler when given same weights

CodeOfKkkkken0502: <p>I’m using WeightedRandomSampler for a long-tailed dataset to get more balanced samples. It works okay, but quite a few samples are not selected. So I give less imbalanced weights to cover more of the dataset, but still many samples are never selected. Then I try the same weight for every sample, but it still samples imbalancedly.<br>
For example,</p>
<blockquote>
<p>samples = list(torch.utils.data.WeightedRandomSampler([0.01 for _ in range(100)], 100, replacement=True))<br>
samples_counter = collections.Counter(samples)</p>
</blockquote>
<p>And len(samples_counter) is usually around 66 (2/3 * 100), which means only 2/3 of all samples are sampled even they share the same weight.<br>
Changing the number of samples, I still got similar result that only 2/3 of all samples are chosen. But I’m expecting that roughly all samples can be selected because they are given the same weights. So how can I cover more of the dataset with the WeightedRandomSampler?<br>
Thanks for any help.</p>

tom: <aside class="quote no-group" data-username="CodeOfKkkkken0502" data-post="1" data-topic="192283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/codeofkkkkken0502/48/65649_2.png" class="avatar"> CodeOfKkkkken0502:</div>
<blockquote>
<p>And len(samples_counter) is usually around 66 (2/3 * 100), which means only 2/3 of all samples are sampled even they share the same weight.<br>
Changing the number of samples, I still got similar result that only 2/3 of all samples are chosen. But I’m expecting that roughly all samples can be selected because they are given the same weights. So how can I cover more of the dataset with the WeightedRandomSampler?</p>
</blockquote>
</aside>
<p>This is an effect of the statistics of sampling 100 values randomly from 1…100.  If you insist on WeightedRandomSampler, you could sample more values or just run it for more epochs.<br>
My go-to solution is to duplicate examples from rarer classes at the Dataset level. This requires some care when using BatchNorm or so, but usually works quite well.</p>
<p>Best regards</p>
<p>Thomas</p>

CodeOfKkkkken0502: <p>I see. Thanks for your answer!</p>
Is there way to build the tensor manipulate subset of pytorch?

Fnypmb_Tqti: <p>There’re some use cases that only need tensor manipulation, no need backprob and other things</p>

tom: <p>For mobile builds it is supported to disable autograd, but if you want it more generally, you might need to dig into the build system. Searching for DISABLE_AUTOGRAD or BUILD_MOBILE might give you some ideas.</p>
<p>Best regards</p>
<p>Thomas</p>
Torch.cdist operation using multiple GPUs

Megh_Bhalerao: <p>I want to do a pairwise distance computation on 2 feature matrices of sizes say <code>n x f</code> and <code>n x f</code>, and get an <code>n x n</code> matrix from this.</p>
<p>I am currently using <code>torch.cdist()</code> for this, and was wondering if there is any way to parallelize this across GPUs, something like how FAISS does - <a href="https://github.com/facebookresearch/faiss" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors.</a>?</p>
<p>Or do I need to write a custom implementation for this using torch multiprocessing?</p>
<p>Thanks for the help in advance!</p>
Attribute Error on adding new attributes to a custom dataset

Kuljeet: <p>I am attempting to add two new attributes to my custom dataset that can give me the shape of input data and a class map of unique classes. I am getting a AttributeError , seemingly because the new attributes are not part of the base class. Is there any way I can achieve this ?</p>
<pre><code>def __getattr__(self, attribute_name):
    if attribute_name in Dataset.functions:
        function = functools.partial(Dataset.functions[attribute_name], self)
        return function
    else:
        raise AttributeError &lt;== ERROR THROWN
</code></pre>
<p>My custom dataset class :<br>
from torch.utils.data import Dataset</p>
<p>class CustomDataset(Dataset):<br>
def <strong>init</strong>(self, data_path, img_dir, labels_csv, transform=None):<br>
self.datapath = data_path<br>
self.img_dir = img_dir<br>
self.labels_csv = labels_csv<br>
self.transform = None<br>
self.labels = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
self.labels_df, self.labels_map = load_labels(<br>
os.path.join(data_path, labels_csv)<br>
)<br>
self.filelist = os.listdir(os.path.join(data_path, img_dir))</p>
<pre><code>def __len__(self):
    return len(self.labels_df)

def __getitem__(self, idx):
    item = self.labels_df["Packet_Number"][idx]
    subs = "PT" + str(item) + "_"
    filename = "".join(filter(lambda x: subs in x, self.filelist))
    image = get_features(os.path.join(self.datapath, self.img_dir, filename))
    class_idx = self.labels_map[self.labels_df["Classification"][idx]
    self.image_shape = image.shape
    if self.transform:
        return self.transform(image), class_idx
    else:
        return torch.from_numpy(image), torch.tensor([class_idx])
</code></pre>
<h1><a name="methods-to-get-two-new-attributes-1" class="anchor" href="#methods-to-get-two-new-attributes-1"></a>methods to get two new attributes</h1>
<pre><code>def get_classes(self):
    return self.labels_map

def get_shape(self):
    return self.image_shape
</code></pre>
<p>Is there any other way by which new attributes and methods can be added to the base Datasets Class ?</p>
<p>Thank you in advance.</p>

ptrblck: <p>Your code works fine for me:</p>
<pre><code class="lang-python">class CustomDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(10, 3, 24, 24)
        self.labels_map = {
            "a": 1,
            "b": 2,
        }
        self.image_shape = self.data.shape
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        return self.data[idx]

    def get_classes(self):
        return self.labels_map
    
    def get_shape(self):
        return self.image_shape
    
dataset = CustomDataset()
print(len(dataset))
# 10
print(dataset.get_classes())
# {'a': 1, 'b': 2}
print(dataset.get_shape())
# torch.Size([10, 3, 24, 24])
</code></pre>

Kuljeet: <p>Thank you so very much, it was my mistake. I was trying to use image.shape after instantiating the Dataset object, but it was getting assigned in <strong>getitem</strong>().</p>
How can I install a low-version pytorch with my high-version CUDA 

vonct: <p>My <code>CUDA version is 11.8</code>. And I want to run a deep learning model which proposed a few years ago.<br>
In the README file of the model, it runs on <code>pytorch1.6</code>, but when I look into the pytorch previous version website(<a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a>). I only found commands of CUDA 9.2 10.1 10.2. In addition, I found that previous version of pytorch(below 1.6), I have to add <code>cudatoolkit</code> in my command. Why I don’t have to add this in recent version?<br>
Anyway, how can I or can I install pytoch1.6 when my CUDA is 11.8, in order to reimplement a deep learning model.</p>
<p>This is a question disturbs me at a long time, hopefully someone could help me.</p>

ptrblck: <aside class="quote no-group" data-username="vonct" data-post="1" data-topic="192213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/3ab097/48.png" class="avatar"> vonct:</div>
<blockquote>
<p>, I have to add <code>cudatoolkit</code> in my command. Why I don’t have to add this in recent version?</p>
</blockquote>
</aside>
<p>Build processes and dependencies as well as package names change over the years. You could check the history in <code>pytorch/builder</code> when and why it was changed.</p>
<aside class="quote no-group" data-username="vonct" data-post="1" data-topic="192213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/3ab097/48.png" class="avatar"> vonct:</div>
<blockquote>
<p>Anyway, how can I or can I install pytoch1.6 when my CUDA is 11.8, in order to reimplement a deep learning model.</p>
</blockquote>
</aside>
<p>Your locally CUDA toolkit won’t be used by the PyTorch binaries as they ship with their own dependencies. If you thus want to use an ancient PyTorch release with its corresponding CUDA version, you could just install it. However, if you want to build an old PyTorch version with a new CUDA toolkit from source, you would need to cherry-pick all needed CUDA-related changes into the old PyTorch branch as the build will most likely fail.</p>

vonct: <p>That sounds rather complicated. I guess a better way is to either only use cpu to do my reimplement or use a high-version pytorch（perhaps change some api）</p>

ptrblck: <p>Yes, the best option would be to make your PyTorch code compatible with the latest release and use PyTorch 2.1.1 or the latest nightly.</p>

vonct: <p>I see. Thank you！Perhaps i would choose to use cpuOnly to do my reimplement <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
Is pytorch SDPA using flash attention V2?

Lelouch1: <p>I wanted to know if Pytorch was using the V2 of flash attention here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <a href="https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html?highlight=scaled_dot_product_attention#torch.nn.functional.scaled_dot_product_attention" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.scaled_dot_product_attention — PyTorch master documentation</a></p>
<p>It is not said in the description of the function, only V1 is mentioned (link above), however it seems to be the case according to the blog :</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/accelerating-generative-ai/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/accelerating-generative-ai/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/blog/accelerating-generative-ai/" target="_blank" rel="noopener nofollow ugc">Accelerating Generative AI with PyTorch: Segment Anything, Fast</a></h3>

  <p>This post is the first part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples of how these...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So is Flash Attention V2 implemented or not ?</p>

daixiangzi: <p>+1，I have same problem</p>

daixiangzi: <p>I think the current torch version(2.1.0) only to support flash attention v1</p>

ptrblck: <p>Nightlies should have it and according to <a href="https://github.com/pytorch/pytorch/pull/105602#issuecomment-1755453126">this post</a> PyTorch 2.2 should be the target release.</p>
Torch Extension Cuda Driver

Jerome_Ku: <p>Working on a custom torch <code>cuda</code> / <code>cpp</code> extension that loads a <code>cubin</code> image using the cuda driver (<code>cuLaunchKernel</code>).</p>
<p>Are there any precautions needed when calling <code>cuda</code> driver functions in the context of <code>torch</code> in an extension?  Getting a segfault when calling the kernel which has the following signature:<br>
<code>f(CUstream stream, CUdeviceptr C, CUdeviceptr A, CUdeviceptr B, int32_t M, int32_t N, int32_t K, int32_t stride_cm, int32_t stride_cn, int32_t stride_am, int32_t stride_ak, int32_t stride_bk, int32_t stride_bn)</code></p>
<p>I’m casting <code>A</code>, <code>B</code>, and <code>C</code> from <code>at::Tensor</code> to <code>CUdeviceptr</code> using <code>reinterpret_cast&lt;CUdeviceptr&gt;(A.data_ptr())</code> and using <code>at::cuda::getCurrentStream()</code> for <code>stream</code>.</p>

eqy: <p>You should not directly cast from <code>at::Tensor</code> to <code>CUdeviceptr</code> but rather use something like <code>tensor.data_ptr()</code> instead.</p>

Jerome_Ku: <p>Can you explain more please?</p>

eqy: <p>Assuming you are using the pointers to access the underlying tensor data, it would be incorrect as that pointer is actually an attribute of <code>at::Tensor</code> rather than the address of <code>at::Tensor</code> itself (which would be a host pointer). There are many examples of this pattern in e.g., APEX, which has many cuda extensions: <a href="https://github.com/NVIDIA/apex/blob/master/csrc/layer_norm_cuda_kernel.cu" rel="noopener nofollow ugc">apex/csrc/layer_norm_cuda_kernel.cu at master · NVIDIA/apex (github.com)</a></p>
Deptwise-separable convolution

Prasan_dh: <p>I am new to pytorch</p>
<p>I am trying to separate the commented code into deptwise and pointwise convolution,<br>
I want to know if my implementation is correct ?</p>
<p><span class="hashtag-raw">#nn</span>.Conv2d(1,32,kernel_size=3, stride=1,padding=0),<br>
nn.Conv2d(1,1,kernel_size=3, stride=1,padding=0, groups=1),  <span class="hashtag-raw">#deptwise</span><br>
nn.Conv2d(1,32,kernel_size=1, stride=1,padding=0),                <span class="hashtag-raw">#point-wise</span><br>
nn.ReLU(),</p>

eqy: <p>EDIT: You seem to only be using one input channel in your depthwise conv, in this case there is no difference between a depthwise convolution and a “regular” convolution.</p>
Is it possible to implement a element wise multiplication as a convolution layer?

Vaatbak: <p>Hey everyone, I was curious if it was possible to implement an elementwise multiplication as a convolutional layer or as a fully connected layer for example.</p>
<p>I have tried the following but it does not appear to give the correct output:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# we create a pytorch conv2d to act as an element wise matrix multiplication and compare it to a standard matrix multiplication
batch_size = 1
input_channels = 1
input_height = 3
input_width = 3
output_channels = 1
kernel_size = 3

#create a random input tensor
input = torch.randn(input_height, input_height, input_channels, batch_size)
print(input.shape)
reginput = torch.squeeze(input)
print(reginput.shape)

#define the convolutional layer
conv_layer = nn.Conv2d(input_height, input_height, bias=False, kernel_size= (1,1), stride= (input_height,input_width))

#create a second tensor to act as the kernel
weights = torch.tensor([[[[1, 2, 3], [4, 5, 6], [7, 8, 9]]]], dtype=torch.float32)

weightsreg = torch.squeeze(weights)

weightsconv = weights.view(input_height, input_height, 1, 1)

conv_layer.weight.data = weightsconv

#perform the convolution
output = conv_layer(input)

#perform the matrix multiplication
output2 = reginput*weightsreg

#compare the outputs
print(output)
print(output2)```

When run it can be seen that the output does not equal the regual elementwise matrix multiplication. Is there anything I am missing? Thank you in advance</code></pre>

eqy: <p>It is doable trivially in the case of a 1x1 input with N channels if you use depthwise convolution:</p>
<pre><code class="lang-auto">import torch

N = 32
a = torch.randn(N)
b = torch.randn(N)
out1 = torch.nn.functional.conv2d(a.reshape(1, N, 1, 1), b.reshape(N, 1, 1, 1), groups=N).reshape(N)
out2 = a * b
print(torch.allclose(out1, out2))
</code></pre>
Only support when num_heads is even in transformer

EvanZ: <p>This is a very strange error I’m getting:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
Only support when num_heads is even in transformer
</code></pre>
<p>It’s being called in my <code>forward</code> method even though I’m not doing inference (just training):</p>
<pre><code class="lang-auto">def forward(self, indices, mask):
        x = self.embed(indices)
        x = self.encoder(x, src_key_padding_mask=mask)
</code></pre>
<p>I have the number of heads set to 1 and it’s never caused an issue in the past. Same code works when <code>nhead=2</code>.</p>
<p>Edit: I have verified that this behavior changed going from v.1.11 to v.1.12. Is this change in behavior documented somewhere? I can’t find it.</p>

kaikai23: <p>I also met the same error, and I solved this by installing a newer pytorch. Specifically, 1.12 doesn’t work but 1.13 works in my case. Hope that helps.</p>

gzx: <p>I met the same error when i am using v2.0. Unfortunately, my system strictly needs num_heads = 1</p>
<ul>
<li>It is strange, because when nn.Module.training is set to True(i’m training my model), <strong>no error occurred</strong>.</li>
<li>When I apply .eval() for <strong>non-dropout</strong> model evaluation, error returns : <em>RuntimeError: Only support when num_heads is even in transformer</em>.</li>
<li>More strangely, when I abort padding_mask input, this error disappears and model can work. However, for attention model, I really need mask!</li>
</ul>
<pre><code class="lang-python">#this would avoid error, but is not a wise choice.
if self.training:
    encode_X = self.transformer_encoder(X,src_key_padding_mask=mask)
else:
    encode_X = self.transformer_encoder(X)
return encode_X
``</code></pre>

Ziheng_Zhang: <p>Did you solve this problem? Can you share your workaround?</p>

alper: <p>I have encountered a similar problem and I <em>guess</em> it is fixed in <code>v2.1.1</code> because my script only raised a warning.</p>
<p><strong>Edit:</strong> I can test it if you have a snippet.</p>
Torch::nn in iOS C++ is unavailable

dnnagy: <p>I started from the HelloWorld example on iOS and everything worked until i tried to add <code>nn</code> functionality like</p>
<pre><code class="lang-auto">auto loss = torch::nn::MSELoss(); // Error: No member named 'nn' in namespace 'torch'
</code></pre>
<p>Is there a way to include <code>torch::nn</code> and <code>torch::nn::functional</code> API in iOS?</p>

xta0: <p>Hi <a class="mention" href="/u/dnnagy">@dnnagy</a> the APIs under <code>nn</code> namespace are being used to support eager mode in Python. Since we’re using torchscript on mobile, we didn’t compile that part of code in our binary. However, I think you can manually enable it by changing the <code>NO_API</code> value to <code>OFF</code> in the root CMakeLists.txt and rebuild the libraries. However, I personally haven’t tried, so not sure if they’ll work or not.</p>

JP-Lisn: <p>Hi <a class="mention" href="/u/xta0">@xta0</a>, I realize this conversation is a little old, but am having a similar issue. I tried your suggestion of changing the <code>NO_API</code> value to <code>OFF</code> but when i do I the build fails with the following:</p>
<p><code>#&gt; BUILD_PYTORCH_MOBILE=1 IOS_PLATFORM=SIMULATOR </code>./scripts/build_ios.sh</p>
<pre><code class="lang-auto">pytorch/csrc/api/src/nn/module.cpp:5:10: fatal error: 'torch/csrc/autograd/generated/VariableType.h' file not found
#include &lt;torch/csrc/autograd/generated/VariableType.h&gt;
</code></pre>
<p>Seems someone has had and resolved this issue… <a href="https://discuss.pytorch.org/t/where-does-the-torch-csrc-autograd-generated-variabletype-h-file-move-to/40136/2">here</a> but the resolution was unclear.</p>
<p>Technically I only need <code>torch::nn::functional::pad()</code> might there be a way to include just this limited requirement?</p>

JP-Lisn: <p><a class="mention" href="/u/xta0">@xta0</a> I was able to build torchlib with <code>NO_API</code> value to <code>OFF</code>, I also had to set <code>BUILD_MOBILE_AUTOGRAD</code> to <code>ON</code> to resolve the missing VariableTypes.h issue above.</p>
<p>However, the <code>nn</code> namespace still does not appear to be included</p>

Cyrill: <p>I had a very similar issue. I wanted to do something like this on iOS Libtorch-Lite:</p>
<pre><code class="lang-cpp">auto inputTensor = torch::nn::functional::pad(inputTensor, padOptions);
</code></pre>
<p>So just copied <code>constant_pad_nd</code> from <a href="https://github.com/pytorch/pytorch/blob/1317dbf17660232d16d40efcbb134d3540446008/aten/src/ATen/native/PadNd.cpp#L29" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/1317dbf17660232d16d40efcbb134d3540446008/aten/src/ATen/native/PadNd.cpp#L29</a> and adjusted the <code>at::</code> namespace.</p>
<p>This avoids the hassle compiling your custom iOS build.</p>