akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Combining classification models and regression (Hierarchical? or MoE modeling?)

neurospacer: <p>Assume, there is a dataset of human poses captured from different angles. For each sample there are 2 level of labels - pose type (standing, sitting, etc) and pose magnitute(continuous).</p>
<p>Defined tasks, are to: 1) classify pose class and 2) pose magnitude (regression). I successfully implemented classification task on dose type with excellent mean accuracy score of 0.95. Now we want to build a model to predict pose magnitude of each class. <strong>Here</strong> is the tricky thing. I can train separate regression models to predict pose magnitude. But if new sample comes, we do not know to which class of pose it belongs, so we would want to first define its class and then gate it to corresponding regression model. Simplest solution, it to just create a decision rule, that if new sample belongs to class that route it to regressor for this class.<br>
<strong>But</strong> is there any intelligent way to do it? How can we train routing function simultaneosly with regression models? From MoE paper I got that authors used same smaller classifier for experts. But I don’t understand how to transfer this approach to my problem.</p>
<p>Would me much appreciated, for any feedback.</p>
Export model from transformers library

mhs4670go: <p>Hello. I’m trying to export <code>.pt2</code> from the model of <code>transformers</code> library.</p>
<p>But, I’ve got <code>module</code> object not callable error.</p>
<p>How can I call the <code>export</code> api for this case?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from transformers import T5Tokenizer, T5Model
model = T5Model.from_pretrained("t5-small")
ep = torch.export(model, model.dummy_inputs)
</code></pre>
<pre data-code-wrap="bash"><code class="lang-bash">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: 'module' object is not callable
</code></pre>

mhs4670go: <p>Very simple answer…</p>
<pre><code class="lang-auto">from torch.export import export
</code></pre>
Torch.index_select op that reuses underlying storage?

pclucas: <p>Hi,</p>
<p>Would it be possible to perform an operation like torch.index_select that does not allocate new memory ?</p>
<p>Thanks!<br>
-Lucas</p>

SimonW: <p>Not really with the strided view of tensors.</p>

pclucas: <p>What exactly do you mean by strided view ? Sorry if it’s a silly question</p>

SimonW: <p>It means that <code>x[..., i, ...]</code> and <code>x[..., i + 1, ...]</code> are fixed number of bytes apart. Specifically, given indices and strides for each dimension, one can calculate the pointer offset as <code>sum_i (stride[i] * index[i])</code>. So it is impossible to have tensor values that are arbitrarily bytes apart. It is also very inefficient if we support that.</p>

_torchbearer: <p>Hey Lucas, any chance you have solved this issue yet? I’m using index_select in my code and I need to make sure I don’t use up memory for this op.<br>
Thanks in advance!</p>
CUDA initialization: Unexpected error from cudaGetDeviceCount()

Ford: <p>A warning appears when<code> torch.cuda.is_available()</code> is executed, which is:</p>
<p><code>CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304:  OS call failed or operation not supported on this OS.</code></p>
<p>Does anyone know how to solve this？ Thanks</p>
Is it safe to open a read-only sqlite connection in a dataset?

Vedant_Roy: <p>From my understanding, a Pytorch dataset is forked by the dataloader. Does this mean the following code is safe to have in a dataset:</p>
<pre><code class="lang-auto">        self.metadata_db = sqlite3.connect(f'file:{metadata_db_path}?mode=ro', check_same_thread=False) if caption_db_path else None
        self.metadata_db_cursor = self.caption_db.cursor() if self.caption_db else None
</code></pre>
<p>and inside <code>__getitem__</code>, we do:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        # a bunch of stuff
        item = get_item_from_idx(idx)
        item_metadata = self.metadata_db_cursor.execute(f"select * from db where id = {item.id}")
        item["metadata"] = item_metadata
        return item
</code></pre>
pytorch autograd on linear combination weights in the parameter space

zhj2022: <p>I’m trying to multiply the parameters of one model (model A) by a scalar $\lambda$ to get another model (model B) which has the same architecture as A but different parameters. Then I feed a tensor into model B and get the output. I want to calculate the gradient of the output on $\lambda$ but the <code>.backward()</code> method doesn’t work. Specifically, I try to run the following program:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class MyBaseModel(nn.Module):
    def __init__(self):
        super(MyBaseModel, self).__init__()
        self.linear1 = nn.Linear(3, 8)
        self.act1 = nn.ReLU()
        self.linear2 = nn.Linear(8, 4)
        self.act2 = nn.Sigmoid()
        self.linear3 = nn.Linear(4, 5)
    def forward(self, x):
        return self.linear3(self.act2(self.linear2(self.act1(self.linear1(x)))))

class WeightedSumModel(nn.Module):
    def __init__(self):
        super(WeightedSumModel, self).__init__()
        self.lambda_ = nn.Parameter(torch.tensor(2.0))
        self.a = MyBaseModel()
        self.b = MyBaseModel()
    def forward(self, x):
        for para_b, para_a in zip(self.a.parameters(), self.b.parameters()):
            para_b.data = para_a.data * self.lambda_
        return self.b(x).sum()

input_tensor = torch.ones((2, 3))
weighted_sum_model = WeightedSumModel()
output_tensor = weighted_sum_model(input_tensor)
output_tensor.backward()

print(weighted_sum_model.lambda_.grad)
</code></pre>
<p>And the printed value is None.</p>
<p>I wonder how can I get the gradient of weighted_sum_model.lambda_ to optimize this parameter?</p>
<p>I tried various ways to get the parameters of weighted_sum_model.b but they all did’t work. And I visualized the computation graph of WeightedSumModel, on which there is only b but not a and lambda.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/zhj2022">@zhj2022</a>,</p>
<p>The <code>.data</code> attribute is depcretated and shouldn’t be used, replace the code with just <code>para_b = para_a + self.lambda_</code></p>
<aside class="quote no-group" data-username="zhj2022" data-post="1" data-topic="202206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhj2022/48/64944_2.png" class="avatar"> zhj2022:</div>
<blockquote>
<pre><code class="lang-auto">            para_b.data = para_a.data * self.lambda_
</code></pre>
</blockquote>
</aside>
How to get image segmentations from flowers102

nforryan: <p>Hi!<br>
I’m using the <a href="https://pytorch.org/vision/stable/generated/torchvision.datasets.Flowers102.html" rel="noopener nofollow ugc">Flowers102 dataset</a> and I’ve seen that the <a href="https://www.robots.ox.ac.uk/~vgg/data/flowers/102/" rel="noopener nofollow ugc">original</a> contains image segmentations for each image. Is there a way to use the Torch dataset to get these images?<br>
Thanks for any help!</p>
CUDA failed suddenly

Sam_Barroso: <p>Good morning!</p>
<p>I have been having a problem with Pytorch lately. I reformated the computer and installed everything I needed to work like torch. Then I ran many experiments which individialy use torch. They are working properly until torch, after some hours, suddenly says “CUDA driver initialization failed”.</p>
<p>The command nvidia-smi still working but something changed: the line under the name of the graphic card is not showing the power used (instead of 15W / 80W it says N/A  /  N/A). When I try to restart, the computer gets frozen so I have to force it. Then after a forced restart, everything is ok again.</p>
<p>I have used it in different conda environments, so, in principle, it is not the installation.</p>
<p>Do you know what to do in this case?</p>
<p>System: Ubuntu 22<br>
Graphic Card: Nvidia 4070</p>
<p>Thank you very much in advance!<br>
Sam.</p>

ptrblck: <p>The issue seems to be unrelated to PyTorch and points to your setup or driver so I would recommend asking in an NVIDIA discussion board.</p>

Sam_Barroso: <p>Thank you for answering!</p>
<p>It is strange since cuda stops working just when torch is running. However, I’ll ask in Nvidia forums about it!</p>
<p>Thanks again!</p>
My Problem about the installation of PyTorch in my computer

MurariV: <p>File “”, line 1, in <br>
File “C:\Users\Murari\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\torch_<em>init</em>_.py”, line 141, in <br>
raise err<br>
OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users\Murari\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\torch\lib\shm.dll” or one of its dependencies.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/murariv">@MurariV</a>,</p>
<p>This issue is being discussed in this thread on the forum: <a href="https://discuss.pytorch.org/t/error-loading-lib-site-packages-torch-lib-shm-dll-or-one-of-its-dependencie/201695/11" class="inline-onebox">Error loading "\lib\site-packages\torch\lib\shm.dll" or one of its dependencie - #11 by ptrblck</a></p>
Alternative to torch.inverse for 16 bit

Bruce_Muller: <p>Hi</p>
<p>torch.inverse() doesn’t work with half precision.</p>
<p>Is there an alternative way we can compute the inverse which could work? I known maybe it’s impossible with stability issues?</p>

tom: <p>I’m not aware of any backends implementing half-precision inverse off the shelve. Part of that might be stability.</p>
<p>Then a practical question would be what your use case is and what you hope to get from it. Is it for a certain problem shape? What is it that makes you prefer half over single precision? What do you need the inverse for?</p>
<p>Anecdotally, way back when I studied numerical linear algebra and analysis at the university, they used to say that when you explicitly compute the inverse, you’re doing it wrong. Now that may not be the case here for you, but I must admit it seems very special-purpose to use the explicit inverse while needing low-precision.</p>
<p>Best regards</p>
<p>Thomas</p>

sprakashdash: <p>I need to use covariance inverse in my optimization. Is there a way I could implement inverse in bfloat16?<br>
And I need low precision for large model training.</p>
Question on forward Parent/Child inheritance for torch.autograd.Function

bmgroth: <p>I have a family of functions that follow the following structure for the forward method.</p>
<pre><code class="lang-auto">class ParentFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx):
        output1 = ParentFunc.my_class_method1()
        output2 = ParentFunc.my_class_method2(output1)
        return output2

    @classmethod
    def my_class_method1(cls):
        return compute1()

    @classmethod
    def my_class_method2(cls, output1):
        return compute2(output1)

    @staticmethod
    def backward(ctx):
       pass # not important right now
</code></pre>
<p>With this structure, I am able to implement a general case that works for a lot of my child functions by simply inheriting the forward() and class methods, which is great. The hope was that when I needed to do edge cases, I would only have to change a few class method and use the other inherited code, rather than copy-paste the entire code block.</p>
<p>See the following edge case example:</p>
<pre><code class="lang-auto">class ChildFunc(ParentFunc):
    @staticmethod
    def forward(ctx):
        output1 = ChildFunc.my_class_method1() # new definiton
        output2 = ParentFunc.my_class_method2(output1)
        return output2

    @classmethod
    def my_class_method1(cls):
        return compute1_child()
</code></pre>
<p>When running ChildFunc, I can’t get it to call the overridden forward() OR my_class_method1(). In VSCode, it is showing that these functions reside in ParentFunc. The function inputs and outputs are the same, which seems to be a requirement of Python overriding.</p>
<p>Looking for options, there is name mangling where you change forward() to _forward() or __forward(), but that doesn’t work with the PyTorch framework to automatically call forward() w/ things like .apply() or <strong>call</strong>. When doing name mangling like _forward() or __forward(), VSCode acknowledges that the new definition resides in ChildFunc.</p>
<p>Is there anything I can do to implement this with inheritance? I am not a Python or PyTorch expert, so I am hoping I am missing something.</p>
PascalVOC `trainaug` and `test` dataset

AreTor: <p>In the <a href="https://arxiv.org/pdf/1802.02611v3.pdf" rel="noopener nofollow ugc">DeepLabv3+ paper</a> the authors state the following:</p>
<blockquote>
<p>The proposed models are evaluated on the PASCAL VOC 2012 semantic segmentation benchmark [1] which contains 20 foreground object classes and one background class. The original dataset contains 1, 464 (train), 1, 449 (val), and 1, 456 (test) pixel-level annotated images. We augment the dataset by the extra annotations provided by [76], resulting in 10, 582 (trainaug) training images.</p>
</blockquote>
<ol>
<li>What’s the exact PyTorch code to retrieve the trainaug dataset?</li>
<li>The size of trainaug considers also the 1464 images of the PASCAL VOC 2012 dataset?</li>
<li>What’s the exact code to retrieve the test dataset?</li>
</ol>

khaoula_boutiche: <p>Hello,<br>
I need also to use trainaug of PASCAL VOC, any suggestions?</p>
<p>Thank you</p>

AreTor: <p>Yes, I asked the authors and they provided me with the data <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Unfortunately, I don’t have them with me, so I suggest you write them!</p>

khaoula_boutiche: <p>Thank you for your response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> . I obtained the augmented masks from this link: <a href="https://www.dropbox.com/s/oeu149j8qtbs1x0/SegmentationClassAug.zip?dl=1" rel="noopener nofollow ugc">SegmentationClassAug.zip (dropbox.com)</a>.</p>
<p>Since you have already used them, could you please tell me why the masks are black and not colored? In this case, how can we utilize them?</p>

AreTor: <p>As far as I remember, the masks contain integers, each corresponding to a class.</p>
Query Auto Suggestion

Sai1: <p>Hello all,<br>
I have a dataset with two columns one of them is a prefix and the other is a suggestion. Eg: Prefix :University of M, Suggestion: University of Montreal. I trained a seq2seq model to achieve this task. But seq2seq models are more general and there can be many number of suggestions for the prefix 'University of '. I was able to extract a set of words from the dataset that provide context. I want to use this set of words to provide the model with context and some how I want the model to prioritize choosing from this set. The context words are all of different nature (Norcross, Trump, Football etc).  I don’t have any labelled data to predict slots. Is there any way I can achieve this task?<br>
Thanks in advance.</p>
Simulating int1 QAT with {-1,1} instead of {0,1}

khn4hi: <p>I am trying to simulate the int1 training and configuring the QConfig manually to do that using the quant_min and quant_max range. However, for int1 training, I want my values to be {-1,1} instead of {0,1}. Is there anyway to do that? Also, since torch does not go natively below qint8 and quint4, converting values into booleans will be a faster and efficient way for storing the int1 representations?</p>
<p>Thanks for your answers in advance!</p>

vadimkantorov: <p>For adding to core bit packing/unpacking utils, please chime in  <a href="https://github.com/pytorch/pytorch/issues/32867" class="inline-onebox" rel="noopener nofollow ugc">[feature request] np.packbits / np.unpackbits, general BitTensors (maybe can be just tensors with dtype torch.bits8 or have a new dtype torch.bits introduced) and bit packed tensors utilities for saving memory / accesses, support for BitTensors wherever BoolTensors are used · Issue #32867 · pytorch/pytorch · GitHub</a> to add info about your usecase <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I think the general sentiment is that a pull request is needed</p>
Forward pass with different weights for the same batch

CCL: <p>I’m implementing a custom forward function for e.g. <code>nn.Conv2d</code> and <code>nn.Linear</code> but I want to do 1 forward pass with N slightly noised versions of the weight for each sample in the mini-batch, without having to break it down into N different forward passes. For example, I might do:</p>
<pre><code class="lang-auto">for p in range(perturbations):
    features.append(F.conv2d(input[p, :], weight+noise[p], bias, stride, padding, dilation, groups))
</code></pre>
<p>So input would be of dimensions <code>B x C_in x H x W</code> whereas the noised-weights would be of dimensions <code>N x C_out x C_in x k x k</code> where <code>N = B</code> in the above example. How can I vectorise the forward passes? Thanks!</p>

KFrank: <p>Hi Ccl!</p>
<aside class="quote no-group quote-modified" data-username="CCL" data-post="1" data-topic="202108" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/dbc845/48.png" class="avatar"> CCL:</div>
<blockquote>
<p>I want to do 1 forward pass with N slightly noised versions of the weight for each sample in the mini-batch<br>
…<br>
So input would be of dimensions <code>B x C_in x H x W</code> whereas the noised-weights would be of dimensions <code>N x C_out x C_in x k x k</code><br>
…<br>
How can I vectorise the forward passes?</p>
</blockquote>
</aside>
<p>Use the <code>groups</code> feature of <code>conv2d()</code>.</p>
<p>In more detail:</p>
<p>Expand your input to create <code>N</code> copies of itself, packaged in the <code>in_channels</code><br>
dimension, so that it has dimensions <code>B x N * C_in x H x W</code>.  Then use<br>
<code>groups = N</code> in your call to <code>conv2d()</code> and package your <code>N</code> noisy copies of your<br>
<code>weight</code> tensor into a single <code>weight</code> tensor with dimensions, similar to as you<br>
described, of <code>N * C_out x C_in x k x k</code>.  You will also have to <code>expand()</code><br>
the <code>bias</code> you pass in to replicate it <code>N</code>-fold to dimensions <code>N * C_out</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

CCL: <p>Looks promising, I’ll test this out very soon, thanks!</p>
Convert Array to Tensor in C++

Jakov_Gaspar: <p>Hi,</p>
<p>I have to give a matrix as input to my network, which is a standard C++ 2d array, and I can’t find a method to transform my data to a tensor. I hope someone can help me.</p>
<p>Thanks</p>

Bill_CX: <p>How did you implement your matrix? Similar question here, is there a way to do</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
torch::Tensor a = torch::tensor(~matrix`)
</code></pre>
<p>Thanks</p>

Jakov_Gaspar: <p>it’s read from a .txt file the matrix is build this way:</p>
<p>float **matrix = new float *[rows];</p>
<p>for (size_t row = 0; row != rows; row++) {<br>
matrix[row] = new float[cols];<br>
}</p>

Jakov_Gaspar: <p>Found the solution. You can use the torch::from_blob() function to read an array and turn it to a tensor. Apparently you also need to call .clone() because of some memory issue.</p>

ptrblck: <p>I think the <code>clone()</code> call is only necessary, if you don’t want to keep the passed array alive, which would invalidate the data?</p>

Jakov_Gaspar: <p>I did not mention here that everything is processed with cuda, i had to insert <code>clone() </code>, otherwise i just got an error.</p>

naveenmarthala: <p>I have a 4D array of shape <code>1x3x224x224</code> which has floats, can I not convert it a torch tensor directly by doing <code>at::Tensor myTensor = torch::from_blob(myArray.clone(), {1, 3, 224, 224}, torch::kFloat);</code></p>
DataLoader, IterableDataset and parallelization

ekrx: <p>I’m using a generator wrapped on a IterableDataset then passed to a DataLoader. The data is being read at demand from the disk because there are thousands of files. So the IterableDataset has some logic to keep track of the open files and the cursors of each file and some funky compurations. If I want to create multiple workers to prefetch the data in the dataloader, I understand this is being created by creating processes. Hence this will cause the data to be read multiple times per epoch.  Is this the right understanding?</p>
<pre><code class="lang-auto">class Dataset(IterableDataset):
    def __init__(self):
        self.x = np.random.rand(100,2)
        self.y = np.random.rand(100,1)

    def __iter__(self):
        return iter(self.generator())
    
    def generator(self):
        for x, y in zip(self.x, self.y):
            yield x, y
</code></pre>
<p>and</p>
<pre><code class="lang-auto">def main():
   train = Dataset()

    count = 0
    for batch in train:
        count += 1

    print(f'count={count}')

    train_loader = DataLoader(dataset=train, batch_size=3)

    count = 0
    for batch in train_loader:
        count += batch[0].shape[0]

    print (f'count={count}')
    train_loader = DataLoader(dataset=train, batch_size=3, prefetch_factor=10, num_workers=3)

    count = 0
    for batch in train_loader:
        count += batch[0].shape[0]

    print (f'count={count}')


if __name__ == '__main__':
    main()
</code></pre>
<p>prints the following:</p>
<pre><code class="lang-auto">count=100
count=100
count=300
</code></pre>
<p>Any recommendations how to prevent this or fix this?</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset">docs</a> show an example using the <code>worker_info</code> inside the <code>__iter__</code> to avoid having duplicate data returned from all workers.</p>
How to use docker image of libtorch?

Diazepam: <p>Dear developers:</p>
<p>So sorry to bother you again and I have trouble in the <a href="https://hub.docker.com/r/pytorch/libtorch-cxx11-builde" rel="noopener nofollow ugc">pytorch/libtorch-cxx11-builde</a>.</p>
<p>I havn’t find any document about this docker image and I tried to build my applications on it, so I pull the docker image:</p>
<pre data-code-wrap="bash"><code class="lang-bash">docker pull pytorch/libtorch-cxx11-builder:cuda121
</code></pre>
<p>And I tried to find libtorch:</p>
<pre data-code-wrap="bash"><code class="lang-bash"># Enter the container
docker run -t -i --rm pytorch/libtorch-cxx11-builder:cuda121 /bin/bash
# Try to find libtorch in the container
find / -name "libtorch.so"  # Nothing to return
find / -name "*torch*"        # Nothing to return
</code></pre>
<p>I’m wondering about the purpose of creating this image:</p>
<ul>
<li>Is it used to build other programs?</li>
<li>Do I need to install libtorch in the container myself?</li>
</ul>
<p>I appreciate your dedication and have a happy and blessed life! <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=12" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:" loading="lazy" width="20" height="20"></p>
<p>John</p>

ptrblck: <p>This container is a builder image used in the <code>pytorch/builder</code> repository to build the binaries.</p>
VGG16 faster at inference than MobileNet V3 Small on GPU

Dot_Pixis: <p>Hi.<br>
I’m doing the inference of VGG16 and MobileNet V3 small models on Google Colab using NVIDIA Tesla T4 GPU, with PyTorch 2.2.1 framework and cuDnn 8906. When measuring the average inference time per image using the code below, I got the follwing results:<br>
VGG16 : 14.38 ms per images<br>
MobileNet V3 Small : 17.39 ms per images<br>
It can be seen that VGG16 is faster than MobileNet V3 small even thogh MobileNet is more efficient and has much less GFLOPS than VGG (0.06 for MobileNet and 15.47 for VGG16).<br>
However when the inference is done on CPU, MobileNet is much faster than VGG16 with 24ms and 906 ms per image respectively.<br>
How could you explain this ?</p>
<p><strong>The used Batch size is 1</strong></p>
<pre><code class="lang-auto">dummy_input = torch.randn(1, 3,256,256, dtype=torch.float).to(device)
starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
timings=np.zeros((len(test_datasets),1))
#GPU-WARM-UP
for _ in range(10):
    _ = model(dummy_input)
# MEASURE PERFORMANCE
loop = tqdm(test_dataloader) # Progress bar
inference_time = 0
rep = 0
for inputs, labels, paths in loop:
  inputs = inputs.to(device)
  labels = labels.to(device)
  starter.record()
  outputs = model(inputs)

  ender.record()
  # WAIT FOR GPU SYNC
  # torch.cuda.synchronize()
  curr_time = starter.elapsed_time(ender)
  timings[rep] = curr_time
  rep = rep + 1

mean_syn = np.sum(timings) / len(test_datasets)
std_syn = np.std(timings)
print('\nAverage time per image : ', mean_syn)
print('Total inference time : ', sum(timings))
</code></pre>

ptrblck: <p>Different optimizations can be used e.g. via cuDNN. You could disable it and profile the code again using fallback kernels.</p>

Dot_Pixis: <p>Hi. Thank you for your reply.</p>
<p>If I understand correctly, I should disable CuDNN and execute the same previous code. I did this using the same GPU and this is what I got:</p>
<p><strong><em>VGG16</em> : 20.9 ms/image</strong> (14.38 ms with CuDNN),<br>
<strong><em>MobileNet-V3-Small</em> : 21.81 ms/img</strong> (17.39 ms with CuDNN).</p>
<p>It can be seen that both models are slower when CuDNN is disabled, and that VGG16 is still faster than MobileNet V3 small, except that the gap between the two inference times is smaller.</p>
<p>Are there any other optimizations I need to disable?</p>
<p>Thanks!!</p>

ptrblck: <p>You could profile the code using e.g. <code>Nsight Systems</code> and check the visual timelines of both models to understand where their bottlenecks are and which kernels are used. The profile might also show that your use case is e.g. CPU-limited if you are seeing a lot of whitespaces between kernels.</p>
Optimizing Model Predictions with Multiple prediction Combinations as correct answer

alireza_samadi: <p>I have a model that predicts values A and B, where A and B are the outputs based on the input. From these predictions, I calculate C_predict using the formula: C = A * (1 - exp(-1 / (B + epsilon))), where epsilon is a small value to prevent division by zero and B=&gt;0 . During training, my ground truth for loss computation is C_groundtruth, but I also have ground truth values for A and B, although they are not directly used for loss computation. They serve to ensure the correctness of predicted values for A and B. The loss is computed using a loss function between C_predict and C_groundtruth.</p>
<p>However, there exist various combinations of A or B that can yield similar results for C. For instance, instead of predicting high values for B (which is the correct answer), my model might predict very low values for A. Is there a method to compel my model to predict the correct values, or is it inherently non-deterministic?</p>
Generalized_box_iou_loss diverge to negative value

Mohammed_Nassar: <p>Hello, PyTorch community,</p>
<p>I’m currently working on an object detection task and I’m interested in implementing the Generalized Intersection over Union (GIoU) Loss instead of the usual MSELoss. While referring to the <code>generalized_box_iou_loss</code> function in PyTorch, I noticed that this loss function expects bounding box values to adhere to the condition <code>0 &lt;= x1 &lt; x2</code>. I have a couple of questions regarding this:</p>
<ol>
<li>
<p>My bounding box regression values are normalized with respect to the image width and height. Should I directly input these normalized values into the GIoU loss function, or is it necessary to denormalize them before use?</p>
</li>
<li>
<p>During the initial training phase, the model’s output can be quite unpredictable. This makes it challenging to ensure that the condition <code>0 &lt;= x1 &lt; x2</code> is always satisfied, potentially leading to negative loss values. In such cases, the optimizer tends to drive the loss towards negative infinity.</p>
</li>
</ol>
<p>I’m seeking guidance on how to effectively train using the GIoU loss function under these conditions. Any insights or recommendations would be greatly appreciated.</p>
<p>Thank you in advance for your assistance!</p>

Valay_Bundele: <p>Hi,</p>
<p>I have the same doubts and was wondering if you have already figured this out. Please let me know.</p>
<p>Thanks in advance!</p>

miotto: <p>There is a couple of ways of going about this problem, here is a suggestion:</p>
<ol>
<li>
<p>Work with bounding boxes in the YOLO format (center_x, center_y, bb_width, bb_height) normalized with respect to (image_width, image_height). The normalization ensures that all target bounding boxes are represented by values between 0 and 1.</p>
</li>
<li>
<p>Call torch.sigmoid() on the output of your model. This keeps all predicted values in the range 0 to 1.</p>
</li>
</ol>
<p>Using this strategy, it is impossible for the model to output an invalid bounding box. Also, the center of the bounding box, and at least a quarter of its area, will be always inside the image.</p>
Performing convolution with large kernels

AreTor: <p>I have very large kernels (from 63 x 63 to 255 x 255) and would like to perform convolutions on an image of size 512 x 512. Currently, I get OOM errors because I think that PyTorch performs an <code>nn.Unfold</code> operation to treat the convolution as a matrix-vector product.</p>
<p>Is there a way to perform such large convolutional operations, for example using a distributed or a sequential approach?</p>

ptrblck: <p>Depending on your setup and if cuDNN is used, internally a matrix multiplication could be used or any other cuDNN algorithm.<br>
Could you post the activation shape, which is causing the OOM?</p>

AreTor: <p>Input and output have size <code>n x 21 x 512 x 512</code>, and I pad the image to get the correct sizes.</p>
<p>I can go with <code>n = 1</code> if needed, however, I prefer to keep <code>&gt; 1</code>. This should be a minor problem since it can be parallelized.</p>
<p>The problem is that the number of input windows (<code>n x 512^2 x k^2</code>) generated by the mini-batch is too large and cannot fit into memory. With <code>n = 1</code> and <code>k = 255</code> (my max kernel size) and working with <code>float32</code>, the kernel windows occupy ~63GB of memory.</p>

AreTor: <p>I was thinking to write a C++ implementation in LibTorch of a customized convolution, that iterates over the pixels of the image and performs the convolution, without the time-efficient / memory-unefficient <code>unfold</code> trick.</p>
<p>Alternatively, I can work in Python by cuting the input image and performing the convolution space-wise (was thinking to use the group parameter of <code>Conv2d</code> but I don’t think it’s possible), taking care of correct padding between the image patches.</p>
<p>The last resort could be to implement a <a href="https://pytorch_sparse" rel="noopener nofollow ugc">pytorch_sparse</a> unfold operator, considering that the images have lot of pixels set to 0.</p>

ashesh-0: <p>I’ve started working with microscopy data which has a use-case for this (convolution with large kernels). Do you know a workable solution for this?</p>

AreTor: <p>Yes! I remember I contributed to a repo for performing convolution in the Fourier space, which was the solution in my case. You can find more information <a href="https://fkodom.substack.com/p/fourier-convolutions-in-pytorch" rel="noopener nofollow ugc">here</a>.</p>
Restrict gradiant values in pytorch

Mohammed_Alruqimi: <p>Can I fore pytorch to gradiant parameter values in restricted bound ?<br>
For example:<br>
I have this torch paramater</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self, args):
        super(Model, self).__init__()
        self.m = nn.Parameter(torch.tensor(0.02), requires_grad=True)
</code></pre>
<p>I want to m values to be between 0.1 and 0.9</p>

KFrank: <p>Hi Mohammed!</p>
<aside class="quote no-group" data-username="Mohammed_Alruqimi" data-post="1" data-topic="202090" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohammed_alruqimi/48/62632_2.png" class="avatar"> Mohammed_Alruqimi:</div>
<blockquote>
<p>Can I fore pytorch to gradiant parameter values in restricted bound ?</p>
</blockquote>
</aside>
<p>You ask about restricting gradients.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        self.m = nn.Parameter(torch.tensor(0.02), requires_grad=True)
</code></pre>
<p>I want to m values to be between 0.1 and 0.9</p>
</blockquote>
</aside>
<p>But then you ask about restricting the value of a <code>Parameter</code> itself.  Let me assume<br>
that you want to restrict the <code>Parameter</code> itself, rather than its gradient.</p>
<p>(Note that in your example, you are asking that <code>m</code> lie between <code>0.1</code> and <code>0.9</code>, but<br>
you are initializing it to <code>0.02</code> – outside of that range.)</p>
<p>My recommendation is to let your trainable <code>Parameter</code>s be unrestricted, that is,<br>
range from <code>-inf</code> to <code>inf</code>, and then map them to a derived variable that satisfies<br>
your desired restriction.  (This is because gradient-descent optimization doesn’t<br>
work for optimizing constrained variables without additional complication.)</p>
<p>So in your case in your <code>__init__()</code> method I would do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">self.m_unconstrained = nn.Parameter (torch.tensor (-1.946),  requires_grad = True)
</code></pre>
<p>and then in your <code>forward()</code> method:</p>
<pre data-code-wrap="python"><code class="lang-python">m_ constrained = 0.1 + 0.8 * self.m_unconstrained.sigmoid()
# use m_constrained in the rest of forward()
</code></pre>
<p>(This makes use of the fact that <code>.sigmoid()</code> maps <code>(-inf, inf)</code> to <code>(0.0, 1.0)</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

Mohammed_Alruqimi: <p>Thank you <a class="mention" href="/u/kfrank">@KFrank</a> . Yes that waht I was meaning</p>
Issues using visual studio code & pytorch

Juarez_Huerta_Jonath: <p>Hi all,<br>
I’m new using jetson nano, and i’m trying to run an object detection model from yolov8. I have succesfully run it on a PC with CUDA, but i’ve encountered numerous issues, one of wich involves the communication between CUDA and the virtual enviroment. This issue prevents my object detector from running as it should<br>
P. S. I’m not using conda<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fcebb7411768ab9a32fde92f871a105aeff6117.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4fcebb7411768ab9a32fde92f871a105aeff6117" title="IMG-20240503-WA0009"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcebb7411768ab9a32fde92f871a105aeff6117_2_690x388.jpeg" alt="IMG-20240503-WA0009" data-base62-sha1="bo0DNDPHH93DV6uvSxOuF4pFf7x" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcebb7411768ab9a32fde92f871a105aeff6117_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcebb7411768ab9a32fde92f871a105aeff6117_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcebb7411768ab9a32fde92f871a105aeff6117_2_1380x776.jpeg 2x" data-dominant-color="454656"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-20240503-WA0009</span><span class="informations">1600×900 263 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks a lot for the help</p>
Unique nn.Parameter computation by shifting paramater values for gradient computation

Aaron_Thomas: <p>Hi everyone,<br>
So to do a particular computation in my neural network training I have a unique way I must compute gradients. This involves shifting the values of the parameters, these params are then processed by a function. Currently, i have a class implemented along these lines:</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super().__init__()

        self.params = nn.ParameterList(
            [
                nn.Parameter(nn.init.uniform_(torch.empty((bits, 2), dtype=torch.float32), a= -pi, b = pi)),
                nn.Parameter(nn.init.uniform_(torch.empty((depth, bits, 2), dtype=torch.float32), a = -pi, b = pi))
            ]
        )
    def grad_computation(self):
        &lt;code_here&gt;
    def forward(self):
        ...
        return ...
</code></pre>
<p>What i want to do is along the following:</p>
<ul>
<li>Make a copy of self.params called something like copy_params</li>
<li>Shift the first value of copy_params by +1 and then -1 leaving the rest of the params untouched</li>
<li>Do my computation using shifted values</li>
<li>Repeat with the next param in self.q_params list</li>
<li>Do until at the end of the self. param list</li>
</ul>
<p>Here is some rough seudo-code:</p>
<pre><code class="lang-auto">copy_params = self.q_params.copy() #Cant copy nn.ParameterList?

#Some sort of loop here to loop over the parameters
# Cant do copy_params[i,j]  due to the shape of self.q_params

copy_params[i,j] += 1 
forward = func(copy_params)

copy_params[i,j] -= 1
backward = func(copy_params)
</code></pre>
<p>Any help would be greatly appreciated!</p>
Trouble making CMake for using LibTorch with c++

Brandon_Fowler: <p>Hi there, I’ve been trying to get set up with LibTorch in c++ using this guide <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html</a> and been having some troubles.</p>
<p>Context</p>
<ul>
<li>using CUDA 12.1</li>
<li>building with cmake tools in vscode</li>
</ul>
<p>Does anyone have a link to a set up for LibTorch using Cuda?</p>
<p>Thanks</p>

Brandon_Fowler: <p>I’m still working on cuda but I got the cpu version working with this cmake (unzipping the libtorch utils into the root of my project):</p>
<pre><code class="lang-auto">
cmake_minimum_required(VERSION 3.18 FATAL_ERROR)
project(main)

set(CMAKE_PREFIX_PATH "${CMAKE_SOURCE_DIR}/libtorch")

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(main main.cpp)
target_link_libraries(main "${TORCH_LIBRARIES}")
set_property(TARGET main PROPERTY CXX_STANDARD 17)

# The following code block is suggested to be used on Windows.
# According to https://github.com/pytorch/pytorch/issues/25457,
# the DLLs need to be copied to avoid memory errors.
if (MSVC)
  file(GLOB TORCH_DLLS "${CMAKE_SOURCE_DIR}/libtorch/lib/*.dll")
  message("DLL files: ${TORCH_DLLS}")
  add_custom_command(TARGET main
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $&lt;TARGET_FILE_DIR:main&gt;
                     )
endif (MSVC)
</code></pre>
<p>I then used these commands to build and make it</p>
<pre><code class="lang-auto">if exist build rmdir /s /q build
mkdir build
cd build
cmake -G "Visual Studio 17 2022" -T v142 -A x64 ..
cmake --build . --config Debug
Debug\main.exe
</code></pre>
<p>My main problems were getting it to use the right compiler since libtorch doesn’t work with the newest visual studio compilers. Here’s versions that worked for me:</p>
<ul>
<li>libtorch: <a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.2.2%2Bcpu.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.2.2%2Bcpu.zip</a></li>
<li>MSVC: MSVC v142 - VS 2019 C++ x64/x86 build tools (…)</li>
</ul>
<p>To install that specific compiler I used the visual studio installer with the following steps</p>
<ul>
<li>open visual studio installer</li>
<li>click ‘modify’ on ‘Visual Studio Build Tools 20XX’</li>
<li>go to the ‘Individual components’ tab and search ‘msvc’</li>
<li>get an older version of ‘MSVC vXXX - VS 20XX C++ x64/x86 build tools (…)’ that might work with whatever they build libtorch with</li>
<li>modify the build command to ‘cmake -G “Visual Studio XX 20XX” -T vXXX -A x64 …’ according to the build tools version you got above</li>
</ul>
Why c10::Dict's insert method marked as const?

fabian_schutze: <p>The API for <code>c10::Dict</code> <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/core/Dict.h#L292" rel="noopener nofollow ugc">insert</a> method is:</p>
<pre><code class="lang-auto">
   * Inserts element(s) into the container, if the container doesn't already contain an element with an equivalent key.
   * May invalidate any references, pointers, or iterators referring to contained elements.
   *
   * @return A pair consisting of an iterator to the inserted element (or to the element that prevented the insertion) and a bool denoting whether the insertion took place.
   */
  template&lt;class Key_, class Value_&gt;
  std::pair&lt;iterator, bool&gt; insert(Key_&amp;&amp; key, Value_&amp;&amp; value) const;
</code></pre>
<p>What’s the meaning of the <code>const</code> qualifier? The dict does not stay const but might be modified.</p>
<p>As reference, the <code>insert</code> method of <code>std::dict</code> is not const: <a href="https://en.cppreference.com/w/cpp/container/map/insert" class="inline-onebox" rel="noopener nofollow ugc">std::map&lt;Key,T,Compare,Allocator&gt;::insert - cppreference.com</a></p>