akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can i inference speed up in libtorch c++

username: SangHoon_LEE
cooked: <p>I tried to improve the inference speed of my model using the C++ programming language, but I encountered slow inference times. Here are the results of my tests:</p>
<ol>
<li>I ran the same LibTorch C++ code on both Windows and Linux, and I noticed that the inference speed was faster on Linux compared to Windows.</li>
</ol>
<ul>
<li>Windows environment:
<ul>
<li>GPU: RTX 3060</li>
<li>CUDA: 11.7</li>
<li>LibTorch version: 2.0.0</li>
<li>Resulting frame rate: 11 fps</li>
</ul>
</li>
<li>Linux environment:
<ul>
<li>GPU: RTX 2080 Ti</li>
<li>CUDA: 12.1</li>
<li>LibTorch version: 2.0.0</li>
<li>Resulting frame rate: 19 fps</li>
</ul>
</li>
</ul>
<ol start="2">
<li>I also tested my LibTorch program on both Visual Studio 2022 and WSL Ubuntu 22.04. Surprisingly, when I tested it on Visual Studio 2022, I achieved a frame rate of 11 fps, while WSL only gave me 9 fps.</li>
</ol>
<ul>
<li>Windows environment:
<ul>
<li>GPU: RTX 3060</li>
<li>CUDA: 11.7</li>
</ul>
</li>
</ul>
<p>Now, I’m looking for suggestions on how to improve my inference times. In conclusion, the inference speed on Linux is approximately twice as fast as on Windows.</p>
<p>Any advice on improving the inference speed would be greatly appreciated.</p>
<p>Thnak you</p>
Issue of install tensordict==0.2.0

username: Zhe_Zhe
cooked: <p>Hello, guys:</p>
<p>There is a quick question. I try to install torchrl which requires tensordict==0.2.0, but I got this error message when I install the latest version of tensordict</p>
<pre><code class="lang-auto">$ pip install tensordict==0.2.0
ERROR: Could not find a version that satisfies the requirement tensordict==0.2.0 (from versions: 0.0.1a0, 0.0.1b0, 0.0.1rc0, 0.0.2a0, 0.0.2b0, 0.0.3, 0.1.0, 0.1.1, 0.1.2)
ERROR: No matching distribution found for tensordict==0.2.0
</code></pre>
<p>My pip version is 23.3, could you provide some quick ideas to solve this issue?<br>
Thanks a lot for the help!</p>

username: Zhe_Zhe
cooked: <aside class="quote no-group" data-username="Zhe_Zhe" data-post="1" data-topic="189981">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhe_zhe/48/46059_2.png" class="avatar"> Zhe_Zhe:</div>
<blockquote>
<p><code>pip install tensordict==0.2.0</code></p>
</blockquote>
</aside>
<p>It works after I clone the code and execute setup.py manually.</p>
Delete elements according to conditional expression

username: swharaday
cooked: <p>Hi,</p>
<p>Here i have tensor<br>
a = torch.tensor(([2,0,3],[2,1,3],[2,3,3],[2,2,3],[3,2,3],[2,5,3],[2,8,3]))</p>
<p>i need to delete elements which have 3&gt;= and &lt;6 in the middle … [2,3,3],[2,5,3]<br>
and to get<br>
torch.tensor(([2,0,3],[2,1,3],[2,2,3],[3,2,3],[2,8,3]))</p>
<p>How can i make it ?</p>
<p>Thanks in advance.</p>

username: akt42
cooked: <p>Create a masking tensor like below and use that to index:</p>
<pre><code class="lang-auto">mask = (a[:, 1] &gt;= 3) &amp; (a[:, 1] &lt; 6)
a[~mask]

#tensor([[2, 0, 3],
#        [2, 1, 3],
#        [2, 2, 3],
#        [3, 2, 3],
#        [2, 8, 3]])
</code></pre>

username: swharaday
cooked: <p>Hi akt42,<br>
thank you for your reply !<br>
your advice seems nice &amp; smart.<br>
i will be out during this weekend. i will let you know after trial.</p>

username: swharaday
cooked: <p>Hi akt42,<br>
It works perfectly !! I really appreciate your advice.</p>
<p>Actually, it took me sometime to check what the ‘~’ is and to review usage of bools.</p>
<p>This will be used to eliminate FP data of YOLOv5 object detection results (results.xyxy[0]) based on x-position.</p>
<p>Thank you very much for your contribution.</p>
Suggestion: Parallel processing of separate neural networks in one GPU

username: TimurIshuov
cooked: <p>Separate deep networks has benefits over one big network with multiple outputs.<br>
there is no correlation or interference between networks.<br>
but the bottleneck is that you cannot process them in parallel if you don’t have several GPU.<br>
e.g., xs = [net(x) for net in nets] makes it processing sequentially.</p>
<p>What I suggest is to add functionality to nn.ModuleList, so that, it has synchronous parallelism, as option or by default, e.g via torch.multiprocessing (if it is possible)</p>
<p>so just first code will turn into: xs = nets(x)</p>
<p>Thank you.</p>
I tried to install Pytorch in vscode, but i am seeing an error?

username: Teddy_Salas
cooked: <p>I trying to install PyTorch in my desktop,but i am seeing an error during the installation, can any one help me to solve it ? i am attaching screen shot of the error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61c4f0ad222b112f5928d6af5b5ff138fd3c36de.png" data-download-href="https://discuss.pytorch.org/uploads/default/61c4f0ad222b112f5928d6af5b5ff138fd3c36de" title="Screenshot 2023-10-15 133844"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c4f0ad222b112f5928d6af5b5ff138fd3c36de_2_690x388.png" alt="Screenshot 2023-10-15 133844" data-base62-sha1="dWUfVGkuZqZ9TuBzCmQrRS76Bxc" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c4f0ad222b112f5928d6af5b5ff138fd3c36de_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c4f0ad222b112f5928d6af5b5ff138fd3c36de_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c4f0ad222b112f5928d6af5b5ff138fd3c36de_2_1380x776.png 2x" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-10-15 133844</span><span class="informations">1920×1080 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Random horizontal translation

username: amy2
cooked: <p>I want to perform data augmentation to my data specificaly I want to perform random horizontal and vertical translation<br>
this is the class that I implemented</p>
<blockquote>
<p>class HorizontalTranslation(object):<br>
def --init–(self, max_translation=10):<br>
self.max_translation = max_translation<br>
def --call–(self, img):<br>
# Randomly choose a horizontal translation value<br>
translation = random.randint(-self.max_translation, self.max_translation<br>
# Translate the image horizontally<br>
img = img.transform(img.size, Image.AFFINE, (1, 0, translation, 0, 1, 0)<br>
return img<br>
def randomize_parameters(self):<br>
pass</p>
</blockquote>
<p>that will be called in a main function<br>
i didn’t get the result I wanted<br>
is that the right way to implement  random translation ?</p>

username: ptrblck
cooked: <p>I assume you are using <code>PIL</code>’s transformation? If so, it sounds like a valid approach. Could you describe why this approach doesn’t match your expected results?</p>

username: amy2
cooked: <p>at first i was only using center cropping and horizontall flip but my model overfit with 0.98 training accuracy and 0.45 vlidation accuracy after adding horizontal translation the accuracy values dropped for both training and validation (got 0.7 for training and 0.1 for validation )<br>
this is the script that calls the functions in the main</p>
<blockquote>
<p>def get_train_utils(opt, model_parameters):<br>
spatial_transform = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
spatial_transform.append(Resize(opt.sample_size))<br>
spatial_transform.append(CenterCrop(opt.sample_size))<br>
if not opt.no_hflip:<br>
spatial_transform.append(RandomHorizontalFlip())<br>
spatial_transform.append(HorizontalTranslation(40))<br>
spatial_transform.append(ToTensor())<br>
spatial_transform = Compose(spatial_transform)</p>
</blockquote>
<p>and after that it will be called by the dataloader<br>
I don’t think I am calling it wrong !! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>Did you check the transformed samples and compared them to an expected result?<br>
Depending on your use case the augmentation might be too aggressive.</p>
How do modules like `nn.Conv2d` allocate memory?

username: omega9123
cooked: <p>I’m working throught the tutorials and I wanted to understand how <code>nn.Conv2d</code> is created. It looks like we can pass in a <code>device</code> param during the init function and I’ve checked that this param create a tensor in the GPU (rather than CPU by default). What I’m wondering is if I wanted to create an empty tensor instead of random values for the layer, how do I do that? It seems like by default random values are setup for that tensor.</p>
<p>A follow up question, If  I use the <code>load_state_dict</code> method to load a model to a GPU allocated tensor, I’m assuming the method would just overwrite the values that were randomly setup?</p>

username: J_Johnson
cooked: <p>PyTorch layers are first set to empty, and then call <code>self.reset_parameters()</code> to initialize them with a kaiming uniform distribution. See the <strong>init</strong> for ConvNd here: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.conv — PyTorch 2.1 documentation</a></p>
<p>As to your second question, that is correct. The previous model values are overwritten by the saved state.</p>

username: ptrblck
cooked: <p>To skip the initialization of parameters avoiding wasting compute as you will overwrite these, you could use <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.skip_init.html#torch.nn.utils.skip_init"><code>torch.nn.utils.skip_init</code></a>.</p>
How to optimise a loop body with a large number of loops？

username: gengxingze
cooked: <p>Here’s how I can optimise this code</p>
<pre><code class="lang-python">a = torch.rand(12800, 100, 3)
b = torch.randint(1, 30000, (12800, 100))
c = torch.zeros(12800, 3)
for i in range(12800):
    c[i] = c[i] + a[b == i].sum(dim=0)
</code></pre>
<p>I tried to use the operator to speed it up, but the need to ask for the gradient of this variable prevents it from being exported using jit</p>

username: KFrank
cooked: <p>Hi Xingze!</p>
<aside class="quote no-group" data-username="gengxingze" data-post="1" data-topic="189906" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gengxingze/48/64597_2.png" class="avatar"> gengxingze:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">a = torch.rand(12800, 100, 3)
b = torch.randint(1, 30000, (12800, 100))
c = torch.zeros(12800, 3)
for i in range(12800):
    c[i] = c[i] + a[b == i].sum(dim=0)
</code></pre>
</blockquote>
</aside>
<p>You can eliminate the for loop by materializing a large numerical (0.0, 1.0)<br>
mask tensor and then using <code>einsum()</code> to compute <code>c</code>.</p>
<p>Here is an example script (reduced in size to fit into my memory):</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2023)

a = torch.rand(6400, 100, 3)   # reduce size from 128000 to 6400 to fit in memory
b = torch.randint(1, 15000, (6400, 100))
c = torch.zeros(6400, 3)
for i in range(6400):
    c[i] = c[i] + a[b == i].sum(dim=0)

print ('c.shape:', c.shape)

mask = (b.unsqueeze (0) == torch.arange (6400).unsqueeze (-1).unsqueeze (-1)).float()   # materialize large mask tensor

print ('mask.shape:', mask.shape)

cB = torch.einsum ('mij, ijn -&gt; mn', mask, a)

print ('cB.shape:', cB.shape)
print ('torch.allclose (c, cB):', torch.allclose (c, cB))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">2.1.0
c.shape: torch.Size([6400, 3])
mask.shape: torch.Size([6400, 6400, 100])
cB.shape: torch.Size([6400, 3])
torch.allclose (c, cB): True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: gengxingze
cooked: <p>You are right,thank you</p>
PyTorch learning rate - possible edition in an external file?

username: wiiiktor
cooked: <p>Can I somehow edit or at least influence the learning rate of a PyTorch model in an external TXT file that is uploaded / checked at every epoch?</p>

username: ptrblck
cooked: <p>I don’t think PyTorch has a built-in util. to do so, but you can of course manually read a file and convert the information into the learning rate.</p>

username: wiiiktor
cooked: <p>I think the optimiser and/or scheduler have a lot of internal statistics - wouldn’t my new learning rate erase them?..</p>
CUDA Kernel Launches

username: solidmacaroni
cooked: <p>Say I do <code>output = model(input)</code> to do a forward pass. Does PyTorch launch a single CUDA kernel for this, or is it possible that multiple kernels are launched for a single forward pass call? Likewise for backward pass, loss calculation and optimizer step. Thank you.</p>

username: ptrblck
cooked: <p>Multiple CUDA kernels will be launched unless the forward pass contains a single operation (unlikely) or unless you’ve used CUDA graphs to capture the entire forward pass as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">here</a>.</p>
.to(device) takes far longer for test data than train data

username: JamesCarr
cooked: <p>I am currently trying to improve the efficiency of my code, but I have found an issue I can’t figure out - the time taken to send tensors to my gpu is far greater for the test data that the training data, despite there being far more training data.</p>
<p>I am using a custom dataset and <code>random_split</code> to create the train_ and test_datasets:</p>
<p><code>train_dataset, test_dataset = random_split(dataset, [0.8, 0.2])</code></p>
<p>before passing the datasets into dataloaders</p>
<pre><code class="lang-auto">    train_dataloader = DataLoader(train_dataset,
                                  batch_size=batch_size,
                                  num_workers=num_workers,
                                  shuffle=True)
    
    test_dataloader = DataLoader(test_dataset,
                                 batch_size=batch_size,
                                 num_workers=num_workers)
</code></pre>
<p>During training and testing, the data is then sent to the device:</p>
<pre><code class="lang-auto">for data1, data2, data3 in dataloader:
    data1, data2, data3 = data1.to(device), data2.to(device), data3.to(device)
</code></pre>
<p>Which is exactly the same for both training and testing loops, the only difference is that the for loop is within <code>with torch.inference_mode()</code> during testing.</p>
<p>I am using cProfile to identify where the inefficiencies in my program are coming from and noticed that .to() was taking up ~1/4 of my total runtime. To further investigate I made the groups of .to(device)s functions, i.e:</p>
<pre><code class="lang-auto">def send_to_device_train(data1, data2, data3, device):
    return data1.to(device), data2.to(device), data3.to(device)
</code></pre>
<p>and a similar function for test data.</p>
<p>When I run a cProfile I get this unusual result:</p>
<pre><code class="lang-auto">   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     6602    5.352    0.001    5.352    0.001 {method 'to' of 'torch._C._TensorBase' objects}
      152    0.001    0.000    2.373    0.016 /home/user/engine.py:13(send_to_device_test)
      581    0.002    0.000    0.179    0.000 /home/user/engine.py:8(send_to_device_train)
</code></pre>
<p>Despite performing .to() one quarter as many times, <code>send_to_device_test</code> takes 13 times longer (at least for cumulative time).</p>
<p>I tried removing the .to()s from the torch.inference_mode() (calling the context manager after the tensors have been sent) and also set shuffle=True, in the hope that might be the culprit, but the problem persists.</p>
<p>I wonder if anyone would be able to help me solve this issue?</p>
GCN using GraphConv in half precision

username: AKT_TARAFDER
cooked: <p>for a simple GraphConv layer, if we get a graph and feed it to GraphConv in half, we get some problems. Same trying to use this layer in GCN.</p>
<pre><code class="lang-auto">
# Case 2: changed datatype
g = dgl.graph(([0,1,2,3,2,5], [1,2,3,4,0,3]))
g = dgl.to_half(g)
print('g idtpye:' , g.idtype)
g = dgl.add_self_loop(g)
feat = th.ones(6, 10).half()
print('feature dtype:', feat.dtype)
conv = GraphConv(10, 2, norm = "right")
res = conv(g, feat)
print('conv dtype: ', res.dtype)
print(res)

# allow_zero_in_degree example
</code></pre>
Help with gaussian splatting torch 4090 torch Version: 2.1.0

username: Marcos_Eng
cooked: <p>Hello guys, I am trying to run the comand line on a Gaussian Splatting test.</p>
<p>python train.py -s </p>
<p>After this comand I got the message:</p>
<p>C:\Users\marcos.costa\AppData\Local\anaconda3\envs\gaussian_splatting\lib\site-packages\torch\cuda_<em>init</em>_.py:106: UserWarning:<br>
<strong>NVIDIA GeForce RTX 4090 with CUDA capability sm_89 is not compatible with the current PyTorch installation.</strong><br>
<strong>The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.</strong><br>
<strong>If you want to use the NVIDIA GeForce RTX 4090 GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></strong></p>
<p>warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))<br>
Output folder: ./output/1ea37bd9-2 [13/10 20:19:08]<br>
Tensorboard not available: not logging progress [13/10 20:19:08]<br>
Reading camera 149/149 [13/10 20:19:09]<br>
Loading Training Cameras [13/10 20:19:09]<br>
[ INFO ] Encountered quite large input images (&gt;1.6K pixels width), rescaling to 1.6K.<br>
If this is not desired, please explicitly specify ‘–resolution/-r’ as 1 [13/10 20:19:09</p>
<p>Can anyone help me?</p>

username: ptrblck
cooked: <p>Based on the error message you’ve installed a PyTorch binary with CUDA 10.2, which is too old for your GPU. Install any current stable or nightly release and it will work.</p>

username: Marcos_Eng
cooked: <p>I have cuda 11.8 and 12.2 installed and the same error proceeds.</p>
<p>Is there a problem to have more than one version?</p>

username: ptrblck
cooked: <p>I doubt it, since e.g. <code>2.1.0+cu121</code> shows these architectures:</p>
<pre><code class="lang-python">torch.__version__
Out[111]: '2.1.0+cu121'

torch.cuda.get_arch_list()
Out[112]: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
</code></pre>
<p>while your error message shows the ones built with CUDA 10.2.</p>
Split an image into a 2 by 2 grid

username: u7122029
cooked: <p>So right now I’m trying to split a tensor (of an image) of dimension 3 x 32 x 32 into a grid, then stack each part of the grid on top of each other to for a 4 x 3 x 16 x 16 tensor. The 0-th element of the tensor in dimension 0 will be the top left piece of the image, the 1-th element will be the top right, the 2-th element will be the bottom left piece, and the 3-th element will be the bottom right piece.</p>
<p>My code so far looks like</p>
<pre><code class="lang-python">def grid_image(image_tensor, grid_length=2):
    n_channels, n_rows, n_cols = image_tensor.shape
    row_length = n_rows // grid_length
    col_length = n_cols // grid_length

    patches = []
    rows = torch.split(image_tensor,row_length,1)
    for row in rows:
        row_patches = torch.split(row, col_length, 2)
        patches += row_patches

    return torch.stack(patches)
</code></pre>
<p>I feel like there is a better way to do this because I am currently splitting the image by rows then columns. I’m looking for an approach that does both at the same time, which would assumedly improve the time complexity of the function. I tried searching around and I didn’t find anything. Could I please have some help?</p>

username: ptrblck
cooked: <p><code>tensor.unfold</code> should work.</p>

username: u7122029
cooked: <p>Thanks for your answer ptrblck, I managed to get the code below for a grid of any size.</p>
<pre><code class="lang-python">patches = (image_tensor
               .unfold(1,row_length,col_length)
               .unfold(2,row_length,col_length)
               .reshape(3,grid_length**2,row_length,col_length)
               .permute(1,0,2,3))
</code></pre>
How can I add an additional output head without losing performance on original head?

username: Abanoub_Ghobrial
cooked: <p>Hi,</p>
<p>I am trying to create a resnet18 model with two different output heads. One head predicts the classification class (classification_head) and the other classifies the rotation (rotation_head) of the image.</p>
<p>I am using a pre-trained model for resnet18.</p>
<p>I have frozen all the model parameters using</p>
<pre><code>for param in model_multi.parameters():
	param.requires_grad = False
</code></pre>
<p>and I only unfreeze the rotation classification head</p>
<pre><code>for param in model_multi.rotation_head.parameters():
	param.requires_grad = True
</code></pre>
<p>Then I train the model to fine-tune the rotation head. The problem is when I swap back to the classification head, there is a drop in the classification accuracy (from 0.85 to 0.79). Theoretically, there should be no drop at all, since the model parameters used for classification of the image class are all the same.</p>
<p>To validate this I have even created a deep copy of the model before fine-tuning for the roation_head to compare the fine-tuned model against. When I calculate the difference between the model parameters of the fine-tuned model and the copied model (i.e. not fine-tuned), I only get a difference in the values for the rotation head whilst all other model parameters do not change.</p>
<p>At this stage, I have no idea why the accuracy for the classification head is changing unless there is something else other than the model parameters that affect classification and I need to freeze it during training of the rotation head. What this other thing be? I would appreciate any help or advice please.</p>
<p>Thanks</p>

username: Abanoub_Ghobrial
cooked: <p>I found the reason why I was getting these changes even though I had all model parameters frozen. The running mean and standard deviation in the Batch normalisation layers were still updating and these do not freeze when you set requires_grad to False. To stop the batch normalisation layers from changing one could use the code below after setting the model to model.train().</p>
<p>for name, module in model.named_modules():<br>
if isinstance(module, nn.BatchNorm2d):<br>
module.eval()</p>
<p>Alternatively, one may save the batch normalisation layer values and reload them as required.</p>
<p>Hope this helps anyone else as well <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Is Conv1D actually a convolution?

username: Ha_St
cooked: <p>Hello all,<br>
I am building a model that needs to perform the mathematical operation of convolution between the batches of 1D input <code>c</code> and a parameter, call it <code>E</code>.<br>
Thus, I want something similar to<code>np.convolve(E,c)</code> but in native pytorch . Am I taking i correctly, that Conv1D is not the right tool for the job? The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d" rel="noopener nofollow ugc">documentation</a> states it uses the valid cross-correlation operator insead of a convolution operator.<br>
I found the following (at least) working solution to give reasonable yet wrong results:</p>
<pre><code class="lang-auto">conv = torch.nn.Conv1d(
    1, #number of batches
    1, #number of features (just 1)
    40, #length of the 1D kernel (E)
    bias=False, 
   padding=n_kernel - 1 #make it a "full" convolution or correlation, I guess
)

sol = conv(c[None, :])

</code></pre>
<p>Am I overlooking something and if so, how can I recreate the expected (numpy) behavior? And also, on a less serious note,why is it not called Cor1D then? <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Thanks so much for your help!</p>

username: ptrblck
cooked: <p>Flip the input or filter and you will perform a true convolution instead of a cross-correlation.</p>
Got different value when use from_blob

username: hedinghong
cooked: <p>Test_R is two dimensional float image data  loaded from a file<br>
and Tensor_R is the expected torch tensor converted from Test_R using from_blob.<br>
Values is different between Test_R and Tensor_R making me so confused<br>
for (int i=0; i&lt;h; i++) {<br>
for (int j = 0; j &lt; w; j++) {<br>
inputR&gt;&gt;Test_R[i][j] ;<br>
inputG&gt;&gt;Test_G[i][j] ;<br>
inputB&gt;&gt;Test_B[i][j] ;<br>
}<br>
}</p>
<p>auto Tensor_R = torch::from_blob(Test_R,{h,w},torch::kFloat).clone();<br>
torch::Tensor Tensor_R_CUDA = Tensor_R.to(torch::kFloat).to(device);<br>
std::cout&lt;&lt;Test_R[0][4]&lt;&lt;std::endl;<br>
std::cout&lt;&lt;Tensor_R[0][4].item().to()&lt;&lt;std::endl;<br>
std::cout&lt;&lt;Tensor_R_CUDA[0][4]&lt;&lt;std::endl;</p>
<p>the output results:<br>
0.0588<br>
-4.18158e-22<br>
-4.18158e-22<br>
[ CUDAFloatType{} ]</p>
<p>Could someone help?</p>

username: gengxingze
cooked: <p>Is your Test_R type also float?</p>
Detecting word boundaries in a speech using LSTMs

username: Vijay_Dubey
cooked: <p>I have a training dataset that contains <strong>speech recording</strong>s (wav files) of several hundred speakers<br>
speaking English sentences. Each wav file is associated with a word file (.wrd file) that would contain the words that are there in the sentence <strong>along with their temporal boundaries</strong>, i.e. the starting and ending time of the word. The sampling is done at 16000Hz.</p>
<p>I wish to <strong>determine the word boundaries</strong> in a new .wav file given the list of words that are spoken.<br>
Please suggest some approaches and references on how should I proceed.</p>
<p>I have some idea that this problem can be solved by using approaches like  <strong>connectionist temporal cost</strong>, but don’t know how t proceed. Any help would be appreciated.<br>
Thanks a lot!!</p>

username: Vijay_Dubey
cooked: <p>Please comment with your suggestions</p>

username: Boumedine_Billal
cooked: <p>Hello Vijay,</p>
<p>there’s a person who has created two models: one for detecting the start of a word and another for detecting the end of a word.</p>
<p>link: <a href="https://github.com/nishadsinghi/word-boundary-detection/blob/master/single_LSTM%20(1).ipynb" rel="noopener nofollow ugc">https://github.com/nishadsinghi/word-boundary-detection/blob/master/single_LSTM%20(1).ipynb</a></p>
Why torch.compile slows down from second run?

username: Nirmai
cooked: <p>The recently released torch 2.0 works great, torch.compile() makes models to run faster. I have tried few experiments where I observed the first run is taking time for compilation and from the second run it speeds up the model.</p>
<p>What is the reason behind it? Even for the second run if we instantiate the model and run the model again from start it takes lesser time. Is torch.compile() stores something at backend?</p>

username: eqy
cooked: <p>If you are saying that “compiling” the model again takes less time, then I believe this is expected as the compiled kernels are cached: <a href="https://discuss.pytorch.org/t/pt-2-0-are-compiled-models-savable/169871/2" class="inline-onebox">PT 2.0 - Are compiled models savable - #2 by smth</a>.</p>

username: Nirmai
cooked: <p>Is there a way to access this cache, from where can we do this?</p>

username: eqy
cooked: <p>I would see if e.g., <a href="https://pytorch.org/docs/stable/_modules/torch/_dynamo.html#reset" class="inline-onebox" rel="noopener nofollow ugc">torch._dynamo — PyTorch 2.0 documentation</a> does what you are looking for</p>

username: Nirmai
cooked: <p>The link above shows how to reset the cache, is there any way to save this cache and use it back by pickling?</p>

username: adal
cooked: <p>I met with the same problem. Are there any updates?</p>
How to add feature fusion to the backbone network

username: YYL-SCAR
cooked: <p>I am learning feature fusion, I want to add the feature fusion part in centernet network, the backbone network used is resnet50.<br>
Is it possible to fuse the downsampled three feature maps x3（64<em>64</em>512）, x4（32x32x1024）, x5 (2048<em>16</em>16)  as an output (2048<em>16</em>16) ?<br>
Is the procedure to convert both l3 and l4 to the size of l5 and then sum them?<br>
The code is as follows:<br>
class ResNet(nn.Module).<br>
def <strong>init</strong>(self, block, layers, num_classes=1000): self.<br>
self.inplanes = 64<br>
super(ResNet, self). <strong>init</strong>()<br>
# 512,512,3 → 256,256,64<br>
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,bias=False)<br>
self.bn1 = nn.BatchNorm2d(64)<br>
self.relu = nn.ReLU(inplace=True)<br>
# Improvement point: replace 7<em>7 with 3</em>3<br>
self.conv2 = nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1, bias=False)<br>
self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=False)<br>
self.conv4 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=False)</p>
<pre><code>    self.conv5 = nn.Conv2d()
    # 256x256x64 -&gt; 128x128x64
    self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=0, ceil_mode=True) # change

    # 128x128x64 -&gt; 128x128x256
    self.layer1 = self._make_layer(block, 64, layers[0])

    # 128x128x256 -&gt; 64x64x512
    self.layer2 = self._make_layer(block, 128, layers[1], stride=2)

    # 64x64x512 -&gt; 32x32x1024
    self.layer3 = self._make_layer(block, 256, layers[2], stride=2)

    # 32x32x1024 -&gt; 16x16x2048
    self.layer4 = self._make_layer(block, 512, layers[3], stride=2)


    self.avgpool = nn.AvgPool2d(7)
        for m in self.modules():
        if isinstance(m, nn.Conv2d):
            n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
        elif isinstance(m, nn.BatchNorm2d):
            m.weight.data.fill_(1)
            m.bias.data.zero_()

def _make_layer(self, block, planes, blocks,stride=1):
    downsample = None
    if stride != 1 or self.inplanes != planes * block.expansion:
        downsample = nn.Sequential(
             nn.Conv2d(self.inplanes, planes * block.expansion,
                 kernel_size=1, stride=stride, bias=False),
           
        nn.BatchNorm2d(planes * block.expansion),
    )

    layers = []
    layers.append(block(self.inplanes, planes, stride, downsample))
    self.inplanes = planes * block.expansion
    for i in range(1, blocks):
        layers.append(block(self.inplanes, planes))

    return nn.Sequential(*layers)

def forward(self, x):
    x = self.conv1(x)
  
    x = self.bn1(x)
    x = self.relu(x)
    x = self.maxpool(x)
    x = self.layer1(x)  # 128x128x64 -&gt; 128x128x256


    xl3 = self.layer2(x)    # 128x128x256 -&gt; 64x64x512

    xl4 = self.layer3(xl3) # 64x64x512 -&gt; 32x32x1024

    xl5 = self.layer4(xl4)# 32x32x1024 -&gt; 16x16x2048
    x = self.avgpool(xl5)
    x = x.view(x.size(0), -1)
    x = self.fc(x)

    return x
</code></pre>
When using an OCR model built with PyTorch for inference, there is an oscillating increase in GPU memory usage as the batch size is increased

username: rjx_3412
cooked: <p>During inference with an OCR model, experimental results indicate a non-linear increase in GPU memory usage as the batch size is raised.</p>
<p>The specific experimental method involved taking a real image and resizing it to various sizes using OpenCV’s resize method with linear interpolation. The batch size was increased by duplicating the numpy matrices of the samples.</p>
<pre data-code-wrap="python"><code class="lang-python">det_model.eval().cuda()
origin_image = cv2.imread(image_path)
with torch.no_grad():
    for tup in itertools.product([i for i in range(640,1000,32)],[i for i in range(640,1000,32)]):
        i = tup[0]
        j = tup[1]
        torch.cuda.empty_cache()
        for z in range(1,100):
            try:
                 img = cv2.resize(origin_image,(i,j), interpolation=cv2.INTER_LINEAR)
                 img = copy_img(batch_size=z, img)
                 input_for_torch = torch.from_numpy(img).cuda()
                 print(det_model(input_for_torch))
            except RuntimeError as e:
                 if "CUDA out of memory" in str(e):
                    print(str(i) + '. ' + "{} × {} batch_size: {}".format(i, j, z) + '    OOM')
</code></pre>
<p>By creating child processes that infinitely loop to monitor GPU usage, you obtained peak memory consumption and inference times for each batch size, as shown in the following graph.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7d1a75f56e85d4e1f1bcdc319b4c46936e69f4e.png" alt="image-1697096452458" data-base62-sha1="nWAUXHrPFf4CFqtIYb9fpfrf58O" width="640" height="480"><br>
Why did the results as shown in the graph occur?</p>

username: ptrblck
cooked: <p>The oscillation is causing by clearing the cache manually in each iteration. Remove <code>torch.cuda.empty_cache()</code> and the memory should be reusable and increase if needed.</p>

username: rjx_3412
cooked: <p>I will experiment with this method, thanks a lot!</p>
Custom Quantization using PT2 -> q/dq representation

username: yshnsh
cooked: <p>I am trying to run a post training quantized linear layer using PT2 quantization flow. However the final quantized model using <strong>convert_pt2e</strong> api result in linear matmul op in fp32 precision and quant/dequant ops are placed in succession in the graph.</p>
<p>Eg. The linear layer I am testing is: nn.Linear(5,10)-&gt;RELU().<br>
Quantization using XNNPACK quantizer result in following quantized model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1774d73a7e5cf32d5df15ef37626eac937cf5d87.png" data-download-href="https://discuss.pytorch.org/uploads/default/1774d73a7e5cf32d5df15ef37626eac937cf5d87" title="Screen Shot 2023-10-12 at 2.43.25 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1774d73a7e5cf32d5df15ef37626eac937cf5d87.png" alt="Screen Shot 2023-10-12 at 2.43.25 PM" data-base62-sha1="3lvjl6PlKLrmeqxqymc0yDe1ziD" width="690" height="85" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-10-12 at 2.43.25 PM</span><span class="informations">1781×221 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The pattern here is q-&gt;dq-&gt;linear_op(fp32)<br>
How can we specify quantization to be q-&gt;linear_op(lower precision)-&gt;dq?</p>

username: jerryzh168
cooked: <p>this is expected, see <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html#convert-the-calibrated-model-to-a-quantized-model" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.1.0+cu121 documentation</a>, you’ll need to lower the model to the target backend in order to get a quantized linear op, please stay tuned for PyTorch Conference that’s happening next week: <a href="https://events.linuxfoundation.org/pytorch-conference/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Conference | Linux Foundation Events</a> we have some announcements in PyTorch Edge that will show how to do this for XNNAPCK.</p>
Custom torch.autograd.Function backward()

username: wcq
cooked: <p>In this example, <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch: Defining New autograd Functions — PyTorch Tutorials 2.1.0+cu121 documentation</a>, specifically in backward(), the gradient is being manually computed and hardcoded, is there a way to make autodiff do the work instead of the manual definition in backward()? I’m trying to extend PyTorch with a custom CUDA lib and make PyTorch aware of it by wrapping a custom CUDA API call within the forward() portion of the new Function, however, I do not wish to manually derive the function defined in forward() which autodiff should be able to handle but it just happens it’s being wrapped inside a new Function() it has to be defined in backward()?</p>

username: ptrblck
cooked: <p>No, you would need to define the <code>backward</code> once you “leave” PyTorch and use a 3rd party library or custom CUDA/C++ code. To allow PyTorch’s Autograd to derive the <code>backward</code> use <code>torch</code> operations.</p>
How to calculate BatchNorm2d manually in Python?

username: Vatsal_Aggarwal
cooked: <p>Hello Everyone</p>
<p>I want to calculate the batchnorm2d manually as I have to use the trained weights and inputs to calculate in C++ for my research. So I was trying to recreate each layer output in Python first.</p>
<p>I have the output of the convolutional layer of size [1, 32, 22, 72] and I want to perform batch normalization mentioned in the link (<a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm2d — PyTorch 2.1 documentation</a>). I also have the weights stored as numpy arrays for gamma (weights), beta (biases), running mean, and running variance obtained from the trained batchnorm2d layer. The shape of each of the parameters above is as follows:<br>
gamma (weights) = [32, 1, 8, 8]<br>
beta (biases) = [32]<br>
running mean = [32]<br>
running variance = [32]</p>
<p>Can someone help me with a Python code to perform batch norm from scratch? I am getting shape or broadcast error when I try to do this. Also, can someone explain why we get the gamma value of the shape [32, 1, 8, 8] and why not [32]? Is it related to the kernel size of the conv layer? Because my kernel size was (8, 8) with 32 filters.</p>

username: ptrblck
cooked: <p><a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">This reference</a> might be helpful.</p>
How to train multiple models on a single big GPU

username: MaxHeuillet
cooked: <p>Hi,</p>
<p>I have to run multiple realizations of the same experiment. I want to use the recent and fast nvidia A100 chips, that have 40go of memory. Each realization of my experiment uses a Pytorch model that is small, i.e. typically works OK with 4Go of GPU memory. Therefore, I would like to know if it is possible to load multiple of these models on the A100 and train them in parallel on the same A100? Should i do this with the regular multiprocessing package or should i use other tools?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You could simply launch multiple Python scripts, which will execute kernels when device resources are free as explained <a href="https://github.com/pytorch/pytorch/issues/109111#issuecomment-1716873219">here</a>.</p>
Transformers(hugging face) module calls layer norm on embedding multiple times

username: legitimate-tee
cooked: <p>In order to understand how exactly fill-mask pipeline works in transformers, I decided to track the computations. The bert embeddings <a href="https://github.com/huggingface/transformers/blob/8b5da9fc6e22a9250691bab688e1465da922176a/src/transformers/models/bert/modeling_bert.py#L239" rel="noopener nofollow ugc">call layer norm</a> once on an embedding. However <a href="https://github.com/pytorch/pytorch/blob/5d5990fc49260cfd717df629017720109c89f1d4/torch/nn/modules/normalization.py#L196" rel="noopener nofollow ugc">F.layer_norm</a> in normalization is called several times right after the above call. I expected F.layer_norm to be called only once. Why does this happen?</p>
ImportError during Stable Diffusion Web UI on Azure ML Compute Instances deployment

username: Velociraptor
cooked: <p>Hi,</p>
<p>some time ago I have successfully deployed Stable Diffusion with AUTOMATIC1111 UI using instance within MLWorkspace at Azure by following tutorial by Vlad Iliescu: <a href="https://vladiliescu.net/stable-diffusion-web-ui-on-azure-ml/" rel="noopener nofollow ugc">https://vladiliescu.net/stable-diffusion-web-ui-on-azure-ml/</a></p>
<p>Recently I have tried to redeploy it and failed due to:<br>
<code> ImportError: cannot import name 'get_innermost_proxy_mode' from 'torch.fx.experimental.proxy_tensor' (/anaconda/envs/a1111-sdwebui/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py)</code></p>
<p>I tried to fix it using different versions of dependencies but failed again.</p>
<p>Any ideas?</p>

username: ptrblck
cooked: <p>The function exists in the current code base as seen <a href="https://github.com/pytorch/pytorch/blob/c4db6076079f23175fb7de63f346b083b1820bff/torch/fx/experimental/proxy_tensor.py#L872">here</a> and was added 10 months ago in <a href="https://github.com/pytorch/pytorch/pull/90628">this PR</a>.<br>
I can also call it in <code>torch==2.1.0</code>:</p>
<pre><code class="lang-python">torch.fx.experimental.proxy_tensor.get_innermost_proxy_mode
Out[15]: &lt;function torch.fx.experimental.proxy_tensor.get_innermost_proxy_mode()&gt;
</code></pre>

username: Velociraptor
cooked: <p>Thank you for your response.</p>
<p>That’s interesting as I have also tried with torch==2.1.0 as well as older implementations from: <a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a>.</p>

username: Moosh
cooked: <p>hi Maks, were you able to figure this out? I am facing the same problem. Tried to upgrade torch to 2.1.0 and that did not help.</p>

username: Moosh
cooked: <p>all right <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  I figured it out. If you use Vlad Iliescu how-to on how to setup this up, I’ve updated the line to:</p>
<p>conda install pytorch=2.0.1 torchvision=0.15.2 torchaudio=2.0.2 chardet pytorch-cuda=11.8 -c pytorch -c nvidia -y</p>
<p>as of 10/12/2023 it works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Run ml job in container(docker) create too many thread and data load very slow

username: m_k
cooked: <p>When I ran a machine learning task in a container, it created a large number of threads, which led to a particularly high load on the cpu, making the data load slow.</p>
<p>but when I run the job in the node(which the container in), It behaves normally and loads data much faster。</p>
<p>my dataload config as:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/891864fef944ade03fb7c3a5cd0ca7a716822750.png" alt="image" data-base62-sha1="jyNG402OzqPvyxt4kpK2ahNyiDS" width="436" height="230"></p>
Loading many images in getitem but returning only one at a time

username: SolErika
cooked: <p>I have a directory full of tfrecords with each tfrecord corresponding to an image which has been subdivided into many tiles, and their labels. For training, I would like to turn this into batches of let’s say 8 random tiles, which can be sampled from various images. My idea is to load as many tiles as possible into RAM in the getitem method (using dareblopy to decode the tfrecords), then to somehow sample them in the get_item function (or alternatively in a custom collate function) and return or possibly yield one batch at a time.</p>
<p>Currently, I first create an array of filenames where each row contains all the filenames to read at a time, then I pass this array into the custom Dataset class, and let the index in get_item refer to the number of rows in the array. Here is sample code</p>
<pre><code class="lang-auto">class TfrecordDataset(Dataset):

    def __init__(self, ids, tfrecords_path):

        self.ids = ids
        self.tfrecords_path = tfrecords_path

    def __getitem__(self, idx):

        # Get list of image IDs.
        image_ids = self.ids[:,idx]

        # Initialize parser for TFRecords.
        features = {'tile_id': db.FixedLenFeature([], db.string),
                    'image': db.FixedLenFeature([], db.string),
                    'label_image': db.FixedLenFeature([], db.string)}
        parser = db.RecordParser(features)


        tile_list = []

        for image_id in image_ids:
            # Decode the tfrecords one by one
            tfr_db = db.RecordReader(self.tfrecords_path + "/" + str(image_id) + "/" + ".tfrecord")
            tfr_length = tfr_db.get_metadata()[2]

            # Gather all their tiles into tile_list
            for i in range(tfr_length):
                example = next(tfr_db)
                tile_id, tile, label = parser.parse_single_example(example)
                image_tile = Image.open(io.BytesIO(tile[0]))
                label_tile = Image.open(io.BytesIO(label[0]))

                image_tile = TF.to_tensor(image_tile)
                label_tile = TF.to_tensor(label_tile)

                tile_list.append((image_tile, label_tile, image_id))

        return tile_list

    def __len__(self):
        return self.ids.shape[1]
</code></pre>
<p>This does what I want in the sense that it successfully loads many images and combines their tiles into a single list which can easily be shuffled, but the get_item return becomes a huge list of thousands of images and I would much prefer if the Dataset iterator could return reasonably sized batches, like 8 tiles or so, since the current approach becomes strange when passing into a dataloader. Is there a way to do this easily using pytorch? If there is a method which accomplishes the same thing without bothering with preparing the array etc., then that’s of course fine too.</p>