akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Support for reading uint16, uint32 type image

ToastCheng: <p>Hi, I am new to libtorch and I’m trying to use it for inferencing. In my case, the input is queried from API as a 1-D pointer image, and then convert it into tensor by <code>torch::from_blob</code>:</p>
<pre><code class="lang-auto">int width = 512;
int height = 512;
uint16* pixelData;
some_initialization(pixelData); // do something to load the data.

auto options = c10::TensorOptions().dtype(torch::kInt16);
at::Tensor t = torch::from_blob(pixelData, { width, height }, options);

</code></pre>
<p>However the supported types seems to only include <code>kUInt8, kInt8, kInt16, kInt32, kInt64, kFloat32 and kFloat64</code> according to the <a href="https://pytorch.org/cppdocs/notes/tensor_creation.html#configuring-properties-of-the-tensor" rel="nofollow noopener">document</a>. But sometimes (rarely) I will get images with type of uint16 or uint32.</p>
<p>The value of tensor might be wrong if I give the option a <code>torch::kInt16</code> type to parse a <code>uint16</code> type image. However in my case the data is mostly signed int16, so it’s hard for me to test. Just want to know is there a way to correctly parse the input into tensor, and also curious about why there isn’t those types in pytorch.</p>
<p>Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

yf225: <p>We don’t have plans to support <code>kUInt16</code> in the short term. What we might want is a conversion utility function. Do you mind opening a feature request on our Github repo for this?</p>

siddhinath_kharade: <p>I want same…is there optimize way to convert it</p>
Backpropagation with dictionaries

tom5: <p>I am trying to write a custom loss function in which dictionary structure is compared. Essentially I call my model multiple times, and the output tensors are put into embedded dictionary structures (i.e.</p>
<p>dict = {‘0’: {‘0’: tensor_a1, ‘1’: tensor_a2}, ‘1’: {‘0’: tensor_b1, ‘1’, tensor_b2}}</p>
<p>etc.), with their position in this structure determined by the value inside the tensor. Is it possible to backpropagate through such a structure? For example, if tensor_b1 should have been in the dictionary with the ‘a’ tensors (based on comparison with an expected dictionary in the training data), I want to define a loss that contains this information and will be compatible with autograd.</p>
<p>Thanks,</p>
How to schedule the autograd order

acplus: <p>Assume that I have model <strong>Model</strong> like this</p>
<pre><code class="lang-auto"> layer1-&gt;layer2-&gt;layer3. 
</code></pre>
<p>I feed a <strong>input</strong>  in the <strong>Model</strong>, then I have</p>
<pre><code class="lang-auto">input -&gt; layer1 -&gt; output1 -&gt; layer2 -&gt; output2 -&gt; layer3 -&gt; output3
</code></pre>
<p>After I call <code>output3.sum().backward()</code>, the default gradient computation is as follows</p>
<pre><code class="lang-auto">output3.grad -&gt; layer3.grad -&gt; output2.grad -&gt; layer2.grad -&gt; output1.grad -&gt; layer1.grad
</code></pre>
<p>However, I want to schedule this gradient computation order like this</p>
<pre><code class="lang-auto">output3.grad -&gt; output2.grad -&gt; output1.grad -&gt; layer1.grad-&gt;layer2.grad-&gt;layer3.grad
</code></pre>
<p>Is it any suggestion about how to achieve this?</p>
Allocates memory and never free it when using Module.cpu()

Xander_C: <p>I create many modules while doing federated learning, and save them in the RAM(cpu), only move to cuda when training and evaluating.</p>
<p>Noticed that the memory was not freed when moving the model to cuda.</p>
<pre><code class="lang-auto">Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    47   3719.9 MiB   3719.9 MiB           1       @profile
    48                                             def train(self):
    49   3719.9 MiB      0.0 MiB           1           self.model = self.model.to(Config.device)
    50   3719.9 MiB      0.0 MiB           1           optimizer = torch.optim.Adam(self.model.parameters(), lr=Config.learning_rate)
    51   3719.9 MiB      0.0 MiB           1           self.model.train()
</code></pre>
<p>Move the model back to cpu when training is complete</p>
<pre><code class="lang-auto">69   3719.9 MiB      0.0 MiB           6               for epoch in range(epochs):
    70   3719.9 MiB      0.0 MiB           5                   batch_loss_list = []
    71   3719.9 MiB      0.0 MiB          60                   for data in self.loader:
    72   3719.9 MiB      0.0 MiB          55                       x = data[0].to(Config.device)
    73   3719.9 MiB      0.0 MiB          55                       y = data[1].to(Config.device)
    74   3719.9 MiB      0.0 MiB          55                       loss, y_ = self.train_batch(x, y)
    75   3719.9 MiB      0.0 MiB          55                       batch_loss_list.append(loss)
    76   3719.9 MiB      0.0 MiB           5                   mean_loss = np.mean(batch_loss_list)
    77   3719.9 MiB      0.0 MiB           5                   if mean_loss &lt; Config.local_loss_threshold:
    78                                                             break
    79   3719.9 MiB      0.0 MiB           5                   self.schedule.step()
    80   3719.9 MiB      0.0 MiB           5                   self.logger.log_client_loss(self.client_id, epoch, np.mean(batch_loss_list).item())
    81   3721.7 MiB      1.8 MiB           1           self.model = self.model.cpu()
    82   3721.7 MiB      0.0 MiB           1           return loss
</code></pre>
<p>Each times move the model from cuda to cpu, 1.6-1.9MiB RAM would be allocated.<br>
In the end, it consumed all my memory and crashed the program</p>
Save normalized tensor to png in a loop

Steier: <p>Hi there,<br>
I am working on a GAN and cannot make it work to save images that I transformed into tensors back to “normal” pngs within a loop. The same goes for the tensors that are generated by the Generator.</p>
<p>I applied the following transformation for the original images I am using for the training in the GAN ( I hope i did it the right way):</p>
<blockquote>
<p>transform = transforms.Compose(<br>
[<br>
transforms.ToPILImage(),<br>
transforms.Resize(img_size),<br>
transforms.CenterCrop(img_size),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),<br>
]<br>
)</p>
</blockquote>
<p>When trying to save the tensors as png images in a loop with the following code they do not come out the right way:</p>
<blockquote>
<p>real_samples = next(iter(train_loader))<br>
for i in range(4):<br>
torchvision.utils.save_image(real_samples[i, :, :, :],<br>
‘Real_Images/real_image{}.png’.format(i))</p>
</blockquote>
<p>On the left is an example of the original image after transformation and on the right an example of the “wrongly” saved ones:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c76e10f009ab06eca3c8e95abe9893b1e49eb5cd.png" alt="Test" data-base62-sha1="ssePBiTUr6oZXDTRRRS1VGr78rH" width="126" height="63"></p>
<p>Can anyone please help me out with saving the images in the right way?</p>

Steier: <p>So I just found the solution, which might be obvious to all of you but me. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> I had to normalize the images again. I thought i had to do the opposite (de-normalize). Accordingly, the code for saving the images is</p>
<blockquote>
<p>torchvision.utils.save_image(<br>
real_samples[i, :, :, :], ‘Training_Punks/Training_{}.png’.format(i), <strong>normalize=True</strong>)</p>
</blockquote>
<p>Can someone let me know why I need to normalize them again?</p>
Aim trainer ,cpu performance

Jermy420: <p>How to improve flicking speed in valorent ?</p>

ptrblck: <p>I think you’ve landed in the wrong discussion board, as this one is about <a href="https://pytorch.org/">PyTorch</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

Andrei_Cristea: <p>But the answer is: lots of practice!</p>

ptrblck: <p>Now with your answer I had to look up what “flicking speed” refers to as I assumed it’s some kind of graphic output measurement (similar to FPS or so), but it seems I was totally ignorant <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

adamsmith34506: <p>Aim training is the answer that you are looking for. You can improve your flicking technique using <a href="https://aimtrainer.com/" rel="noopener nofollow ugc">https://aimtrainer.com/</a> . I am an avid gamer myself and i use aim trainers for the same thing.</p>
Issues with building libtorch without python dependencies

grmnptr: <p>Hello!<br>
I am trying to build libtorch on an HPC with very limited python support (no dependencies). Since I am trying to utilize the C++ frontend only and</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/docs/libtorch.rst</a></h4>


    <pre><code class="lang-rst">libtorch (C++-only)
===================

The core of pytorch does not depend on Python. A
CMake-based build system compiles the C++ source code into a shared
object, libtorch.so.

Building libtorch using Python
------------------------------

You can use a python script/module located in tools package to build libtorch
::
   cd &lt;pytorch_root&gt;

   # Make a new folder to build in to avoid polluting the source directories
   mkdir build_libtorch &amp;&amp; cd build_libtorch

   # You might need to export some required environment variables here.
   Normally setup.py sets good default env variables, but you'll have to do
   that manually.
</code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>states that I don’t need python to build libtorch, I am wondering if anybody managed to do it without any python installation before. When I try to configure cmake, I get the following error:</p>
<pre><code class="lang-auto">File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/raid/projects/pytorch/tools/codegen/gen.py", line 3, in &lt;module&gt;
    from typing_extensions import Literal
ModuleNotFoundError: No module named 'typing_extensions'
</code></pre>
<p>I see that cmake is actually checking my python directories, which is super weird since I don’t want to do anything with python. This is the cmake command that I tried in my build directory:</p>
<pre><code class="lang-auto">cmake -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_BUILD_TYPE:STRING=Release -DPYTHON_LIBRARY='' -DCMAKE_INSTALL_PREFIX:PATH=../pytorch-install -DBUILD_PYTHON:BOOL=OFF -DBUILD_TEST=OFF ../pytorch
</code></pre>
<p>Any help would be much appreciated. Thanks!</p>

kostas: <p>Hi Peter, did you solve that?</p>
Ignoring invalid distribution while import torch_xla

naychelynn: <p>I’m trying to use a vision transformer and I need to import torch_xla. Firstly, I tied to install it with" pip install torch_xla " but it doesn’t work. So, I search on the PyTorch discussion, GitHub, etc and I found some suggestions but I can’t still import torch_xla with the error message " ModuleNotFoundError: No module named ‘torch_xla’ ".<br>
The warning message to me while I’m trying to install torch_xla is as below:<br>
WARNING: Ignoring invalid distribution -orch (c:\users\shimalab\anaconda3\lib\site-packages)<br>
WARNING: Ignoring invalid distribution -orch (c:\users\shimalab\anaconda3\lib\site-packages)<br>
ERROR: torch_xla-1.9-cp37-cp37m-linux_x86_64.whl is not a supported wheel on this platform.<br>
WARNING: Ignoring invalid distribution -orch (c:\users\shimalab\anaconda3\lib\site-packages)<br>
WARNING: Ignoring invalid distribution -orch (c:\users\shimalab\anaconda3\lib\site-packages)<br>
WARNING: Ignoring invalid distribution -orch (c:\users\shimalab\anaconda3\lib\site-packages)</p>
<p>Welcome all the suggestions and thanks in advance!</p>
Each element in list of batch should be of equal size

HJX-zhanS: <p>When I try to use ConcatDataset, I get the error: “each element in list of batch should be of equal size”</p>
<p><code>ConcatDataset([dataset1,dataset2,dataset3])</code></p>
<p>Data dimensions dataset1, dataset2 and dataset3 are different.</p>
<p>Can you help me? Thanks</p>
NVIDIA GeForce RTX 3050 Ti Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation

yolo: <p>miniconda3/envs/smr_env1/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py:106: UserWarning:<br>
NVIDIA GeForce RTX 3050 Ti Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.<br>
If you want to use the NVIDIA GeForce RTX 3050 Ti Laptop GPU GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>All the other tickets specify to use CUDA version 11, but I am already on CUDA Version: 11.4 and still face this error. Any solutions?</p>

ptrblck: <p>You are most likely installing a PyTorch pip wheel or conda binary with the CUDA 10.2 runtime.<br>
Note that your locally installed CUDA toolkit would only be used if you are building PyTorch from source or a custom CUDA extension.</p>

yolo: <p>I most likely ran <code>pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html  </code> to install PyTorch and specify CUDA version.</p>
<p>The output for PyTorch CUDA version is -</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.version.cuda)
11.3
</code></pre>
<p>What would be a likely solution in this case?</p>

ptrblck: <p>I don’t think that’s the case, as the <code>1.10.0+cu113</code> pip wheel ships with these compute capabilities:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.0+cu113'
&gt;&gt;&gt; torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
</code></pre>
<p>while yours returns:</p>
<pre><code class="lang-python">sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37
</code></pre>
<p>which is the CUDA 10.2 support.</p>
<p>If you get stuck, try to create a new virtual env, reinstall the right version, and verify the right versions in the install logs.</p>

yolo: <p>You were right. The env I was using had an incorrect version. Thanks go it working.</p>

nguy_n_m_nh: <p>So what is the python version you choose, I’m having the same problem</p>

hoanhvong: <p><a href="https://github.com/pytorch/pytorch/issues/45028#issuecomment-695978783" class="inline-onebox" rel="noopener nofollow ugc">GeForce RTX 3080 with CUDA capability sm_86 is not compatible with the current PyTorch installation. · Issue #45028 · pytorch/pytorch · GitHub</a> I think this will help you<br>
<code>conda install pytorch torchvision cudatoolkit=11 -c pytorch-nightly</code></p>
Reshape the tensor with a certain fashion

miraboreasu: <pre><code class="lang-auto">import torch
import numpy as np
a = torch.tensor([[1, 4], [2, 5],[3, 6]])
b = a.view(6).detach().numpy()

</code></pre>
<p>The b is[ 1 4 2 5 3 6]</p>
<p>how to use reshape or other function, without for loop, so that b is [1 2 3 4 5 6]</p>
<p>This is just an example, want some generic answers, even 3D</p>

miraboreasu: <p>Here is the solution but only for Numpy (from other forum), can anyone helps with Torch</p>
<pre><code class="lang-auto">In [73]: a.ravel(order='F')
Out[73]: array([1, 2, 3, 4, 5, 6])
</code></pre>

ptrblck: <p>You could transpose the tensor and flatten it afterwards:</p>
<pre><code class="lang-python">a = torch.tensor([[1, 4], [2, 5],[3, 6]])
b = a.t().contiguous().view(-1)
print(b)
# tensor([1, 2, 3, 4, 5, 6])
</code></pre>
<p>I can’t give a generic answer as the reshape operations would depend on the actual data and what your expectation would be. Transposing could certainly work, but you would have to provide examples.</p>
Std and mean not clear at all

jhoanmartinez: <p>Hello fellas,<br>
What is the purpose of find mean and std, is from each image or whole dataset, what is use for, how to find it? not clear yet and i have see that is a must in all models or am I wrong? thanks a lot</p>

ptrblck: <p>Normalizing the data is often beneficial and the standardization or whitening of the data by subtracting the mean and dividing by the stddev shows commonly a superior convergence of the model. I’m sure there might be exceptions, but it’s quite the standard now. I couldn’t find a quick reference but am sure that Bishop explains it in <code>Pattern Recognition and Machine Learning</code>.</p>
Fed Avg-two implementations give very different results

Huzaifa_Capricorn: <h4>
<a name="implementation-1-1" class="anchor" href="#implementation-1-1"></a>Implementation 1</h4>
<h3>
<a name="fed-averaging-and-loading-weights-same-loop-2" class="anchor" href="#fed-averaging-and-loading-weights-same-loop-2"></a>Fed Averaging  and Loading weights same loop</h3>
<pre><code>for k in global_dict.keys(): 

    global_dict[k] = torch.stack([client_models[i].state_dict()[k].float() for i 
    in range(len(client_models))], 0).mean(0)
    global_model.load_state_dict(global_dict)
    for  model in client_models:
        model.load_state_dict(global_model.state_dict())
</code></pre>
<h4>
<a name="implementation-2-3" class="anchor" href="#implementation-2-3"></a>Implementation 2</h4>
<h3>
<a name="fed-averaging-and-loading-weights-different-loop-4" class="anchor" href="#fed-averaging-and-loading-weights-different-loop-4"></a>Fed Averaging and Loading weights different loop</h3>
<pre><code>for k in global_dict.keys(): 

    global_dict[k] = torch.stack([client_models[i].state_dict()[k].float() for i 
    in range(len(client_models))], 0).mean(0)
    global_model.load_state_dict(global_dict)
    
global_model.load_state_dict(global_dict)
for model in client_models:
  model.load_state_dict(global_model.state_dict())
</code></pre>
<p>Implementation 2 gives me a test accuracy of the global model similar to the client model but implementation 1 does not. I don’t know the reason behind it ?</p>
<p>Also, I am using a pre-trained model with additional layers. I want to keep the layers of the pre-trained model frozen during training for each client.</p>

ptrblck: <p>I’m a bit confused about the logic of both loops. It seems you are manipulating the <code>global_dict</code> in the outer loop (<code>for k in global_dict.keys()</code>) but are also loading it into the <code>global_model</code> as well as all <code>client_models</code>. Check if you are overwriting some of the states in this loop as I don’t fully understand why you would call the <code>load_state_dict</code> operation before the <code>gobal_dict</code> is finalized.</p>

Huzaifa_Capricorn: <p>Yeah so the preamble to this code is this:<br>
global_dict = global_model.state_dict()<br>
This is a temp dictionary that is initialized as the previous state of the global dictionary.<br>
This new dictionary is then loaded into the global model and client models as this is what happens after aggregation. So whether I am changing the states at every iteration or not, it should not matter, right?</p>

ptrblck: <aside class="quote no-group" data-username="Huzaifa_Capricorn" data-post="3" data-topic="155916">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huzaifa_capricorn/48/51194_2.png" class="avatar"> Huzaifa_Capricorn:</div>
<blockquote>
<p>it should not matter, right?</p>
</blockquote>
</aside>
<p>I don’t know as it apparently does change the execution. Feel free to post a minimal, executable code snippet reproducing the issue in case you get stuck.</p>
Tensor raw memory organization

Kubci: <p>Howdy!</p>
<p>I am developing models that are exported through ONNX and used by TensorRT. However, I am not observing the same results (not even similar) for the same images processed by the two executors (native pytorch and tensorrt). My first hunch was data loading and memory organization.</p>
<p>Goal:<br>
I want to compare the memory fed into executors and be sure that different executors work over the same data.</p>
<p>The questions:<br>
How is tensor organized in raw memory?<br>
Where is up in the image loaded by torchvision.io.read_image or is origin in upper left corner or lower left corner?<br>
How do I print out raw memory of tensor? One of the things I tried print(X.data_ptr().to_bytes(batch<em>channels</em>height<em>width</em>(4 bytes per float), sys.byteorder)) however, I got a lot of \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00… which seems incorrect</p>
<p>Note:<br>
In TensorRT the memory is organized following:</p>
<ul>
<li>batch offset → w*h*c*i; where i is batch number</li>
<li>channel offset → batch offset + w*h*i; where i is channel index</li>
<li>row offset → channel offset + w*i; where i is image row index</li>
<li>data element (column element) → row offset + i; where i is element index<br>
Assuming that the pointer is of matching data type.<br>
Where w is width, h is height, c is number of channels, and origin of the image is in upper left corner.</li>
</ul>

ptrblck: <p>By default PyTorch uses the channels-first layout as <code>[batch_size, channels, height, width]</code>.<br>
I’m not familiar with your use case, but wouldn’t the <a href="https://github.com/pytorch/TensorRT"><code>torch_tensorrt</code> module</a> work?</p>
Why is detach() used in this return statement?

oskean: <p>I saw the following bit of code on Github and was curious about the behavior of the return statement in the forward() function. Specifically I’m wondering why something is subtracted, detached, and then added back. Is this a standard technique or some hack to get the computation graph to behave a certain way?</p>
<pre><code class="lang-auto">import math
import torch
import torch.nn as nn

from torch.distributions.multivariate_normal import MultivariateNormal


class DoE(nn.Module):

    def __init__(self, dim, hidden, layers, pdf):
        super(DoE, self).__init__()
        self.qY = PDF(dim, pdf)
        self.qY_X = ConditionalPDF(dim, hidden, layers, pdf)

    def forward(self, X, Y, XY_package):
        hY = self.qY(Y)
        hY_X = self.qY_X(Y, X)

        loss = hY + hY_X
        mi_loss = hY_X - hY
        return (mi_loss - loss).detach() + loss
</code></pre>

AlphaBetaGamma96: <p>Do you have the link to the original Github repo? It might be explained there?</p>

ptrblck: <p>The code snippet would return the <code>mi_loss</code> in the forward pass (the <code>loss</code> tensor is subtracted) and would use <code>loss</code> the backward pass (the left-hand side is detached).</p>
Split an image into four equal coordinates

alameer: <p>Is there a function in PyTorch that allows me to split an image into four equal coordinates?</p>
<p>Thanks a lot.</p>

ptrblck: <p>Could you explain a bit what coordinates mean in the context of an image? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

alameer: <p>I’m trying to write a code which count objects in an image. I would like to spilt the image into 4 equal areas then run a count on each part then sum them all. Many Thanks for your help.</p>

ptrblck: <p>To create 4 patches of your input, you could use <code>unfold</code> as shown here:</p>
<pre><code class="lang-python">x = torch.randn(3, 24, 24) # channels, height, width
kernel_size, stride = 12, 12
patches = x.unfold(1, kernel_size, stride).unfold(2, kernel_size, stride)
patches = patches.contiguous().view(patches.size(0), -1, kernel_size, kernel_size)
print(patches.shape) # channels, patches, kernel_size, kernel_size
&gt; torch.Size([3, 4, 12, 12])
</code></pre>
<p>Once you have these patches, you could pass them to your model separately or as a batch.</p>

alameer: <p>I am quite new to PyTorch, based on what have you selected the values please of the channels, height, width and the kernal_size and stride. Many Thanks</p>

ptrblck: <p>I selected the channels randomly as 3, since it would correspond to an RGB image.<br>
The kernel size and stride of 12 were chosen to create 4 patches of the <code>24x24</code> spatial input shape.<br>
You might need to change it, if your input shape is different.</p>

alameer: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Many Thanks for your help. I would like to implement this into a training model. My aim by doing this, is to increase the number of images for training. The idea i have is to split each image into 4 patches prior to the model training, which works by counting the number of cells in a microscopy image. After splitting each image into 4 patches i would like to do a count of each patch then sum the total. Is this feasible to do or would you recommend a different method? Many Thanks</p>

favour_chianumba: <p>Hey ptrblck, I have used your code to create patches in fashion-mnist and I am struggling with understanding how to use them in my model or,  do i create a new array of tensors or mini batches or do i separate the patches to create a longer array and how would i do that</p>

ptrblck: <p>Could you explain your use case a bit more and in particular how you would like to use these patches in the model?<br>
The reshaping of these patches might depend on the actual model architecture, so I would need more information about your use case.</p>

xabder_zhan: <p>Hi ptrblck, thanks for your code. I am now wondering how can I fold back to the original image x after some computations on the patches. could this be implemented with <code>fold</code>?</p>

ptrblck: <p>Yes, <code>fold</code> can be used to recreate the input tensor assuming no overlapping patches were used before.</p>
Prune.global_unstructured usability

Ilya_Kamenshchikov: <p>I am finding <code>prune.global_unstructured</code> unnecessarily difficult to use - I have a torch.nn.Module that is a Sequential model of layers and other Sequential objects. I’d like to experiment with global pruning, so naturally starting point is to try pruning all parameters. To do so I need to pass a list of tuples of format [module, key] where key would be the name of a module parameter.</p>
<p>So I do:</p>
<pre><code class="lang-python">parameters_to_prune = []

for name, val in model.named_parameters():
    parameters_to_prune.append((model, name))
</code></pre>
<p>Yet the implementation of  <code>prune.global_unstructured</code> works not via torch.get_parameter, which would play nicely with keys from <code>named_parameters</code>, but tries <code>getattr(model, name)</code> which fails - the actual parameter lives under sequence of objects, like model[0].weight for '0.weight` parameter.</p>
<p>Is there a reason for this design? Is there a better way than writing a fancy recurrent function to find all leaf submodules of model?</p>
<p>Specific way I call prune and output:</p>
<pre><code class="lang-python">
def ConvMixer(dim, depth, kernel_size=5, patch_size=2, n_classes=10):
    return nn.Sequential(
        nn.Conv2d(3, dim, kernel_size=patch_size, stride=patch_size),
        nn.GELU(),
        nn.BatchNorm2d(dim),
        *[nn.Sequential(
            Residual(nn.Sequential(
                nn.Conv2d(dim, dim, kernel_size, groups=dim, padding="same"),
                nn.GELU(),
                nn.BatchNorm2d(dim)
            )),
            nn.Conv2d(dim, dim, kernel_size=1),
            nn.GELU(),
            nn.BatchNorm2d(dim)
        ) for i in range(depth)],
        nn.AdaptiveAvgPool2d((1, 1)),
        nn.Flatten(),
        nn.Linear(dim, n_classes)
    )
model = ConvMixer(args.hdim, args.depth)

parameters_to_prune = []

for name, module in model.named_parameters():
    parameters_to_prune.append((model, name))

prune.global_unstructured(
    parameters=parameters_to_prune, pruning_method=prune.L1Unstructured, amount=0.05
)

""" Output
Traceback (most recent call last):
   File "python3.9/site-packages/torch/nn/utils/prune.py", line 1088, in global_unstructured
    [
  File "python3.9/site-packages/torch/nn/utils/prune.py", line 1089, in &lt;listcomp&gt;
    getattr(module, name + "_mask", torch.ones_like(getattr(module, name)))
  File "python3.9/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Sequential' object has no attribute '0.weight'
"""
</code></pre>

mingzhe_lv: <p>I face the same problem as you, dude.</p>
How are probs from logits computed in Categorical?

leckofunny: <p>Hello everybody!</p>
<p>Just by looking at the code of Categorical and its base class, I was not able to find a line that assigns something to self.probs, if logits are supplied as the only argument. While using the debugger to step through the code, I observed that self.probs is assigned magically once the line</p>
<pre><code class="lang-auto">self.logits = logits - logits.logsumexp(dim=-1, keepdim=True)
</code></pre>
<p><a href="https://github.com/pytorch/pytorch/blob/master/torch/distributions/categorical.py#L60" rel="noopener nofollow ugc">categorical.py#L60</a></p>
<p>is run.</p>
<p>It would be great if someone could shed some light onto this matter!</p>
<p>Thanks</p>

gphilip: <p>There is something wrong with the URL you posted.</p>

leckofunny: <p>The link is fixed now.</p>
<p>I think I know the answer now.</p>
<p>It is due to the lazy property.</p>

vwxyzjn: <p>For those stumbling across this issue, I found <a href="https://gregorygundersen.com/blog/2020/02/09/log-sum-exp/" class="inline-onebox" rel="noopener nofollow ugc">The Log-Sum-Exp Trick</a> to be very helpful.</p>
CUDA_ERROR_NOT_INITIALIZED when setting num_workers > 0 in DataLoader running with CUDA

imayachita: <p>Hi,<br>
I’m getting an issue <code>F tensorflow/stream_executor/cuda/cuda_driver.cc:153] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error</code> when calling torch geometric <code>DataLoader</code> (which is inherited from <code>torch.utils.data.DataLoader</code>) when setting <code>num_workers</code> &gt; 0 when running using CUDA (e.g. <code>CUDA_VISIBLE_DEVICES=0 python script.py</code>). After that, the script failed with <code>DataLoader worker (pid(s) 31890) exited unexpectedly</code>. Any explanation/solution? I read that the most common solution is to use <code>num_workers</code> = 0 but I want to parallelise the dataloading since reading the dataset is the current bottleneck (my dataset is saved in TFRecords). Thanks</p>

ptrblck: <p>The error seems to be raised from TensorFlow and I would guess that your use case tries to re-initialize a CUDA context and thus might break. Try to use the <code>spawn</code> method and it might work.</p>

imayachita: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for replying!<br>
Do you mean calling <code>torch.multiprocessing.set_start_method('spawn')</code> before calling the function?<br>
If I do something like this:</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    torch.multiprocessing.set_start_method('spawn')
    training_func()
</code></pre>
<p>It would throw an error:<br>
<code>I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory</code></p>

ptrblck: <p>The initial error seems to be solved and TF fails to allocate device memory now.<br>
I guess you might want to change TF’s behavior to allocate memory if needed and not to grab all device memory, but the TF discussion board might be a better place as you would find the experts there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

imayachita: <p>Ok I’m trying to sort out that issue now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">. In the case that the GPU memory is fully occupied because the model takes a lot of memory, can I still use GPU to run multiprocessing in dataloader? Is there any equivalent way like tf.data.Dataset where we can fetch and the process happens in CPU?</p>

ptrblck: <p>The <code>DataLoader</code> uses the <code>Dataset.__getitem__</code> to fetch each sample which would use the CPU by default unless you are explicitly using the GPU. PyTorch will not use your GPU(s) by default.</p>
Using nn.Parameter in Opacus

ChrisWaites: <p>Hey there, we are trying to train a private version of a particular model which uses nn.Parameter and getting the error<code> torchdp.dp_model_inspector.IncompatibleModuleException</code>.</p>
<p>In particular, the parameters are defined for the model <a href="https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/vit.py#L98-L99" rel="noopener nofollow ugc">here</a> and used in the forward function <a href="https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/vit.py#L117-L118" rel="noopener nofollow ugc">here</a>. To the best of our knowledge these parameters and associated operations preserve privacy because they don’t compute any aggregate batch statistics. What would be the recommended way about being able to train with this model definition.</p>
<p>Is there some sort of workaround we could do to wrap these lines in a valid module? Do we need to wait for the team to add an accepted module to <code>opacus.SUPPORTED_LAYERS</code>?</p>

Darktex: <p>Hi Chris!</p>
<p>Luckily you don’t need to fork nor wait for the team to change that for you. You will however have the responsibility of writing a function to compute grad_sample for your layer. Once you write it, you simply register it for your layer using <a href="https://opacus.ai/api/per_sample_gradients.html#opacus.grad_sample.utils.register_grad_sampler" rel="noopener nofollow ugc">@register_grad_sampler</a> and you should be good to go!</p>

anibadde: <p>Hi! I’ve been running into a similar issue, <a class="mention" href="/u/chriswaites">@ChrisWaites</a> did you manage to get this working?</p>

Huzaifa_Capricorn: <p>Hi Chris, were you able to write a compute grad_sample for the nn.Parameter ? I was stuck on this problem as well</p>
Is LayerNorm deterministic?

divinho: <p>It isn’t for me. I want to confirm whether that is normal or not.</p>
<p>More specifically what I see happening is that in different builds I get different results for the same input.</p>

ptrblck: <aside class="quote no-group" data-username="divinho" data-post="1" data-topic="155854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9dc877/48.png" class="avatar"> divinho:</div>
<blockquote>
<p>in different builds I get different results for the same input.</p>
</blockquote>
</aside>
<p>I don’t think there is a guarantee to yield bitwise identical results between different PyTorch and library versions (e.g. different cublas, cuDNN etc.).<br>
The results should be deterministic using the same setup (PyTorch + lib versions, GPU etc.) if you stick to the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>.</p>

divinho: <p>Sorry with builds I meant different training runs. The GPU could be different though.</p>
<p>I guess that explains it (different GPU).</p>
DDP training process hanging on when some subprocess didn't compute loss

DogeWatch: <p>I use multiprocessing and Distributed Data Parallel to train on multi gpus, my code:</p>
<pre><code class="lang-auto"> for graphs in self.dataloader:
                graphs = [self._data_to_gpu(data, self.device) for data in graphs]
                graph_labels = torch.cat(list(map(lambda x: x.pop('label'), graphs)))
                try:
                    embeddings = torch.cat(self.model(graphs))
                except:
                    pass
                else:
                    optimizer.zero_grad()
                    loss = criterion(embeddings, graph_labels, miner(embeddings, graph_labels) if miner is not None else None)
                    loss.backward()
                    if self.rank == 0:
                        total_loss.append(loss.item())
                    optimizer.step()
                    torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.args.max_norm)
                if self.rank == 0:
                    bar.update()
</code></pre>
<p>I use the try…except… to catch the out of gpu memory error because of some big data. but it didn’t work. when code running into the except part, the whole training process would hang on.</p>

ptrblck: <p>If you are dealing with use cases where not all parameters are used in the forward pass, <code>find_unused_parameters</code> might help and avoid running into hangs. However, I’m unsure if this would/should also work if the loss is not created, since this particular model wouldn’t be able to contribute to the current training step at all.</p>

Harry1: <p>I also met this issue, did you solve it?</p>
DDP Hanging on when some iter have no GT for loss.backward()

zhao-yiqun: <p>When I use the DDP for training, some iter may have no ground truth for compute the loss. Following the method proposed here <a href="https://github.com/pytorch/pytorch/issues/23425" rel="noopener nofollow ugc">DDP hanging up</a>,  I just skip them in the model.forward(x) and use a loss = torch.zeros(1, required_grad=True) and loss.backward(). while also set find_unused_parameters=True.<br>
In his setting, only part of his network was not updated by loss.backward(). But in my setting, I use the some preprocess method  in the forward() to compute whether that is a useful feature after passing it from backbone, If  there is no feature, I just use the code like this,  And will not continue for the following networks. It means, in some iters of some epochs, there are no parameters to be updated.</p>
<blockquote>
<p>Blockquote<br>
def forward(x):<br>
feat = self.backbone(x)<br>
pairs, nopair = preprocess(feat, x)<br>
if nopair: # ret here !<br>
output_dict = {}<br>
loss_dict.update({‘total_loss’: torch.zeros(1, requires_grad=True)})<br>
return output_dict, loss_dict<br>
else:<br>
out = self.main_network(feat, x, pairs)<br>
output_dict.update({‘outputs’: out })<br>
loss_dict.update({‘total_loss’: self.compute_loss(out, ground_truth)})<br>
return   output_dict, loss_dict</p>
</blockquote>
<p>out, loss = model(x)<br>
loss = loss{‘total_loss’}<br>
loss.backward()</p>
<p>I’m looking forward to a solution! Thank you all.</p>

Harry1: <p>I also met this issue, have you found the solution to it?</p>
Which loss function to optimize the recall?

Liano: <p>I train some pre trained models for a binary classification task. Therefore I want to optimize on the recall value. It seems like the nn.CrossEntopyLoss Function is the best one for the job, but is there another one or should I define my own loss function?? (I’m new to PyTorch)<br>
some help would be great <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Kilian!</p>
<aside class="quote no-group" data-username="Liano" data-post="1" data-topic="155900" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ecb155/48.png" class="avatar"> Liano:</div>
<blockquote>
<p>Therefore I want to optimize on the recall value.</p>
</blockquote>
</aside>
<p><em>Recall</em> (in the context of making classification predictions for a given<br>
dataset) is the percentage of positive samples that you correctly classify<br>
as positive.  Note, it doesn’t care whether you classify negative samples<br>
correctly or not.</p>
<p>It is a useful performance metric, but, in isolation, problematic.  You can<br>
achieve perfect, 100% recall by having your model classify <em>all</em> samples<br>
as positive – hardly useful.</p>
<p>(<em>Precision</em> – often used as a companion metric – is the percentage of<br>
samples that you classified as positive that actually are positive.  This<br>
metric does care whether you incorrectly classify negative samples as<br>
positive.)</p>
<aside class="quote no-group">
<blockquote>
<p>It seems like the nn.CrossEntopyLoss Function is the best one for the job,</p>
</blockquote>
</aside>
<p>For a binary classification task, I would recommend that you use<br>
<a href="https://pytorch.org/docs/1.12/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> as your loss criterion.  (It is a form of cross entropy,<br>
but specifically tailored to the case of binary classification.)</p>
<p>You can bias your training to achieve improved recall – at the cost of<br>
reduced precision – by using <code>BCEWithLogitLoss</code>’s <code>pos_weight</code> constructor<br>
argument, setting it to a value greater than one.</p>
<p>Best.</p>
<p>K. Frank</p>

Liano: <p>thanks Frank! I got a good solution now</p>
How to raise each element of a tensor to a power of another tensor of different size?

Andrew_Legend: <p>Sorry if the title is a bad explanation of what I want, but I had no better way of describing it</p>
<p>I asked a similar question yesterday, at the time of posting this and I got a solution. However it turned out that is not what I want. What I want instead is something like this example:</p>
<pre><code class="lang-auto">A= [2, 3]

B= [1, 2, 3]

C= Function(A, B)
</code></pre>
<p>and then have it return:</p>
<p>C=[[2, 4, 8], [3, 9, 27]]</p>
<p>But I looked at the documentation as best as I could and still I can’t figure out how to do it without using a fancy for loop. Any ideas are appreciated</p>

ptrblck: <p>Broadcasting should work:</p>
<pre><code class="lang-python">A = torch.tensor([2, 3])
B = torch.tensor([1, 2, 3])

ret = A.unsqueeze(1) ** B
print(ret)
# tensor([[ 2,  4,  8],
#         [ 3,  9, 27]])
</code></pre>

Andrew_Legend: <p>In yourr particular example it works pretty well. However it does not work when tensor B is in the form of [B, C, H, W]</p>