akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Predictions from DataLoader: RuntimeError if using cpu(), TypeError if using cuda()

AJV: <p>I’m learning PyTorch basics. I have trained a ResNet50-based model successfully in a Jupyter Notebook using cuda. It achieves around 85% accuracy, and I’d like to explore whether there are patterns to which classes it struggles with. Adapting code from a Microsoft Azure tutorial to obtain the predictions and labels so that I can go on to make a confusion matrix to begin the exploration, I have</p>
<pre><code class="lang-auto">#Pytorch doesn't have a built-in confusion matrix metric, so we'll use SciKit-Learn
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
%matplotlib inline

# Set the model to evaluate mode
model.eval()

# Get predictions for the test data and convert to numpy arrays for use with SciKit-Learn
print("Getting predictions from test set...")
truelabels = []
predictions = []
#probabilities = []
for data, target in test_loader:
    for label in target.cpu().data.numpy():
        truelabels.append(label)
    for prediction in model.cpu()(data).data.numpy().argmax(1):
        predictions.append(prediction) 
</code></pre>
<p>The problem I’m encountering is that when I use target.cpu() and model.cpu() in the above code, it starts running very slowly then throws the following error:</p>
<p>RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:79] data. DefaultCPUAllocator: not enough memory: you tried to allocate 62914560 bytes.</p>
<p>Or crashes the Google Chrome window that the Jupyter Notebook is open on.</p>
<p>I trained the model using cuda, but if I switch to target.cuda() and model.cuda() it throws the following error:</p>
<p>TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.</p>
<p>Since I adapted this from a tutorial that used a much smaller dataset as a proof of concept, I’m not sure if this is even a proper way to explore the results. Can someone please advise me on best practice in this regard, or recommend a workaround for the issues I’m encountering?</p>

AJV: <p>Update: I used the new pre-trained vit-b-16 model and did not get the RuntimeError from the posted code snippet when using that model. Still interested in using cuda() if possible to speed up the process and hearing if there is a better way to accomplish my objective.</p>

ptrblck: <p>The <code>RuntimeError</code> in the <code>CPUAllocator</code> is raised if you are running out of host RAM and you would need to reduce the memory usage e.g. by decreasing the batch size of the training.</p>
<p>The <code>TypeError</code> is raised as you are trying to convert a <code>CUDATensor</code> to a numpy array, which doesn’t work without pushing the tensor to the CPU first as described in the error message.</p>

AJV: <p>I tried pushing the tensor to the CPU and it wouldn’t work, but then I found that for whatever reason, the data from the loader had different Tensor types for the input and weights. After explicitly sending the data to cuda before prediction then explicitly pushing it back to the cpu, it now works as expected. Thanks for the help.</p>
What is the correct Pytorch resnet50 input normalization intensity range

einsteinxx: <p>I’m using resnet50 pre-trained as my backbone for faster-rcnn and am trying to normalize the data for fine-tuning. The data original intensity is 0 to 1, then I do some contrast equalization and then convert it back to 0,1 range and apply the Resnet norm (from pytorch page). This results in an odd range (see image below). When I apply a generic normalization (not the resnet preferred) and get it to 0,1 range, the final results are worse after fine-tuning/training.</p>
<p>Should the resnet input images always use the prescribed norm setup and if so, what should the final intensity range be? This looks to be ~zero centered, but would 0,1 provide any advantages. Would there be advantages to normalizing specific to your fine-tuning data instead of using resnet info?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51159a33b8ebb4d0fff8f907ff91d2b3c052ab32.jpeg" alt="image" data-base62-sha1="bziXohTpIq3ujNtWGXBbBupjNHc" width="519" height="381"></p>
<pre><code class="lang-auto">Pytorch Resnet information:
All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 
3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. 
The images have to be loaded in to a range of [0, 1] and then normalized using mean = 
[0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].
</code></pre>
<pre><code class="lang-auto">Generic function used to do the ImageNet norm---
#Imagenet requires a specific Norm to 0,1 then norm with the mean and std from 
#the large imagenet dataset
##### NOTE: ToTensor() converts to float32 tensor and adjusts range to 0,1

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]) ])
</code></pre>

ptrblck: <p>I would assume calculating the new dataset stats would be beneficial especially if you are changing the data domain (e.g. from “natural” ImageNet images to medical images).<br>
As a quick check you could calculate the <code>mean</code> and <code>std</code> of the batches after using the ImageNet <code>Normalize</code> and see how strongly it diverges from a zero-mean and unit-variance.</p>
Shuffling 2 downloaders in the same way

Anna_Inberg: <p>Good afternoon!<br>
I’m building a DL model in PyTorch, based on 2 types of input - images and the corresponding numerical coordinates of 50 landmarks on the image. Both types of input have labels 0 or 1.<br>
The file names are aligned: image name is filename1.png and the corresponding landmarks filename1.txt<br>
I need to shuffle both images and landmarks in unison to keep the labels aligned. What is the best way to do it?<br>
I tried importing the data in NumPy arrays first, then turning the arrays into torch tensors and then turning the tensors into dataloaders, however, this approach corrupts the tensors’ shape and makes a mess.</p>

ptrblck: <p>The cleanest approach would be to create a custom <code>Dataset</code> (e.g. as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>) and to load the data sample with its corresponding target in the <code>__getitem__</code>. This would avoid creating two separate <code>DataLoader</code>s and making sure the <code>sampler</code> shuffles both in the same way.</p>
Observer schemes for activations in Dynamic Quantization of LSTMs

srana: <p>We are attempting to replicate the logic of Dynamic Quantization for LSTMs. While the observer scheme for the weights if clear from the default_qconfig for Dynamic Quantization, the scheme used for the activations isn’t clear to me. I have attempted to use the MinMaxObserver, MovingAverageMinMaxObserver, and the HistogramObserver but the results I obtain do not match the ones I get from using the DynamicQuantizedLSTM directly. I would appreciate any inputs on the scheme used for activations.</p>
<p>P.S. The overarching objective of attempting this exercise is to be able to implement Static Quantization and QAT for LSTMs which currently does not exist in PyTorch, and this would be a very welcome feature.</p>

deeplearner20: <p>Hello OP, did you find a fix for this yet?</p>
How can I change the Bias value in nn.Linear ? can I customize the bias?

Sam_Fst: <p>nn.Linear(num_in_features, num_out_features) ?</p>
<p>I have to make a network where the initial hidden layer bias is constant, how can I do that ?</p>

ptrblck: <p>If you don’t want to update the bias parameter, you could set the <code>requires_grad</code> attribute of the <code>bias</code> to <code>False</code> and don’t pass it to the optimizer:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
lin.bias.requires_grad = False

optimizer = torch.optim.Adam([lin.weight], lr=1.)
output = lin(torch.randn(1, 1))
output.backward()
lin.bias.grad
&gt; 
lin.weight.grad
&gt; tensor([[-0.0095]])
</code></pre>

Sam_Fst: <p>Thanks, but how can I set the Bias to a constant tensor array ?</p>

ptrblck: <p>These methods should work:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
with torch.no_grad():
    lin.bias.fill_(1.)
    # or
    lin.bias = nn.Parameter(torch.randn(1))
</code></pre>

Rainbow995: <p>How can I set the bias of the classifier of Mobilenetv2 to 55? Can I use this method? sth like this:</p>
<p>model = torchvision.models.mobilenet_v2(pretrained=pretrained)<br>
model.classifier[1] = nn.Linear(model.last_channel, 1)<br>
model.classifier[1].bias = nn.Parameter(55)</p>

ptrblck: <p>No, your code snippet work work since <code>nn.Parameter</code> expects a tensor input not an <code>int</code> (see my code snippet).<br>
Besides that, <code>model.classifier[1].bias</code> contains <code>1000</code> values but I would assume that a single <code>bias</code> value would also work as internally a broadcast might be used.</p>
Rfftn with real output values

mojzaar: <p>Hi,<br>
When I tried the rfft function at Pytorch a few years ago, instead of getting a complex tensor, I could get a real tensor with an addition dimension that had the real and complex values as two real value channels in the outputs. I’m curious as to how I can get the output of the rfftn in this manner. I know I can concatenate the torch.real and torch.imag into a new dimension, but I was hoping there was a faster method to achieve it.</p>
Mean std of merged dataset

sakh251: <p>Hello,</p>
<p>I have a model trained on a merged datasets. I know mean and std of each dataset. Now I want to know how to should normalize new data for this model.<br>
For I think we can calculate the average of means?<br>
But for std since it could depend on covariance of them I don’t have any idea. Datasets two different dataset based on some labeled images.</p>
<p>Thank you.</p>

Andrei_Cristea: <p>Could you normalize the two datasets first, and then merged the normalized datasets?</p>

tom: <p>Yes, you can calculate this.</p>
<p>If the relative sizes of the datasets are <code>w_1</code> and  <code>w_2</code> (i.e. <code>w_1 + w_2 == 1</code>) and the means <code>m_1</code>, <code>m_2</code>, and stds <code>s_1</code>, <code>s_2</code> (with a tiny bit of inaccurary if they are unbiased std, should not matter if you just merge two datasets, though), you have</p>
<p>mean: <code>m = w_1 m_2 + w_2 m_2</code></p>
<p>std: <code>s = ( (w_1 (s_1**2 + m_1**2) + w_2 (s_2**2 + m_2**2) - m**2)**0.5</code></p>
<p>(This computes the uncentered second moments from the stds, takes the weighted average to get that of the entire set, and then computes the std  of the entire set from that.)</p>
<p>Best regards</p>
<p>Thomas</p>

sakh251: <p><a class="mention" href="/u/tom">@tom</a> thank you very much Thomas<br>
I am a bit confused about how did you compute merged std. what do you mean by s_12 or m_22.<br>
Are these joint std and mean of set 1 and 2 ? Or set 2 with itself?</p>
<p>Best Regards</p>

tom: <p>I had the formatting wrong. Fixed it now, hopefully its more clear now.</p>
<p>Best regards</p>
<p>Thomas</p>
TorchData image loading

austinmw: <p>Hi, I’m learning to use TorchData and have run into some questions regarding image loading. I put together a few-line example like this:</p>
<pre><code class="lang-python">from torchvision.prototype.datasets.utils import DatasetConfig
from torchvision.prototype import datasets as new_datasets
from torch.utils.data import DataLoader

ds = new_datasets._api.find('coco')
config = DatasetConfig(split='val', year='2017', annotations='instances')
ds = ds.load('./root', config=config)

dataloader = DataLoader(dataset=datapipe, batch_size=1, shuffle=True)

batch = next(iter(dataloader))
print(batch.keys())
</code></pre>
<p>A couple things that I noticed and have questions about though, are:</p>
<ol>
<li>
<p>The image tensor is output as 1-D. How can I either output as a rectangle, or determine the original image dimensions for resizing?</p>
</li>
<li>
<p>How best can I apply transformations such as resizing so that I can batch? I saw <code>ImageTransformer</code> mentioned in a TorchData youtube video, but this doesn’t seem to exist in mainline.</p>
</li>
</ol>
<p>Thanks for any advice!</p>
How to change the bias of the classifier of Mobilenetv2

Rainbow995: <p>How can I set the bias of the classifier of Mobilenetv2 to 55? Can I use this method? sth like this:</p>
<p>model = torchvision.models.mobilenet_v2(pretrained=pretrained)<br>
model.classifier[1] = nn.Linear(model.last_channel, 1)<br>
model.classifier[1].bias = nn.Parameter(55)</p>

Shashank_Agnihotri: <p>If you want to set all the zeros(default values of the bias, in case of a normal torchvision model, it is a tensor of len 1000 for 1000 classes of imagenet) to 55, then the best way to do it is:<br>
model.classifier[1].bias=torch.nn.Parameter(torch.FloatTensor([55]*len(model.classifier[1].bias)))</p>
<p>In your case, you are changing the number of output channels to 1, so you can avoid the “len(model.classifier[1].bias)” part, however including that will just make it more dynamic.</p>
<p>Important thing to note here is the torch.FloatTensor([55]), because torch.nn.Parameter() requires a tensor, and since you want the gradient to be updated on the bias, as it learns, it should be a float tensor.</p>

Rainbow995: <p>Thank you for your answer. I applied what you said but it is really strange, my output is still negative. do you have any suggestion? I think the bias didn’t have any effect on the output.</p>

Shashank_Agnihotri: <p>You are welcome.<br>
For that, I will need much more details, like the task, the network etc.<br>
It might be easier for you to debug it by looking at the output of each layer and where it is deviating from the expected.</p>
Embedding method for continuous variables

Federico_Ottomano: <p>I’ve understood that <code>nn.Embedding</code> can be used to associate an embedding to some categorical label corresponding to high-dimensional inputs such as images, for example during the training of a conditional generative model. What if my label is <em>continuous</em> instead?<br>
Is there a way inside <code>PyTorch</code> to create a high-dimensional embedding of a continuous number, for example given that I want to train a generative model associating to each image representing a person’s face, his/her age.</p>
Compatibility Issue

Hannibal046: <p>Hi, I build pytorch from source in a docker container where the CUDA is 11.1. But I can not use it in A100,  I got the following error:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:145: UserWarning: 
A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_75.
If you want to use the A100-SXM4-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd466a6cf772a448abb85e12e95729e5d1e64936.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd466a6cf772a448abb85e12e95729e5d1e64936" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd466a6cf772a448abb85e12e95729e5d1e64936_2_406x500.png" alt="image" data-base62-sha1="vzunVEc23cAcGzV6eTbPfTVZgge" width="406" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd466a6cf772a448abb85e12e95729e5d1e64936_2_406x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd466a6cf772a448abb85e12e95729e5d1e64936_2_609x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd466a6cf772a448abb85e12e95729e5d1e64936_2_812x1000.png 2x" data-dominant-color="0D310D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1762×2168 649 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Hannibal046: <p>OK, I solved this by passing exact CUDA_ARCH when building pytorch from source <code>TORCH_CUDA_ARCH_LIST="8.0" python setup.py install </code></p>
How to flatten vertically

Keeva: <p>I have this tensor:</p>
<pre><code class="lang-auto">torch.tensor([[1, 1, 1],
           [1, 1, 1],
           [1, 1, 1],
           [2, 2, 2],
           [2, 2, 2],
           [2, 2, 2],
           [3, 3, 3],
           [3, 3, 3],
           [3, 3, 3]])
</code></pre>
<p>I want to flatten it vertically like this:<br>
<code>a = torch.tensor([[1, 1, 1, 2, 2, 2, 3, 3, 3,1, 1, 1, 2, 2, 2, 3, 3, 3,1, 1, 1, 2, 2, 2, 3, 3, 3]])</code></p>
<p>I have also another problem, when I reshape it by a.reshape(3,-1), the output:</p>
<pre><code class="lang-auto">tensor([[1, 1, 1, 2, 2, 2, 3, 3, 3],
        [1, 1, 1, 2, 2, 2, 3, 3, 3],
        [1, 1, 1, 2, 2, 2, 3, 3, 3]])
</code></pre>
<p>But I thought it would be:</p>
<pre><code class="lang-auto">tensor([[1, 2, 3,1, 2, 3, 1, 2, 3],
        [1, 2, 3,1, 2, 3, 1, 2, 3],
        [1, 2, 3, 1, 2, 3, 1, 2, 3]])
</code></pre>
<p>Why, and how I can get the last output?</p>
<p>Thank you for your help.</p>

Matias_Vasquez: <p>When you <a href="https://pytorch.org/docs/stable/generated/torch.reshape.html" rel="noopener nofollow ugc">reshape</a> a tensor, you are basically changing how the view of the tensor is done. The values actually remain on the same positions and are still contiguous. You might want to look at how the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.view.html#torch.Tensor.view" rel="noopener nofollow ugc">view</a> works.</p>
<p>This should work for both of the views you requested.</p>
<p>Hope it helps <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-python">a = torch.flatten(t.T).unsqueeze(0)
print(a)

b = torch.stack(torch.split(t, 3), dim=2).reshape(3, -1)
print(b)
</code></pre>
Normalize to maximum to [0,1] range

keepmoving4444: <p>Is there any way that I can normalize the tensor to the maximum (in range of 0,1) in my network layers? Because the label images I have are normalized to the max pixel values of each image. I would like to have an output in the same scales.<br>
Can I write a layer use torch.max(), like a/torch.max(a,0)?</p>
Why I got 3 other processes on GPU 0 when use DDP to train model?

dachun: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bc77b3f17e392187dbabbe9bbe5a7acc1f798c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c7bc77b3f17e392187dbabbe9bbe5a7acc1f798c" title="5a40b07f981587ffaa0b7728e265c01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bc77b3f17e392187dbabbe9bbe5a7acc1f798c_2_690x219.jpeg" alt="5a40b07f981587ffaa0b7728e265c01" data-base62-sha1="suWO0rCV0GKI9lDr3y4PVKOXuEs" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bc77b3f17e392187dbabbe9bbe5a7acc1f798c_2_690x219.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bc77b3f17e392187dbabbe9bbe5a7acc1f798c.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bc77b3f17e392187dbabbe9bbe5a7acc1f798c.jpeg 2x" data-dominant-color="363732"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">5a40b07f981587ffaa0b7728e265c01</span><span class="informations">882×280 87.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I guess you are initializing a additional CUDA contexts on the default device (most likely from the other processes). Check if you have any CUDA operations working on the default device instead of the one used in the DDP launch.</p>
Converting pytorch model to numpy/ python

Anime_Drive: <p>Hello,</p>
<p>We are trying to build a custom hardware for binary-matrix-mulitplicaton, we stubled up on xnor-net which uses binary values for the neural network… we were trying to cutomize the implementation used here<br>
“<a href="https://github.com/jiecaoyu/XNOR-Net-PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jiecaoyu/XNOR-Net-PyTorch: PyTorch Implementation of XNOR-Net</a>” , for now we just want to change the matrix multiplication used in nn.Linear to our custom module. But the problem is that the module only works with numpy arrays.<br>
We train the model without using the module , but we want to use our module for prediction with the trained weights. Is there any way to get the trained weights to numpy array and use it to create the model.predict only using numpy.?</p>

ptrblck: <aside class="quote no-group" data-username="Anime_Drive" data-post="1" data-topic="147498">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anime_drive/48/47892_2.png" class="avatar"> Anime_Drive:</div>
<blockquote>
<p>Is there any way to get the trained weights to numpy array</p>
</blockquote>
</aside>
<p>Yes, as described in your <a href="https://discuss.pytorch.org/t/create-a-custom-predict-function/147285/2">double post</a>.</p>
<aside class="quote no-group" data-username="Anime_Drive" data-post="1" data-topic="147498">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anime_drive/48/47892_2.png" class="avatar"> Anime_Drive:</div>
<blockquote>
<p>and use it to create the model.predict only using numpy.?</p>
</blockquote>
</aside>
<p>Assuming you want to use the numpy arrays only for inference (i.e. no <code>backward</code> calls would be allowed or any gradient calculations) you could write a custom <code>nn.Module</code> and use numpy in the <code>forward</code> method directly.<br>
On the other hand, if you want to calculate gradients, you would need to write a custom <code>autograd.Function</code> and implement the <code>backward</code> method manually.</p>
Torchrun launched jobs hang on multiple machines

Rhett_Ying: <p>Hi,</p>
<p>I am  trying to launch RPC-based jobs on multiple machines via <code>torchrun</code>, but it gets stuck: <code>PRINT</code> is not printed.</p>
<p>machineA: <code>MASTER_ADDR='xxxxx' MASTER_PORT=12348  torchrun --nnodes=2 --nproc_per_node=2 --node_rank=0 demo.py</code><br>
machineB: <code> MASTER_ADDR='xxxxx' MASTER_PORT=12348 torchrun --nnodes=2 --nproc_per_node=2 --node_rank=1 demo.py</code></p>
<p>Run on single machine withe same <code>demo.py</code> works as expected.<br>
<code>MASTER_ADDR='xxxxx' MASTER_PORT=12348 torchrun --nnodes=1 -nproc_per_node=2</code></p>
<p><code>demo.py</code> is almost empty which just contains <code>print(os.environ['RANK'])</code> only.</p>

zizhao.mo: <p>Hi, I am facing the same problem. Can you deal with this problem now?</p>

Rhett_Ying: <p>run on machine A: <code>torchrun --nnodes=2 --nproc_per_node=4 --node_rank=0 --rdzv_id=0  --rdzv_endpoint=IP:12348  demo.py</code></p>
<p>run on machine B: <code>torchrun --nnodes=2 --nproc_per_node=4 --node_rank=1 --rdzv_id=0  --rdzv_endpoint=IP:12348  demo.py</code></p>

zizhao.mo: <p>thank you. Do I need to change the training script from the example of torch.distributed.launch, besides the local_rank loading? I cannot find any example of torchrun in Internet.</p>

Rhett_Ying: <p>pls refer to <a href="https://pytorch.org/docs/master/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 1.12 documentation</a>.</p>
Normalize dicom pixel values

MightyStud: <p>I’m working on a dicom files and after investigating the pixel values range of the images it turned out to be around -2000 to 6000  range, which I assume corresponds to Hounsfield unit range HU, I will be using this code to normalize it,</p>
<pre><code class="lang-auto">image_transforms = transforms.Compose([transforms.ToTensor(), transforms.Resize((128, 128), interpolation=transforms.InterpolationMode.NEAREST), transforms.Normalize()])
</code></pre>
<p>however I’m not sure what normalize parameters(mean and std) I should use for this case.</p>

ptrblck: <p>In common use cases <code>ToTensor</code> will output an already normalized tensor with values in the range <code>[0, 1]</code> such that the <code>mean</code> and <code>std</code> values of the following <code>Normalize</code> transformation would respect this range (and are thus set to values in <code>[0, 1]</code>).<br>
However, given that you are working with DICOM data, I don’t know which format you are passing to <code>ToTensor</code> and if this transformation is already changing the range.<br>
In any case, check the range of the tensor created by <code>ToTensor</code> and adapt the stats of <code>Normalize</code> to this range.</p>

MightyStud: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your time,<br>
As you mentioned <code>ToTensor</code> does normalize by default if the the input is one of the stated formats in <a href="https://pytorch.org/vision/main/generated/torchvision.transforms.ToTensor.html" rel="noopener nofollow ugc">pytorch documentation of ToTensor</a>, however in my case I’m passing a numpy array of float64 data type, so it is not doing any kind of normalization.<br>
Right now I believe the best way to figure it out is to find the <code>mean</code> and <code>std</code> of a batch of the data and normalize upon it. I’m just not sure if this is the most efficient way.</p>
Indexed assignemet has a significant overhead

andrea_bragagnolo: <p>Hi all, I’ve recently noticed that an indexed assignment (i.e. <code>r[idx] = something</code>) leads to quite an overhead. Does anyone have some insight into why does this happen and any mitigation of this problem?</p>
<p>Here a snippet to reproduce:</p>
<pre><code class="lang-auto">torch.set_num_threads(1)

r = torch.zeros(1000, 512)
m1 = torch.randn(1, 1000)
m2 = torch.rand(1, 512)

idx = torch.arange(start=0, end=1000, step=1).long()

times = []
for i in range(1000):
    start = time.perf_counter_ns()
    r = m1.t().mm(m2)
    end = (time.perf_counter_ns() - start) / 1e+6
    times.append(end)
print("nothing")
print(f"{np.mean(times)}ms pm {np.std(times)}ms\n")

times = []
for i in range(1000):
    start = time.perf_counter_ns()
    r = m1[:, idx].t().mm(m2)
    end = (time.perf_counter_ns() - start) / 1e+6
    times.append(end)
print("m1 indexing")
print(f"{np.mean(times)}ms pm {np.std(times)}ms\n")

times = []
for i in range(1000):
    start = time.perf_counter_ns()
    r[idx] = m1[:, idx].t().mm(m2)
    end = (time.perf_counter_ns() - start) / 1e+6
    times.append(end)
print("r and m1 indexing")
print(f"{np.mean(times)}ms pm {np.std(times)}ms\n")
</code></pre>
<p>which results in</p>
<blockquote>
<p>nothing<br>
0.8633705ms pm 0.1330834445742595ms</p>
<p>m1 indexing<br>
0.9316685ms pm 0.26098909218155075ms</p>
<p>r and m1 indexing<br>
1.4100757999999998ms pm 0.4815578560197726ms</p>
</blockquote>
<p>Thank you all.</p>

ptrblck: <p>I’m not sure I understand the question completely, but indexing tensors would add some overhead which is expected.</p>

andrea_bragagnolo: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> what I’d like to do actually is to only compute the <code>mm</code> only for a subset of the tensor and have the resulting tensor be nonzero only for the corresponding subset.</p>
<pre><code class="lang-auto">r = torch.zeros(10, 5)
m1 = torch.randn(1, 10)
m2 = torch.rand(1, 5)

idx = torch.tensor([0, 1, 3, 4, 6])

r[idx] = m1[:, idx].t().mm(m2)
</code></pre>
<p>Here <code>r</code> will be zero at indexes <code>2, 5, 7, 8, 9</code> and nonzero at the remaining.</p>
<p>What I’ve noticed is that the indexed <code>mm</code> (<code>m1[:, idx].t().mm(m2)</code>) has no overhead (is actually faster than the full, non subset <code>mm</code>), but the indexed assignment (<code>r[idx] = ...</code>) has significant overhead (more or less doubles the total execution time).</p>
<p>Is there any way to accomplish this operation without such increase in time?</p>
Manually extracted feature maps to input

neurospacer: <p>Hi, I am new to MONAI<br>
I have 5 manually extracted brain features volumetric maps (thickness, sulc, curv, etc)  and ROI target label mask with size (241, 336, 283) for a number of subjects. How can I stack all feature in inputs channels? and what classification model best fits for label detection?</p>

ptrblck: <aside class="quote no-group" data-username="neurospacer" data-post="1" data-topic="147448">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neurospacer/48/47962_2.png" class="avatar"> neurospacer:</div>
<blockquote>
<p>How can I stack all feature in inputs channels?</p>
</blockquote>
</aside>
<p>Assuming each of your features has a shape as <code>[241, 336, 283]</code> which is given as the <code>[dept, height, width]</code>, then using <code>torch.stack</code> should work:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.randn(241, 336, 283)
b = torch.randn(241, 336, 283)
c = torch.randn(241, 336, 283)

features = torch.stack((a, b, c), dim=0)
print(features.shape)
# torch.Size([3, 241, 336, 283])

# add batch dimension
features = features.unsqueeze(0)
print(features.shape)
# torch.Size([1, 3, 241, 336, 283])
</code></pre>
<aside class="quote no-group" data-username="neurospacer" data-post="1" data-topic="147448">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neurospacer/48/47962_2.png" class="avatar"> neurospacer:</div>
<blockquote>
<p>and what classification model best fits for label detection?</p>
</blockquote>
</aside>
<p>I don’t know, but maybe <a class="mention" href="/u/monai">@MONAI</a> would know good resources.</p>
How I can avoid the for loops in a 3D tensor problem?

J2M2: <p>Hi guys!</p>
<p>I have a tensor called <em>classification</em> with size (B,1,2048) with B the batch size.</p>
<p>From it I extract the indices of the top k1 and k2 elements for each batch, so I have another two tensors of size (B,1,k1) and (B,1,k2).</p>
<p>In this case k1 and k2 are both 1024 but in general they can be also different.</p>
<p>Then I have another tensor called <em>grasp_labels</em> of shape (B,N,2) with N pairs of indices. I have to build a new tensor <em>combined_labels</em> with shape (B,k1,k2) that contains 1 in position (b,i,j) if (b,i,j) is in <em>grasp_labels</em> and i is in top_k1 and j is in top_k2 (in the right batch), 0 otherwise.</p>
<p>The problem I just can concibe this using for loops, what is not a good idea because I lose the gradient flow. I would need to find a <strong>pytorch</strong> vecrorized way to do this.</p>
<p>Pseudocode with for loops:</p>
<pre><code class="lang-auto">for b in range(0,B):
     for i in range(0,k1):
        for j in range(0,k2):
            a = topk1[b,1,i]
            c = topk2[b,1,j]
            if [a,c] in grasps_labels[b,:,:]:
                combined_labels[b,i,j] = 1
</code></pre>
<p>Any idea?? Thinking in more than two dimensions sometimes blow my brain.</p>
<p>Thanks a lot in advance!</p>

eqy: <p>I’m a bit confused as to why gradients need to flow as it looks like this is computing labels here. Here’s a sketch of an alternative implementation that doesn’t use Python for loops (although iteration is still heavily used). The main change is that <code>[a,c] in grasps_labels[b,:,:]</code> (which doesn’t seem to be valid PyTorch) is instead done via a lookup with a precomputed <code>grasps_labels_matrix</code>instead.</p>
<p>Note that this is untested as I don’t have any reference input/output to compare with:</p>
<pre><code class="lang-auto">import torch

B = 32
k1 = 1024
k2 = 1024
N = 2048

classification = torch.randn(B, 1, 2048)
topk1 = torch.topk(classification, k1).indices
topk2 = torch.topk(classification, k2).indices

# create a mapping from [b,a,c] to 0,1
grasps_labels = torch.randint(0, 1024, (B, N, 2)).reshape(B*N,2)
grasps_indices = torch.unbind(torch.cat((torch.arange(B).repeat_interleave(N).unsqueeze(1), grasps_labels), axis=1), axis=1)
grasps_labels_matrix = torch.zeros(B, N, N)
grasps_labels_matrix.index_put_(grasps_indices, torch.tensor(1.0))

# collect indices to check
indices_k1_repeat = torch.repeat_interleave(topk1, k2, axis=2).reshape(B*k1*k2)
indices_k2_repeat = torch.repeat_interleave(topk2, k1, axis=2).reshape(B*k1*k2)
indices_k1k2 = torch.stack((indices_k1_repeat, indices_k2_repeat), dim=1)
indices_batch = torch.arange(B).repeat_interleave(k1*k2).unsqueeze(1)
indices_combined = torch.unbind(torch.cat((indices_batch, indices_k1k2), axis=1), axis=1)
print(len(indices_combined[0]), B*k1*k2) # B * k1 *k2
combined_labels = grasps_labels_matrix[indices_combined].reshape(B, k1, k2)
</code></pre>
'auto' grad with 'custom' grad function

Bokyeong1001: <p>hello<br>
I have a torch model which first pass the torch Conv2d module and then pass my custom transformer encoder block(made into torch module).<br>
And I made a custom grad function for transformer encoder block.<br>
This transformer encoder block works well both forward and backward.<br>
What I want is that, for transformer encoder, it uses my custom grad function and for conv net, it uses auto grad.<br>
I tried train my model but it seems conv params are not updating.<br>
Is there any way to use both autograd and custom grad?</p>

ptrblck: <p>Could you explain your custom autograd approach a bit more?<br>
Did you write a custom <code>autograd.Function</code> or changed anything else?</p>

Bokyeong1001: <p>Yes, I used autograd.Function</p>

ptrblck: <p>In that case <code>Autograd</code> will use your custom <code>Function</code> and I’m not sure how it would break the conv net.<br>
Could you add more information about your use case and post a minimal, executable code snippet to reproduce the issue, please?</p>
How to use multi-cpu or muti-cpu core to train

Ardeal: <p>Hi,</p>
<p>as we know, we could use CUDA_VISIBLE_DEVICES env and torch.distributed to run training on multi-GPU.<br>
is there any way to use multi-CPU or multi-CPU core to run parallel training?</p>

ptrblck: <p>Multiple CPU cores can be used in different libraries such as MKL etc. and can be set via the env variables:</p>
<pre><code class="lang-auto">OMP_NUM_THREADS=nb
MKL_NUM_THREADS=nb
</code></pre>
<p>or via:</p>
<pre><code class="lang-python">torch.set_num_interop_threads() # Inter-op parallelism
torch.set_num_threads() # Intra-op parallelism
</code></pre>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>When we train model with multi-GPU, we usually use command:<br>
CUDA_VISIBLE_DEVICES=0,1,2,3 WORLD_SIZE=4 python -m torch.distributed.launch --nproc_per_node=4 train.py --bs 16.</p>
<p>if we use the upper command and corresponding in code, we could run parallel training on multi-GPU. My question is: is there any similar method to run training on CPU like GPU?</p>

ptrblck: <p>Based on <a href="https://pytorch.org/docs/stable/distributed.html#which-backend-to-use">this doc</a> it seems <code>gloo</code> might be the right choice, but I’m not deeply familiar with it:</p>
<blockquote>
<p>Rule of thumb</p>
<ul>
<li>Use the NCCL backend for distributed <strong>GPU</strong> training</li>
<li>Use the Gloo backend for distributed <strong>CPU</strong> training.</li>
</ul>
</blockquote>

fduwjj: <p>Thanks for your question, I guess <a class="mention" href="/u/jiayisuse">@jiayisuse</a> might also want to comment on this.</p>

jiayisuse: <p>We will need to use Gloo for CPU training. Need to call torch.distributed.init_process_group(backend=‘gloo’) in your code</p>

Ardeal: <p><a class="mention" href="/u/jiayisuse">@jiayisuse</a> ,<br>
Thank you!<br>
Thank <a class="mention" href="/u/fduwjj">@fduwjj</a> , <a class="mention" href="/u/ptrblck">@ptrblck</a> !<br>
Is there any further doc or sampler code about how to use Gloo for CPU training?</p>

fduwjj: <p>I think the API looks similar after you have created the process group. Here we have all supported collectives for gloo (and all others): <a href="https://pytorch.org/docs/stable/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.11.0 documentation</a>.</p>
Eval the saved qat model get a different results with the results in training processing

shupinghu: <p>Hey, I am new to Quantization Aware Training using PyTorch. I have been trting to train a qat model for my “rknn devices” so I use ‘qnnpack’ during the training and eval processing. During training processing, the eval result is map=0.4, and during the individual eval processing, the eval result is 0.3, but I used the same eval dataset and eval codes. When I trying to do the expriments without Quantization Aware Training (Float model), th eval results is equal in the training processing and the individual eval processing. I wonder what is wrong here?</p>
<p>The traing codes are as follows:<br>
net = mymodel()<br>
checkpoint = torch.load(float_model_path)<br>
net.load_state_dict(net, checkpoint)<br>
net.qconfig = torch.quantization.get_default_qat_qconfig(‘qnnpack’)<br>
torch.quantization.prepare_qat(self, inplace=True)<br>
net.train()<br>
‘’’<br>
traing<br>
‘’’<br>
torch.quantization.convert(net.eval().cpu(), inplace=True)<br>
torch.save(net.cpu().state_dict(), qat_model_path)<br>
evaluate(net.eval().cpu(), … )</p>
<p>The individual eval codes are as follows:<br>
net = mymodel().eval()<br>
net.qconfig = torch.quantization.get_default_qat_qconfig(‘qnnpack’)<br>
torch.quantization.prepare_qat(self, inplace=True)<br>
torch.quantization.convert(net, inplace=True)<br>
checkpoint = torch.load(qat_model_path, map_location=‘cpu’)<br>
net.load_state_dict(checkpoint)<br>
evaluate(net, … )</p>
<p>Both of these codes are run in my server with CentOS system and V100 GPUs. pytorch version: torch 1.7.0</p>

suraj.pt: <p>Hi <a class="mention" href="/u/shupinghu">@shupinghu</a> , welcome!</p>
<p>Why are you using <code>torch.quantization.convert(net.eval().cpu(), inplace=True)</code> instead of simply <code>torch.quantization.convert(net, inplace=True)</code>? <code>net.eval().cpu()</code> is a different object as <code>net</code>, so the inplace operation on <code>net.eval().cpu()</code> won’t actually change <code>net</code></p>

terry_chen: <p>what’s your evaluation quantization engine?<br>
Try to print torch.backends.quantized.engine on eval and train mode to see if they are matched.</p>
Why does torch.jit.load(quantized_model) spend much longer time than torch.jit.load(fp32_model)?

mkcho: <p>Just wondering why INT8 quantized model requires much longer time than fp32 model when loading via torch.jit.load(). I think it’s very weird because the saved file size (using torch.jit.save()) of INT8 quantized model is 4x smaller than that of fp32 model.</p>
<p>Is there anyone who has the same issue? or is there any solution to reduce the loading time of quantized models like fp32 models?</p>

terry_chen: <p>quantized model may take longer time due to packing linear conv weighs</p>
Export INT8 Quantized model to ONNX/Openvino

AyanKumarBhunia: <p>Can you suggest a way regarding how I can export an INT8 Quantized PyTorch model to ONNX/Openvino?</p>
<p>I found <a href="https://github.com/openvinotoolkit/nncf" class="inline-onebox" rel="noopener nofollow ugc">GitHub - openvinotoolkit/nncf: Neural Network Compression Framework for enhanced OpenVINO™ inference</a></p>
<p>Do you have a better suggestion?</p>
Stage-wise training

Mohammad_Farazi: <p>Hi all,</p>
<p>I have a deep neural network which consists of a U-Net and two CNN layers afterwards. The U-net is just to learn point-wise features for a graph. Then feed the learned features to new CNN layers, but this seems not working as there is a huge bottleneck. I wonder how I can learn the U-net first then after converging learn the second part for fine-tuning. This is called stage-wise learning but I have not found a good resource for Pytorch.</p>
Image file reading：TypeError: batch must contain tensors, numbers, dicts or lists; found <class 'PIL.Image.Image'>

dingwang: <p>I have image dataset. when I use my dataloader, I get the following error:</p>
<blockquote>
<p>File “resnet_cub.py”, line 138, in test<br>
for i,(input_x, target) in enumerate(val_loader):<br>
File “/root/Util/miniconda/envs/py3.5/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 179, in <strong>next</strong><br>
batch = self.collate_fn([self.dataset[i] for i in indices])<br>
File “/root/Util/miniconda/envs/py3.5/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 109, in default_collate<br>
return [default_collate(samples) for samples in transposed]<br>
File “/root/Util/miniconda/envs/py3.5/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 109, in <br>
return [default_collate(samples) for samples in transposed]<br>
File “/root/Util/miniconda/envs/py3.5/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 112, in default_collate<br>
.format(type(batch[0]))))<br>
TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class ‘PIL.Image.Image’&gt;</p>
</blockquote>
<p>My dataloader code is this:</p>
<blockquote>
<p>class MyDataset(Dataset):<br>
def <strong>init</strong>(self, root, set, transform=None, target_transform = None):<br>
if set == ‘train’:<br>
fname = os.path.join(root, ‘train1.txt’)<br>
elif set ==‘test’:<br>
fname = os.path.join(root, ‘test1.txt’)<br>
fn = open(fname, ‘r’)<br>
imgs = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for line in fn.readlines():<br>
words = line.split()<br>
imgs.append((words[0],int(words[1])))<br>
self.root = root<br>
self.imgs = imgs<br>
self.transform = transform<br>
self.target_transform = transform</p>
</blockquote>
<pre><code>def __getitem__(self, index):
	fn, label = self.imgs[index]
	img = Image.open(os.path.join(self.root,'images',fn)).convert('RGB')
	if self.transform is not None:
		img = self.transform(img)
	return img, torch.Tensor(label)

def __len__(self):
	return len(self.imgs) 
</code></pre>
<blockquote>
<p>def test():</p>
</blockquote>
<pre><code>val_dataset = MyDataset(dataroot, 'train')

val_loader = DataLoader(dataset = val_dataset, batch_size = 8, shuffle = False)

pre_label = []
for i,(input_x, target) in enumerate(val_loader):
	input_var =  torch.autograd.Variable(input_x)
	target_var = torch.autograd.Variable(target)
	print('.....')
</code></pre>

BartolomeD: <p>The error states that the DataLoader receives a PIL image. This is because there are no transforms made (<code>transform=None</code>) on the image. The <code>__getitem__</code> method of MyDataset passes an unprocessed PIL image to the DataLoader, whereas it should receive a tensor.</p>
<p>You can add a transform that creates a tensor from the PIL image by adding <code>transform</code>:</p>
<pre><code>from torchvision import transforms

transform = transforms.Compose([
    # you can add other transformations in this list
    transforms.ToTensor()
])
</code></pre>
<p>as an argument to MyDataset.</p>

dingwang: <p>The issue  was solved in the way you show.Thank you! (*￣︶￣)</p>
How to do matmul with [batch, nodes, in] and [nodes, in, out], and get [batch, nodes, out]

MrAlbedo: <p>So I would like to perform a special matmul operation between</p>
<p><strong>A</strong> in shape [batch, num_nodes, in_channels],</p>
<p>and</p>
<p><strong>W</strong> in shape [num_nodes, in_channels, out_channels].</p>
<p>The shape of the result should be [batch, num_nodes, <strong>out_channels</strong>].</p>
<p>I tried my best and I can only get one solution, using a for loop (which of course is so slow…):</p>
<pre><code class="lang-auto">assert A.shape[1]==W.shape[0]

result = []

for i in range(A.shape[1]):
    result.append(A[:, i, :] @ W[i, :, :])

return torch.stack(result, dim=1)
</code></pre>
<p>Anyone can give me a hand? Thanks in advance!</p>

JuanFMontesinos: <p>torch.einsum(‘bni,nio-&gt;bno’, A, W)</p>

MrAlbedo: <p>Thanks for your timely reply! May I ask Can PyTorch’s Autograd handle this fancy operation well?</p>

JuanFMontesinos: <blockquote>
<p>Thanks for your timely reply! May I ask Can PyTorch’s Autograd handle this fancy operation well?</p>
</blockquote>
<p>yesd <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

MrAlbedo: <p>Great! Thank you sooooo much <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Make further prediction with argmax in a end2end model

stevelu: <p>Hi,</p>
<p>I want to first make a prediction with <code>r_index = x.argmax()</code>. The <code>argmax()</code> result <code>r_index</code> is the index of different rates. For example, there are 3 predefined rates: <code>rates = torch.Tensor([0.1, 0.5, 0.7])</code>.<br>
Then I need this index for further prediction.  <code>output = rates[r_index] * ...</code></p>
<p>As <code>argmax</code> is not differentiable. How to make it work in pytorch? I want to do it in an end-to-end model.</p>
<p>Thanks a lot if anyone can help!</p>

Andrei_Cristea: <p>Hi Steve,</p>
<p>Could you brute force this by training a model of the type:<br>
Linear(len(x), len_intermediary)<br>
Linear(len_intermediary, 1)</p>
<p>And train it to output the desired result (e.g. 0.1 if argmax(x) == 0, 0.5 if argmax(x) == 1, and so forth). Then simply freeze these parameters and load them into your end2end model in the appropriate place.</p>
<p>In other words, perhaps the transformation you want – from argmax to specific value – can be learned by a series of fully connected layers. This would allow you to piggyback on the differentiable tools already available.</p>
<p>You probably want to make sure that the range of x used in this separate training matches what would be coming in from your end2end model. Perhaps applying some normalization function like sigmoid on the x layer is required to cast everything to a comparable range.</p>
<p>Curious if that works,<br>
Andrei</p>