akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cuda Reserve Memory

Rami_Ismael: <p>I don’t know what this means.</p>
<p>If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>

ptrblck: <p>Take a look at the <a href="https://pytorch.org/docs/stable/notes/cuda.html#memory-management">Memory Management docs</a> which explain how the caching memory allocator works.<br>
The last section explains how the env variable can be used to prevent fragmentation in case your workload is suffering from it.</p>

david08170322: <p>RuntimeError: CUDA out of memory. Tried to allocate 3.23 GiB (GPU 0; 16.00 GiB total capacity; 4.87 GiB already allocated; 6.53 GiB free; 7.32 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
<p>can you help me?</p>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing this issue, please?</p>
Catching Exceptions in Libtorch

jonahclarsen: <p>Hi all,</p>
<p>When I run .forward&lt;&gt;() on an AnyModule in Libtorch, if it ends up using too much GPU memory it calls a c10::CUDAOutOfMemoryError exception followed by a c10::Error exception, as expected.</p>
<p>However, if I wrap that function in a try/catch statement, it doesn’t get caught. The program just crashes and shows the exceptions arising in their respective Libtorch files.</p>
<p>How can I catch this exception and prevent the program from crashing?</p>

jonahclarsen: <p>Anyone have any ideas?</p>
The official documentation clearly specifying the highest python version which PyTorch-1.11.0 requires

songyuc: <p>Hi, guys!<br>
I want to know whether there is an official documentation which clearly specifies the highest python version which PyTorch-1.11.0 can work with?</p>
<p>Your answer and guide will be appreciated!</p>

ptrblck: <p>The current <code>1.11.0</code> release seems to be already built for Python 3.10 as seen <a href="https://download.pytorch.org/whl/cu113/torch_stable.html">here</a>.</p>

songyuc: <p>Thanks very much! I hope there could be documentation about the requirement of the python version for us to look up, which would be more convenient and friendly.</p>
Sorting similar to lexsort

najam: <p>Hi</p>
<p>Lets say I have tensor x</p>
<p>x = torch.tensor([<br>
[1, 2, 3, 4, 3, 3, 4],<br>
[1, 6, 3, 5, 3, 5, 4]<br>
], dtype=torch.long)</p>
<p>and tensor y which are the associated scores</p>
<p>torch.tensor([<br>
[1, 1, 1, 1, 2, 1, 1],<br>
[1, 1, 1, 1, 2, 2, 1]<br>
], dtype=torch.long)</p>
<p>I want the output such that each row of x is sorted and the duplicates are sorted based upon scores. So the final output will be</p>
<p>x = torch.tensor([<br>
[1, 2, <strong>3</strong>, 3, 3, 4, 4],<br>
[1, 3, 3, 4, 5, 5, 6]<br>
], dtype=torch.long)</p>
<p>The bold 3 correspond to the score 2 in tensor y. Is there any way I can do it efficiently by just using the PyTorch functions?</p>
<p>Thank you</p>
Add argmax in a model that wraps up a pytorch pretrained model (deeplabv3)

buneloo: <p>Is there any possibility of adding an argmax operation at the end of the pretrained deeplabv3_mobilenet_v3_large pytorch model?<br>
Basically I want to know if this thing is possible without any training, just squeezing the original output of the model from [1,21,513,513] tensor output to [1,513,513] to decrease the grpc latency on a model-server solution.</p>
<p>I tried something like this in the code below. I managed to include the normalization in the model but the argmax is throwing and error I don’t know how to handle.</p>
<p>Code:</p>
<pre><code class="lang-auto">class ArgMax(torch.nn.Module):
    def __init__(self):
        super(ArgMax,self).__init__()

    def forward(self, input):
        x = torch.argmax(input, dim=1)
        x = x.to(torch.int8)
        return x    


class Normalize(torch.nn.Module):
    def __init__(self, mean, std):
        super(Normalize, self).__init__()
        self.mean = torch.tensor(mean).reshape(1,3,1,1)
        self.std = torch.tensor(std).reshape(1,3,1,1)

    def forward(self, input):
        x = input / 255.0
        x = x - self.mean
        x = x / self.std
        return x

new_model = torch.nn.Sequential(
    Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
    model,
    ArgMax()
)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">forward(__torch__.ArgMax self, Tensor x) -&gt; (Tensor):
Expected a value of type 'Tensor (inferred)' for argument 'x' but instead found type 'Dict[str, Tensor]'.
Inferred 'x' to be of type 'Tensor' because it was not annotated with an explicit type.
</code></pre>
<p>I don’t know if I am doing the things right or if this thing is possible, so any input in very welcomed!<br>
Thanks!</p>

ptrblck: <p>Based on the error message it seems that your custom <code>ArgMax</code> module expects a tensor input while the model returns a <code>dict</code>.<br>
You could change your custom module to accept the output <code>dict</code> and index the desired outputs.</p>

buneloo: <p>I found out after searching about the dict you mentioned. Now the custom ArgMax looks like this:</p>
<pre><code class="lang-auto">class ArgMax(torch.nn.Module):
    def __init__(self):
        super(ArgMax,self).__init__()

    def forward(self, x: Dict[str, torch.Tensor]) -&gt; torch.Tensor:
        t = x["out"]        
        t = torch.argmax(t, dim=1)
        t = t.to(torch.int8)
        return t
</code></pre>
<p>Thanks for your help!</p>
Vectorized version of function

Robinbux: <p>I need to calculate the lambda i score for two given tensors.</p>
<pre><code class="lang-auto">def lambda_ij_score(score_i, score_j, label_i, label_j):
    if label_i &gt; label_j:
        S_i_j = 1
    elif label_j &gt; label_i:
        S_i_j = -1
    else:
        S_i_j = 0
    return 0.5 * (1 - S_i_j) - (1 / (1 + np.exp(score_i - score_j)))

def compute_lambda_i(scores, labels):
    all_costs = []
    for i in range(len(scores)):
        current_costs = []
        for j in range(len(scores)):
            if i == j:
                continue
            cost = lambda_ij_score(scores[i], scores[j], labels[i], labels[j])
            current_costs.append(cost)
        all_costs.append(np.sum(current_costs))

    return torch.FloatTensor(all_costs).unsqueeze(1)
</code></pre>
<p>scores is a tensor of size [N, 1] and labels a tensor of size [N].<br>
While this function does work, it seems very inefficient.</p>
<p>I am very new to PyTorch and would like to find out if there is a neat vectorized version to calculate the desired result.</p>
<p>I am happy about any hints on how to do that!</p>
Dataset to connect to a database

kboroszko: <p>Hi all,</p>
<p>Are you aware of any pytorch extensions made available in the internet (preferably as a pypi package) that provide connection to a database?</p>
<p>I would like to be able to provide a database credentials and a query and create a dataset/dataloader from that.</p>
<p>Kajetan</p>

ptrblck: <p>It’s rather an example not an extension, but <a href="https://github.com/ptrblck/pytorch_misc/blob/master/pytorch_redis.py">here</a> you can find a simple <code>Dataset</code> implementation reading data from redis.</p>

jsetty: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, Do you see any bottlenecks in using databases like Redis or MongoDB for training? Any insights into how databases compare with a file system?</p>

ptrblck: <p>I don’t have any performance comparisons between databases and a filesystem loading, sorry. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
Do i need to quantize pretrained vgg16 model before converting to onnx for Tflite?

Mukesh1729: <p>I want to use my PyTorch pre-trained vgg16 model on a rasberry pi. I am converting it to onnx and then TFLITE. Inference on onnx and tflite work well on my computer but on the pi with the same pre-processing the output are scaled differently. Do i need to quantize the model before converting to onnx format to tflite. Any hints would be helpful. Thanks</p>

HDCharles: <p>my understanding is that onnx export to caffe2 is the only thing supported for pytorch quantized models and even then its spotty. you may be better off asking in a specific onnx or tflite group since it doesn’t sound like you’re using the pytorch quantization apis currently and your existing process sounds like it mostly works.</p>

Mukesh1729: <p>I am facing an issue with the preprocessing step as far as i have tried to find the issue. Upon conversion of my model to say onnx runtime my values are scaled differently at inference, if i use the same preprocessing steps without pytorch transforms. Whereas, when i use pytorch tranforms. Compose() to make the totensor and normaliztion of my input data then my outputs on the onnx works like a charm. Its is the same for my tflite model too. Do you know why this could be the case? Any hints. Thanks</p>

HDCharles: <p>No idea, sorry, pytorch quantization primarily refers to the APIs described here: <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.11.0 documentation</a></p>
<p>There’s some work to convert models quantized with the above APIs to onnx but that doesn’t sound like your issue, it sounds like you have a onnx model and converted it to TFLITE and something’s gone wrong somewhere during that process. Its possible someone with experience in those steps but it doesn’t sound like you need help with pytorch quantization so it may be faster to ask in an onnx or tflite group.</p>
CUDA is not avaliable after building from source

artimonkey: <p>Hello! I just built torch from source for cuda Compute capability = 3.5. But <em>torch.cuda.is_available()</em> returns <em>False</em>. I tried this command before with pytroch from pytorch_org and it returned True, but because of <em>compute capability = 3.5</em>, it can’t work properly. I have CUDA toolkit 11.6, that supports 3.5 compute capability, cudnn 8.3.2.44. And videodriver 463.15-data-center-tesla. Despite the failure of some tests compilation started succesfully. Summary configuration: <a href="https://drive.google.com/file/d/145gOu2i5Ipp7LG6T6ZZLiWN5afyQZ_K8/view?usp=sharing" rel="noopener nofollow ugc">Summary configuration</a><br>
Output: <a href="https://drive.google.com/file/d/1k-t2k1loN5quHkxMO1zUnNf_KwUEetDS/view?usp=sharing" rel="noopener nofollow ugc">CMakeOutput.log</a></p>
<p>Idk where to start searching for a reason of this problem.<br>
I appreciate any help! Thank you.</p>

artimonkey: <p>In CMakeError.log found the line:<br>
Determining if the prototype magma_get_sgeqrf_nb exists for MAGMA_V2 failed with the following output:<br>
Change Dir: G:/pytorch/build/CMakeFiles/CMakeTmp</p>
<p>Run Build Command(s):G:/condaEnv/Library/bin/ninja.exe cmTC_8b1dc &amp;&amp; [1/2] Building C object CMakeFiles\cmTC_8b1dc.dir\CheckPrototypeDefinition.c.obj</p>
<p>FAILED: CMakeFiles/cmTC_8b1dc.dir/CheckPrototypeDefinition.c.obj</p>
<p>E:\Apps\vsbuild\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64\cl.exe  /nologo   /DWIN32 /D_WINDOWS /w /bigobj -openmp:experimental -IE:/Apps/vsbuild/VC/Tools/MSVC/14.29.30133/include -DNDEBUG  /MDd /Z7 /Ob0 /Od /RTC1 /w /bigobj -std:c11 /showIncludes /FoCMakeFiles\cmTC_8b1dc.dir\CheckPrototypeDefinition.c.obj /FdCMakeFiles\cmTC_8b1dc.dir\ /FS -c G:\pytorch\build\CMakeFiles\CMakeTmp\CheckPrototypeDefinition.c<br>
G:\pytorch\build\CMakeFiles\CMakeTmp\CheckPrototypeDefinition.c(1):<strong>fatal error C1083: Cannot open include file: magma.h: No such file or directory,</strong><br>
<strong>ninja: build stopped: subcommand failed.</strong></p>
<p>but I created variable <em>MAGMA_HOME=G:\pytorch\requirements\magma</em><br>
So magma.h is here: <em>G:\pytorch\requirements\magma\include</em><br>
And why CheckPrototypeDefinition.c ends with that error?</p>
Find PCIe/network throughput per worker

abs5688: <p>Hello - Is there a way to find out PCIe/network throughput available to a worker for forward pass and backward pass? Please note that I am only interested in throughput per worker and not in a system wide throughput.</p>

rvarm1: <p>Hi, there are no utilities within PyTorch Distributed package that do this out of the box, but here are some suggestions:</p>
<ol>
<li>NVIDIA offers NCCL tests: <a href="https://github.com/NVIDIA/nccl-tests" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/nccl-tests: NCCL Tests</a> to evaluate performance/throughput of collectives which you could use to estimate the throughput available to a particular worker.</li>
<li>Another option is <code>lscpi</code> command: <a href="https://linux.die.net/man/8/lspci" class="inline-onebox" rel="noopener nofollow ugc">lspci(8): all PCI devices - Linux man page</a>
</li>
</ol>
Text Generation with LSTM-RNN

Ansh: <p>Hi guys can anybody help me in converting this code from Tensorflow to PyTorch:</p>
<pre><code class="lang-auto"># define the LSTM model
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')


# define the checkpoint
filepath="weights-improvement-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]

model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)
</code></pre>

AfonsoSalgadoSousa: <p>Hi. Something like this should work (it does not contemplate batching):</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch import optim

X = torch.randint(1, 100, (32, 20))
y = torch.randint(0, 2, (32, 1))

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = nn.Embedding(100, 64)
        self.lstm = nn.LSTM(64, 256)
        self.dropout = nn.Dropout(p=0.2)
        self.linear = nn.Linear(256, 1)
    
    def forward(self, x):
        x = x.transpose(0, 1)
        out = self.emb(x)
        out, _ = self.lstm(out)
        out = self.dropout(out)
        out = self.linear(out)
        return torch.sigmoid(out)
    
model = Model()
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters())
for epoch in range(20):
    out = model(X)
    out, _ = torch.max(out, dim=0)
    loss = criterion(out, y.float())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
How to train MLM model XLM Roberta large on google machine specs fastly with less memory

pchhapolika: <p>I am fine tuning masked language model from <code>XLM Roberta large</code> on google machine specs.<br>
I made couple of experiments and was strange to see few results.</p>
<pre><code class="lang-auto">"a2-highgpu-4g" ,accelerator_count=4, accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 4 Running ( 4 data*4 GPU=16 data points)
"a2-highgpu-4g" ,accelerator_count=4 , accelerator_type="NVIDIA_TESLA_A100"on 4,12,672 data batch size 8 failed
 "a2-highgpu-4g" ,accelerator_count=4, accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 16 failed
"a2-highgpu-4g" ,accelerator_count=4.,accelerator_type="NVIDIA_TESLA_A100" on 4,12,672 data batch size 32 failed
</code></pre>
<p>I was not able to train model with <code>batch size </code> more than 4 on # of GPU’s. It stopped in mid-way.</p>
<p>Here is the code I am using.</p>
<pre><code class="lang-auto">training_args = tr.TrainingArguments(
#     disable_tqdm=True,
    output_dir='/home/pc/Bert_multilingual_exp_TCM/results_mlm_exp2', 
    overwrite_output_dir=True,
    num_train_epochs=2,
    per_device_train_batch_size=4,
#     per_device_train_batch_size
#     per_gpu_train_batch_size
    prediction_loss_only=True
    ,save_strategy="no"
    ,run_name="MLM_Exp1"
    ,learning_rate=2e-5
    ,logging_dir='/home/pc/Bert_multilingual_exp_TCM/logs_mlm_exp1'        # directory for storing logs
    ,logging_steps=40000
    ,logging_strategy='no'
)

trainer = tr.Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_data
    
)
</code></pre>
<p><strong>My Questions</strong></p>
<p>How can I train with larger batch size on <code>a2-highgpu-4g</code> machine?</p>
<p>Which parameters can I include in <code>TrainingArguments</code> so that training is fast and occupies small memory?</p>
<p>Thanks in advance.</p>
<h3>
<a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<pre><code class="lang-auto">torch==1.11.0+cu113 

torchvision==0.12.0+cu113  

torchaudio==0.11.0+cu113 

transformers==4.17.0
</code></pre>
nn.Embedding and one-hot nn.Linear produce different results

AfonsoSalgadoSousa: <p>I have a seq2seq task and a model starting with an Embedding layer to process the tokenized input. I noticed that the performance using nn.Embedding is not great, but when using nn.Linear it is.<br>
Shouldn’t this:</p>
<pre><code class="lang-auto">self.embedding_layer = nn.Embedding(self.vocab_size, 512)
embedding_out = self.embedding_layer(input)
</code></pre>
<p>Be the same as this?:</p>
<pre><code class="lang-auto">self.embedding_layer = nn.Sequential(
   nn.Linear(vocab_size, 256),
   nn.Linear(256, 512)
)
embedding_out = self.embedding_layer(F.one_hot(input))
</code></pre>
<p>Yet, the latter produces much better results. Does anyone have any insights on why this is? I would much prefer to use the Embedding layer as it is more readable, but I do not seem to figure out why it does not perform well.</p>

vdw: <p>For a fair comparison, shouldn’t it look like this:</p>
<pre><code class="lang-auto">self.embedding_layer = nn.Sequential(
   nn.Linear(vocab_size, 512),
)
embedding_out = self.embedding_layer(F.one_hot(input))
</code></pre>
<p>or simply</p>
<pre><code class="lang-auto">self.embedding_layer = nn.Linear(vocab_size, 512)

embedding_out = self.embedding_layer(F.one_hot(input))
</code></pre>
<p><code>nn.Embedding</code> is essentially just Linear Layer, but your current “replacement” contains 2 Linear Layers</p>

AfonsoSalgadoSousa: <p>Yes, you are right. It is not a fair comparison. Yet, I do not understand how can one produce good results, while the other does not. There should not be a significant difference between them.</p>
Can’t find necessary C compiler for Pytorch in Windows 10

zydjohn: <p>Hello:<br>
I want to run some pre-trained pytorch model to make some predication.<br>
I have installed the following Pytorch package:<br>
D:\Pytorch\Cartoon-StyleGan2&gt;pip show torch<br>
Name: torch<br>
Version: 1.11.0+cu113</p>
<p>D:\Pytorch\Cartoon-StyleGan2&gt;<br>
But when I want to run some python script to make predication for one pytorch pre-trained model:<br>
D:\Pytorch\Cartoon-StyleGan2&gt;python MakeCartoonPicture.py<br>
C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\cpp_extension.py:322: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified<br>
warnings.warn(f’Error checking compiler version for {compiler}: {error}’)<br>
INFO: Could not find files for the given pattern(s).<br>
Traceback (most recent call last):<br>
File “D:\Pytorch\Cartoon-StyleGan2\MakeCartoonPicture.py”, line 7, in <br>
from model import Generator</p>
<p>I read the python file: cpp_extension.py<br>
And I also installed some MS C compiler, as the following:<br>
\x64&gt; cl.exe<br>
Microsoft (R) C/C++ Optimizing Compiler Version 19.31.31104 for x64<br>
Copyright (C) Microsoft Corporation.  All rights reserved.</p>
<p>usage: cl [ option… ] filename… [ /link linkoption… ]</p>
<p>\x86&gt;cl.exe<br>
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30141 for x86<br>
Copyright (C) Microsoft Corporation.  All rights reserved.</p>
<p>usage: cl [ option… ] filename… [ /link linkoption… ]</p>
<p>I think the installed MS C compiler versions:<br>
Either 19.31.31104(x64) or 19.29.30141(x86) is higher than min C compiler version:<br>
MINIMUM_MSVC_VERSION = (19, 0, 24215)</p>
<p>But I can’t install the minimum C compiler version: 19.0.24215.  It seems to be an old C compiler version, it is now difficult to install this.<br>
I also tried to install GCC according to the instruction from this python file:<br>
But I don’t know how to find the GCC version from Windows 10.</p>
<p>I also installed NVIDIA cuda 11.6 for Windows 10.<br>
Please advise on how I can fix this issue.<br>
By the way, I am using Python 3.9 on Windows 10.<br>
Thanks,</p>
Copy array to cuda tensor

Jakub_Mitura: <p>Hello I am trying to copy the boolean vector into pytorch tensor and generate appropriate accessor Hovewer I get some error that I do not understand</p>
<pre><code class="lang-auto">    bool* onlyBladderBoolFlat;
//... populating onlyBladderBoolFlat with data - it works well
    auto options = torch::TensorOptions().dtype(torch::kBool).device(torch::kCUDA, 1);
    torch::PackedTensorAccessor32&lt;bool, 1, torch::RestrictPtrTraits&gt; onlyBladderBoolFlatTensor = torch::from_blob(onlyBladderBoolFlat, { lengg }, options)
        .packed_accessor32&lt;bool, 1, torch::RestrictPtrTraits&gt;();

</code></pre>
<p>give error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a8a05416f0e556672dfc85d1acd68bff3f9c80.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8a8a05416f0e556672dfc85d1acd68bff3f9c80" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a8a05416f0e556672dfc85d1acd68bff3f9c80.png" alt="image" data-base62-sha1="o41uHtrIeZui1aWzH7TSNevWHOo" width="690" height="21" data-dominant-color="3E3C3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×23 1.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>configuration details</p>
<pre><code class="lang-auto">torch==1.10.2+cu113 torchvision==0.11.3+cu113 torchaudio===0.10.2+cu113
Python 3.6
</code></pre>
Getting each vision transformer's block prediction

binbbaz: <p>I’m curious to know if it’s possible to get prediction from each vision transformer block if I’m using a pretrained model</p>
<pre><code class="lang-auto">import timm
model = torch.hub.load('facebookresearch/deit:main', 'deit_small_patch16_224', pretrained=True)
model.eval()
#load data (imagenet)
prediction = model(data)
</code></pre>
<p>There are 12 blocks in the loaded model with 384 in and out features.<br>
Will it be possible to get the predictions from each of the 12 blocks?</p>
How to normalize all feature maps to a range of [0, 1]

spadel: <p>I have as an output of a convolutional network a tensor of shape [1, 20, 64, 64]. I want to normalize all feature maps to a range of [0, 1]. I found out, that I can get all the means with <code>means = torch.mean(features, (2, 3))</code>, but I don’t know how to proceed further. Any help?</p>

ptrblck: <p>You could subtract the min. value from the tensor and divide by the max value.<br>
This would make sure that all values are in the range <code>[0, 1]</code>.</p>

spadel: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, is there an efficient way to to that? I know how to do that with a for loop iterating over all 20 feature map but that seems very ugly.</p>

KaiHoo: <p>I do not understand why you need torch.mean to normalize all feature maps to a range of [0, 1].</p>
<p>If you want standardized normalization：</p>
<pre><code class="lang-auto">X = torch.randn(1,20,64,64)
X -= X.mean((2,3),keepdim=True)
X /= X.std((2,3),keepdim=True)
</code></pre>
<p>If you want them in [0,1]:</p>
<pre><code class="lang-auto">X = torch.randn(1,20,64,64)
X -= X.min()
X /= X.max()
</code></pre>
<p>Further, if you want to do this for each feature map independently:</p>
<pre><code class="lang-auto">X = torch.randn(1,20,64,64)
min_val = X.min(-1)[0].min(-1)[0]
max_val = X.max(-1)[0].max(-1)[0]
X = (X-min_val[:,:,None,None])/(max_val[:,:,None,None]-min_val[:,:,None,None])
</code></pre>

spadel: <p>Thank you <a class="mention" href="/u/kaihoo">@KaiHoo</a>, one question:</p>
<p>Why is the <code>- min_val[:, :, None, None]</code> within the denominator necessary? I just tested it and it works perfectly, but I don’t understand why. If we don’t want to normalize all feature maps independently we can just use <code>X /= X.max()</code>.</p>
<p>Edit: Ahh I understand, it’s because in your first example the max values are determined by the min-subtracted tensor.</p>

Jo-w: <p>What if normalize the feature for a whole dataset? It looks like these operations above only normalize features within the same batch.<br>
Thank you.</p>
Learning rate vs AUC

Pau: <p>Hi,</p>
<p>Instead of trying different learning rates, Leslie N. Smith proposed a method to find the most optimal lr. However, I don’t understand why we can’t just plot the lr vs AUC to find the best value. Is it because this method is computationally demanding?</p>
<p>Since my dataset is very small, in just 4 minutes I have already trained all the epochs. In my case, would it be better to decide which is the best lr based on the AUC it provides?</p>
<p>In this case, what values of lr should I try and how far apart? That is, for example from 0.1 to 0.000001 each time dividing by 10, or by 5.</p>
<p>Thanks for your help.</p>
GPU memory usage during backward pass for sparse parameters

Nora_Verplaetse: <p>Hi,</p>
<p>I’m building a sparse feedforward layer for a large input (each input node only connected to 1 or 2 nodes of next layer). I input the sparse weights into a torch.sparse_coo_tensor together with the coordinates of the sparse connections (edges) and use torch.sparse.mm in the forward function. The whole idea of the sparse connections is of course to keep memory usage low. However, during the loss.backward() the GPU memory usage goes up significantly to a level I would expect for calculating gradients of the same layer fully connected (±10Gb).</p>
<p><span class="hashtag">#Input</span> nodes: 147.931<br>
<span class="hashtag">#Output</span> nodes: 14.670<br>
<span class="hashtag">#Sparse</span> connections: 150997 =&gt; estimated gradient memory usage: 603.988 bytes<br>
<span class="hashtag">#Connections</span> if fully connected: 2.170.148.000 =&gt; estimated gradient memory usage: 8.680.591.000 bytes</p>
<pre><code class="lang-auto">class SparseConnectionsLayer(t.nn.Module):
    def __init__(self, bioNet,  name = "SCLNN"):
        super(SparseConnectionsLayer, self).__init__()      
        self.edges, self.size = bioNet.getDataForCOO()#edges contain coordinates for sparse connections, size equals t.Size([#nodes1, #nodes2])
        self.weight = t.normal(0, 0.1, [self.edges.size()[1]])
        self.weight = t.nn.Parameter(self.weight)
        self.bias = t.nn.Parameter(t.ones([self.size[1]])*0.1)

    def forward(self, x):
        adjWeight = t.sparse_coo_tensor(self.edges.to(self.weight.device), self.weight, self.size).to(self.weight.device)   
        adjWeight = adjWeight.t()
        o = t.sparse.mm(adjWeight, x.t()).t()
     
        return o + self.bias
</code></pre>
<p>Training loop:</p>
<pre><code class="lang-auto">while e &lt;= epochs  + warmStart:
			errTot = 0
			for sample in loader:
				optimizer.zero_grad()
				x, y = sample
				x=t.tensor(x, dtype=t.float, device=device)
				y=t.tensor(y, dtype=t.float, device=device)
				yp = self.model(x)
				loss = lossfn(t.squeeze(yp.float()), y)
				norm = sum(p.pow(2.0).sum() for p in self.model.parameters())
				loss = loss + weight_decay*norm
				loss.backward()
				optimizer.step()
				t.cuda.empty_cache()
				errTot += loss.data.item()

			scheduler.step(errTot)
</code></pre>
<p>config: torch 1.8.1, CUDA 11.4, Python 3.8.2</p>
<p>Any ideas on how to lower the memory usage?<br>
Thanks!</p>
Shifting Output for Transformer

JanoschMenke: <p>Hi Guys,</p>
<p>I was wondering how I would go about shifting the Output for a Transformer when I also want to use the <code>&lt;sos&gt;</code> token encodings produced by the Enocder for additional Classification. So I am training reconstruction as well as predictions at the same time.<br>
I know I have to shift the input to the right. But in this case I would also require the <code>&lt;sos&gt;</code> token of the source sequence.</p>
<p>This would be my idea but I am not if that is correct</p>
<p><code>src = &lt;sos&gt; I am fine &lt;eos&gt; </code><br>
<code>trg = &lt;sos&gt; I am fine</code></p>
How to convert cvxpy's values to torch tensor?

ADONAI_TZEVAOT: <p>I have variable values that the cvxpy has solved. How do I convert D to torch tensor in the following code:</p>
<pre><code class="lang-auto">import cvxpy as cp
import numpy as np
import torch

# Problem data.
a = 28
b = 2
c = 1

np.random.seed(1)
X = np.random.randn(a, b)
R = np.random.randn(b, c)

Z = X @ R
Z.shape

# Construct the problem.
D = cp.Variable((c,b))
objective = cp.Minimize(cp.sum_squares(Z*D - X))
constraints = [0 &lt;= D, D &lt;= 1]
prob = cp.Problem(objective, constraints)
prob.solve()
# The optimal objective value is returned by `prob.solve()`.
result = prob.solve()
# The optimal value for x is stored in `x.value`.
print(D.value)
# The optimal Lagrange multiplier for a constraint is stored in
# `constraint.dual_value`.
print(constraints[0].dual_value)

</code></pre>
Pytorch-forecasting: Applying prediction plotting on in-sample data

PaulNoah: <p>I am very new to Pytorch and Pytorch-forecasting and I was wondering how I can access the predictions made in-sample (on the training set) for plotting purposes? More concretely, I am following this <a href="https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html#Predict-on-new-data" rel="noopener nofollow ugc">tutorial</a> for time-series forecasting, and I can see the code that makes the predictions on out-sample (validation) data but can’t find the in sample predictions.</p>

ptrblck: <p>I guess you should be able to call <code>best_tft.predict</code> on the training data.</p>
How to stack tensors?

ADONAI_TZEVAOT: <p>i have  a tensor x = [1,2,3]. I have another tensor y = [4,5,6]. I want to stack x on top of y such that the new tensor z = [[1,2,3], [4,5,6]]. How to do this?</p>

ptrblck: <p><code>torch.stack</code> should work:</p>
<pre><code class="lang-python">x = torch.tensor([1,2,3])
y = torch.tensor([4,5,6])
z = torch.stack((x, y), dim=0)
print(z)
# tensor([[1, 2, 3],
#         [4, 5, 6]])
</code></pre>
What op in torch produce NonZero op in onnx

Kracozebr: <p>Hi. I’m trying to covert pytorch model to tensorrt through onnx. I can export from pytorch to onnx but when I try to export to tensorrt and got  the error. For now the latest version of tensorrt does not support NonZero op of tensorrt.<br>
So I wanted to try to change operations in pytorch that produce NonZero to smth else and try export.<br>
I found that two operations torch.nonzero and torch.where can produce NonZero op after export. I checked my code and had not found it.<br>
How can I check what operations in pytorch produce the NonZero op in onnx?</p>
Dictionary and Data-Parallel

KanZa: <p>I have implemented models in the dictionary but now for Data-Parallel it is not supporting dictionary, any alternate solution in DataParallel</p>
<pre><code class="lang-auto">if method =='lloss':
                    models = {'backbone': resnet18, 'module': loss_module}
torch.backends.cudnn.benchmark = True

#If I use this 
models = torch.nn.DataParallel(models, device_ids=[0,1,2])
#or this both methods are giving error 
models['backbone'] = torch.nn.DataParallel(models['backbone'], device_ids=[0,1,2])
models['backbone'].cuda()
</code></pre>
Having Trouble Installing PyTorch on Windows

TitoBernier: <p>Hello all. Yesterday evening, I was trying to install PyTorch from <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a> on my Windows 10 PC. While I was installing it, I encountered an error message ‘Access is Denied’. Does anyone have any idea why I encountered this issue? Please guide me to resolve it.</p>

ptrblck: <p>Maybe you are hitting <a href="https://stackoverflow.com/questions/31172719/pip-install-access-denied-on-windows">this pip/Windows issue</a>?<br>
Based on the description I don’t think it’s a PyTorch-specific issue, but rather a setup one.</p>

TitoBernier: <p>Yeah, the issue was might be a windows settings issue. I did some google research about this and found some articles regarding the described issue, I temporarily disabled UAC according to <a href="https://www.thewindowsclub.com/access-is-denied-software-installation-windows" rel="noopener nofollow ugc">this</a> article and then tried installing PyTorch again. That time, I did not face any issue.</p>
Keep pytorch tensor in GPU memory after program exit

marsupialtail: <p>I would like to keep a Pytorch tensor in GPU memory after the program is gone.</p>
<p>The use case is I have some process that uses Torch to move stuff to GPU. Then I want another process to read those memory regions into torch.Tensor. Ideally I would just pass a list of device ptrs and memory sizes/types between those two processes.</p>

ptrblck: <aside class="quote no-group" data-username="marsupialtail" data-post="1" data-topic="146528">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marsupialtail/48/47587_2.png" class="avatar"> marsupialtail:</div>
<blockquote>
<p>Ideally I would just pass a list of device ptrs and memory sizes/types between those two processes.</p>
</blockquote>
</aside>
<p>This sounds like a security concern and I don’t think it would be possible. Each process uses a different memory space and its address space is local. This is why “out of bounds” errors (segmentation faults) are raised if you are trying to access memory outside of the valid memory space.<br>
You could try to use <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#interprocess-communication">CUDA IPC</a> instead or any other IPC approach which would fit your use case.</p>
Affine parameter in batchnorm

kko: <p>From the documentation of batchnorm, “affine – a boolean value that when set to true, gives the layer learnable affine parameters. Default: True”. So, when I set affine=False, does gamma and beta in Ioffe’s paper is 1 and 0 or propagated standard deviation and mean?</p>

fmassa: <p>When <code>affine=False</code> the output of <code>BatchNorm</code> is equivalent to considering <code>gamma=1</code> and <code>beta=0</code> as constants.</p>

coincheung: <p>By assigning <code>affine=False</code>, are the parameters of <code>gamma</code> and <code>beta</code> still learnable or are they fixed to constant values of <code>gamma=1</code>, <code>beta=0</code> ?<br>
By the way, how could I assign the initial gamma and beta values if assigining <code>affine=False</code> does not mean to initial <code>gamma</code> and <code>beta</code> but to fix the values?</p>

nmakes: <p><code>affine = False</code> is equivalent to simply computing:</p>
<p><code>y = (x - mu) / sqrt(var + eps)</code></p>
<p>where, <code>mu</code> is the running (propagated) mean and <code>var</code> is the running (propagated) variance. Equivalently, this can be interpreted as fixing <code>gamma=1</code> and <code>beta=0</code> (These will then be non-trainable. Since they don’t appear in the equation above, no gradients will be calculated for those).</p>
<p>If you rather want to initialize gamma, beta to (1, 0) and train them, you’d want to perform something like:</p>
<pre><code class="lang-auto">bn = nn.BatchNorm1d(num_c, affine=True)
bn.weight = 1
bn.bias = 0
</code></pre>

ptrblck: <p>The formula you’ve given would be used for <code>affine=False</code>. I guess you have a typo in your post. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

nmakes: <p>Ah. My bad. Thanks for pointing that out. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Jo-w: <p>Hi there,<br>
What if I init the model parameters from a pre-trained model (e.g. .pth file) and want to keep the gamma and beta frozen while changing mean and var when resuming the model?<br>
Thank you.</p>

ptrblck: <p>You could set the <code>.requires_grad</code> attributes of the <code>.weight</code> and <code>.bias</code> parameters to <code>False</code> and keep this layer in <code>.train()</code> mode.</p>