akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Indexing variable axis

Sam_Lerman: <p>Say I want to index a tensor along an axis, but I don’t know a priori which axis. It could be:</p>
<p><code>tensor[:, idx, :]</code></p>
<p>Or it could be:</p>
<p><code>tensor[idx, :, :]</code></p>
<p>Or:</p>
<p><code>tensor[:, :, idx]</code></p>
<p>In these examples, the tensor has 3 possible axes. Is there a way to do this kind of indexing where both the axis and the index are variable?</p>

ptrblck: <p>You can use <code>tensor.index_select</code> as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, 10)
idx = torch.randint(0, 10, (2,))

ref1 = x[idx]
ref2 = x[:, idx]
ref3 = x[:, :, idx]

out1 = x.index_select(0, idx)
out2 = x.index_select(1, idx)
out3 = x.index_select(2, idx)

print((ref1 == out1).all())
# tensor(True)
print((ref2 == out2).all())
# tensor(True)
print((ref3 == out3).all())
# tensor(True)
</code></pre>
Adam.step() is a single step or multiple

robinho: <p>I wonder if adam.step() is a single step or multiple steps towards convergence=minimization of loss? If it’s the latter, how is convergence judged?</p>

ptrblck: <p>It’s a single weight update step as seen <a href="https://github.com/pytorch/pytorch/blob/07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98/torch/optim/adam.py#L106-L174">here</a> and <a href="https://github.com/pytorch/pytorch/blob/07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98/torch/optim/adam.py#L230-L308">here</a>.</p>

robinho: <p>thank you! If so, why is adam.step() called only once for <strong>each</strong> sample point in minibatch training? Don’t we need convergence/minimization of loss making full use of <strong>each</strong> sample point?</p>

ptrblck: <aside class="quote no-group" data-username="robinho" data-post="3" data-topic="156492">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c6cbf5/48.png" class="avatar"> robinho:</div>
<blockquote>
<p>If so, why is adam.step() called only once for <strong>each</strong> sample point in minibatch training?</p>
</blockquote>
</aside>
<p>I’m not sure I understand the “each sample point” reference, but mini-batch training is a common approach in machine learning. While some ML approaches are able to use the full training dataset for a single update, it’s usually not feasible in deep learning due to the size of the model, data etc.</p>

robinho: <p>If we talk about gradient descent where minibatch=fullsample, iterations=1. Then <strong>epochs</strong> determine convergence=min(loss) as the following:<br>
for epoch in epochs<br>
adam.step()<br>
end</p>
<p>Then in minibatch training, once again <strong>epochs</strong> determine convergence=min(loss):<br>
for epoch in epochs<br>
for i in iterations<br>
adam.step()<br>
end<br>
end</p>
<p>Correct?</p>

ptrblck: <p>Assuming <code>fullsample</code> represents the entire dataset, then yes, the second approach is the common one and represents the mini-batch training.<br>
Take a look at the <code>Optimization</code> chapter of the Deep Learning Book and in particular <a href="https://www.deeplearningbook.org/contents/optimization.html">8.1.3 Batch and Minibatch Algorithms</a> for more information.</p>
Calculating RAM size of network

Omroth: <p>I’d really like a call to print the RAM footprint of my nn.Module (including gradients) - does this exist?</p>
<p>I know that there will be more requirements for training data etc, but it does seem possible to calculate just the Module requirements deterministically.</p>
<p>Thanks.</p>

thecho7: <p>This post (<a href="https://discuss.pytorch.org/t/finding-model-size/130275" class="inline-onebox">Finding model size</a>) might help you.<br>
If you want to print the RAM occupancy for each module, here’s an example snippet</p>
<pre><code>for m in model.modules():
    for param in m.parameters():
        print(param.nelement() * param.element_size())
</code></pre>

Omroth: <p>Thanks - would that give KB?</p>

thecho7: <p>It would be byte.<br>
Divide the output with 1024 gives you KB</p>
Where is the real implementation of codes of Sigmoid_()?

111466: <p>I can not find the real full implemetnation codes of function Sigmoid_()?<br>
Colud you help me ? Thank you!</p>

ptrblck: <p>The CPU implementation should dispatch to <a href="https://github.com/pytorch/pytorch/blob/2c43876f64603d6ccaedcf6a95f3a10319c754e6/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp#L37-L68">this kernel</a> and the CUDA implementation to <a href="https://github.com/pytorch/pytorch/blob/c3e67d8a8c1217c6c65c0539866e6c59ee2bd9a4/aten/src/ATen/native/cuda/UnarySpecialOpsKernel.cu#L123-L158">this one</a>.</p>

111466: <p>Thany you very much. And I’m sorry to disturb you once more.<br>
If I want to read other codes of some specific functions, how can I find their implementations?<br>
Besides, as far as I’m concerned, the two codes shared by you are beyond the mathematical meaning of Sigmoid function. It provides more difficulty for me to understand.</p>

ptrblck: <p>Generally, you could either directly search for the function names in the repository and then track the calls to the actual kernel implementation or alternatively you could also start by searching for the operation in <a href="https://github.com/pytorch/pytorch/blob/07f4dc9b2c4cf6a00aa12cfb1d5db9056475ae98/aten/src/ATen/native/native_functions.yaml"><code>native_functions.yaml</code></a> and then continue with the operation names.</p>
<aside class="quote no-group" data-username="111466" data-post="3" data-topic="156417">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111466/48/51407_2.png" class="avatar"> 111466:</div>
<blockquote>
<p>Besides, as far as I’m concerned, the two codes shared by you are beyond the mathematical meaning of Sigmoid function. It provides more difficulty for me to understand.</p>
</blockquote>
</aside>
<p>The <code>sigmoid</code> function is used in both kernels as seen in e.g.:</p>
<pre><code class="lang-auto">return static_cast&lt;float&gt;(1) / (static_cast&lt;float&gt;(1) + std::exp((-a0)));
...
return scalar_t{1} / (scalar_t{1} + std::exp(-a));
...
</code></pre>
<p>However, you are right that e.g. the <code>TensorIterator</code> is used to speed up the code which might not be trivial to understand.</p>
Why are the detected result on images better than video or stream with yolov5 in the same frame?

JS_Transportation: <p>I deploy YOLOv5 Object Detection Model with OpenCV. I got bad results when I used it on a video but then I screenshot of that video and used a model on that screenshot picture, and I got a good result. Why are the detected result on images better than video or stream in the same frame? How should I do it better?</p>

ptrblck: <p>I would recommend to check the processing of the frames and make sure the same normalization etc. is applied to the video stream as well as the single input image. Storing a video frame and comparing it to the single frame might also be a good idea to check the value ranges etc.</p>
How cuda is initialized in DDP

czhxiaohuihui: <p>As I know, when we start a process, we can prevent init cuda many times with following code in<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/Context.h#L102" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Context.h at master · pytorch/pytorch · GitHub</a></p>
<p>THCState* lazyInitCUDA() {<br>
std::call_once(thc_init,[&amp;] {<br>
thc_state = detail::getCUDAHooks().initCUDA();<br>
});<br>
return thc_state.get();<br>
}</p>
<p>By the time, we use ddp,  we set start_method like (spawn, fork, forkserver),  there will be multi-process, how we can prevent the CUDA to be initialized many times???</p>
<p>And I want to know how cuda is initialized in DDP???</p>
<p>Thanks a lot!</p>

ptrblck: <p>The CUDA context is initialized in DDP in the same way as in a single process Python script, i.e. in its first CUDA operation. You could set the device via <code>torch.cuda.set_device</code>, mask the other devices etc. to make sure the default device (or any other unwanted devices) are used in the current Python process.</p>
Increase Learning Rate using lr_scheduler

pabloruizp: <p>I am trying to progressively increase the lr using a lr_scheduler but it is not possible due to their implementation:</p>
<pre><code class="lang-auto">class ConstantLR(_LRScheduler):
    def __init__(self, optimizer, factor=1.0 / 3, total_iters=5, last_epoch=-1, verbose=False):
        if factor &gt; 1.0 or factor &lt; 0:
            raise ValueError('Constant multiplicative factor expected to be between 0 and 1.')
   ...
</code></pre>
<p>Is there any way to achieve my goal  (without implementing my own _LRScheduler?</p>

ptrblck: <p>I think a <code>LambdaLR</code> object should work:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)

factor = 2.
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda epoch: (epoch+1) * factor, last_epoch=- 1, verbose=False)

for epoch in range(10):
    print(optimizer.param_groups[0]['lr'])
    optimizer.step()
    scheduler.step()
</code></pre>
How to use Ensemble Learning for balancing data

farahnaz: <p>HI. i have a imbalanced dataset with number of sample in class1:98,class2:68,class3:21 .<br>
now i want to run my classification code A thousand times and every time i want to randomly choose 21 samples for each class.<br>
i think i should use Ensemble Learning but i don’t know how? and which technique of Ensemble Learning is suitable for my problem?<br>
if you have PyTorch code for my problem i will be very thankful if you share it with me.</p>

ptrblck: <p>I’m not sure I understand the use case completely but assuming you want to use a model ensemble with each trained model, this would mean your final model would run the forward pass 1000 times (using the corresponding models) to classify a single input batch. Is this indeed what you are planning to do?</p>
Multiclass Image Segmentation setting

KevinChuehb05102135: <p>Hi, I’m currently working on a multiclass image segmentation project. There are 4 classes (including background 0) in the dataset. My input image loader size is (10,1,244,244), and the output of Unet is (10,4,244,244). As you can see, i set the output channel number as 4 because there are 4 classes. However, the mask (target) size is (10,1,244,244), which has only 1 channel although 4 classes.</p>
<p>I’m using dice_loss from here <a href="https://kornia.readthedocs.io/en/v0.1.2/_modules/torchgeometry/losses/dice.html" class="inline-onebox" rel="noopener nofollow ugc">torchgeometry.losses.dice — PyTorch Geometry documentation</a>,<br>
whose target size should be (B, H, W). Thus i squeeze the first dim of mask(target) so now it’s (10,244,244). To sum up, my image input: (10,1,244,244), network output: (10,4,244,244), Target mask : (10,244,244).</p>
<p>I’m wondering is the setting right for multiclass segmentation? I’m training at about 100 epochs but the training loss is stuck at about 0.55, so i’m thinking probably something is wrong… Thank you!</p>
How can I use O2 optimization with torch.cuda.amp like apex

chenrui17: <p>it looks like there’s no opt_level option in <code>torch.cuda.amp</code> , so how to use opt_level=O2 in pytorch ？</p>

ptrblck: <p><code>torch.cuda.amp</code> supports the <code>O1</code> mode and an <code>O2</code> mode is not available.</p>
TypeError: Can't convert object to 'str' for 'filename'

NagaYu: <pre><code class="lang-auto">with torch.no_grad():
    for i in test_data:
        test_data = cv2.imread(i)
        
        #transform = ImageTransform(size, mean, std)
        #img = transform(img)
        #img = img.unsqueeze(0)
        #img = img.to(device)

        # 推論モード
        model.eval()

        # 推論
        outputs = model(test_data)
        # ソフトマックス関数を使って確率を出力
        preds = F.softmax(outputs, dim=1)[:, 1].tolist()
        
        image_paths.append(img)
        predictions.append(preds[0])
    
res = pd.DataFrame({
    'ImageID': image_paths,
    'predictions': predictions})

res.sort_values(by='img', inplace=True)
res.reset_index(drop=True, inplace=True)

# CSV出力
res.to_csv('test_data/sample_submission.csv', index=False)
</code></pre>
<p>TypeError                                 Traceback (most recent call last)<br>
Input In [11], in &lt;cell line: 1&gt;()<br>
1 with torch.no_grad():<br>
2     for i in test_data:<br>
----&gt; 3         test_data = cv2.imread(i)<br>
5         <span class="hashtag">#transform</span> = ImageTransform(size, mean, std)<br>
6         <span class="hashtag">#img</span> = transform(img)<br>
7         <span class="hashtag">#img</span> = img.unsqueeze(0)<br>
8         <span class="hashtag">#img</span> = img.to(device)<br>
9<br>
10         # 推論モード<br>
11         model.eval()</p>
<p>TypeError: Can’t convert object to ‘str’ for ‘filename’</p>

ptrblck: <p>It seems you are trying to overwrite the <code>test_data</code> while also iterating it, so I would recommend to use another variable name and check if you are still hitting the issue.</p>
Pytorch and CUDA 11.4

a.bac: <p>Hello everyone, i’m aware that there are already a few post for this topic, but none of the answer i considered seems to help.</p>
<p>I’m trying to use pytorch on a machine with CUDA and CUDNN already installed.<br>
I have not the permission to change the CUDA and CUDNN versions.<br>
nividia-smi produces:<br>
“NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4”</p>
<p>I can’t find the right pytorch version for using gpus.<br>
“torch.cuda_is_available() returns False”.</p>
<p>I really need that system to be able to work.<br>
A help is really needed, thank you.</p>

ptrblck: <p>I assume you are installing the PyTorch binaries and are selecting a proper CUDA runtime version.<br>
If so, note that your local CUDA toolkit will only be used if you are building custom CUDA extensions or build PyTorch from source. The binaries will use their own CUDA runtime.</p>
<p>Based on the output of <code>cuda.is_available()</code> it seems you’ve either installed the CPU-only binaries (the install log would show it) or PyTorch is unable to communicate with the NVIDIA driver.</p>
What is the specific reason to get different results on cuda and without cuda?

krishna511: <p>Can I get a straight answer why this is happening on a simple <code>MLPClassifer</code>?<br>
Is it possible even? I am using the same seed same data same method.</p>
<p>Regards</p>

ptrblck: <p>I don’t know how you are comparing the models and outputs as you haven’t shared any code, but note that the random number generators are not the same on CPU and the GPU, so seeing won’t create the same values.<br>
If you want to compare the outputs of different devices, store and load a <code>state_dict</code> so that all parameters and buffers of the model will be equal.</p>

ParGG: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Would it be possible that, since Linear on GPU (Ampere) makes use of TensorFloat32, there are some discrepancies due to the smaller mantissa?</p>
<ul>
<li>GPU: 10 bits of mantissa with TensorFloat32 + FMA in float32</li>
<li>CPU: 23 bits of mantissa + FMA in float64 before rounding to float32</li>
</ul>

ptrblck: <p>The computation is independent from the initialization, but yes if TF32 is enabled (note that it’s disabled in linear layers starting in <code>1.12.0</code>) the output would be different. However, even in <code>float32</code> (or any other <code>dtype</code>) you would expect to see numerical errors according to the limited floating point precision as most likely different algorithms are used.</p>

ParGG: <p>I am not sure if that’s always the case, but in most scenarios the data are generated on CPU and then copied to the GPU using <code>.to("cuda:0")</code>. In that case, when using the same seed, you should have the same data. Therefore, the discrepancy would come either from the quantization error due to the smaller mantissa and/or a different algorithm used in the computation (I think stochastic algorithms are used by default <a href="https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility" class="inline-onebox" rel="noopener nofollow ugc">cuBLAS :: CUDA Toolkit Documentation</a> + CPU &amp; GPU might not use both FMA).</p>
<p>Please <a class="mention" href="/u/ptrblck">@ptrblck</a> correct me if I am wrong, by reading your answers I learned really a lot in the past years.</p>

ptrblck: <p>Yes, your description look correct. If you sample all data (and parameters) on the CPU and push it to the GPU, you would expect to reuse the same values. However, even then the pseudorandom number generators will use different implementations and e.g. dropout layers would behave differently (the CUDA implementation has a vectorized code path for example).<br>
Also you are correct that different algorithms could create different small errors due to the limited floating point precision.</p>
Varationale autoencoder

Berny: <p>Hi. I’m implementing a varational autoencoder and I used the following code for the training part:</p>
<p>model.train()<br>
train_loss = 0<br>
for epoch in range(1, NUM_EPOCHS + 1):<br>
for batch_idx, data in enumerate(train_loader):<br>
<a class="hashtag" href="/c/data/37">#<span>data</span></a> = np.asarray(data)<br>
<a class="hashtag" href="/c/data/37">#<span>data</span></a> = torch.from_numpy(data.astype(‘long’))<br>
<a class="hashtag" href="/c/data/37">#<span>data</span></a> = data.detach().numpy()<br>
<a class="hashtag" href="/c/data/37">#<span>data</span></a> = data.numpy()<br>
<a class="hashtag" href="/c/data/37">#<span>data</span></a> = torch.FloatTensor(data)<br>
data = data.to(device)<br>
optimizer.zero_grad()<br>
recon_batch, mu, logvar = model(data)</p>
<p>I get the error: ‘list’ object has no attribute ‘to’. How can I fix it? Thank you for your help.</p>

ptrblck: <p><code>data</code> seems to be a <code>list</code> instead of a tensor so you might need to index it first and call <code>.to()</code> on the tensor(s).</p>

Berny: <p>So, I replaced the 4th line with:</p>
<p>for batch_idx, (data, _) in enumerate(train_loader):</p>
<p>and now it doesn’t give me error. Does it make sense?</p>

ptrblck: <p>Yes, this could work as you are not unwrapping the return value from the <code>train_loader</code>.</p>
Extracting the output of attention heads from a pretrained Vit?

alireza_khatami: <p>Hello good people .<br>
Let’s say I have loaded the Vit with pretrained = True. And I want to extract the output of softmax of the attention heads from this model. How can  I achieve that? should I create a new model and inherit from this model and then change the forward method. Or should  I use the modules for the Vit and load each layer to the new model?</p>
Replace default operator run time implementation with custom code to make use of DSP HW

Able_Eldhose: <p>I am trying to run a tflite model in an ARM based micro controller<br>
I was able to generate a .o file and corresponding header files specific to a model</p>
<p>My micro controller also has a DSP engine which can do faster matrix multiplication and it is a custom HW which is not currently supported by GLOW.</p>
<p>How can I make a specific operator of my choice to compile my custom code instead of the default implementation.</p>
<p>For eg in a model every fully_connected layer I want to run my custom code instead of default fully_connected layer implementation, so that I can make use of the custom DSP HW present in my micro controller.</p>
Compiling pytorch with CUDA11.1+CUDNN 8.0.5, NO AVX with Ampere architecture

ys231: <p>Hello everyone,</p>
<p>I am trying to compile pytorch  on an Ubuntu 20 machine that does not have AVX support. The GPU on the server is based on the Ampere architecture with SM_86. I know that Cuda 11.1 and CUDNN 8.0.5 have SM_86 support and are not dependent on AVX (I successfully compiled Tensorflow, for example, and it works with the hardware).</p>
<p>I am looking for a combination of pytorch version and its dependencies that I compile with no AVX+the aforementioned CUDA/CUDNN versions. I cloned the github (recursively) and tried compiling different releases with the following options:</p>
<p>export TORCH_CUDA_ARCH_LIST=“8.6”<br>
sudo cmake CC=/usr/bin/clang CXX=/usr/bin/clang++  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DCMAKE_BUILD_TYPE=Debug -DPYTHON_EXECUTABLE=/usr/local/bin/python3 -DDISABLE_AVX2:BOOL=TRUE -DCXX_AVX2_FOUND:BOOL=FALSE -DC_AVX2_FOUND:BOOL=FALSE -DDISABLE_AVX512F:BOOL=TRUE -DDISABLE_AVX:BOOL=TRUE -DCXX_AVX_FOUND:BOOL=FALSE -DC_AVX_FOUND:BOOL=FALSE -DCMAKE_CXX_FLAGS:STRING=-std=c++14  …</p>
<p>I’ve tried both clang and gcc, but the compilation stops at different points with different release versions of pytorch. I get errors at pytorch-cpu or ATEN and sometimes one of the dependencies seems to assume that a later version of CUDA is present (judging by the header constants it is unable to find).</p>
<p>Does anyone have any suggestions?</p>
<p>Many thanks!</p>
How to Normalize 4d Tensor?

Bled_Clement: <p>Hi Folks.</p>
<p>I have a tensor of size:<br>
[B, C, dimA, dimB, h, w]</p>
<p>The above tensor is supposed to be a batch of B images, with C channels (currently 1 grayscale channel). dimA and dimB are the result of unfolding each images into smaller blocks and then we have the pixels in each block, h and w.</p>
<p>Using convNd I have applied a 4D convolution. I would now like to apply 4D batch normalization across this tensor. Is this possible? Is there a more efficient alternative?</p>
<p>Thanks guys!</p>

Kushaj: <p>You can try two things</p>
<ol>
<li>Use <code>view</code> to change the size of tensor to that of 1d and then apply <code>BatchNorm1d</code>. After that again use <code>view</code>.</li>
<li>Define BatchNorm4d. batch norm internally uses <code>torch.batch_norm</code> and it can work on input of any size. But you might not get CUDNN speedup (I am not sure about this).</li>
</ol>

Bled_Clement: <p>Thanks for the response! I’ll implement the view method for now. Do I need to use contiguous in conjunction with view() to keep my tensor in the correct order?</p>
<p>I looked up the definition of batch_norm and it seems to have two different functions, one for CPU and one for GPU as you mentioned. They’re also written in C++ which I haven’t worked with in a while!</p>
<p>I’ll report back later.</p>

Kushaj: <p>I was also wondering about contiguous. I don’t think you have to. But you might have to double check this, if you get an error without contiguous.</p>

Bled_Clement: <p>The 1D normalize solution seems to work for me. Thank you!</p>

NilsMi: <p>This is the way 1d, 2d and 3d batchnorm are done. If you want N dimensional batchnorm (or dont care about dimension)  you can change _check_input_dim to whatever you need. This way you dont need to worry about changing view etc.</p>
<pre><code class="lang-auto">class BatchNorm4d(torch.nn.modules.batchnorm._BatchNorm):
    def __init__(self, num_features, **kwargs):
        super().__init__(num_features=num_features, **kwargs)
    def _check_input_dim(self, input):
        if input.dim() != 6:
            raise ValueError("expected 6D input (got {}D input)".format(input.dim()))
</code></pre>
Can I re-register a pytorch hook after removing it?

kocchop: <p>Hi,</p>
<p>For one of my studies I want to dynamically register and unregister a pytorch forward hook. For example, I only need to access the activation outputs at certain steps. I have created a handle for the forward hook:</p>
<pre><code class="lang-auto">handle_1 = model.fc_1.register_forward_hook(act_hook())
</code></pre>
<p>Now if I remove the handle using:</p>
<pre><code class="lang-auto">handle_1.remove()
</code></pre>
<p>Is there anyway I can re-attach it somehow? Thanks in advance!</p>

ptrblck: <p>Yes, I think calling <code>register_forward_hook</code> another time would register the hook again and would return the corresponding handle.</p>

kocchop: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, It actually worked! Thanks so much <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
Multitask model for text (sentence) classification using PyTorch

kripa: <p>Dear all,</p>
<p>I am a beginner in PyTorch and am looking to write a multitask model to optimize two text classification tasks, one being a binary classification and the other being a three-class classification. Can someone help me with a starting point reference (an example running code, tutorial etc.)? I have used Text CNN for binary and multi-class text classification. However, I find Multi Task Learning (MTL) difficult. I think that in the neural model, the MTL loss will be some combination (e.g. sum) of the individual task losses. However, I am unable to proceed further. I am curious to know how the MTL backpropagation is implemented, how the common layers are shared between the tasks etc.</p>
<p>Thanks in advance.</p>
Gradient is always zero with torch.cdist?

vnx_int: <p>Hi everyone! I’m new to pytorch and I’m working on a custom model based on <a href="https://towardsdatascience.com/how-to-use-pytorch-as-a-general-optimizer-a91cbf72a7fb" rel="noopener nofollow ugc">this medium article</a>, but for some reason my weights aren’t updating. When I try to print out the gradients of the weights as they are training I simply end up seeing zero tensors. My model and the training loop is shown below:</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        weights = torch.zeros([n_vec, 2])
        self.weights = nn.Parameter(weights)        
        
    def forward(self):
        return torch.cdist(self.weights, self.weights, p=2)
    
def training_loop(model, optimizer, n=1000):
    losses = []
    for i in range(n):
        preds = model()
        loss = torch.sum((preds - x) ** 2)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        losses.append(loss)  
    return losses
m = Model()
opt = torch.optim.Adam(m.parameters(), lr=0.005)
losses = training_loop(m, opt)
</code></pre>
<p>I can’t figure out why this is, the only thing I can think of is that torch.cdist is somehow messing up the autograd. Any help would be greatly appreciated!</p>

ptrblck: <p>You are initializing the <code>weights</code> with zeros and <code>torch.cdist</code> should thus also return zeros only.<br>
I don’t know what <code>x</code> is, but I would assume if you initialize the <code>self.weights</code> parameters randomly, you should see valid gradients.</p>
Adam with multiprocessing

ArchieGertsman: <p>I am following the Hogwild example from <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">multiprocessing best practices</a>:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters

if __name__ == '__main__':
    num_processes = 4
    model = MyModel()
    # NOTE: this is required for the ``fork`` method to work
    model.share_memory()
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>and I’ve adapted it in the context of my reinforcement learning code. The way I construct my optimizer determines whether or not the model learns correctly.</p>
<p>What works: I define <code>optim = Adam(model.parameters(), lr=.005)</code> in the main process and pass it into <code>train</code> when creating the processes, i.e. <code>p = mp.Process(target=train, args=(model, optim))</code>. Then, at the start of <code>train</code>, I make a copy of <code>optim</code> local to each subprocess, i.e. <code>optim = deepcopy(optim)</code>.</p>
<p>What doesn’t work: I directly define <code>optim = Adam(model.parameters(), lr=.005)</code> inside <code>train</code>. Note: when I do this with <code>SGD</code>, the model <em>does</em> learn!</p>
<p>Is there an obvious explanation as to why doesn’t <code>Adam</code> work in the second approach?</p>

kwen2501: <p>Cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a>, wondering if you have any idea, thanks!</p>
Map_location not working with mps

Robin_Lobel: <p>Given a <code>device = torch.device("mps")</code>, I can’t use functions that have a map_location parameter (while this is working fine with “cpu” or “cuda:0” devices).</p>
<p><code>model = torch.jit.load(buffer, map_location=device)</code> will give the following error:</p>
<pre><code class="lang-auto">   model = torch.jit.load(buffer, map_location=device)
  File "/Users/divide/TorchStudio/python/lib/python3.9/site-packages/torch/jit/_serialization.py", line 164, in load
    cpp_module = torch._C.import_ir_module_from_buffer(
RuntimeError: supported devices include CPU and CUDA, however got MPS
</code></pre>
<p>and <code>model = torch.package.PackageImporter(buffer).load_pickle('model', 'model.pkl', map_location=device)</code> will give the following error:</p>
<pre><code class="lang-auto">    model = torch.package.PackageImporter(buffer).load_pickle('model', 'model.pkl', map_location=device)
  File "/Users/divide/TorchStudio/python/lib/python3.9/site-packages/torch/package/package_importer.py", line 256, in load_pickle
    result = unpickler.load()
  File "/Users/divide/TorchStudio/python/lib/python3.9/pickle.py", line 1212, in load
    dispatch[key[0]](self)
  File "/Users/divide/TorchStudio/python/lib/python3.9/pickle.py", line 1253, in load_binpersid
    self.append(self.persistent_load(pid))
  File "/Users/divide/TorchStudio/python/lib/python3.9/site-packages/torch/package/package_importer.py", line 213, in persistent_load
    load_tensor(
  File "/Users/divide/TorchStudio/python/lib/python3.9/site-packages/torch/package/package_importer.py", line 201, in load_tensor
    loaded_storages[key] = restore_location(storage, location)
  File "/Users/divide/TorchStudio/python/lib/python3.9/site-packages/torch/serialization.py", line 973, in restore_location
    return default_restore_location(storage, str(map_location))
  File "/Users/divide/TorchStudio/python/lib/python3.9/site-packages/torch/serialization.py", line 178, in default_restore_location
    raise RuntimeError("don't know how to restore data location of "
RuntimeError: don't know how to restore data location of torch.storage._TypedStorage (tagged with mps)
</code></pre>

aw632: <p>Claims to be fixed in <a href="https://github.com/pytorch/pytorch/issues/79384" class="inline-onebox" rel="noopener nofollow ugc">torch.load() fails on MPS backend ("don't know how to restore data location") · Issue #79384 · pytorch/pytorch · GitHub</a> but I get the same problem</p>

kulinseth: <p>The serialization code was added back in and hopefully works <a class="mention" href="/u/robin_lobel">@Robin_Lobel</a> with your example. Can you please check the latest nightly.</p>
Torchvision.transforms.functional.resize bilinear incorrect results

tareq: <p>Hi there,</p>
<p>torchvision.transforms.functional.resize bilinear produces a result that does not match with the bilinear algorithm nor does it match with tf.image.resizeBilinear. I tried with multiple test cases, yet I always keep getting the wrong resize results. Perhaps there is something I’m missing? Thanks.</p>

ptrblck: <p>You might need to enable the <code>antialias</code> argument to reduce the error between <code>torchvision</code> and <code>PIL</code>:</p>
<pre><code class="lang-python">x = torch.randint(0, 256, (3, 224, 224)).byte()

out = torchvision.transforms.functional.resize(
    x,
    size=(100, 100),
    interpolation=torchvision.transforms.functional.InterpolationMode.BILINEAR,
    antialias=True
)

img = torchvision.transforms.functional.to_pil_image(x)
ref = img.resize(size=(100, 100), resample=PIL.Image.Resampling.BILINEAR)
ref = torch.from_numpy(np.array(ref)).permute(2, 0, 1)

print((ref.float() - out.float()).abs().max())
# tensor(1.)
</code></pre>
Replace torch.optim.Adam by torch.optim.LBFGS

DRISS_ELALAOUI: <p>please, how can I replace torch.optim.Adam by torch.optim.LBFGS in this code?</p>
<pre><code class="lang-auto">model = Net().to(device)
crit = nn.BCELoss(reduce=True)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

def train():
    model.train()
    loss_all = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        output = model(data)
        label = data.y
        loss = crit(output, label)
        loss.backward()
        loss_all += data.num_graphs * loss.item()
        optimizer.step()
    return loss_all / len(train_dataset)

for epoch in tqdm(range(100), desc="Epochs: ", leave=True):
    loss = train()
    print('Epoch: {:03d}, Loss: {:.5f}.format(epoch, loss))
</code></pre>
<p>and thank you</p>

ptrblck: <p>You would need to move the loss calculation and update to a <code>closure</code> as seen in this example:</p>
<pre><code class="lang-python">import torch
from torch.optim import LBFGS

x = torch.tensor([100.], device='cuda', requires_grad=True)
optimizer = LBFGS([x])
def f(x):
	return (x - 1) * (x - 1)

n_step = 2

for _ in range(0, n_step):
    def closure():
        optimizer.zero_grad()
        output = f(x)
        output.backward()
        return output
    optimizer.step(closure)
    print(x)
</code></pre>
Transformer to reorder sequences

MagMueller: <p>My goal is to reorder input sequences like [1,1,1,2,2,3,4,5,6,6] to e.g. [0,0,6,6,0,0,6,2,2,1,5,0,0,1,12,3,4] to simulate an industrial process.<br>
Each token in the input should also be in the output + some 0s at which time step the process has no output.</p>
<p>I did it similar to <a href="https://github.com/matsuolab/transformer_tutorial/blob/master/main.py" rel="noopener nofollow ugc">that Transformer</a> and it learns pretty good.</p>
<p>But especially with long sequences, sometimes tokens are missing from the output or sometimes there are some that are not in the input. This is really bad for my application.</p>
<p>Is there any way I can restrict the output so that it can only generate each token exactly once from the input sequence?</p>
<p>What I was thinking about:</p>
<ul>
<li>Using beam search for decoding and cutting out all paths that are not valid (Is this possible during training? (My model outputs logits, which I am currently using for CrossEntropyLoss))</li>
<li>use different losses, e.g. JARO_WINKLER_LOSS or LEVENSHTEIN_LOSS (edit - distance) or just something that counts the number of same and different tokens, but with all these comparison methods I have the problem that they do not have gradients.</li>
</ul>
<p>I would be very grateful for any ideas, tips or links on how I could solve this.</p>
Extracting transformer attention output weights

lambda42: <p>Hi there,<br>
first of all, thanks for the nice modular implementation of the transformer architecture in pytorch!</p>
<p>I recently tried to extract the attention output weights for some layers in a TransformerEncoder for an input sample and have some suggestions on how to improve this.</p>
<ul>
<li>
<p>The option to get the attention weights is currently only present in MultiheadAttention with the need_weights argument. This returns already the mean attention weights instead of the attention weights per head.<br>
First, it would be better to return the attention weights per head and let the user do the sum over heads if needed (some users want to see the attention weights per head).<br>
Second, it would be good to introduce the same argument to TransformerEncoderLayer and TransformerDecoderLayer in their forward methods and use that argument when evaluating self.self_attn and self.multihead_attn (currently, the default value True is used, which calculates the attention weights but then they are discarded, which is clearly not optimal)</p>
</li>
<li>
<p>There should be a way to get the forward keyword arguments in forward hooks! Then it would be simple to install a forward hook in each TransformerEncoderLayer, evaluate with needs_weights=True and store the results somewhere. For me, this is currently not possible because the attention mask is passed as keyword argument. This is a severe shortcoming of hooks! They should have access to all the arguments that forward has access to. Keyword arguments could be passed as a separate dict.</p>
</li>
</ul>
<p>What is your opinion on this?</p>

pacert: <aside class="quote no-group" data-username="lambda42" data-post="1" data-topic="99058">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e480ec/48.png" class="avatar"> lambda42:</div>
<blockquote>
<p>TransformerEncoderLayer</p>
</blockquote>
</aside>
<p>Hey <a class="mention" href="/u/lambda42">@lambda42</a>, I am fairly new to transformers and try to find a way to access the attention weights at the end of my nn.transformerEncoder  did you manage to do that? If so, I would highly appreciate any advice how I can get the weights, for my purposes it would be sufficient to get the means over all heads. I agree thou that a more modular option would be generally helpful.<br>
Thanks for your support in the matter.</p>
Parametrization moves the parameter to device, but not the gradient?

tueboesen: <p>I am trying to use the new parametrization tool, talked about here: <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html#torch.nn.utils.parametrize.register_parametrization" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.utils.parametrize.register_parametrization — PyTorch 1.12 documentation</a></p>
<p>More specifically I’m trying to use the ortogonal module: <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.utils.parametrizations.orthogonal — PyTorch 1.12 documentation</a></p>
<p>I have created the following minimal example to show the problem:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class network_test(nn.Module):
    def __init__(self, low_dim, high_dim):
        super().__init__()

        self.lin = torch.nn.Linear(low_dim,high_dim)
        self.ortho = torch.nn.utils.parametrizations.orthogonal(self.lin)
        self.register_buffer("K", self.lin.weight)

    def forward(self,x):
        y = x @ self.K.T
        return y

low = 3
high = 6
batch_size = 10
model = network_test(low,high)
device = 'cuda:0'



model.to(device)
lr = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

x = torch.randn(batch_size,low,device=device)
y = model(x)
loss = y.sum()
loss.backward()
optimizer.step()
</code></pre>
<p>This code will crash because the gradient of the parameter is somehow not on cuda, while the parameter itself is. I’m not sure how this is happening or what I should be doing differently?</p>

ParGG: <p>When you perform this: <code>self.register_buffer("K", self.lin.weight)</code> you store in the buffer a tensor that is on CPU. This is also written in the docs of <code>register_buffer</code>: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_buffer" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 2.1 documentation</a>.</p>
<blockquote>
<p><strong>tensor (Tensor or None)</strong> – buffer to be registered. If None, then operations that run on buffers, such as cuda, are ignored. If None, the buffer is not included in the module’s state_dict.`</p>
</blockquote>
<p>To make your code work, just use any other parameter but the registered buffer <code>K</code> for your operations. You can call <code>self.lin(x)</code>, <code>self.ortho(x)</code>, or even use their weight.</p>
After extracting activation map from intermediate layers how can I pass it to the submodules (skip connection blocks) of another network?

space_ps: <p>I have extracted activation maps from unet (<a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="noopener nofollow ugc">https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix</a>)  and this is discussed here (<a href="https://discuss.pytorch.org/t/how-to-extract-intermediate-feature-maps-from-u-net/27084">https://discuss.pytorch.org/t/how-to-extract-intermediate-feature-maps-from-u-net/27084</a>). Now I want to pass the values to UnetSkipConnectionBlock module.</p>
<p>I was unable to pass it by this manner<br>
out_image = self.model(input, activation=self.activation, hook_idx=self.hook_idx).</p>
<p>Many thanks in advance.</p>

ParGG: <p>Hi,</p>
<p>I checked the two links and the implementation of the UnetSkipConnectionBlock. Unfortunately I cannot really tell what your are trying to do and what your <code>self.model</code> actually implements. Could you elaborate more and share some code?</p>

space_ps: <p>Thank you for your response…</p>
<p>I want to pass the ‘activation’ and ‘hook_idx’ variables to each UnetSkipConnectBlock so that it can be accessed at each layer and modified. The parameters are common for every (UnetSkipConnectBlock) layer. I think if I make down, up and submodule (in UnetSkipConnectBlock) global and change it in forward pass (of UnetSkipConnectBlock) it will work…? Currently trying something similar…</p>

ParGG: <p>If you register the hooks to the other network, would’t that do the job? You would be able to access the <code>activation</code> dictionary (which is global) and index it by setting the name you want when you register the hook.</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model = UnetGenerator(1, 1, 5)
model.unet_block_1.register_forward_hook(get_activation('block1'))
model.unet_block_2.register_forward_hook(get_activation('block2'))
model.unet_block_3.register_forward_hook(get_activation('block3'))

model2 = UnetGenerator(1, 1, 5)
model2.unet_block_1.register_forward_hook(get_activation('block1'))
model2.unet_block_2.register_forward_hook(get_activation('block2'))
model2.unet_block_3.register_forward_hook(get_activation('block3'))
</code></pre>
<p>Since the blocks of the 2 models have the same name, they will index the same activation in the dictionary. Then, if you want to modify it in a way that is not overriding, you can just adapt the <code>hook</code> logic accordingly.</p>

space_ps: <p>That makes sense! Thank you for the reply!</p>
Create tensor from dataloader directly on gpu?

jayz: <p>Hey, is it possible to use default_collate such that it directly creates tensors on the cuda device? Then, the .to(‘cuda’) step can be saved, which will cost less time, right?</p>
<p>Thanks!</p>
<p>Best, JZ</p>

ptrblck: <p>Yes, it’s possible, but note that you have to make sure no additional CUDA contexts are created if you are using multiprocessing (i.e. multiple workers) and you would need to profile the use case and see if it’s really speeding up the training. By default the CPU will be used and will create the batches in the background (if multiple workers are used again) while the GPU is busy training. If you directly push the samples to the GPU, you would of course need GPU resources, which might slow down your actual training.</p>

jayz: <p>Ah, yes, I understand. If all the cpu workers create the tensors directly on the gpu, then, this will much overutilize the GPU memory, right? From that viewpoint, it seems more reasonable to use .to().</p>
Matrix Multiplication

kiranpurohit: <p>How to do matrix multiplication on two matrices of sizes<br>
torch.Size([32, 32, 3, 3]) and torch.Size([32, 64, 1, 1]) to get a matrix of size<br>
torch.Size([32, 64, 3, 3])  ??</p>

ParGG: <p>You can use einsum: <a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" class="inline-onebox" rel="noopener nofollow ugc">torch.einsum — PyTorch 1.12 documentation</a>. It is a very cool notation that, if understood, can let you do operations really efficiently.</p>
<pre><code class="lang-python"># A = tensor of torch.Size([32, 32, 3, 3])
# B = tensor of torch.Size([32, 64, 1, 1]) 

C = torch.einsum("ijkl,jmno-&gt;imkl", A, B)
</code></pre>

rad: <p>While the multiplication is indeed “possible” in terms of transpositions/broadcasting, I think we could use a bit more context to see if what he’s trying to do is correct. To me it doesn’t look like those two matrices “should” be multiplied (i.e. what does the operation itself represent?).</p>

kiranpurohit: <p>Thanks a lot. It worked. What’s the logic behind this?</p>

ParGG: <p>In simple terms, you name each dimension of the tensors with a letter.</p>
<p>The einsum notation corresponds of two parts:</p>
<ol>
<li>the first one in which you specify the dimensions of each tensor separated by comma</li>
<li>what comes after “-&gt;”, that specified the operation.</li>
</ol>
<p>Of the tensors you have, assign the same letter to the dimensions that you want to multiply, and remove from the output the dimensions along which you want to accumulate.</p>
<p>E.g.<br>
Element wise multiplication between two matrices: “ij,ij-&gt;ij”</p>
<p>Matrix multiplication:<br>
“mn,np-&gt;mp” (multiply rows with columns (n) and accumulate (n))</p>
<p>In your example I have multiplied the dimension j and accumulated over j, n and o (since n and o are one-dimensional, you could reduce the number of letters and multiply those dimensions instead of accumulating them, this should be less efficient though).</p>
<p>In general you can also transpose the dimensions and do other fancy stuff with it.</p>
<p>You can find more information online.</p>
<p>Hope it was enough for an introduction.</p>

kiranpurohit: <p>Thanks for such a detailed explanation.</p>
How to efficiently remove inplace operations?

tueboesen: <p>I have a pytorch code which performs batched gradient descent, but the problem is that it contains an inplace operation which prevents me from using the autograd.</p>
<p>More specifically I have determined that this line is the problem:<br>
<code>y[idx] = y[idx] - alpha[idx,None,None] * dy</code></p>
<p>I  have managed to replace the operation with the following:</p>
<pre><code class="lang-auto">            ysel = y[idx] - alpha[idx,None,None] * dy
            yall = []
            count = 0
            for i in range(y.shape[0]):
                if i in idx:
                    yall.append(ysel[count])
                    count = count + 1
                else:
                    yall.append(y[i])
            y = torch.stack(yall,dim=0)
</code></pre>
<p>But this is horribly inefficient so I’m wondering whether anyone has a better solution for how to extract elements from a tensor and insert new values back into it, without looping through each value and determining whether to update it or keep the original value.</p>
<p>Any ideas for this particular problem, or general information on how to replace inplace operations efficiently would be much appreciated.</p>

ptrblck: <p>You could check the newly added <a href="https://pytorch.org/functorch/stable/generated/functorch.experimental.functionalize.html#functorch.experimental.functionalize"><code>functorch.experimental.functionalize</code></a> method which was added in PyTorch 1.12.0 and might help for your use case.</p>

tueboesen: <p>I haven’t tested this yet, but from reading about it, it looks perfect, thank you very much!</p>