akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to compute the MelSpectrogram on batches of data?

vgoklani: <p>Hey there,</p>
<p>I can’t figure out how to run the MelSpectrogram (from torchaudio) on batches of data.</p>
<p>I checked the torchaudio docs:</p>
<p><a href="https://pytorch.org/audio/_modules/torchaudio/transforms.html#MelSpectrogram" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/audio/_modules/torchaudio/transforms.html#MelSpectrogram</a></p>
<p>and the waveform needs to be in this format: (channel, time)</p>
<p>which makes sense for <em>one</em> wave, but what if we have batches of data?</p>
<p>Thanks!</p>

ptrblck: <p>This transformation might only work on a single sample (like <code>torchvision.transforms</code>), as it’s usual use case would probably be to apply it in the <code>__getitem__</code> method on each data sample.</p>
Pad_packed_sequence and pack_padded_sequence

XiaoQQin: <p>I want to know does <code>pad_packed_sequence</code>  and <code>pack_padded_sequence</code> is necessary when using the biLSTM?</p>

ironv: <p>Please see <a href="https://discuss.pytorch.org/t/understanding-pack-padded-sequence-and-pad-packed-sequence/4099/15" class="inline-onebox">Understanding pack_padded_sequence and pad_packed_sequence</a>  and <a href="https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html" rel="nofollow noopener">https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html</a>  (referred to in that article).</p>

XiaoQQin: <p>I got it. Thank you so much!</p>
Concatenate and sort tensor

zahra: <p>Hi,</p>
<p>I have 2 tensors like below:</p>
<pre><code class="lang-auto">hc1 = torch.randn(5,1, 1, 1)
hc2 = torch.randn(5,1, 1, 1)
</code></pre>
<p>I want to concatenate these 2 tensors as hc3 and then sort hc3 based on amounts of hc1.<br>
How could I do it?</p>
<p>Thanks</p>

ptrblck: <p>Could you post a simple example with some values?<br>
You could concatenate these tensor using <code>out = torch.cat((hc1, hc2), dim=dim)</code>, however I’m not sure, how you would like to sort the tensors.</p>

zahra: <p>Dear ptrblck,</p>
<p>Thanks for your reply. Sure. I simplify my question with changing the dimensions of tensors to 2. Suppose we have these 2 tensors:</p>
<pre><code class="lang-auto">hc1 = tensor([[30],
              [20],
              [10],
              [50],
              [40]])

hc2=tensor([[1],
            [2],
            [3],
            [4],
            [5]])
</code></pre>
<p>1, 2, 3, 4, 5  are, subsequently, related to 30, 20, 10, 50, 40.<br>
After sorting hc1 we would have 10, 20, 30, 40, 50 and 3, 2, 1, 5, 4. Because hc1 and hc2 are 1 to 1.<br>
I thought that I should concatenate them as hc3 and sort it based on hc1.<br>
I hope that I explained my question as well as possible.</p>
<p>Thanks</p>

phan_phan: <p>Hi,<br>
You can get the permutation by calling <code>perm = hc1.argsort(dim=0).squeeze()</code>.<br>
In your example it returns <code>tensor([2, 1, 0, 4, 3])</code></p>
<p>Then you can rearrange hc2 with this permutation : <code>hc2_rearranged = hc2[perm]</code>.<br>
In your example it returns</p>
<pre><code class="lang-auto">tensor([[3.],
        [2.],
        [1.],
        [5.],
        [4.]])
</code></pre>
<p>You can always concatenate those afterwards</p>

zahra: <p>Thanks for your reply.<br>
I can not understand how to have sorted hc1 while you achieved sorted indices of hc1? Is there any way?</p>
<p>Thanks</p>

zahra: <aside class="quote no-group" data-username="phan_phan" data-post="4" data-topic="57036">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phan_phan/48/51960_2.png" class="avatar"> phan_phan:</div>
<blockquote>
<p><code>hc2_rearranged = hc2[perm]</code> .</p>
</blockquote>
</aside>
<p>Dear Phan_Phan,</p>
<p>Oh, how funny question I asked. It just needs to write:</p>
<pre><code class="lang-auto">print(hc1[perm])
</code></pre>
<p>Many thanks</p>
What's the most efficient way to convert a vector into a triangular matrix?

Alaya-in-Matrix: <p>For example, I want to convert  torch.tensor([1,2,3,4,5,6] to torch.tensor([[1,0,0],[2,3,0],[4,5,6]]), what is the most efficient way to do the conversion?</p>

SimonW: <p>Currently, take <code>tril_indices</code> from numpy, and use advanced indices to assign to an all zeros matrix.</p>

CNguyen: <p>Beside numpy, PyTorch has now implemented <code>tril_indices</code>.</p>
<pre><code class="lang-auto">x = torch.tensor([1., 2., 3., 4., 5., 6.])
m = torch.zeros((3, 3))

tril_indices = torch.tril_indices(row=3, col=3, offset=0)
m[tril_indices[0], tril_indices[1]] = x
</code></pre>
The Gather problem in DataParallel: dimension are not matched

Shaobo_Min: <h2><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h2>
<p>During my application, a strange bug is that my model works well with single GPU but fails in multi-GPUs by:<br>
<strong>RuntimeError: Gather got an input of invalid size: got [24, 10, 448，448], but expected [24, 11, 448，448] (gather at /pytorch/torch/csrc/cuda/comm.cpp:239</strong><br>
My input size is [48,3,448,448], and two GPUs are used. Thus, it is ok to split 24 images into each gpu, but it is strange why exists 10 and 11 channels?</p>
<p>After debug, the problem is found out:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b9fa427028107de72bbaf580dbeb1fca29affe3.png" alt="image" data-base62-sha1="d4xtJnFxxC2AJxq1GQjEhYdP8fF" width="477" height="91"><br>
The self.embed_arr is an input semantic labels, whose size is [21,300].<br>
self.embed_arr will affect the image feature channels.<br>
Under single GPU setting, each image will meet a same self.embed_arr, thereby having the same image channels.<br>
However, under multi-GPUs settings, self.embed_arr will be split into multi-parts, e.g., [10,30] and [11,30], thereby leading to different image channels and a bug during feature gathering.<br>
( If self.embed_arr has a size of [20,300], this problem will not appear, and I may think that the bad performance is attributed to my algorithm, which is terrible!)</p>
<h2><a name="solution-2" class="anchor" href="#solution-2"></a>Solution</h2>
<p>An alternative solution is to duplicate the input so that the scattered inputs in each GPU is the same by:<br>
self.embed_arr = self.embed_arr.repeat(len(range(torch.cuda.device_count())),1)</p>
<h2><a name="suggest-3" class="anchor" href="#suggest-3"></a>Suggest</h2>
<p>So, I suggest pytorch to add a function that can ontrol which inputs should be scattered to different GPUs. Or, is there any better solutions?</p>
To slice up sequence of data into shorter sequences

Dudly01: <p>Hello!<br>
I have sequences of data created from multiple agents playing a game for a given amount of timesteps in synchron, thus with a shape of (<span class="hashtag">#agents</span>, <span class="hashtag">#timesteps</span>, x, …), where the data denoted by ‘x, …’ can be of any shape.</p>
<p>What would be an easy way to slice the ‘timestep’ long sequences of data into smaller sequential parts?</p>
<p>A specific example (with a picture as data):<br>
From (32, 1024, 3, 28, 28) into (128, 256, 3, 28, 28), as we create 256 long sequences from 1024 long sequences and as a result increase our “number” of data from 32 to 4*32=128.</p>
<p>My only guess would be to make a fancy indexing such as one would do on a list of lists of lists, but there must be a better way.</p>
<p>Thanks is advance.</p>

Dudly01: <p>I might have come up with a solution, but i am not sure whether the sequence order would be intact:</p>
<pre><code class="lang-auto"> # Long sequenced data
a = torch.randn((32, 1024, 3 , 28, 28)) 

# Get the original shape and modify it
b = [i for i in a.size()]  
b[0] = -1
b[1] = int(b[1] / 4) 

# Desired length sequences
c = a.view(b) 
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt;c.shape
torch.Size([128, 256, 3, 28, 28])
</code></pre>

spanev: <p>Hi <a class="mention" href="/u/dudly01">@Dudly01</a>,</p>
<p>Yes this is the way to go, why would the sequence order not be intact?<br>
<code>view</code> doesn’t modify the underlying sequence, just how you “see” it.</p>
<p>and you can simply do this to get the shape list:</p>
<pre><code class="lang-auto">b = list(a.size())  
</code></pre>