akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Transformer learning gets worse with more encoder blocks

Gabor_Erdos: <p>I am following this tutorial and tweaked it slightly to my data:<br>
<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>In my data the input is a set of letters (80 amino acid codes) where the vocabulary is 21 (20 amino acids and one padding letter) and the output is a single continuous number. What I did was I dropped masking, and changed the decoder to output a single value instead of ntokens.</p>
<p>The parameters that I used:<br>
ntokens = 22  # size of vocabulary, needs to be even for some reason<br>
emsize = 30  # embedding dimension<br>
d_hid = 256  # dimension of the feedforward network model in nn.TransformerEncoder<br>
nlayers = 2  # number of nn.TransformerEncoderLayer in nn.TransformerEncoder<br>
nhead = 2  # number of heads in nn.MultiheadAttention<br>
dropout = 0.1  # dropout probability</p>
<p>I have around 1 million data points, which is not that much. Now when I train this network I get a relatively nice loss curve, it trains to around ~1.5 MSE.</p>
<p>However, when I try to increase the nlayers to a higher number (I read in other tutorials that it should be around 6) my network fails to train. Even with 4 layers the MSE loss stay the same for 100 epochs.</p>
<p>What am I missing? I thought if I have low amount of data, the network should over train and I should get a very nice train loss, and bad validation/test loss, however this is not the case.</p>
Single forwarding vs double forwarding the same input

cneyang: <p>“Why do single forward and double forwarding the same input to compute the same losses make difference?”</p>
<p>Suppose we have two loss terms for updating the model.<br>
We normally forward the input a single time and compute the two loss terms using the output. Then the gradients for each loss term will be added to the model parameters. Like code below.</p>
<pre><code class="lang-auto">model = models.resnet50().cuda()
model.fc = nn.Identity()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

x = torch.randn(128, 3, 224, 224).cuda()
y = torch.randn(128, 2048).cuda()

# single forward as usually
for _ in range(10):
    z1 = model(x)
    loss1 = torch.mean(z1)
    loss2 = F.mse_loss(z1, y)
    loss = loss1 + loss2
    print(f"loss1: {loss1.item():.4f}, loss2: {loss2.item():.4f}, loss: {loss.item():.4f}")
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Suppose, for some reason (although it is inefficient and not desired), we forward the same input twice to compute each loss term. As code below.</p>
<pre><code class="lang-auto"># double forward
for _ in range(10):
    z1 = model(x)
    z2 = model(x)
    loss1 = torch.mean(z1)
    loss2 = F.mse_loss(z2, y)
    loss = loss1 + loss2
    print(f"loss1: {loss1.item():.4f}, loss2: {loss2.item():.4f}, loss: {loss.item():.4f}")
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>I thought both ways would accumulate the same gradient, however, they behave in a different way even though the random seed is fixed and cuda deterministic is on.<br>
Why do they make difference?</p>

ptrblck: <p><code>(z1 - z2).abs().max()</code> returns <code>tensor(0., device='cuda:0', grad_fn=&lt;MaxBackward1&gt;)</code> for me so you would need to explain your use case and issue in more detail and what exactly you are comparing.</p>
Does torch.compile use FlashAttention？

dancedpipi: <p>I tested the performance of <code>torch.compile</code> on the <code>bert-base</code> model on the <code>A100</code> machine, and found that the training performance has been greatly improved.<br>
I wonder if <code>flashattention</code> is used under <code>torch.compile</code>. Is there an option to make <code>torch.compile </code>disabled <code>flashattention</code></p>

marksaroufim: <p>Take a look at this tutorial <a href="https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html#beta-implementing-high-performance-transformers-with-scaled-dot-product-attention-sdpa" class="inline-onebox" rel="noopener nofollow ugc">(Beta) Implementing High-Performance Transformers with Scaled Dot Product Attention (SDPA) — PyTorch Tutorials 2.0.0+cu117 documentation</a></p>
<p>Specifically take a look at the <code>backend_map</code> dict, pass them in to an <code>sdp_kernel</code> context manager and then <code>torch.compile</code> your model within that scope to disable</p>
<p>But to answer your first question, yes I believe it is enabled by default now with or without <code>torch.compile</code></p>

dancedpipi: <p>Thank you for your reply.<br>
I didn’t use <code>torch.nn.functional.scaled_dot_product_attention</code> and <code>torch.nn.MultiheadAttention</code>, I used the self-attention structure of transformers, looks like this:</p>
<pre><code class="lang-python">mixed_query_layer = self.query(hidden_states)
query_layer = self.transpose_for_scores(mixed_query_layer)
key_layer = self.transpose_for_scores(self.key(hidden_states))
value_layer = self.transpose_for_scores(self.value(hidden_states))

attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
attention_scores = attention_scores / math.sqrt(self.attention_head_size)
attention_probs = nn.functional.softmax(attention_scores, dim=-1)

context_layer = torch.matmul(attention_probs, value_layer)
context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
context_layer = context_layer.view(new_context_layer_shape)
</code></pre>
<p>I wonder if torch.compile will use flash attention in this case? Is it possible to find the generated triton code locally</p>

marksaroufim: <p>cc <a class="mention" href="/u/drisspg">@drisspg</a> who might know how to generate the triton kernels - I know how to do this for inductor only</p>

HamidShojanazeri: <p><a class="mention" href="/u/dancedpipi">@dancedpipi</a>, torch.compile() does not generate flash or memory-efficient kernels at this point. They are custom kernels that get called for now if you are using <code>torch.nn.functional.scaled_dot_product_attention</code> or <code>torch.nn.MultiheadAttention</code>.</p>

dancedpipi: <p>Does that mean that if I don’t use <code>torch.nn.functional.scaled_dot_product_attention</code> or <code>torch.nn.MultiheadAttention</code> and only use torch.compile, it won’t use flash attention or memory-effective kernel?</p>
Shuffling the input before the model and shuffling the output after the model being not consistent on CUDA

cneyang: <p>Shuffling the input before feeding it into the model and shuffling the output the model output produces different outputs.<br>
I think it has something to do with GPU and batch norm since the problem only happens in train mode only on CUDA not CPU.<br>
Does anyone know why this is happening?</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models

model = models.resnet50()

model = model.cuda()
idx = torch.randperm(128).cuda()
x = torch.randn(128, 3, 224, 224).cuda()

print("input indexing", model(x[idx]))
print("output indexing", model(x)[idx])
</code></pre>

cneyang: <p>input indexing tensor([[-0.8570,  0.8326, -0.5860,  …,  0.8055, -0.0580,  0.1908],<br>
[-1.0155,  0.6950, -0.3823,  …,  0.7518,  0.2098,  0.2948],<br>
[-0.9844,  0.6219, -0.2615,  …,  0.8177,  0.1248,  0.2177],<br>
…,<br>
[-0.9578,  0.5173, -0.1964,  …,  0.6827,  0.0789,  0.2514],<br>
[-1.0569,  0.5965, -0.3957,  …,  0.6497,  0.0286,  0.3090],<br>
[-1.0232,  0.6058, -0.2303,  …,  0.6736,  0.1143,  0.4659]],<br>
device=‘cuda:0’, grad_fn=)<br>
output indexing tensor([[-0.8633,  0.8343, -0.5875,  …,  0.8065, -0.0620,  0.1971],<br>
[-1.0178,  0.6932, -0.3813,  …,  0.7535,  0.2155,  0.2961],<br>
[-0.9846,  0.6146, -0.2619,  …,  0.8111,  0.1284,  0.2190],<br>
…,<br>
[-0.9474,  0.5150, -0.1974,  …,  0.6781,  0.0821,  0.2537],<br>
[-1.0614,  0.5949, -0.3894,  …,  0.6516,  0.0253,  0.3094],<br>
[-1.0255,  0.6015, -0.2268,  …,  0.6716,  0.1218,  0.4665]],<br>
device=‘cuda:0’, grad_fn=)</p>
<p>Output is like this above.</p>

cneyang: <p>But the mean and variance for each feature channel will not be changed whether the input is shuffled or not.<br>
If <code>idx</code> is only selecting a subset of the batch, the batch statistics will change but <code>idx</code> is just permuting the entire batch.</p>

ptrblck: <p><code>nn.BatchNorm2d</code> layers will take the stats from dims <code>[0, 2, 3]</code> and indeed shuffling the samples should not change the applied function. However, you are changing the order of operations (or rather samples in this case), which might show small errors due to the limited floating point precision order.<br>
Here is a small example using only the <code>sum</code> operation:</p>
<pre><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)
s3 = x[torch.randperm(x.size(0))].sum()
print(s1 - s2)
# tensor(-5.7220e-06)
print(s1 - s3)
# tensor(1.1444e-05)
</code></pre>
<p>All three approaches calculate the same desired sum but due to a change in the operation/sample order the outputs show small errors and you might see the same effect in your model.</p>
Question about the calculation of adding certain SNR background noise to audio in tutorial

summernstar: <p>The tutorial address is as followed:<br>
<a href="https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html#adding-background-noise" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html#adding-background-noise</a><br>
The part confuses me most is in this picture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/597107e3244635fed77464a631c31244f73b070b.png" data-download-href="https://discuss.pytorch.org/uploads/default/597107e3244635fed77464a631c31244f73b070b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/597107e3244635fed77464a631c31244f73b070b.png" alt="image" data-base62-sha1="cLeF3JLxqNajCw6b2FnnYsThmEr" width="594" height="500" data-dominant-color="EEECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">719×605 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My question is: how does this code work? what’s the principle of mathmatics and derivative process behind it?<br>
Any reference is appreciated.</p>

nateanl: <p>According to <a href="https://en.wikipedia.org/wiki/Signal-to-noise_ratio" rel="noopener nofollow ugc">Wikipedia</a></p>
<p><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/7a7b11f600ce6e7defb4f20ffa8071818b6baa14" alt="{athrm {SNR_{{dB}}}}=10og {{10}}eft({rac {P{{athrm {signal}}}}{P_{{athrm {noise}}}}}ight)." width="204" height="44"></p>
<p>Given a desired SNR value, you just need to scale your speech signal so that the output of the above formula is equal to the desired one.</p>

summernstar: <p>Thanks, I understand now. But why divided by 2 at last?</p>

nateanl: <p>It’s just a practical operation to avoid value overflow. If the dtype of the waveform is float, the value range must be [-1, 1]. I would recommend dividing the noisy speech signal by the max absolute value, which can guarantee such range.</p>
How does PyTorch dist ranks map to NCCL ranks?

Olivier-CR: <p>when doing an allreduce with PT distributed, NCCL is used. In case we use NCCL ring, is the order of the NCCL ring defined by the order of the ranks set in PyTorch <code>dist.init_process_group(rank=rank)</code> ?</p>

marksaroufim: <p>It is not controlled from PyTorch directly and NCCL doesn’t provide control to this ordering via enivornment variables but you can take a look at <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html" class="inline-onebox" rel="noopener nofollow ugc">Environment Variables — NCCL 2.17.1 documentation</a> and see if anything resonates</p>
Is there an equivalent of jax.lax.scan (eg in torch.func)?

murphyk: <p>I would like to translate the following jax code (that implements a Kalman filter) to torch.</p>
<pre><code class="lang-auto">def kf(params, emissions, return_covs=False):
    F, Q, R = params['F'], params['Q'], params['R']
    def step(carry, t):
        ll, pred_mean, pred_cov = carry
        H = params['Ht'][t]
        y = emissions[t]
        ll += MVN(H @ pred_mean, H @ pred_cov @ H.T + R).log_prob(y)
        filtered_mean, filtered_cov = condition_on(pred_mean, pred_cov, H, R, y)
        pred_mean, pred_cov = predict(filtered_mean, filtered_cov, F, Q)
        carry = (ll, pred_mean, pred_cov)
        if return_covs:
            return carry, (filtered_mean, filtered_cov)
        else:
            return carry, (filtered_mean, None)     
    
    num_timesteps = len(emissions)
    carry = (0.0, params['mu0'], params['Sigma0'])
    (ll, _, _), (filtered_means, filtered_covs) = lax.scan(step, carry, jnp.arange(num_timesteps))
    return ll, filtered_means, filtered_covs
</code></pre>
<p>The torch code below works, but is ~5x slower (on CPU), presumably because of the for loop.<br>
Is there an equivalent of scan?</p>
<pre><code class="lang-auto">def kf_pt(params, emissions, return_covs=False, compile=False):
    F, Q, R = params['F'], params['Q'], params['R']
    def step(carry, t):
        ll, pred_mean, pred_cov = carry
        H = params['Ht'][t]
        y = emissions[t]
        #ll += MVN(H @ pred_mean, H @ pred_cov @ H.T + R).log_prob(y)
        filtered_mean, filtered_cov = condition_on_pt(pred_mean, pred_cov, H, R, y)
        pred_mean, pred_cov = predict_pt(filtered_mean, filtered_cov, F, Q)
        carry = (ll, pred_mean, pred_cov)
        if return_covs:
            return carry, (filtered_mean, filtered_cov)
        else:
            return carry, filtered_mean
    
    if compile:
        step = torch.compile(step)
    num_timesteps = len(emissions)
    D = len(params['mu0'])
    filtered_means = torch.zeros((num_timesteps, D))
    if return_covs:
        filtered_covs = torch.zeros((num_timesteps, D, D))
    else:
        filtered_covs = None
    ll = 0
    carry = (ll, params['mu0'], params['Sigma0'])
    for t in range(num_timesteps):
        if return_covs:
            carry, (filtered_means[t], filtered_covs[t]) = step(carry, t)
        else:
            carry, filtered_means[t] = step(carry, t)
    return ll, filtered_means, filtered_covs
</code></pre>

marksaroufim: <p>Doesn’t exist quite yet but feel free to add your support in this thread where it’s being discussed <a href="https://github.com/pytorch/pytorch/issues/50688" class="inline-onebox" rel="noopener nofollow ugc">[feature request] torch.scan (also port lax.fori_loop / lax.while_loop, lax.cond / lax.switch?) · Issue #50688 · pytorch/pytorch · GitHub</a></p>
Should I use GPU or TPU on Google Colab?

raushankm: <p>On Google colab they provide options for both gpu and the tpu? Which is better to use in terms of memory usage? Does TPU usage memory the same way as GPU? In GPU I am frequently getting out of memory error? Can that be avoided with TPU?</p>

marksaroufim: <p>This is a hard question to answer without more details on your workload. Google Colab might not provide you to the same GPU or TPU every time you login, typically it’s best to benchmark and see. So benchmark and see!</p>
Appending dimension only

ammar_naich: <p>I have two tensors <strong>a = torch.Size([1, 566757, 3])</strong> and <strong>b =  torch.Size([1, 566757, 16])</strong> and want the last feature dimension to be merged to result in tensor <strong>c = torch.Size([1, 566757, 19])</strong>.</p>
<p>Will appreciate the help if any one can suggestion a work around.</p>
<p>Thanks</p>

arunppsg: <p>You could use <code>torch.cat</code>.</p>
<p>Example</p>
<pre><code class="lang-auto">import torch

a = torch.randn(5, 10, 1)
b = torch.randn(5, 10, 6)

torch.cat((a, b), dim=2).shape
</code></pre>

ammar_naich: <p>that works, many thanks <a class="mention" href="/u/arunppsg">@arunppsg</a></p>
Torchaudio.load ignores normalize=False for 8 bit ulaw

robz: <p>According to <a href="https://pytorch.org/audio/stable/backend.html#load" rel="noopener nofollow ugc">the docs</a>, torchaudio.load(normalize=False) shouldn’t convert to floats when loading wav files. But it appears to ignore normalize=False when the file uses 8 bit mu-law encoding:</p>
<pre><code class="lang-auto">audio = torch.sin(2 * torch.pi * torch.linspace(0, 1, 100))
torchaudio.save('temp.wav', audio[None, :], format='wav', encoding='ULAW', bits_per_sample=8, sample_rate=16_000)
info = torchaudio.info('temp.wav')
print(info) # sample_rate=16000, num_frames=100, num_channels=1, bits_per_sample=8, encoding=ULAW
data = torchaudio.load('temp.wav', normalize=False)
print(data[0].dtype) # torch.float32 ?! expected uint8
</code></pre>
<p>Testing with <code>torchaudio.__version__ == 0.13.1+cu116</code></p>

nateanl: <p>Hi <a class="mention" href="/u/robz">@robz</a>, I can reproduce the issue.<br>
To broadcast it to torchaudio maintainers, could you post it to the GitHub issue page? <a href="https://github.com/pytorch/audio/issues/new/choose" class="inline-onebox" rel="noopener nofollow ugc">Sign in to GitHub · GitHub</a></p>
<p>Thanks.</p>

robz: <p>Sure thing, posted here: <a href="https://github.com/pytorch/audio/issues/3256" class="inline-onebox" rel="noopener nofollow ugc">Torchaudio.load ignores normalize=False for 8 bit ulaw · Issue #3256 · pytorch/audio · GitHub</a></p>
Distributed training stucks with CUDA12.0

hubert233: <p>I am using torch2.0 with cuda12.0. The env is installed by <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code>. However, I found the distributed training stucks with no output. The GPU usage is normal, as I increase the batch size , the GPU usage will also rise. However, the model has no outputs for even one batch. The training script is here <a href="https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py" rel="noopener nofollow ugc">transformer_mlm</a>. I found it stucks at <code>outputs = self.parallel_apply(replicas, inputs, kwargs)</code> in data_parallel.py during and thus there are no outputs.<br>
I did a simple verification with this <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">data parallel example</a>. Also, I got no outputs for <code>output = model(input)</code>.</p>
<p>I am just curious, does pytorch2.0 does not support data parallel with cuda12.0? Do I need to downgrade cuda or pytorch?</p>
<p>Thanks a lot in advance.</p>

ptrblck: <p>The posted install command will use the CUDA 11.8 runtime with their corresponding math libs, not 12.0.<br>
Are you able to execute the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html"><code>DDP</code> tutorial</a> successfully?</p>

hubert233: <p>Thanks for replying! In my env, I can run the DDP tutorial, but I still cannot get the data parallel example passed. I just tested on another machine with the same env but with CUDA11.8 installed. It works. So I guess the issue may be raised by the CUDA version.</p>

ptrblck: <p>The PyTorch binaries ship with their own CUDA libs as already mentioned and I doubt your locally installed CUDA toolkit is related to the mentioned issue, as it won’t even be used unless you build PyTorch from source or a custom CUDA extension.</p>
<p>Good to hear <code>DDP</code> is working as it’s the recommended way since <code>DataParallel</code> is in maintenance mode.</p>
Using torch.onnx.export came across "collections. OrderedDict' object is not callable”

chenpirate: <p>Hello, I have some problems, can anyone help me?</p>
<p>I am using my trained Resnet34 network, and my attempt to convert the trained network to ONNX model fails, and the error is as follows.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/private/hrrp/NanHu/resnetV2/model.py", line 216, in &lt;module&gt;
    torch.onnx.export(model,
  File "/home/private/anaconda3/envs/resnet34/lib/python3.8/site-packages/torch/onnx/utils.py", line 504, in export
    _export(
  File "/home/private/anaconda3/envs/resnet34/lib/python3.8/site-packages/torch/onnx/utils.py", line 1506, in _export
    with exporter_context(model, training, verbose):
  File "/home/private/anaconda3/envs/resnet34/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/private/anaconda3/envs/resnet34/lib/python3.8/site-packages/torch/onnx/utils.py", line 176, in exporter_context
    with select_model_mode_for_export(
  File "/home/private/anaconda3/envs/resnet34/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/private/anaconda3/envs/resnet34/lib/python3.8/site-packages/torch/onnx/utils.py", line 137, in disable_apex_o2_state_dict_hook
    for module in model.modules():
TypeError: 'collections.OrderedDict' object is not callable
</code></pre>
<p>Then I tried the official sample code and the same error.</p>
<pre><code class="lang-python">import torch
import torchvision


dummy_input = torch.randn(10, 3, 224, 224, device="cuda")
model = torchvision.models.alexnet(pretrained=True).cuda()

input_names = [ "actual_input_1" ] + [ "learned_%d" % i for i in range(16) ]
output_names = [ "output1" ]

torch.onnx.export(model, dummy_input, "alexnet.onnx", verbose=True, input_names=input_names, output_names=output_names)
</code></pre>
<p>My Pytorch is 1.13.1。</p>

ptrblck: <p>I cannot reproduce the issue using <code>torch==2.0.0+cu118</code> and <code>onnx==1.13.1</code> and get:</p>
<pre><code class="lang-python">dummy_input = torch.randn(10, 3, 224, 224, device="cuda")
model = torchvision.models.alexnet(pretrained=True).cuda()

input_names = [ "actual_input_1" ] + [ "learned_%d" % i for i in range(16) ]
output_names = [ "output1" ]

torch.onnx.export(model, dummy_input, "alexnet.onnx", verbose=True, input_names=input_names, output_names=output_names)

import onnx

model = onnx.load("alexnet.onnx")
onnx.checker.check_model(model)
# Print a human readable representation of the graph
print(onnx.helper.printable_graph(model.graph))
# graph torch_jit (
#   %actual_input_1[FLOAT, 10x3x224x224]
# ) initializers (
#   %learned_0[FLOAT, 64x3x11x11]
#   %learned_1[FLOAT, 64]
#   %learned_2[FLOAT, 192x64x5x5]
# ...
</code></pre>

chenpirate: <p>Thank you, I found out that my pytorch was not installed completely for some reason, and it can be used after I reinstall it.</p>
[fixed] (actually pillow-simd crashed)Torchvision.transform.resize crashed python kernel

yan564693678: <p>To reproduce : python 3.9<br>
torchvision 0.15.1+cu117<br>
pytoch 2.0.0+cu117<br>
pillow-simd 9.0<br>
I tried when using pytorch0.41, python 3.6.9 and normal pillow, no such bug happend.<br>
I tried uninstalling pillow-simd and using pillow 9.2 instead, the bug fixed.<br>
bugged code:</p>
<pre><code class="lang-auto">from PIL import Image  

import torch
import torch.utils.data as data
import torchvision
from torchvision import transforms
print(torchvision.__version__)
print(torch.__version__)
datapath = r'M:/datas/kitti_raw\2011_09_26/2011_09_26_drive_0014_sync\image_02/data\0000000016.png'
im = Image.open(datapath)
im = im.convert('RGB')
im = transforms.Resize((200,200))(im)
print (type(im))
</code></pre>
<blockquote>
<p>Windows fatal exception: code 0xc000001d</p>
<p>Main thread:<br>
Current thread 0x00005894 (most recent call first):<br>
File “C:\Users\yan56\AppData\Roaming\Python\Python39\site-packages\PIL\Image.py”, line 1980 in resize<br>
File “C:\Users\yan56\AppData\Roaming\Python\Python39\site-packages\torchvision\transforms_functional_pil.py”, line 250 in resize<br>
File “C:\Users\yan56\AppData\Roaming\Python\Python39\site-packages\torchvision\transforms\functional.py”, line 490 in resize<br>
File “C:\Users\yan56\AppData\Roaming\Python\Python39\site-packages\torchvision\transforms\transforms.py”, line 361 in forward<br>
File “C:\Users\yan56\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py”, line 1501 in _call_impl<br>
File “C:\Users\yan56\Desktop\monocular-depth-prediction\monodepth2-master\datasets\untitled0.py”, line 22 in <br>
File “C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 465 in exec_code<br>
File “C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 585 in runfile<br>
File “C:\Users\yan56\AppData\Local\Temp\ipykernel_13700\484932580.py”, line 1 in </p>
</blockquote>

ptrblck: <p>Since the issue was isolated to <code>PIL-SIMD</code> I would recommend posting it with the minimal code snippet in their GitHub repository so that the developers could try to reproduce and fix the issue.</p>
How to train a classifier to return logits for three separate labels

mttbrv: <p><strong>Preamble</strong><br>
Hello, I’m trying to solve a classification task where, given a text sample, I have to infer three labels: <code>label_1_binary</code>, <code>label_2_binary</code> and <code>label_3_multiclass</code>. Two of them are binary i.e. they can only take one of two values. The third label is multi-class i.e. it can take one of <em>n</em> possible values.</p>
<p>My idea was to fine-tune BERT, retrieve the CLS token, feed it to a “common layer” and then use the output of the latter in three separate linear layers to retrieve the logits for the classification. Unfortunately, something is not working. The F1 scores I get are extremely low, even after taking care of the unbalanced classes and tuning the hyper-parameters with Ray-Tune.</p>
<p><strong>Question</strong><br>
Being new to NLP and Pytorch, I’d appreciate your feedback on the following points.</p>
<ol>
<li>Is the structure of the model conceptually correct i.e. can the model be trained to return three separate logits for the labels I’m trying to infer?</li>
<li>Or would it be better to have the model return a single logit and then process it with three separate classifiers e.g. MLP?</li>
</ol>
<p><strong>The model</strong></p>
<pre><code class="lang-python">class CustomModel(nn.Module):
    def __init__(self, drop_rate: float = 0.15) -&gt; None:
        super().__init__()

        self._language_model = BertModel.from_pretrained(
            "dbmdz/bert-base-italian-uncased"
        )
        self._common_layer = nn.Sequential(
            nn.Linear(768, 768),
            nn.Tanh(),
            nn.Dropout(drop_rate)
        )
        self._label_1_bin_layer = nn.Sequential(
            nn.Linear(768, 1)
        )
        self._label_2_bin_layer = nn.Sequential(
            nn.Linear(768, 1)
        )
        self._label_3_mul_layer = nn.Sequential(
            nn.Linear(768, 4)
        )

    def forward(
        self,
        input_ids: torch.Tensor,
        token_type_ids: torch.Tensor,
        attention_mask: torch.Tensor
    ) -&gt; Dict[str, torch.Tensor]:
        CLS_representation = self._language_model(
            input_ids=input_ids,
            token_type_ids=token_type_ids,
            attention_mask=attention_mask
        )[0][:, 0, :] # type: ignore

        common_layer = self._common_layer(CLS_representation)

        return {
            "label_1_binary": self._label_1_bin_layer(common_layer),
            "label_2_binary": self._label_2_bin_layer(common_layer),
            "label_3_multiclass": self._label_3_mul_layer(common_layer)
        }
</code></pre>

ptrblck: <p>Yes, creating multiple classifier “heads” is a valid approach. I don’t know how your data is distributed but would recommend scaling down the use case to try to overfit a small dataset (e.g. just 10 samples) by adapting the hyperparameters.</p>
Unrolling the model graph in a static fashion

neel_g: <p>I’m using pytorch on TPUs, and wish to implement an early exit for my layers to stop execution.</p>
<p>Say for simplicity’s sake, I have a single block - <code>myblock()</code> that I want to forward through multiple times:</p>
<pre><code class="lang-py">iters = randrange(1,10)
input : torch.tensor = ...

outputs  = ... # preallocate `iters` lenght tensor, to be filled at each iteration
for i in range(iters):
    interim_tensor = myblock(input)
    outputs[i] = interim_tensor

if is_training:
    return interim_tensor

if is_inferring:
    return outputs # we want a log of intermediate results too, just in case
</code></pre>
<p>Now, because <code>iters</code> is stochastic, this graph is generated dynamically.</p>
<p>However, on <strong>TPUs</strong> this might trigger a recompilation everytime <code>iters</code> changes slowing down the training.</p>
<p>So how do I convert this to a static graph? I <em>do</em> know the maximum number of iterations I can use - defined through <code>randrange</code>.</p>
<p>So perhaps I can forward through every layer and extract <code>i</code>th tensor I want? I could use <code>torch.where</code> but I’m worried I’ll be proprogating the wrong gradients elsewhere since those ops would be recorded on the graph too - whereas I <em>want</em> them on the graph (so it remains static) just that they shouldn’t have any effect on the backprop because I used an intermediate tensor instead.</p>
<p>Anyone have any ideas?</p>

neel_g: <p>Anyone any ideas <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"> ? could use some help here</p>
Music demixing (spectrogram2spectrogram)

deadman: <p>Has anyone made spectrogram 2 spectrogram models for music analysis, specifically demixing such as isolating vocals? I’m currently using a Mel spectrogram for the input and output but struggling to get good results. Using a hop length=512,n_fft=2048, no_mels=128. My model is currently a bi directional GRU model with 3 layers and a 256 hidden size.Does anyone know a good model type to use and/or good audio transformations for this project?</p>
<p>Cheers</p>

JuanFMontesinos: <aside class="onebox allowlistedgeneric" data-onebox-src="https://ipcv.github.io/Acappella/">
  <header class="source">

      <a href="https://ipcv.github.io/Acappella/" target="_blank" rel="noopener nofollow ugc">Acappella</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://ipcv.github.io/Acappella/" target="_blank" rel="noopener nofollow ugc">Acappella</a></h3>

  <p>A cappella: Audio-visual Singing Voice Separation Juan F. Montesinos *   Venkatesh S. Kadandale * Gloria Haro Universitat Pompeu Fabra, Barcelona, Spain Paper Code + Weights Dataset Demos The paper has been accepted to BMVC 2021! Abstract The...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It’s called source separation (if isolating all sources) or enhancement if isolating a specific one.<br>
I think you can google about it.</p>

nateanl: <p>Juan’s model is very fascinating <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<p>If you only have audio signal for demixing, you can also try the Hybrid Demucs model, which is built in torchaudio: <a href="https://pytorch.org/audio/stable/tutorials/hybrid_demucs_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Music Source Separation with Hybrid Demucs — Torchaudio 2.0.1 documentation</a></p>
<p>There are other tools that can also do music separation, such as</p>
<ul>
<li>Open-Unmix <a href="https://github.com/sigsep/open-unmix-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - sigsep/open-unmix-pytorch: Open-Unmix - Music Source Separation for PyTorch</a>
</li>
<li>AudioShake <a href="https://www.audioshake.ai/" rel="noopener nofollow ugc">https://www.audioshake.ai/</a>
</li>
<li>lalal.ai <a href="https://www.lalal.ai/" rel="noopener nofollow ugc">https://www.lalal.ai/</a>
</li>
</ul>