Question:
Hi,
Thanks for this great project! I want to adjust the learning rate in opacus. When I create scheduler before attaching optimizer just like the following code.
model = Net()
optimizer = SGD(model.parameters(), lr=0.05)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[30,80], gamma=0.1)
privacy_engine = PrivacyEngine(
    model,
    batch_size,
    sample_size,
    alphas=[10, 100],
    noise_multiplier=1.3,
    max_grad_norm=1.0,
)
privacy_engine.attach(optimizer)

It raises UserWarning: Seems like optimizer.step() has been overridden after learning rate scheduler initialization. What’s the potential risk in the above code?
I also try to attach optimizer before learning rate scheduler initialization, thus this warning goes away. Is it necessary?
Any help is appreciated!

Context:
Hello @hkz,
Thank you for surfacing this quirk.
Opacus does indeed override the optimizer.step() when privacy engine is attached to it. In the current implementation, we perform the per-sample gradient clipping and subsequently call the original step() function of the optimizer (opacus/privacy_engine.py at master · pytorch/opacus · GitHub 1). Therefore, your code should work correctly regardless of when you initialize the scheduler.
However, this is risky as it might not work correctly should the implementation of optimizer.step() change in a manner inconsistent with the scheduler’s expectations; although, I don’t see this happening.
Bottom line: heeding the warning is ideal and recommended.


Some questions about (ϵ,δ)-differential privacy guarantee

Question:
Hi all.
I apply the opacus package to Unet training, I don`t know how to judge the privacy guarantee level of Unet. Is it related to the output of ϵ? Does ϵ less than a certain value mean that privacy guarantee is good.
However, ϵ keeps increasing during the training process. Does this mean that the longer the training time, the lower the privacy protection level?
Sincerely looking forward to receiving your reply.

Context:
Hi @liuwenshuang0211.
First of all, thanks for trying out opacus, we do appreciate it.
To better understand the concept of (ε,𝛿) - differential privacy, I suggest starting with FAQ section on our website, we have a paragraph on that: FAQ · Opacus 9
tl;dr - Epsilon defines the multiplicative difference between two output distributions based on two datasets, which differ in a single example. In other words, epsilon is a measure of how much difference a single example can make under the worst possible circumstances.
Delta, in turn, defines the probability of failure, in which we don’t uphold privacy guarantee defined by epsilon.
Now, the question about threshold values for “good privacy guarantee” is tricky and hard.
For delta you typically want it to be orders of magnitude less than 1/N, where N is the number of entries in your dataset. Otherwise you don’t protect against full release of one entry.
For epsilon, the answer is “it depends”.
I’d suggest to look at papers to see what’s considered reasonable for your task.
For example, on MNIST we’re able to get acceptable results with ε=1.19 and good results with ε=7.
In another example, paper “Tempered Sigmoid Activations for
Deep Learning with Differential Privacy” by Papernot et al uses ε=3 for MNIST and ε=7 for CIFAR10.
There’re also some evidence, that epsilon values reported by DP-SGD are on a very pessimistic end, and the observed privacy is much better (roughly shrinks epsilon by a factor of 10)
To summarise this part, you’d want to set your 𝛿 << 1/N, and to have your ε in single digits (but it depends)
As for the training process, the short answer is yes - the more training iterations you perform, the weaker privacy guarantee is. Bear in mind, however, that the relation is sub-linear.
For more math behind it I would suggest looking into the original DP-SGD paper (https://arxiv.org/pdf/1607.00133.pdf 3), specifically section 3.1 Differentially Private SGD Algorithm, subsection “Privacy Accounting”


Question:
Hi Opacus community,
I am looking for experiences / best practices for using DP with transfer learning. Let’s say a hospital decides to build a DP image classification model based on patient data.
3 scenarios com to mind:

baseline network pre-trained on public dataset (from a similar domain, e.g. x-rays)
baseline network pre-trained on existing data of the hospital (classic, non private way)
baseline network pre-trained with DP on existing data of the hospital

I would assume that generally all approaches could make sense because probably less epochs are required compared to training from scratch (hence, spending less privacy budget).
I would appreciate learning your experiences / thoughts about these scenarios.
Thanks
Andreas

Context:
Hello @Andreas_Kopp !
With transfer learning, there are two separate datasets to consider: one used for training the baseline model and another for fine-tuning/training your final model.
In scenarios 1, the privacy of the public dataset is not preserved, but the privacy of the final model is preserved if you train with DP.
In scenario 2, just as in scenario 1, the privacy of the dataset used to train your baseline model is NOT preserved, but only that of the data used to train your final model is. If preserving the data used to train your baseline model is not important for your use case, this might result in better accuracy than that of scenario 1 (depending, ofcourse, on your task as well as the size and distribution of this data compared the public dataset)
In scenario 3, the privacy of all the data is preserved, but the accuracy will be very low.
Hope this adds some clarity.


Question:
Hi all,
I am wondering if anyone has figured out any “standard” hyperparameter configurations for obtaining the best accuracy at a given epsilon privacy level on CIFAR-10?
If so, it might be a good idea to include them in the tutorial scripts?
Thanks in advance!

Context:
I don’t think these experiments are published yet, but am sure that any contribution is welcome.
Would you be interested in working on such a tutorial?


Question:
(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters. Until you edit this description or create topics, this category won’t appear on the categories page.)
Use the following paragraphs for a longer description, or to establish category guidelines or rules:

Why should people use this category? What is it for?

How exactly is this different than the other categories we already have?

What should topics in this category generally contain?

Do we need this category? Can we merge with another category, or subcategory?

Context: It would be nice to have some links or something useful


Torchaudio.functional.lfilter runs very slow on gpu, but fast on cpu

Question:
As the title said, lfilter on cpu cost about 1 ms to run. But on the gpu, it is 1000x slower!  I posted my code and test result below. Could anyone tell me why and how to accelerate the speed on gpu?
device="cuda"
waveform = torch.rand([16000]).to(device)
a = torch.tensor([1.0, 0.0]).to(device)
b = torch.tensor([1.0, -0.97]).to(device)
for i in range(5):
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    waveform = torchaudio.functional.lfilter(waveform, a, b)
    end.record()
    torch.cuda.synchronize()
    print("run on %s cost %.3f ms" % (device, start.elapsed_time(end)))

Test result
run on cpu cost 1.473 ms
run on cpu cost 1.531 ms
run on cpu cost 0.905 ms
run on cpu cost 0.774 ms
run on cpu cost 1.007 ms
run on cuda cost 961.567 ms
run on cuda cost 955.971 ms
run on cuda cost 962.749 ms
run on cuda cost 957.605 ms
run on cuda cost 965.437 ms

My env is v100 gpu, cuda 11, torch ‘1.10.1+cu113’ and torchaudio ‘0.10.1+cu113’
Thank you very much!

Context:
If I’m not mistaken, this code 1 will be used, which will launch a lot of GPU kernels (nsys shows 160045 cudaLaunchKernel calls) which should create a lot of CPU overhead.
To reduce the overhead caused by the kernel launches, you could try to use CUDA graphs as described here 3 and see if it would match your use case.
Example:
import torch
import torchaudio
import time

device = 'cuda'
nb_iters = 100

# Placeholder input used for capture
waveform = torch.rand([16000]).to(device)
a = torch.tensor([1.0, 0.0]).to(device)
b = torch.tensor([1.0, -0.97]).to(device)

# warmup
for _ in range(10):
    waveform = torchaudio.functional.lfilter(waveform, a, b)

# profile
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    waveform = torchaudio.functional.lfilter(waveform, a, b)
torch.cuda.synchronize()
t1 = time.perf_counter()

print('Eager, {}s/iter'.format((t1 - t0)/nb_iters))

# CUDA graphs

g = torch.cuda.CUDAGraph()

# Warmup before capture
s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for _ in range(3):
        waveform = torchaudio.functional.lfilter(waveform, a, b)
torch.cuda.current_stream().wait_stream(s)

# Captures the graph
# To allow capture, automatically sets a side stream as the current stream in the context
with torch.cuda.graph(g):
    waveform = torchaudio.functional.lfilter(waveform, a, b)

torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    g.replay()
torch.cuda.synchronize()
t1 = time.perf_counter()

print('CUDA graphs, {}s/iter'.format((t1 - t0)/nb_iters))

Output:
Eager, 0.6360926309000934s/iter
CUDA graphs, 0.07975950390042272s/iter


Mechanism of represneting PCM16 by float32

Question:
Hello everyone, I am the novice to audio coding. I simply use “torchaudio” to load WAV files whose format is PCM16 with sample rate=16kHz and bit depth=16bits. In addition, datatype of loaded tensor is torch.float32.
My questions are:
(1) What is the mechanism of representing PCM16bits by float32?
(2) Since original data has been quantized in terms of 16bits already,  finer representation(>16bits) cannot specify more details yet consume more hardware memories. Thus, what are necessity and benefit of using float32 to represent PCM16 by “torchaudio”?
Sincerely thanks for your reply!

Context:
Hey @Felix_astw , welcome to the forum!
In audio file containing 16 bit PCM samples, the values are restricted to the range [-32768, 32767]. The conversion from from 16bit to to float is done by dividing each sample value by 32768. This results in signal represented in floating point values in the range [-1.0, 1.0].
Using float is the canonical way of representing audio samples. There are many advantages, for example: getting full 32-bit precision, normalisation - regardless of whether data in the file is 8, 16, 24 or 32 bit wide, simple representation of clipping and easy application of operations (e.g. mixing signals is as easy as adding sample values together). At the end, it is also fast to scale/convert to the target bit depth - it’s just matter of multiplying by target range.


Question:
Hi,
I was evaluating the stft function in pytorch vs librosa. I found that the precision is only upto 2 decimal places. I am wondering if that can be improved to atleast 4-5 decimal places.
Also, regarding the speed of execution,
librosa is ~2x faster than pytorch on CPU (Didn't expect this, thought CPU times should be similar.)
pytorch is ~4x faster than librosa on GPU (Expected this.)

Rig: TitanX pascal, 4-core CPU, pytorch 0.5.0a0+8fbab83

Do these numbers look ok? Or is there a possibility I am doing something wrong?
Regards
Nabarun

Context: I found an answer from stackoverflow said that “The difference is from the difference between their default bit. NumPy’s float is 64bit by default. PyTorch’s float is 32bit by default.”


Question:
Hi,
I am trying to train a VQVAE. I used this 5 implementation.
As for the input, I tried various types of input, either taking some part of the original audio or taking that audio and converting it to a 2D matrix and applying minmax normalization. I tried various input types and even different loss functions but the model isn’t able to reconstruct the original audio. I have tried MSE and L1 loss. I have even tried UNet, but none of them seem to work. This is how I load the data:
class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):

        self.data = files

    def __getitem__(self, i):

        tmp = self.data[i]
        x = np.load('/aud_images/'+tmp)
        noise = np.random.normal(0.3,0.01,(256,256))
        x_n = x+noise
        x = np.reshape(x,(1,x.shape[0],x.shape[1]))
        x = torch.from_numpy(x).float()
        x_n = np.reshape(x_n,(1,x_n.shape[0],x_n.shape[1]))
        x_n = torch.from_numpy(x_n).float()

        return x_n, x

    def __len__(self):

        return len(self.data)

What should I do since almost everywhere I read that MSE loss works fine for audio data. Let me know.

Context:
As I work with Speech Synthesis, from what I have seen people generally have a hard time reconstructing the raw waveforms and I have not seen a lot of people trying to do it as you lose a lot of information in the frequency domain which could be useful for the properties of the Audio data, I think a good idea will be to turn it into more audio friendly representation’s first like Spectrogram or MFCC or Mel-Spectrogram if it is speech.
More information could be found regarding this in torchaudio.transforms.Spectrogram and torchaudio.transforms.MFCC.
Then you can use any vocoder to reconstruct it into the raw waveforms.
I hope it helps


How to restore the full signal from non-centered stft?

Question:
Posting for the first time, please tell me if I made a mistake.
Hi, I am working on a speech enhancement problem, with a STFT → modification in the frequency domain → iSTFT workflow.
My problem is, I have only managed to reconstruct the full signal that I passed into torch.stft when using the center=True option. Using the librosa implementation it seems like it could work. Maybe I am missing something, how I can achieve the same behavior in pytorch.
The following is a dummy setup to transform a sine wave and transform it back.
import torch
import matplotlib.pyplot as plt
import numpy as np
import librosa

n_fft = 32
# Example singal:
signal = (torch.linspace(0, 2*n_fft, 2*n_fft) * 2 * np.pi).sin()

# Parameters

# stft parameters with and without center
centered = {
    'n_fft': n_fft,
    'hop_length': n_fft // 2,
    'win_length': n_fft,
    'window': torch.hann_window(n_fft),
    'center': True,
    'return_complex': True,
}
uncentered = centered.copy()
uncentered['center'] = False

# parameters for librosa stft for comparison
lr_centered =  {
    'n_fft': n_fft,
    'hop_length': n_fft // 2,
    'win_length': n_fft,
    'window': 'hann',
    'center': True,
}

lr_uncentered = lr_centered.copy()
lr_uncentered['center'] = False

# parameters for the istft
i_centered = {
    'n_fft': n_fft,
    'hop_length': n_fft // 2,
    'win_length': n_fft,
    'window': torch.hann_window(n_fft),
    'center': True,
    'return_complex': False,
    # 'length': len(signal),
}
i_uncentered = i_centered.copy()
i_uncentered['center'] = False

i_lr_centered = {
    'hop_length': n_fft // 2,
    'win_length': n_fft,
    'window': 'hann',
    'center': True,
}
i_lr_uncentered = i_lr_centered.copy()
i_lr_uncentered['center'] = False

# stfts
stft_centered = signal.stft(**centered)
stft_uncentered = signal.stft(**uncentered)
stft_lr_centered = librosa.stft(signal.numpy(), **lr_centered)
stft_lr_uncentered = librosa.stft(signal.numpy(), **lr_uncentered)

# istfts
i_stft_centered = stft_centered.istft(**i_centered)
# i_stft_uncentered = stft_uncentered.istft(**i_uncentered) # ! this causes an error!
# RuntimeError: istft(CPUComplexFloatType[17, 3], n_fft=32, hop_length=16, win_length=32,
# window=torch.FloatTensor{[32]}, center=0, normalized=0, onesided=None, length=None,
# return_complex=0) window overlap add min: 0
i_stft_lr_centered = librosa.istft(stft_lr_centered, **i_lr_centered)
i_stft_lr_uncentered = librosa.istft(stft_lr_uncentered, **i_lr_uncentered)

# I used the centered parameters to see what happens:
i_stft_uncentered = stft_uncentered.istft(**i_centered)

With the code above, I get the following results:
The input Signal:
[Had to delete this picture, because new users can post only one]
The stfts:
[Had to delete this picture, because new users can post only one]
And the reconstructed signals:

As we can see, the librosa implementation can handle both centered and uncentered stfts with complete reconstruction, but the pytorch implementation fails at i_stft_uncentered = stft_uncentered.istft(**i_uncentered) (see above).
Is this a limitation of the pytorch implementation, or am I missing a way to get the same behavior in pytorch?

Context:
The STFTs:

I hope I am not breaking any rules by circumventing the 1 media element restriction like this


Question:
I have saved a model as torch script, to calculate mel spectrograms. Saving the model, and using the model at python gives no error and works as expected. However when I try to use model in c++ with libtorch I have received the following error, which is unintuitive for me. I would be happy, if one explains what is wrong and how to fix it. Thanks
RuntimeError: stft(CUDAFloatType[3708000, 1], n_fft=400, hop_length=80, win_length=400, window=CUDAFloatType{[400]}, normalized=0, onesided=1, return_complex=1) : expected 0 < n_fft < 1, but got n_fft=400
]

Context:
It would appear that the input might not have the right shape (should it be 1x3708000 or even 1x1x3708000 or something like this? If so, you need to use torch::Tensor::permute or somesuch).
For me, making a habit of checking the input shapes and possible a small portion of the tensor (say slicing the first 5 elements in each dimension) has reduced a great deal of confusion with the outputs when moving between Python and C++.
Best regards
Thomas


Torchaudio: how to frame and append long wav file?

Question:
torchaudio: how to frame and append long wav files using different durations?
e.g., I have 40G of 100,000 wav files (5 labels) with different length.

I wish to do 2, 3 and 4 durations of frames among these labels. Append the frames (I do not want miss the file)
How to append them without save to my disk >> 100,000 files?

Context: Hey, if by framing you mean slicing audio into series of optionally overlapping frames, see librosa.util.frame() 2 or soundfile.blocks() 2.


Question:
Hi,
I am converting Hubert model to onnx format with this script:
import torch
import torchaudio
import numpy as np
import soundfile as sf
import torch.nn.functional as F

import onnx
import onnxruntime

device="cpu"
# https://pytorch.org/audio/stable/pipelines.html#hubert-large
bundle = torchaudio.pipelines.HUBERT_LARGE
model = bundle.get_model().to(device)

audio_file = "sample.wav"   # shape: torch.Size([1, 101467])
x, sr = sf.read(audio_file, dtype='float32')
x = torch.Tensor(x).unsqueeze(0).cpu()
x = F.layer_norm(x, x.shape)

model_path = "torchaudio_hubert_large.onnx"
torch.onnx.export(model, x, 'torchaudio_hubert_large.onnx', input_names=['input'], output_names=['output'])
model = onnx.load(model_path)
model.graph.input[0].type.tensor_type.shape.dim[1].dim_param = '?'
onnx.save(model, model_path.replace(".onnx", "_dyn.onnx"))

Then I am trying to infer a sample with this code:
model_path = "torchaudio_hubert_large_dyn.onnx"
ort_session = onnxruntime.InferenceSession(model_path)
feat = ort_session.run(None, {'input': x.numpy().astype(np.float32)})

In this step, this error occurs:
---------------------------------------------------------------------------
RuntimeException                          Traceback (most recent call last)
/tmp/ipykernel_13203/1463404001.py in <module>
      1 print(x.shape)
----> 2 feat = ort_session.run(None, {'input': x.numpy().astype(np.float32)})[0]
      3 print(feat.shape)

/path/to/miniconda3/envs/onnx/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    190             output_names = [output.name for output in self._outputs_meta]
    191         try:
--> 192             return self._sess.run(output_names, input_feed, run_options)
    193         except C.EPFail as err:
    194             if self._enable_fallback:

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_208' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&, bool) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,316,1024}, requested shape:{1,49,16,64}

How can I solve this?

Context:
Hi @yunusemre, thanks for raising this issue. I have tried the code on my local machine and I can successfully run your script.
Could you post your onnx and torch version and PyThon version as well? You can get the environment information by this script: https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
Thanks!


Question:
Hi
Is there an integrated solution to compose torchaudio transforms in the same style as torchvision do?

Context: torchaudio doesn’t provide a dedicated compose transformation since 0.3.0 (see release notes 23). Instead, one can simply apply them one after the other x = transform1(x); x = transform2(x), or use nn.Sequential(transform1, transform2).


Inner product 2-D weight matrix with 3-D torch of spectrograms

Question:
Hi.
I have the following setup:
[49, 49] matrix, where each row is a probabilities vector (obtained from softmax over logits). overall it has 49 probability vectors, each with 49 examples.
[49, x, y] matrix, containig 49 spectrograms of size [x,y] each.
I try to obtain a 49 different weighted spectrograms, from each of the 49 probability vectors and 49 spectrograms.
Output size shall be [49, x, y].
I tried my best to search the net, and tried many configurtations of torch matmul, bmm, etc…

Context:
Noy_Uzrad:

Output size shall be [49, x, y].

If I understand correctly, since you want differently weighted spectrograms with each of the 49 probability vectors and 49 spectrograms, the output size should be [49, 49, x, y]. No?
i.e., In the spectrogram size [49, x, y], batch size = 49. The same batch size as in the probability matrix of [49, 49]. For a particular spectrogram, you have 49 probability values corresponding to different classes.


Torchaudio.transforms doesn’t have log scale frequency spectrogram

Question:
Hi,
I’ve found Spectrogram and MelSpectrogram in torchaudio.transforms, but there seems to be no implementation of log scale frequency spectrogram.
Why is that?

Context: Hi. Look at AmplitudeToDb https://pytorch.org/audio/stable/transforms.html#amplitudetodb 2torchaudio.transforms — Torchaudio 0.10.0 documentation 2


Question:
I try to do some DSP thing and I need to convert frame into single bin and then restore the frame from that bin. Here’s minimal example:
import torch
import matplotlib.pyplot as plt

frame = torch.rand(1536)
bin = torch.stft(frame, n_fft=frame.shape[0], hop_length=frame.shape[0], center=False,
                     return_complex=True, window=torch.hann_window(frame.shape[0]))
rest_frame = torch.istft(bin, n_fft=frame.shape[0], hop_length=frame.shape[0], center=False,
                        return_complex=True, window=torch.hann_window(frame.shape[0]))

plt.plot(rest_frame-frame)
plt.show()

I expected to get the same frame, but instead I got
RuntimeError: Cannot have onesided output if window or input is complex
in istft. torch.stft produces the bin that I need, but I still can’t make istft work. What do I do wrong?

Context:
Hi @jBloodless, thanks for sharing. In this case, I suggest changing the return_complex to False in torch.istft.
Also the n_fft and hop_length and window length need to be carefully selected, as torch.istft requires the settings to pass NOLA check to make the signal invertible. Check this doc for more information: Invertibility of overlap-add processing


Question:
Hi,
I am trying to install the torchaudio library in google Colaboratory notebook. However I get this dependency error:
running install
running bdist_egg
running egg_info
creating torchaudio.egg-info
writing torchaudio.egg-info/PKG-INFO
writing dependency_links to torchaudio.egg-info/dependency_links.txt
writing top-level names to torchaudio.egg-info/top_level.txt
writing manifest file 'torchaudio.egg-info/SOURCES.txt'
reading manifest file 'torchaudio.egg-info/SOURCES.txt'
writing manifest file 'torchaudio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building '_torch_sox' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/torchaudio
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.6-sXpGnM/python3.6-3.6.3=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/include/python3.6m -c torchaudio/torch_sox.cpp -o build/temp.linux-x86_64-3.6/torchaudio/torch_sox.o -DTORCH_EXTENSION_NAME=_torch_sox -std=c++11
x86_64-linux-gnu-gcc: error: torchaudio/torch_sox.cpp: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Is there any workaround to fix this issue ?
Best Regards

Context:
I just installed it via pip on Google Colab directly from the GitHub repo and it works fine.
pip install git+git://github.com/pytorch/audio

Screen Shot 2018-05-22 at 1.57.46 PM.png880×193 26 KB

Note that you need to install the required dependencies via apt-get first as mentioned in the GitHub repo. Also you need to have PyTorch installed, otherwise you will get an error during the install.
EDIT:
The dependencies I mentioned above are the ones listed for Ubuntu Linux (https://github.com/pytorch/audio#dependencies 33). Worked fine for me in the Colaboratory Notebook:
apt-get install sox libsox-dev libsox-fmt-all


Use torchaudio cause torch jit load model error,help!

Question:
we use pytorch version of 1.7.0+cu101 to turn torchscript，and I successfully loaded the torchscript model in torch version of 1.7.0+cu101 to inference, but when I use pytorch version of 1.8.1+cu101 to inference, there are some error
python3 torch_test_jit_1109.py
Traceback (most recent call last): File "torch_test_jit_1109.py", line 35, in <module> main(sys.argv[1], sys.argv[2]) File "torch_test_jit_1109.py", line 17, in main e2e_model = torch.jit.load(script_model) File "/data3/users/suziyi/Miniconda3/lib/python3.8/site-packages/torch/jit/_serialization.py", line 161, in load cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files) RuntimeError: Class Namespace cannot be used as a value: Serialized File "code/__torch__/torchaudio/sox_effects/sox_effects.py", line 5 effects: List[List[str]], channels_first: bool=True) -> Tuple[Tensor, int]: in_signal = __torch__.torch.classes.torchaudio.TensorSignal.__new__(__torch__.torch.classes.torchaudio.TensorSignal) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE _0 = (in_signal).__init__(tensor, sample_rate, channels_first, ) out_signal = ops.torchaudio.sox_effects_apply_effects_tensor(in_signal, effects) 'apply_effects_tensor' is being compiled since it was called from 'SynthesizerTrn.forward' Serialized File "code/__torch__/models_torchaudio_torchscript.py", line 41 sr: Tensor, vol: Tensor) -> Tensor: _0 = __torch__.torchaudio.sox_effects.sox_effects.apply_effects_tensor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE _1 = torch.tensor([(torch.size(x))[-1]], dtype=None, device=None, requires_grad=False) x_lengths = torch.to(_1, 4, False, False, None)

we use torchaudio to change audio speed,volume and so on, please help me
when we create torchscript the pytorch version is 1.7.0+cu101 , the torchaudio version is 0.7.0

Context:
Hi @c9412600, thanks for sharing the issue. Could you share your script that uses sox_effect?
Also you can post it on torchaudio’s GitHub page. This helps other people who also meet this problem.


There is no sampling occuring when I apply offset and n_frames in the load function

Question:
Is n_frames equivalent to librosa’s duration?
pytorch_version: 1.9.0+cu111
torchaudio_version: 0.9.0
I’m currently working in audio classification task where I found that if the audio file is provided entirely in load_audio() function, it provides the stereo output of the sampled version with default sampling_rate
But, if i provide offset and n_frames into the function, it doesn’t resample it.
Actually I have a doubt that is n_frames is equivalent to librosa’s duration.
In my case the offset is 30 and duration is 10 seconds
reading the audio file with offset and n_frames

Screenshot from 2021-10-30 16-09-331920×1080 202 KB

Context:
Hi @ThiruRJST, thanks for sharing it. In torchaudio, the frame_offset and num_frames are in frames or samples, thus it’s different from librosa’s duration argument.
If you want to extract 10 seconds of audio from the 30th second, you can call the method torchaudio.load(file_path, frame_offset=30*sample_rate, num_frames=10*sample_rate).


What are the stages for training a neural net for speech recognition?

Question:
is that:

creating a mel spectrogram
feed it to the neural net
backprop(and by the way what loss function should I use?)
repeat until the loss decreases

or should I pass in batches or something? i’m super newbie and its my first project, so I dont know how to create an audio dataser and pass in batches… the tutorials that pytorch provides cant help me unfortunately… maybe can someone explain the steps in the comments or provide a good tutorial?

Context:
There also are models working directly on waveforms, e.g. wav2letter 1.
For an even simpler entry, you might look at the Speech Commands tutorial 3 where the waveform is translated into class predictions for keywords (a much simpler problem and relatively fast to train).
Best regards
Thomas


Question:
Hello, I am tryig to create neural network for audio recognition. I want to recognize 6 types of speech. I have several problems maybe they could relate. I copied and a bit modified my nn from here. I want to put in neural network 8000 samples and get number from 0 to 5.

Problem is that I cant put there array of size [64, 8000] and I need to put there [64, 1, 8000]. Why is this? I think that I undestand that 64 is my batch size and 8000 is number of samples but why i need 3rd dim?
NN returns array (predicted) has size [64, 6] (64 is batch and 6 is full of zeros). Second dim should be only 1 and there should be number from 0 to 5 (my categories).

I am new in pytorch before that I used Matlab where it is from my view much easier.
# My model
class NeuralNetwork(nn.Module):
    def __init__(self, n_input=1, n_output=6, stride=16, n_channel=32): # nchalnnesl 32
        super().__init__()
        self.conv1 = nn.Conv1d(n_input, n_channel, kernel_size=80, stride=stride)
        self.bn1 = nn.BatchNorm1d(n_channel)
        self.pool1 = nn.MaxPool1d(4)
        self.conv2 = nn.Conv1d(n_channel, n_channel, kernel_size=3)
        self.bn2 = nn.BatchNorm1d(n_channel)
        self.pool2 = nn.MaxPool1d(4)
        self.conv3 = nn.Conv1d(n_channel, 2 * n_channel, kernel_size=3)
        self.bn3 = nn.BatchNorm1d(2 * n_channel)
        self.pool3 = nn.MaxPool1d(4)
        self.conv4 = nn.Conv1d(2 * n_channel, 2 * n_channel, kernel_size=3)
        self.bn4 = nn.BatchNorm1d(2 * n_channel)
        self.pool4 = nn.MaxPool1d(4)
        self.fc1 = nn.Linear(2 * n_channel, n_output)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(self.bn1(x))
        x = self.pool1(x)
        x = self.conv2(x)
        x = F.relu(self.bn2(x))
        x = self.pool2(x)
        x = self.conv3(x)
        x = F.relu(self.bn3(x))
        x = self.pool3(x)
        x = self.conv4(x)
        x = F.relu(self.bn4(x))
        x = self.pool4(x)
        x = F.avg_pool1d(x, x.shape[-1])
        x = x.permute(0, 2, 1)
        x = self.fc1(x)
        return F.log_softmax(x, dim=2)

# My training
criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    n_iterations = int(len(train_dataset)/batch_size)
    for epoch in range(num_epochs):
        running_loss = 0.0
        for i, (inputs, labels) in enumerate(train_loader):
            inputs, labels = inputs.cuda(), labels.cuda()
            optimizer.zero_grad()
            outputs = model(inputs)
            loss = criterion(outputs.squeeze(), labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
            if (i+1) % 5 == 0:
                print(f'epoch {epoch+1}/{num_epochs}, step {i+1}/{n_iterations}, input {inputs.shape}')

correct = 0
    total = 0
    model = NeuralNetwork()
    model.load_state_dict(torch.load("./model.pth"))
    correct_pred = {classname: 0 for classname in v.LABELS}
    total_pred = {classname: 0 for classname in v.LABELS}
    # since we're not training, we don't need to calculate the gradients for our outputs
    with torch.no_grad():
        for data in test_loader:
            inputs, labels = data
            # calculate outputs by running images through the network
            outputs = model(inputs)
            # the class with the highest energy is what we choose as prediction
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted[:, 0] == labels).sum().item()

            for label, predict in zip(labels, predicted[:,0]):
                if label == predict:
                    correct_pred[v.LABELS[label]] += 1
                total_pred[v.LABELS[label]] += 1

    print('Accuracy: %d %%' % (100 * correct / total))
    # print accuracy for each class
    for classname, correct_count in correct_pred.items():
        accuracy = 100 * float(correct_count) / total_pred[classname]
        print("Accuracy for class {:5s} is: {:.1f} %".format(classname,
                                                    accuracy))

Context:
Hi @martin.hajek1

Problem is that I cant put there array of size [64, 8000] and I need to put there [64, 1, 8000]. Why is this?

So your model uses nn.Conv1d layers that requires input_channel and output_channel arguments. 64 is the batch size, 8000 is the number of audio samples in your batch. The input channel is 1 as there is only one channel of audio. That’s why you need your input to be of size [64, 1, 8000].


Is there anything similar to grid_sample for 1D input?

Question: /is there any way I can make index selection diff wrt indices?

Context: I have the same question. I think it’s important for tasks like stereo matching.


Question:
Hi,
I’m trying to finish a multi-classification problem. In detail, I’m trying to determine the speech either contains some specific noise or no noise.
After training my model 100 epochs, the Cuda utilization suddenly jumps down to 0 - 2%. In the previous epochs, that was about 60 -70%.
Even though the error message reports a format issue, I don’t think that’s the file format issue because in the previous epochs I have already successfully loaded the same file with the same code.
I guess this issue just pops out suddenly.
Does anyone have some thoughts on that?
I’ll really appreciate it!

a07a4e226ae93ade256148e5f732e941840×936 78.2 KB

Context:
Rui_Liu:

Even though the error message reports a format issue, I don’t think that’s the file format issue because in the previous epochs I have already successfully loaded the same file with the same code.
I guess this issue just pops out suddenly.
Does anyone have some thoughts on that?

One thing you could check if it is a memory issue. My thinking around this is that “unknown format” might really be something like “something went wrong while decoding” and if your training leaks memory, it might end up not being able to load whatever it needs for decoding.
It’s just a guess, though.
Best regards
Thomas


Question:
I’m using the WavAugment 3 package, which imports torchaudio. I’m receiving the following warning:
UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.
    '"sox" backend is being deprecated. '

I tried adding the following line as recommended:
torchaudio.set_audio_backend("sox_io")
But I still get the warning. What am I doing wrong?

Context: Hi @Tzeviya, what’s your version of torchaudio? Upgrading to 0.9.0 should resolve this issue.


How to used custom dataset in Conv-TasNet

Question:
I created a music dataset including vocal and instrument, and I tried to use Conv-tasnet to sepatate the vocal and instrument. I have successfully used other people’s datasets for training first, but when I use my dataset to train, it keeps failing and I can’t find the problem. Here is the error and my datasets. Did I miss something?

2021-10-21 16-50-20 的螢幕擷圖1920×1080 237 KB

2021-10-21 16-51-29 的螢幕擷圖1920×1080 101 KB

2021-10-21 16-51-33 的螢幕擷圖1920×1080 102 KB

2021-10-21 16-59-34 的螢幕擷圖1920×1080 294 KB

Context:
The code in your screenshots is quite hard to read, so please post code snippets by wrapping them into three backticks ```, as it’s easier to read/debug, and the search engine would be able to index it as well if other users hit similar issues.
Based on what I can see it seems that a local variable is references before its assignment, so you would need to check where its definition is.


Question:
I am running the same code on two machines running Ubuntu 20.04. In one machine the code runs fine, in the other I get the error reported below. The only difference I can see are the graphic cards (NVIDIA Titan Xp in the machine that works and NVIDIA GeForce RTX 3090 in the one that doesn’t). The version of torch and torchaudio are 1.5.0 and 0.5.0 (these are required for the experiment to be reproducible). The machine that works runs python 3.7.9, the one that doesn’t runs python 3.8.10. Both machines have cuda V10.1.243.
Let me know if there is other information that I missed and that is relevant to this question.
Do you have any suggestion on what to look for to solve this problem?
Thank you!
Giampiero
Using device ‘cuda:0’
Creating Datasets
Calculating mean and std from dataset
Traceback (most recent call last):
File “main.py”, line 665, in
main(args)
File “main.py”, line 621, in main
task_and_more, dataloaders, model_and_more = prepare_for_task(args)
File “main.py”, line 555, in prepare_for_task
datasets = create_datasets(args, paths[‘data_path’], flags[‘load_dataset_extra_stats’])
File “main.py”, line 252, in create_datasets
datasets = set_transforms_on_datasets(args, datasets, transforms_device)
File “main.py”, line 303, in set_transforms_on_datasets
stats = get_dataset_stats_and_write(datasets[‘train’], args[‘device’],
File “…/preprocessing.py”, line 79, in get_dataset_stats_and_write
mean, std, min_val, max_val = calc_dataset_stats(dataloader, device=device)
File “…/processing.py”, line 72, in calc_dataset_stats
input_channel = dataloader.dataset[0][‘image’].shape[0]
File “…/CLEAR_dataset.py”, line 433, in getitem
game_with_image = self.transforms(game_with_image)
File “…/venv/lib/python3.8/site-packages/torchvision/transforms/transforms.py”, line 61, in call
img = t(img)
File “…/data_interfaces/transforms.py”, line 36, in call
specgram = self.spectrogram_transform(sample[‘image’])[0, :, :]
File “…/venv/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “…/venv/lib/python3.8/site-packages/torchaudio/transforms.py”, line 81, in forward
return F.spectrogram(waveform, self.pad, self.window, self.n_fft, self.hop_length,
File “…/venv/lib/python3.8/site-packages/torchaudio/functional.py”, line 276, in spectrogram
spec_f /= window.pow(2.).sum().sqrt()
RuntimeError: CUDA error: invalid device function

Context:
Your Ampere GPU (RTX 3090) needs CUDA>=11.0 and I don’t know which exact PyTorch binary (or source build) you’ve installed.
In any case, the CUDA11 support came on PyTorch ~1.7 so your 1.5.0 installation will most likely be using CUDA10, which is causing the error.

 giampierosalvi:

Both machines have cuda V10.1.243.

The pip wheels and conda binaries ship with their own CUDA runtime and your local CUDA toolkit is used for a source build or to build custom CUDA extensions.


Does torchaudio.transforms.spectrogram work correctly if n_fft > win_size

Question:
I’m using a custom loop to get the STFT of an audio signal, while doing some processing on a frame by frame basis. The number of resulting frames corresponds to the formula
number_of_samples = 1440000 + winsize I pad either end of the signal with winsize //2
number_of_frames = (number_of_samples)//(window_size - window_size//2)
%  = (1440000 + 1024)//(1024 - 512)
%  = 2814
During the loop to prevent aliasing I zero pad the end of both the window and input frame by length winsize but my hopsize still remains at 512. This extra zero padding shouldn’t affect the number of frames as it’s done on a frame by frame basis to the current time domain segment under analysis.
I then want to use torchaudio.transforms.Melspectrogram on the original audio signal - this would result in me having two STFT signals, one with linearly spaced frequency bins and one with mel spaced. However the number of frames outputted from the transform is not as expected depending on the value of n_fft. With the n_fft  = winsize and center=True it outputs 2816 frames and with center=False it outputs the expected 2814. However if n_fft = 2048 and winsize = 1024 it outputs 2812 frames. I can’t work out why n_fft would effect the number of total frames if frames are based on signal length, window size, and hopsize.

Context:
Hi @Mole_m7b5, thanks for posting the question. The number of frames is hard coded in torch.stft
  int64_t n_frames = 1 + (len - n_fft) / hop_length;
  // time2col
  input = input.as_strided(
    {batch, n_frames, n_fft},
    {input.stride(0), hop_length * input.stride(1), input.stride(1)}
  );
  if (window_.defined()) {
    input = input.mul(window_);
  }

If you increase the value of n_fft, the number of frames will be decreased.


Question:
Hello peeps,
I have some audio data for which i computed the audio features. The labels for each input feature is 20 binary classes. i.e. for each input element there are 20 classes (1 or 0).
My initial understanding is that  this is a multi-label classification that can be addressed using nn.BCELoss.
Can you guys let me know if I got this right and to explain how nn.BCELoss works?
Thank you

Context: If you are dealing with multi-label classification sigmoid + nn.BCELoss should be what you are looking for; and if you are looking for multi-class classification (each of your sample will belong to only one of the 20 classes) then softmax + nn.CrossEntropyLoss are the things.


Question:
Hello,
import torchaudio.sox_effects as sox_effects
wav, sr = sox_effects.apply_effects_file(
wav_file,
[[‘speed’, str(speed)],[‘pitch’, ‘0’], [‘rate’, str(sample_rate)]])
This method apply_effects_file is used to change the speed of speech, but the pitch is changed even I set the pitch shift is zero.  I want to know how to change the speed of audio but not change the pitch.
In addition, I found another method “torchaudio.transforms.TimeStretch” to stretch on the stft and keep the pitch unchanged. If the “torchaudio.transforms.TimeStretch” can make the speed change but the pitch unchanged?
Thanks very much for your help.

Context:
Hi @Alva-2020, you can apply tempo argument for changing the speed while keeping the pitch. As described in sox document.
Yes, torchaudio.transforms.TimeStretch can change the speech while keeping the pitch. Note that the method applied is different from sox, the values may not be fully identical.


Question:
Hi, I noticed there is a difference in the values from mp3 file when loaded using torchaudio.load vs librosa.load.  Also, the shapes of the tensors are different. I am loading an mp3 file with 44.1kHz sampling frequency of 1 sec. duration and I am getting the following output.
librosa_audio, sr_librosa = librosa.load(os.path.join(root, path), sr=44100)
torch_audio, sr_torch = torchaudio.load(os.path.join(root, path))
print(librosa_audio.shape, sr_librosa)
print(torch_audio.shape, sr_torch)
# (44100,) 44100
# torch.Size([1, 46040]) 44100

I am loading a one second audio and I expect the shape to be 44100 for both the case. Can someone please explain what is happening here?
Thanks.

Context: I think you might be hitting this issue 5 so feel free to comment on this issue with your use case and description of the difference.


Loss stuck. Model for speech-to-text system

Question:
I’m trying to build a speech-to-text system my data is (4 - 10 seconds audio wave files) and their transcription (preprocessing steps are char-level encoding to transcription and extract mel-Spectrograms from audio files).
this is my model architecture is ( a 3 conv1d layers with positional encoding to the audio file - embedding and positional encoding to encoded transcription and then use those as input to transformer model and lastly a dense layer)
the loss function is cross entropy and optimizer is Adam.
the problem is that the loss is always stuck at some point it starts around 3.8 (I have 46 classes) and after some batches it decreases to (e.g.  2,8) and stuck their. it bounces around that value and never decrease again.
I tried changing parameters of the model, I’ve changed the optimizer and learning rate always result the same problem.
I don’t understand what I’m doing wrong
Screenshot 2021-09-28 1925021423×611 56.9 KB

Context: Hi @Abdulrahman_Adel, what’s the format of your label? Usually the length of the text label is not the same as the feature sequence length, hence there is a CTC loss for addressing the issue. The output of the model contains “” symbol and it’s excluded when evaluating the word error rate.


Wav2Vec2: Forward() takes 2 positional arguments but 3 were given

Question:
I am trying to use wav2vec2_base model for audio feature extraction. I am getting the error in the title for this code:
class Identity(nn.Module):
    @overrides
    def forward(self, input_):
        return input_

            model = wav2vec2_base(num_out=32)
            model.load_state_dict(torch.load("wav2vec2-base-960h.pt"))
            model.encoder = Identity()

            sample_rate, samples = wavfile.read(wav_file)
            samples = samples.reshape((1,samples.shape[0]))
            samples = torch.from_numpy(samples)
            #samples = samples.type(torch.short)
            print("Start")
            samples = samples.float()

            output = model(samples.float())

I have tried to select the related parts in the code but there is not much left besides these. What is wrong with my forward function? Thanks for any help!

Context: Could you post the entire model definition as well as the error message including the stack trace?


Can someone recommend for a tutorial about speech recognition in pytorch?

Question: I made a neural network with the urban sound dataset according to a tutorial, but now I wanna create my own dataset and network, which will recognize a wake word for sound assistant (hopefully). where should I start?

Context:
There is a voice activity detection example in torchaudio. I used that for inspiration when doing my raspberry lightbulb control.
Best regards
Thomas


Question:
Hi,
I’ve been looking into using a Constant Q Transform in my pipeline, which I’m currently doing with librosa. I would like to rewrite this function, so that I only need to use pytorch/torchaudio for my application, and also so that it can be written in c++ like torch.stft. I am however unsure on how to get started. Where is the c++ part of torch.stft defined, so that I can get a sense of how to proceed with writing a VQT function. (VQT and CQT are essentially the same).
Thanks!

Context:
Hi @lewiswolf! Thanks for the proposal. It’ll be great to add the VQT and CQT functions. Here is the reference for torch.stft 3.
I also recommend adding the functions to torchaudio repo 2, it already supports several audio processing functions like MFCC, PitchShift, and so on. Feel free to create an issue for feature proposal and add your PR there


How to set weight randomly for TCN output

Question:
hello everyone,
I am using multi slice-TCN model in my program. So each TCN generate the output and then multiply with weight and finally the combine all TCN’s result. equation like this (output = TCN’output * weight). So my question is that “How can i randomly initial the weight for TCN”. I am sending the screen shot. I am sending also my hand draw picture because i need like this picture.Thank your very much.

Screenshot from 2021-08-20 18-41-541920×1200 417 KB

2cb2dba4-dfd6-4ce2-9754-e7336da4404c (1)1960×2374 251 KB

Context: Hi @Kuldeep_Rana! You can use torch.nn.init.uniform_ method to initialize your paramter. See this link 1


I am plotting three plots per audio file using subplots. How to made axis invisible?

Question:
path=glob.glob('............./*.wav')

fig, ax =  plt.subplots(nrows=4, ncols=3, sharex=True)

for i in range(4) :

    y, sr = librosa.load(path[i], sr=16000)
    plt.axis('off')

    librosa.display.waveplot(y, sr, ax=ax[i, 0])  # put wave in row i, column 0

    mfcc=librosa.feature.mfcc(y)
    plt.axis('off')
    librosa.display.specshow(mfcc, x_axis='time', ax=ax[i, 1]) # mfcc in row i, column 1

    S = librosa.feature.melspectrogram(y, sr)
    plt.axis('off')
    librosa.display.specshow(librosa.power_to_db(S), x_axis='time', y_axis='log', ax=ax[i, 2])  # spectrogram in row i, column 2

I tried different positions for plt.axis(off) But none is working only on the last plot it’s working. any suggestions @ptrblck sir plz help, I know this is out of forum discussion but if you can guide me,
Regards

Context:
Just to close the conversation and help others , I got the answer . After defining subplot axis off is used like this:
fig, ax =  plt.subplots(nrows=4, ncols=3, sharex=True)
[axi.set_axis_off() for axi in ax.ravel()]

and problem is solved

image946×625 51.6 KB


How do I train on variable length audio?

Question: How do I allow my neural network to train on an audio stream? In particular, what I want to achieve is to take in, for example, an audio that is 4 seconds long and pass a sliding analysis window wherein I classify the audio in every window and return the class with the highest average probability in all the windows. I’m not sure how to do this in PyTorch. Does this have something to do with the data loaders? Or is it more on the design of the network?

Context: A classic solution in this case would be to simply pad the audio to the longest length possible.


Batch of audio with unequal length and transformations

Question: I understand that when dealing with audio with unequal length, one could define a collate function that would pad all audio to the same  length, i.e. the length of the longest audio. However, if I were to do some transformations in my custom dataset’s __getitem__, like taking the log-Mel spectrogram, how do I pad audio of unequal length? My guess is to still use a collate function and pad along the last dimension of the batch of audio transformed to a log-Mel spectrogram, but I want to know what the best practices are with regards to this matter. Thanks!

Context: What you mentioned is the best practice as per my knowledge and referring to the Nvidia’s Tacotron2 source code Tacotron2/data_utils.py 7 the same is done here.


Question:
Dear Sir/Madam,
I am a beginner at the pytorch. I have occurred a problem. I want to update the weight after 15 epoch and keep the same weight till the end of training. How can I do this? Thank you very much.

Context:
Show code and tell what you mean
EDIT: Or simply make if statement and after your 15th epoch dont call
optimizer.zero_grad()     loss.backward()     optimizer.step()


Invalid argument 0: Sizes of tensors must match except in dimension 1. Got 29 and 30 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71

Question:
Hello, I am trying to train a speech enhancement CNN based on Wave-U-Net but I get this error
invalid argument 0: Sizes of tensors must match except in dimension 1. Got 29 and 30 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
class DownSamplingLayer(nn.Module):
def __init__(self, channel_in, channel_out, dilation=1, kernel_size=15, stride=1, padding=7):
    super(DownSamplingLayer, self).__init__()
    self.main = nn.Sequential(
        nn.Conv1d(channel_in, channel_out, kernel_size=kernel_size,
                  stride=stride, padding=padding, dilation=dilation),
        nn.BatchNorm1d(channel_out),
        nn.LeakyReLU(negative_slope=0.1)
    )

def forward(self, ipt):
    return self.main(ipt)

class UpSamplingLayer(nn.Module):
def __init__(self, channel_in, channel_out, kernel_size=5, stride=1, padding=2):
    super(UpSamplingLayer, self).__init__()
    self.main = nn.Sequential(
        nn.Conv1d(channel_in, channel_out, kernel_size=kernel_size,
                  stride=stride, padding=padding),
        nn.BatchNorm1d(channel_out),
        nn.LeakyReLU(negative_slope=0.1, inplace=True),
    )

def forward(self, ipt):
    return self.main(ipt)

class SE_Model(nn.Module):
def __init__(self, n_layers=12, channels_interval=24):
    super(SEModel, self).__init__()

    self.n_layers = n_layers
    self.channels_interval = channels_interval
    encoder_in_channels_list = [1] + [i * self.channels_interval for i in range(1, self.n_layers)]
    encoder_out_channels_list = [i * self.channels_interval for i in range(1, self.n_layers + 1)]
    self.encoder = nn.ModuleList()
    for i in range(self.n_layers):
        self.encoder.append(
            DownSamplingLayer(
                channel_in=encoder_in_channels_list[i],
                channel_out=encoder_out_channels_list[i]
            )
        )

    self.middle = nn.Sequential(
        nn.Conv1d(self.n_layers * self.channels_interval, self.n_layers * self.channels_interval, 15, stride=1,
                  padding=7),
        nn.BatchNorm1d(self.n_layers * self.channels_interval),
        nn.LeakyReLU(negative_slope=0.1, inplace=True)
    )

    decoder_in_channels_list = [(2 * i + 1) * self.channels_interval for i in range(1, self.n_layers)] + [
        2 * self.n_layers * self.channels_interval]
    decoder_in_channels_list = decoder_in_channels_list[::-1]
    decoder_out_channels_list = encoder_out_channels_list[::-1]
    self.decoder = nn.ModuleList()
    for i in range(self.n_layers):
        self.decoder.append(
            UpSamplingLayer(
                channel_in=decoder_in_channels_list[i],
                channel_out=decoder_out_channels_list[i]
            )
        )

    self.out = nn.Sequential(
        nn.Conv1d(1 + self.channels_interval, 1, kernel_size=1, stride=1),
        nn.Tanh()
    )

def forward(self, input):
    tmp = []
    o = input

    # Up Sampling
    for i in range(self.n_layers):
        o = self.encoder[i](o)
        print(o.shape)

        tmp.append(o)
        # [batch_size, T // 2, channels]
        o = o[:, :, ::2]
    print(o.shape)

    o = self.middle(o)
    print(o.shape)

    # Down Sampling
    for i in range(self.n_layers):
        # [batch_size, T * 2, channels]
        print(o.shape)
        o = F.interpolate(o, scale_factor=2, mode="linear", align_corners=True)
        print(o.shape)
        # Skip Connection
        o = torch.cat([o, tmp[self.n_layers - i - 1]], dim=1)
        o = self.decoder[i](o)
        print(o.shape)

    o = torch.cat([o, input], dim=1)
    o = self.out(o)
    return o

Context: These shape errors are often created for odd input shapes (or generally shapes, which weren’t used while developing the model). You could add custom shape checks and either slice the larger activation or pad the smaller one before concatenating them. Alternatively (and if possible) you could also use the expected shapes (often these kind of models work for the default shapes such as [batch_size, channels, 224, 224]).


Expected 4-dimensional input for 4-dimensional weight [8, 1, 10, 2], but got 5-dimensional input of size [32, 1, 1, 128, 81] instead

Question:
Hi there
I am using below function for feature extraction. Using mfcc and melspectrograms  in my model. Both outputs should be of similar dimension. There is not problem with mfcc alone but I am not getting why model is taking 5 dimensions inmelspectrogram. Can anyone figure It out plz
Regards
class FeatureExtractor(object):
    def __init__(self, rate):
        self.rate = rate

    def get_features(self, features_to_use, X):
        X_features = None
        accepted_features_to_use = ('mfcc', ''melspectrogram')
        if features_to_use not in accepted_features_to_use:
            raise NotImplementedError("{} not in {}!".format(features_to_use, accepted_features_to_use))

        if features_to_use in ('mfcc'):
            X_features = self.get_mfcc(X,26)

        if features_to_use in ('melspectrogram'):
            X_features = self.get_melspectrogram(X)

        return X_features

    def get_mfcc(self, X, n_mfcc=13):
        def _get_mfcc(x):
            mfcc_data = librosa.feature.mfcc(x, sr=self.rate, n_mfcc=n_mfcc)

            return mfcc_data

        X_features = np.apply_along_axis(_get_mfcc, 1, X)
        return X_features

    def get_melspectrogram(self, X):
        def _get_melspectrogram(x):
            mel = librosa.feature.melspectrogram(y=x, sr=self.rate, n_fft=800, hop_length=400)[np.newaxis, :]
            delta = librosa.feature.delta(mel)
            delta_delta = librosa.feature.delta(delta)
            out = np.concatenate((mel, delta, delta_delta))
            return mel

        X_features = np.apply_along_axis(_get_melspectrogram, 1, X)
        return X_features

Here the shapes of X is (5984, 32000)

Context:
@ptrblck sir please have a look in this code,
Regards


Applying Attention (Single and MultiHead Attention)

Question:
Applying Attention from paper

      danielpovey.com

2018_interspeech_xvector_attention.pdf 13

266.24 KB

Suppose my Hidden audio representation  shape is  (after few CNN operations/layers)
 H = torch.Size([128, 32, 64])    [Batch Size X FeatureDim X Length]

and I want to apply self-attention weights to the audio hidden frames as
A = softmax(ReLU(AttentionWeight1 * (AttentionWeight2 * H))

In order to learn these two self attention weight matrices. Do I need to register these two weights as Parameters in the init function like below

class Model(nn.Module):
        def __init__(self, batch_size):
           super(Model, self).__init__()
           self.attention1   = nn.Parameter(torch.Tensor(self.batch_size,16, 32))
           self.attention2   = nn.Parameter(torch.Tensor(self.batch_size,1, 16))

and in the forward do I need to do like this
def forward(self, input):
    ....
   H = CNN(input)     #[B X Features X length]
   attention = nn.Softmax(nn.ReLU(torch.mm(self.attention2, torch.mm(self.attention1, H))
   H = H*attention
   return H

Please help. How can we apply attention here. A the above code is throwing error
RuntimeError: matrices expected, got 3D, 3D tensors at /opt/conda/conda-bld/pytorch_1591914985702/work/aten/src/TH/generic/THTensorMath.cpp:36

Context:
torch.mm expects two matrices (2D tensors), while you seem to use two 3D tensors.
You could use torch.bmm or torch.matmul instead, which would work for these tensors.
However, usually the parameters are not depending on the batch size.
Are you sure you want to initialize them with the batch size in dim0?


Question:
If i get the below error what should my input tensor look like to the inverse mel scale function:
inv_mel = torchaudio.transforms.InverseMelScale(n_stft=1025)
inv_mel(s)

TypeError                                 Traceback (most recent call last)
 in ()
2 plt.imshow(s)
3 inv_mel = torchaudio.transforms.InverseMelScale(n_stft=1025)
----> 4 inv_mel(s)
1 frames
/usr/local/lib/python3.7/dist-packages/torchaudio/transforms.py in forward(self, melspec)
388         “”"
389         # pack batch
 → 390         shape = melspec.size()
391         melspec = melspec.view(-1, shape[-2], shape[-1])
392
TypeError: ‘int’ object is not callable

Context:
Based on the error message it seems that you are using a np.array instead of a tensor, as the former returns an int from the .size attribute:
x = np.random.randn(10)
x.size()
> TypeError: 'int' object is not callable

x = torch.randn(10)
x.size()
> torch.Size([10])

so you might want to use tensors.


Question:
I am Training a Pytorch model. After some time, even if on shuffle, the model contains, besides a few finite tensorrows only NaN values:
tensor([[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         ...,
         [ 1.4641,  0.0360, -1.1528,  ..., -2.3592, -2.6310,  6.3893],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],
       device='cuda:0', grad_fn=<AddBackward0>)

The detect_anomaly functions return:

RuntimeError: Function ‘LogSoftmaxBackward’ returned nan values in its 0th output.

in reference to the next line output = F.log_softmax(output, dim=2)
A normal tensor should look like this:
tensor([[[-3.3904, -3.4340, -3.3703,  ..., -3.3613, -3.5098, -3.4344]],

        [[-3.3760, -3.2948, -3.2673,  ..., -3.4039, -3.3827, -3.3919]],

        [[-3.3857, -3.3358, -3.3901,  ..., -3.4686, -3.4749, -3.3826]],

        ...,

        [[-3.3568, -3.3502, -3.4416,  ..., -3.4463, -3.4921, -3.3769]],

        [[-3.4379, -3.3508, -3.3610,  ..., -3.3707, -3.4030, -3.4244]],

        [[-3.3919, -3.4513, -3.3565,  ..., -3.2714, -3.3984, -3.3643]]],
       device='cuda:0', grad_fn=<TransposeBackward0>)

Please notice the double brackets, if they are import.
Code:
spectrograms, labels, input_lengths, label_lengths = _data
spectrograms, labels = spectrograms.to(device), labels.to(device)
optimizer.zero_grad()

output = model(spectrograms)

Additionally, I tried to run it with a bigger batch size (current batch size:1, bigger batch size: 6) and it run without errors until 40% of the first epoch in which I got this error.

Cuda run out of memory

Also, I tried to normalize the data torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128, normalized=True)
And reducing the learning rate from 5e-4 to 5e-5 did not help either.
Additional information: My dataset contains nearly 300000 .wav files and the error came at 3-10% runtime in the first epoch.
I appreciate any hints and I will gladly submit further information.
Also asked on Stackoverflow: python - Why does my Pytorch tensor size change and contain NaNs after some batches? - Stack Overflow 7

Context: Something was wrong with my labels. See Stackoverflow for more


Stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1

Question:
Hi all, I have pre-processed my dataset to obtained three sets as train test and validation. The shapes and type of each of them are as follows.
Shape of X_train:  (3441, 7, 1, 128, 128)
type(X_train): numpy.ndarray
Shape of X_val:  (143, 7, 1, 128, 128)
type(X_val): numpy.ndarray
Shape of X_test:  (150, 7, 1, 128, 128)
type(X_test): numpy.ndarray

The class Dataset is created as given:
import torch
from torch.utils.data import Dataset

class Dataset(Dataset):
    def __init__(this, X=None, y=None,  mode="train"):
        this.mode = mode
        this.X = X
        if mode == "train":

            this.y = y

    def __len__(this):
        return this.X.shape[0]

    def __getitem__(this, idx):
        if this.mode == "train":
            return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])
        else:
            return torch.FloatTensor(this.X[idx])

Then I have use PytorchDataloader as
train_set = Dataset(X=X_train, y=Y_train,mode="train")
tr_loader = DL(train_set, batch_size=32, shuffle=True)

test_set  = Dataset(X=X_test, y=Y_test,mode="train")
ts_loader = DL(test_set, batch_size=32, shuffle=False)

Everything is working fine uptill now, (I think so :)). Then for training I used the code below.
verbose=True
Losses = []
Accuracies = []
epochs=10
DLS = {"train": tr_loader, "valid": ts_loader}

start_time = time.time()
for e in range(epochs):
    epochLoss = {"train": 0, "valid": 0}
    epochAccs = {"train": 0, "valid": 0}

    for phase in ["train", "valid"]:
        if phase == "train":
            model.train()
        else:
            model.eval()

        lossPerPass = []
        accuracy = []

        for X, y in DLS[phase]:
            X, y = X.to(device), y.to(device).view(-1)

            optimizer.zero_grad()
            alpha=1.0
            beta=1.0
            with torch.set_grad_enabled(phase == "train"):

                pred_emo= model(X)
                emotion_loss = criterion(pred_emo,y)

                total_loss = alpha*emotion_loss
                if phase == "train":
                    total_loss.backward()
                    optimizer.step()
            lossPerPass.append(total_loss.item())
            accuracy.append(accuracy_score(torch.argmax(torch.exp(pred_emo.detach().cpu()), dim=1), y.cpu()))
            torch.save(model.state_dict(),"E:/Python_On_All_Dataset/IEMO/long_codes/Kosta.jo/model_checkpoint_spec/Epoch_{}.pt".format(e+1))
        epochLoss[phase] = np.mean(np.array(lossPerPass))
        epochAccs[phase] = np.mean(np.array(accuracy))

    Losses.append(epochLoss)
    Accuracies.append(epochAccs)

    if verbose:
        print("Epoch : {} | Train Loss : {:.5f} | Valid Loss : {:.5f} \
| Train Accuracy : {:.5f} | Valid Accuracy : {:.5f}".format(e + 1, epochLoss["train"], epochLoss["valid"],
                                                        epochAccs["train"], epochAccs["valid"]))

Then I am getting this error traceback:
File “”, line 21, in
for X, y in DLS[phase]:
File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 517, in next
data = self._next_data()
File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 557, in _next_data
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py”, line 47, in fetch
return self.collate_fn(data)
File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in default_collate
return [default_collate(samples) for samples in transposed]
File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in
return [default_collate(samples) for samples in transposed]
File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 55, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1
Can someone comment on this, Is there any problem with class Dataset or in training. The tittle of the problem can be changed on suggestions.
please guide
Regards

Context:
I cannot reproduce the issue using:
X_train = torch.randn(3441, 7, 1, 128, 128)
y_train = torch.randint(0, 10, (3441,))

dataset = Dataset(X_train, y_train)
loader = DataLoader(dataset, batch_size=16)

for data, target in loader:
    print(data.shape)
    print(target.shape)

Could you compare my code snippet to yours and check, what the difference might be?


Question:
I have a train input of size [3082086, 7, 30], that is 3082086 frames, a window of 7 frames, each single frame contains 30 numbers.
The train labels is of size [3082086, 1], that is one label for every window of 7 frames.
I tried to stack them to create a train set as follow:
trainset = torch.hstack(train_input, train_labels)
…but I got an error RuntimeError: Sizes of tensors must match except in dimension 1. Got 30 and 1 in dimension 2 (The offending index is 1)
Any way I can create the dataset with the above dimensions?

Context: You dont want to mix inputs and labels into the same tensor, it doesnt make sense to merge them. Instead create a dataset, have separate tensors self.inputs, self.labels in the datasets __init__ and In the __getitem__(idx), just return self.inputs[idx], self.labels[idx]


Question:
This is a repost from the computer vision category as I didn’t see that there was an audio category
Here is my problem:
I am currently working on building a CNN for sound classification. The problem is relatively simple: I need my model to detect whether there is human speech on .wav recording sounds of a tropical ecosystem. I made a train / test set containing records of 3 seconds on which there is human speech (speech) or not (no_speech). From these 3 seconds fragments I get a mel-spectrogram of dimension 128 x 128 that is used to feed the model.
I checked if the script Audiodataset (given below) gives an expected output and I can’t find any problems. For the category “speech” the mel-spectrograms look similar to:

While the mel-spectrograms for “no-speech” look like:

Since it is a simple binary problem I thought the a CNN would easily detect human speech but I may have been too cocky. However, it seems that after 1 or 2 epoch the model doesn’t learn anymore, i.e. the loss doesn’t decrease and the number of correct prediction stays roughly the same. I tried to play with the hyperparameters but the problem is still the same. I tried a learning rate of 0.1, 0.01 … until 1e-7. Here are the runs displayed in Tensorboard:

image1117×655 33 KB

Then I thought it could be due to the script itself but I cannot find anything wrong. I would be glad you could have a quick look at the script and let me know what could go wrong! If you have other ideas of why this problem may occur I would also be glad to receive some advice on how to best train my CNN
I based the script on the LunaTrainingApp from “Deep learning in PyTorch” by Stevens as I found the script to be elegant. Of course I modified it to match my problem.
Here is the script creating the input for the model (from .wav file to mel-spectrogram):

"""
Define a class AudioDataset which take the folder of the training / test data as input
and make normalized mel-spectrograms out of it
Labels are also one hot encoded
"""

from torch.utils.data import Dataset
from pydub import AudioSegment
from sklearn.preprocessing import LabelEncoder
from fs import open_fs
from fs.osfs import OSFS

import numpy as np
import librosa
import torch
import os

class AudioDataset(Dataset):
    def __init__(self, data_root, n_fft, hop_length, n_mels):
        self.data_root_fs = open_fs(data_root)
        self.samples = []
        self.n_fft = n_fft
        self.hop_length = hop_length
        self.n_mels = n_mels
        self.class_encode = LabelEncoder()
        self._init_dataset()

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):

        audio_filepath, label = self.samples[idx]

        with self.data_root_fs.open(audio_filepath, 'rb') as audio_fd:
            audio_file = AudioSegment.from_file(audio_fd)

        #audio_file = AudioSegment.from_file(audio_filepath)
        array_audio = np.array(audio_file.get_array_of_samples(), dtype=float)

        mel = self.to_mel_spectrogram(array_audio)
        #mel_norm = self.normalize_row_matrix(mel)
        mel_norm_tensor = torch.tensor(mel)
        mel_norm_tensor = mel_norm_tensor.unsqueeze(0)

        label_encoded = self.one_hot_sample(label)
        label_class = torch.argmax(label_encoded)

        return (mel_norm_tensor, label_class)

    def _init_dataset(self):

        folder_names = set()
        for match in self.data_root_fs.glob("*/*.wav"):
            folder = match.path.split('/')[-2]
            folder_names.add(folder)
            self.samples.append((match.path, folder))
        self.class_encode.fit(list(folder_names))

    def to_mel_spectrogram(self, x):

        sgram = librosa.stft(x, n_fft=self.n_fft, hop_length=self.hop_length)
        sgram_mag, _ = librosa.magphase(sgram)
        mel_scale_sgram = librosa.feature.melspectrogram(S=sgram_mag, sr=16000, n_mels=self.n_mels)
        mel_sgram = librosa.amplitude_to_db(mel_scale_sgram)
        return mel_sgram

    def normalize_row_matrix(self, mat):

        mean_rows = mat.mean(axis=1)
        std_rows = mat.std(axis=1)
        normalized_array = (mat - mean_rows[:, np.newaxis]) / std_rows[:, np.newaxis]
        return normalized_array

    def to_one_hot(self, codec, values):
        value_idxs = codec.transform(values)
        return torch.eye(len(codec.classes_))[value_idxs]

    def one_hot_sample(self, label):
        t_label = self.to_one_hot(self.class_encode, [label])
        return t_label

The model itself, which basically performs 2D convolutions on the mel-spectrogram:
import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class VADNet(nn.Module):
    def __init__(self, in_channels=1, conv_channels=8):
        super().__init__()

        self.tail_batchnorm = nn.BatchNorm2d(1)

        self.block1 = ConvBlock(in_channels, conv_channels)
        self.block2 = ConvBlock(conv_channels, conv_channels * 2)
        self.block3 = ConvBlock(conv_channels * 2, conv_channels * 4)
        self.block4 = ConvBlock(conv_channels * 4, conv_channels * 8)

        self.head_linear = nn.Linear(8 * 8 * conv_channels * 8, 2)

        self._init_weights()

    def _init_weights(self):
        for m in self.modules():
            if type(m) in {
                nn.Linear,
                nn.Conv3d,
                nn.Conv2d,
                nn.ConvTranspose2d,
                nn.ConvTranspose3d,
            }:
                nn.init.kaiming_normal_(
                    m.weight.data, a=0, mode='fan_out', nonlinearity='relu',
                )
                if m.bias is not None:
                    fan_in, fan_out = \
                        nn.init._calculate_fan_in_and_fan_out(m.weight.data)
                    bound = 1 / math.sqrt(fan_out)
                    nn.init.normal_(m.bias, -bound, bound)

    def forward(self, input_batch):
        bn_output = self.tail_batchnorm(input_batch)

        block_out = self.block1(bn_output)
        block_out = self.block2(block_out)
        block_out = self.block3(block_out)
        block_out = self.block4(block_out)

        conv_flat = block_out.view(block_out.size(0),-1)
        linear_output = self.head_linear(conv_flat)

        return linear_output

class ConvBlock(nn.Module):
    def __init__(self, in_channels, conv_channels):
        super().__init__()

        self.conv1 = nn.Conv2d(
            in_channels, conv_channels, kernel_size=3, padding=1, bias=True,
        )
        self.relu1 = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(
            conv_channels, conv_channels, kernel_size=3, padding=1, bias=True,
        )
        self.relu2 = nn.ReLU(inplace=True)

        self.maxpool = nn.MaxPool2d(2, 2)

    def forward(self, input_batch):
        block_out = self.conv1(input_batch)
        block_out = self.relu1(block_out)
        block_out = self.conv2(block_out)
        block_out = self.relu2(block_out)

        return self.maxpool(block_out)

And the script that I use for training the model:
import torch
import torch.nn as nn
import argparse
import numpy as np

import logging
logging.basicConfig(level = logging.INFO)
log = logging.getLogger(__name__)

from torch.optim import SGD
from torch.utils.tensorboard import SummaryWriter
from torch.utils.data import DataLoader
from sklearn.metrics import confusion_matrix

from dataset_loader.audiodataset import AudioDataset
from models.vadnet import VADNet
from utils.earlystopping import EarlyStopping

class VADTrainingApp:

    def __init__(self, sys_argv=None):

        parser = argparse.ArgumentParser()

        parser.add_argument("--train_path",
                           help='Path to the training set',
                           required=True,
                           type=str,
        )

        parser.add_argument("--test_path",
                           help='Path to the testing set',
                           required=True,
                           type=str,
        )

        parser.add_argument("--save_path",
                           help='Path to saving the model',
                           required=True,
                           type=str,
        )

        parser.add_argument("--save_es",
            help='Save the checkpoints of early stopping call',
            default="checkpoint.pt",
            type=str,
        )

        parser.add_argument('--num-workers',
            help='Number of worker processes for background data loading',
            default=8,
            type=int,
        )

        parser.add_argument("--batch_size",
                            help='Batch size to use for training',
                            default=32,
                            type=int,)

        parser.add_argument('--epochs',
            help='Number of epochs to train for',
            default=50,
            type=int,
        )

        parser.add_argument('--lr',
            help='Learning rate for th stochastic gradient descent',
            default=0.001,
            type=float,
        )

        self.cli_args = parser.parse_args(sys_argv)

        # related to the hardware
        self.use_cuda = torch.cuda.is_available()
        self.device = torch.device("cuda" if self.use_cuda else "cpu")

        # directly related to the neural network
        self.model = self.initModel()
        self.optimizer = self.initOptimizer()

        # For early stopping
        self.patience = 7

        # For metrics
        self.METRICS_LABELS_NDX = 0
        self.METRICS_PREDS_NDX = 1
        self.METRICS_LOSS_NDX = 2
        self.METRICS_SIZE = 3

    def initModel(self):
        """Initialize the model, if GPU available computation done there"""
        model = VADNet()
        model = model.double()

        if self.use_cuda:
            log.info("Using CUDA; {} devices".format(torch.cuda.device_count()))
            if torch.cuda.device_count() > 1:
                model = nn.DataParallel(model)
            model = model.to(self.device)
        return model

    def initOptimizer(self):

        return SGD(self.model.parameters(), lr=self.cli_args.lr)#, momentum=0.8, weight_decay=0.01)

    def adjust_learning_rate(self):
        """Sets the learning rate to the initial LR decayed by a factor of 10 every 20 epochs"""
        self.cli_args.lr = self.cli_args.lr * (0.1 ** (self.cli_args.epochs // 20))
        for param_group in self.optimizer.param_groups:
            param_group['lr'] = self.cli_args.lr

    def initTrainDL(self):

        trainingset = AudioDataset(self.cli_args.train_path,
                                   n_fft=1024,
                                   hop_length=376,
                                   n_mels=128)

        batch_size = self.cli_args.batch_size
        if self.use_cuda:
            batch_size *= torch.cuda.device_count()

        trainLoader = DataLoader(trainingset,
                                batch_size = batch_size,
                                shuffle=True,
                                num_workers=self.cli_args.num_workers,
                                pin_memory=self.use_cuda)
        return trainLoader

    def initTestDL(self):

        testset = AudioDataset(self.cli_args.test_path,
                                   n_fft=1024,
                                   hop_length=376,
                                   n_mels=128)

        batch_size = self.cli_args.batch_size
        if self.use_cuda:
            batch_size *= torch.cuda.device_count()

        testLoader = DataLoader(testset,
                                batch_size = batch_size,
                                shuffle=True,
                                num_workers=self.cli_args.num_workers,
                                pin_memory=self.use_cuda)
        return testLoader

    def main(self):

        log.info("Start training, {}".format(self.cli_args))

        train_dl = self.initTrainDL()
        test_dl = self.initTestDL()

        trn_writer = SummaryWriter(log_dir='runs' + '-trn')
        val_writer = SummaryWriter(log_dir='runs' + '-val')

        early_stopping = EarlyStopping(patience=self.patience, path=self.cli_args.save_es, verbose=True)

        for epoch_ndx in range(1, self.cli_args.epochs + 1):
            log.info("Epoch {} / {}".format(epoch_ndx, self.cli_args.epochs))

            # Adjust the new learning rate
            self.adjust_learning_rate()

            # Train the model's parameters
            metrics_t = self.do_training(train_dl)
            self.logMetrics(metrics_t, trn_writer, epoch_ndx)

            # Test the model
            metrics_v = self.do_val(test_dl, val_writer)
            self.logMetrics(metrics_v, val_writer, epoch_ndx, train=False)

            # Add the mean loss of the val for the epoch
            early_stopping(metrics_v[self.METRICS_LOSS_NDX].mean(), self.model)

            if early_stopping.early_stop:
                print("Early stopping")
                break

        # Save the model once all epochs have been completed
        torch.save(self.model.state_dict(), self.cli_args.save_path)

    def do_training(self, train_dl):
        """Training loop"""
        self.model.train()

        # Initiate a 3 dimension tensor to store loss, labels and prediction
        trn_metrics = torch.zeros(self.METRICS_SIZE, len(train_dl.dataset), device=self.device)

        for batch_ndx, batch_tup in enumerate(train_dl):

            if batch_ndx%100==0:
                log.info("TRAINING --> Batch {} / {}".format(batch_ndx, len(train_dl)))

            self.optimizer.zero_grad()

            loss = self.ComputeBatchLoss(batch_ndx,
                                        batch_tup,
                                        self.cli_args.batch_size,
                                        trn_metrics)

            loss.backward()
            self.optimizer.step()

        return trn_metrics.to('cpu')

    def do_val(self, test_dl, early_stop):
        """Validation loop"""
        with torch.no_grad():
            self.model.eval()

            val_metrics = torch.zeros(self.METRICS_SIZE, len(test_dl.dataset), device=self.device)

            for batch_ndx, batch_tup in enumerate(test_dl):

                if batch_ndx%100==0:
                    log.info("VAL --> Batch {} / {}".format(batch_ndx, len(test_dl)))

                loss = self.ComputeBatchLoss(batch_ndx,
                                            batch_tup,
                                            self.cli_args.batch_size,
                                            val_metrics)

        return val_metrics.to('cpu')

    def ComputeBatchLoss(self, batch_ndx, batch_tup, batch_size, metrics_mat):
        """
        Return a tensor the loss of the batch
        """

        imgs, labels = batch_tup
        imgs = imgs.to(device=self.device, non_blocking=True)
        labels = labels.to(device=self.device, non_blocking=True)

        outputs = self.model(imgs)
        _, predicted = torch.max(outputs, dim=1)

        loss_func = nn.CrossEntropyLoss(reduction="none")
        loss = loss_func(outputs, labels)

        start_ndx = batch_ndx * self.cli_args.batch_size
        end_ndx = start_ndx + labels.size(0)

        metrics_mat[self.METRICS_LABELS_NDX, start_ndx:end_ndx] = labels.detach()
        metrics_mat[self.METRICS_PREDS_NDX, start_ndx:end_ndx] = predicted.detach()
        metrics_mat[self.METRICS_LOSS_NDX, start_ndx:end_ndx] = loss.detach()

        return loss.mean()

    def logMetrics(self, metrics_mat, writer, epoch_ndx, train=True):
        """
        Function to compute custom metrics: accurracy and recall for both classes
        and % of correct predictions. Log the metrics in a tensorboard writer
        """

        # Confusion matrix to compute precision / recall for each class
        tn, fp, fn, tp = torch.tensor(confusion_matrix(metrics_mat[self.METRICS_LABELS_NDX],
                                                       metrics_mat[self.METRICS_PREDS_NDX],
                                                       labels=[0,1]).ravel())

        precision_no_speech = tp / (tp + fp)
        recall_no_speech = tp / (tp + fn)

        # class speech is labelled 0, so true positive = true negative for speech
        precision_speech = tn / (tn + fn)
        recall_speech = tn / (fp + tn)

        # % of correct predictions - optional metrics that are nice
        no_speech_count = (metrics_mat[self.METRICS_LABELS_NDX] == 0).sum()
        speech_count = (metrics_mat[self.METRICS_LABELS_NDX] == 1).sum()

        no_speech_correct = ((metrics_mat[self.METRICS_PREDS_NDX] == 0) & (metrics_mat[self.METRICS_LABELS_NDX] == 0)).sum()
        speech_correct = ((metrics_mat[self.METRICS_PREDS_NDX] == 1) & (metrics_mat[self.METRICS_LABELS_NDX] == 1)).sum()

        correct_all = (speech_correct + no_speech_correct) / float(speech_count + no_speech_count) * 100
        correct_speech = speech_correct / float(speech_count) * 100
        correct_no_speech = no_speech_correct / float(no_speech_count) * 100

        loss = metrics_mat[self.METRICS_LOSS_NDX].mean()

        writer.add_scalar("loss", loss, epoch_ndx)

        writer.add_scalar("precision/no_speech", precision_no_speech, epoch_ndx)
        writer.add_scalar("recall/no_speech", recall_no_speech, epoch_ndx)
        writer.add_scalar("precision/speech", precision_speech, epoch_ndx)
        writer.add_scalar("recall/speech", recall_speech, epoch_ndx)

        writer.add_scalar("correct/all", correct_all, epoch_ndx)
        writer.add_scalar("correct/speech", correct_speech, epoch_ndx)
        writer.add_scalar("correct/no_speech", correct_no_speech, epoch_ndx)

        if train:
            log.info("[TRAINING] loss: {}, correct/all: {}% , correct/speech: {}%, correct/no_speech: {}%".format(loss,
                                                   correct_all,
                                                   correct_speech,
                                                   correct_no_speech))
        else:
            log.info("[VAL] loss: {}, correct/all: {}% , correct/speech: {}%, correct/no_speech: {}%".format(loss,
                                                   correct_all,
                                                   correct_speech,
                                                   correct_no_speech))

if __name__ == "__main__":
    VADTrainingApp().main()

I tried to fit the model on a small subset of the data but the problem remains. I would be very grateful for any help as this problem is getting frustrating
Thank you!

Context:
Well ijt’s difficult to say.
I would use Binary Cross entropy rather than cross entropy as you have a binary case.
I also would do a pooling in after the CNN. Think that those features are spatial and you don’t really care about where the speech is located in the original mel. It also forces each neuron in the linear layer to distinguish between speech/non-speech.
Rather than taking a subset, take a single batch, load it in the gpu and backprop with a simple clean script. This can help you to debug whether there is a bad network desing or a bug in the code.
If everything is okay then I would start from checking the dataset and making sure it works as supposed. If it does you can go for a simpler problem. For example, two clean different categories. Using enhanced speech where the human voice is clearly predominant. In short you have to be sure the task is suitable.


Question: Hi everyone, is it possible to use Kaldi Voice Activity Detection (VAD) in Pytorch?

Context:
I’m not deeply familiar with Kaldi, but how would you like to use it and what have you tried so far?
Are you stuck at a specific point?


Question: I want to convert ogg-file to torch.Tensor using torchaudio.load function, but in deployment I need to do it in C++ (I converted original model to TorchScript). Can I use it from C++? Or what can I do as alternative?

Context: torchaudio does not yet have a C++ interface. However, torchaudio.load is a wrapper around https://github.com/pytorch/audio/blob/master/torchaudio/torch_sox.cpp 129 which you could use.


Model.eval() accuracy is 0 and running_corrects is 0

Question:
I’m having an issue with my DNN model.
During train phase, the accuracy is 0.968 and the loss is 0.103, but during test phase with model.eval(), the accuracy is 0 and the running corrects is 0.
def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):
    model.train()
    liveloss = PlotLosses()
    data_len = len(train_loader.dataset)
    with experiment.train():

        logs = {}
        running_loss = 0.0
        running_corrects = 0

        for batch_idx, _data in enumerate(train_loader):
            features, labels = _data[:][:,:,:-1], _data[..., -1]
            features = features.permute(0, 2, 1)

            features, labels = features.to(device), labels.to(device)

            optimizer.zero_grad()

            output = model(features)
            loss = criterion(output, torch.max(labels, 1)[1])
            loss.backward()

            experiment.log_metric('loss', loss.item(), step=iter_meter.get())
            experiment.log_metric('learning_rate', scheduler.get_last_lr(), step=iter_meter.get())

            optimizer.step()
            scheduler.step()
            iter_meter.step()

            _, preds = torch.max(output, 1)
            running_loss += loss.detach() * features.size(0)
            running_corrects += torch.sum(preds == torch.max(labels, 1)[1])
        epoch_loss = running_loss / len(train_loader.dataset)
        epoch_acc = running_corrects.float() / len(train_loader.dataset)
        logs['log loss'] = epoch_loss.item()
        logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()

iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)

The evaluation script is:

def test(model, device, tst_loader, criterion, epoch, iter_meter, experiment):
    print('\nevaluating...')
    model.eval()
    test_loss = 0
    liveloss = PlotLosses()
    data_len = len(tst_loader.dataset)
    with experiment.test():
        with torch.no_grad():

            logs = {}
            running_loss = 0.0
            running_corrects = 0

            for batch_idx, _data in enumerate(tst_loader):
                features, labels = _data[:][:,:,:-1], _data[..., -1]
                features = features.permute(0, 2, 1)

                features, labels = features.to(device), labels.to(device)

                output = model(features)
                loss = criterion(output, torch.max(labels, 1)[1])
                test_loss += loss.item() / len(tst_loader)

                experiment.log_metric('loss', loss.item(), step=iter_meter.get())

                iter_meter.step()

                _, preds = torch.max(output, 1)
                running_loss += loss.detach() * features.size(0)
                running_corrects += torch.sum(preds == torch.max(labels, 1)[1])

            epoch_loss = running_loss / len(tst_loader.dataset)
            epoch_acc = running_corrects.float() / len(tst_loader.dataset)
            logs['log loss'] = epoch_loss.item()
            logs['accuracy'] = epoch_acc.item()
    liveloss.update(logs)
    liveloss.send()

iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    test(model, device, tst_loader, criterion, epoch, iter_meter, experiment)

Is there something with the code?

Context:
I want to know what version of python this is.
Some previous versions of python do not support +=
This is not likely the issue tho coz it’ll flag an error.
Now I noticed u are not saving the model after training.
U are passing the model into the training function as an argument and u are not returning it after training.
When u then pass the model into the testing function, it doesn’t pass the already trained model.
So try making model a global variable so that whatever changes u make to it from the training function, it’ll reflect on the testing function.
Or
After training the model, return the already trained model and make model = train()


Question: Does Pytorch have an equivalent implementation to Tensorflow’s tf.signal.frame  |  TensorFlow Core v2.5.0 6 ? I’ve been searching everywhere and cannot seem to find anything, even in torchaudio. I know that librosa has an equivalent function, but it requires that the inputs be converted to numpy arrays first

Context: Based on the description I guess that tensor.unfold 27 would perform the same operation (with an additional F.pad before if needed).


Question:
I have a dataset of N audio frames of size (N , 13), and for the task of phoneme recognition
train_data = torch.hstack((train_feat, train_labels))
train_loader = torch.utils.data.DataLoader(train_data, batch_size= 128, shuffle=True)

print(train_data.shape)
torch.Size([3082092, 14])

How can I stack 7 frames each time to feed to the DNN?

Context: I’m not sure which dimension refers to the “frame dimension”, but you could probably either use torch.stack or torch.cat to create the new stacked/concatenated tensor.


RuntimeError: Given groups=1, weight of size [512, 13, 5], expected input[128, 1, 13] to have 13 channels, but got 1 channels instead

Question:
I’m having issue with input/output sizes.
This is my dataloader:
train_data = torch.hstack((train_feat, train_labels))
train_loader = torch.utils.data.DataLoader(train_data, batch_size= 128, shuffle=True)

This is my dataset:
print(len(train_loader))
24079

print((train_feat.shape))
torch.Size([3082092, 13])

print((train_labels.shape))
torch.Size([3082092, 1])

This is my model:
class TDNN(nn.Module):

    def __init__(self, feat_dim=13, embedding_size=512, num_classes=51,
                 config_str='batchnorm-relu'):
        super(TDNN, self).__init__()

        self.network = nn.Sequential(OrderedDict([
            ('tdnn1', TDNNLayer(feat_dim, 512, 5, dilation=1, padding=0,
                                config_str=config_str)),
            ('tdnn2', TDNNLayer(512, 512, 3, dilation=2, padding=0,
                                config_str=config_str)),
            ('tdnn3', TDNNLayer(512, 512, 3, dilation=3, padding=0,
                                config_str=config_str)),
            ('tdnn4', DenseLayer(512, 512, config_str=config_str)),
            ('tdnn5', DenseLayer(512, 1500, config_str=config_str)),
            ('stats', StatsPool()),
            ('affine', nn.Linear(3000, embedding_size))
        ]))
        self.nonlinear = get_nonlinear(config_str, embedding_size)
        self.dense = DenseLayer(embedding_size, embedding_size, config_str=config_str)
        self.classifier = nn.Linear(embedding_size, num_classes)

        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight.data)
                nn.init.zeros_(m.bias)

    def forward(self, x):
        x = self.network(x)
        if self.training:
            x = self.dense(self.nonlinear(x))
            x = self.classifier(x)
        return x

The input should be batches of 128. I’m probably making an error with the data here:
class IterMeter(object):
    """keeps track of total iterations"""
    def __init__(self):
        self.val = 0

    def step(self):
        self.val += 1

    def get(self):
        return self.val

def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment):
    model.train()
    data_len = len(train_loader.dataset)
    with experiment.train():
        for batch_idx, _data in enumerate(train_loader):
            features, labels = _data[:, :-1], _data[:, -1]
            features, labels = features.to(device), labels.to(device)
            features = features.unsqueeze(dim=1)

            optimizer.zero_grad()

            output = model(features)  # (batch, n_class)

            loss = criterion(output, labels)
            loss.backward()

            experiment.log_metric('loss', loss.item(), step=iter_meter.get())
            experiment.log_metric('learning_rate', scheduler.get_lr(), step=iter_meter.get())

            optimizer.step()
            scheduler.step()
            iter_meter.step()
            if batch_idx % 100 == 0 or batch_idx == data_len:
                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                    epoch, batch_idx * len(features), data_len,
                    100. * batch_idx / len(train_loader), loss.item()))

iter_meter = IterMeter()
for epoch in range(1, epochs + 1):
    train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)

If I remore the features.unsqueeze(dim=1), I get:
RuntimeError: Expected 3-dimensional input for 3-dimensional weight [512, 13, 5], but got 2-dimensional input of size [128, 13] instead

Context: @ptrblck any idea what’s causing the error?


Question:
So I have an input tensor with shape [16, 1, 125, 256] and a selector tensor with shape [124, 2]. Is there any PyTorch equivalent of tf.gather(input, selector, axis=2)? To begin with, how does Tensorflow handle both tensors despite not having the same number of dimensions (unlike torch.gather wherein they must be equal)? Furthermore, what must dim in torch.gather be to be similar to axis=2 in tf.gather.
For more context, I’m trying to make PyTorch version of this 3, which defines a frame function that expands signal's axis dimension into frames of frame_length.

Context:
It’s still not 100% clear to me what this does  . But I guess that would work:
dim = 2
new_size = inp.size()[:dim] + selector.size() + inp.size()[dim+1:]
out = inp.index_select(selector.view(-1), dim=dim)
out.view(new_size)