Create dataset out of x_train and y_train

Question:
How to put the x_train and y_train into a model for training?
the x_train is a tensor of size (3000, 13).
the y_train is of size (3000, 1)
That is for each element of x_train (1, 13), the respective y label is one digit from y_train.
if I do:
train_data = torch.hstack((train_feat, train_labels))
print(train_data[0].shape)
print(train_data[1].shape)

torch.Size([3082092, 13])
torch.Size([3082092, 1])
train_loader = data.DataLoader(dataset=train_data,
                                batch_size= 7,
                                shuffle=True)

The dataloader  does not return the batch size, but returns the whole dataset instead

Context:
I’m not sure what you’re trying to do.
You want to apply a convolution with a kernel size of 7 to a 1d vector of size 13, is that correct ?
In that case your input should be a tensor with 1 channel containing a 1D vector of size 13.
So your input size should be [7,  1,  13] (batch_size, channels, dim).
And in your model, the first conv must have an one input channel instead of 13.
Given your error, you’re giving to your model an input of size [3082092, 13] so it seems you didn’t followed my first comment. Starting from what I wrote, we have:
features, labels = batch[:, :-1], batch[:, -1] # features size is [7, 13]

In order to obtain the needed dimension you simply need to create the channel dim:
features = features.unsqueeze(dim=1) # feature size is now [7, 1, 13]

Then you can apply your model (with the first conv corrected to have 1 input channel).
Then after this first convolution your tensor will be of shape [7, 1024, 7] (batch_size, output_dim of the fist conv, output_size in function of padding, dilation, and stride)
As you seem to apply two convolutions with a kernel size of 1, the output dim won’t change. So at the end of your model, the size is [7, 50, 7].
If you wanna feed that to a linear classifier, you can flatten the last two dims and feed the result to your classifier, and correct your classifier input size which should be 50 * 7.
Here is a complete example:
model = torch.nn.Sequential(
    torch.nn.Conv1d(1, 1024, kernel_size=7, stride=1, padding=0, dilation=1, groups=1, bias=True),
    torch.nn.BatchNorm1d(1024),
    torch.nn.ReLU(inplace=True),
    torch.nn.Conv1d(1024, 1024, kernel_size=1, stride=1, padding=0, dilation=1, groups=1, bias=True),
    torch.nn.BatchNorm1d(1024),
    torch.nn.ReLU(inplace=True),
    torch.nn.Conv1d(1024, 50, kernel_size=1, stride=1, padding=0, dilation=1, groups=1, bias=True)
    )

classifier = torch.nn.Linear(50 * 7, 50)

dummy_features = torch.randn(3000, 13)
dummy_labels   = torch.randint(2, (3000, 1))  # integers in [0, 1]

train_data = torch.hstack((dummy_features, dummy_labels))

train_loader = torch.utils.data.DataLoader(train_data, batch_size= 7, shuffle=True)

batch = next(iter(train_loader))
features, labels = batch[:, :-1], batch[:, -1]

features = features.unsqueeze(dim=1)

outputs = model(features)

outputs = outputs.view(outputs.size(0), -1)

scores = classifier(outputs) # size (7, 50)


How to create a custom dataset for audio recognition

Question:
Hi,
I have a question, I have a dataset of audiofiles that I’d like to convert into melspectogram and I want to use tourchaudio library to convert audio into a tensor directly. I’ve seen some people doing this by saving as an image, and I’d like to bypass that step, and train directly as a tensor. My question is, how should I do regarding, creating a Dataloader so that I can do this computational expensive operation and taking advantage of the fact that each folder is the label that contains all the training data for that class. The end goal is create a classification algorithm
Thanks so much!

Context: You could create a custom Dataset as explained in this tutorial 49 and apply the transformation on each sample in the __getitem__ of this class. To create the class indices based on the folder structure you could reuse the logic from DatasetFolder 7.


Question:
Hey guys,
I need to install the torchaudio library via source because I had to install pytorch via source too(NPACK problem).
So my question is how to install  torchaudio via source becuase when I try to install this library via conda or pip  my pytorch source build is being deleted and the normal pytorch is being downloaded.
In order to avoid the deinstallation of my pytorch build I have to install torchaudio via source.
Thank’s for every suggestion and help in advance:)

Context: The install instructions are found here 85. Let us know, if you encounter any issues.


Question: Is there any difference in training on the spectrogram of a waveform as compared to training on the waveform itself? Does training on the waveform generate better results in general?

Context: I think the difference would be quite large, as the sampling rate could be high in the time domain and make the training quite challenging. If I remember correctly, one way to make some models work on waveforms directly was to use a stack of conv layers with a specific dilation, so that the input size was feasible to process in the end.


Training model with large dataset on a GPU with insufficient Memory (effeciently)

Question:
Hi.
I am working with audio data. The code that I am running takes the X and label data from the dataloader, sends it to GPU with X.cuda() and label.cuda() and then pass them through the forward graph. The dataset size in .npy files is around 8GB. My machine is RTX 2060 which has 6 gb memory. So if i run it on my GPU, it processes some batches and runs out of memory, although the code runs fine on Colab, which has tesla T4 with 15 GB memory. The reason for this behavior is I am guessing, by the first iteration, the code tries to force all the tensors on the gpu, but the size of tensors is greater than GPU RAM, That’s why cuda runs out of memory.
Here is what I want to do. I want to load A tensor to the GPU, compute the forward and backward graph, and then send it back to CPU again. But I know that these operations like data.cuda() andn data.cpu() are expensive. So I want them to be parallel to the forward and backward pass (model(input) to be specific).
Is it possible? If so, how?
Thanks!

Context:
Write a custom data loader to do lazy loading, that is, at __init__ just load the list of data ( this could be path information, row/column/line information of each data. The actually .npy data is not loaded yet and with the help of that list, load data batch-wise using the __getitem__.
Check the link below, they do something very similar!

    Loading huge data functionality

    Just define a Dataset object, that only loads a list of files in __init__, and loads them every time __getindex__ is called. Then, wrap it in a torch.utils.DataLoader with multiple workers, and you’ll have your files loaded lazily in parallel.
class MyDataset(torch.utils.Dataset):
    def __init__(self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__(self, idx):
        return load_file(self.data_files[idx])

    def __len__(self):
        r…


Efficient way of accessing N multiples of k indexes and performing an operation over it

Question:
Hi all,
Im trying to implement a filter that runs over a spectrogram frequencies for each time frame and filters out inharmonic frequencies.
What I need is, given a time frame, for each frequency k in this timeframe, to access frequencies multiple of k (the harmonics), then perform some operation over them. So for frequency k, k=1,…,F, I need to access indexes, k, 2k, 3k,…, N*k (N in total), perform some operation and set the result to the position k (original frequency bin). Same calculation for all timeframes.
In resume, it would be something like this:
spec[k, :] = operation(spec[mutiple_indexes,:])
Basically its like running a operation sliding window starting in each frequency, but for each frequency this sliding window is more space (spaced by the frequency value).
I can do this iteratively, but Im wondering if there is a vectorized way of doing this.
I was thinking about torch.unfold, but it can only work on sequential patches, not of increasing stride.
Any suggestions?
Thanks!

Context:
I’m gonna assume you want to run 1D convolutions just in those specific bins rather than 2D convolutions around the harmonics.
If that were the case you can preset an slice operator which retrieves those harmonics, then you can run a 2D convolution with kernel size 1 for the frequency. With a proper padding you can keep the shape and then using the preset slice to “paste” back the values.


PyTorch WaveNet implementation complications

Question:
Currently, we are a group doing a project about implementing WaveNet in a Tacotron2 → WaveNet → ASR (Given by firm) for midterm project. We are all novices to PyTorch, but recommended to try this library for constructing our WaveNet. We have a problem with the padding and the F.cross_entropy problem for a given .wav-file.
The main issue is when we compute the loss function. Our output (from WaveNet) is a tensor of shape:
output = tensor([1, 256, 225332]) # [batch_size, sample_size, audio_length]
input = tensor([1, 256, 225360])

There is a problem here, and from what I can see and talk to my supervisor about it is padding the input of the WaveNet. (Cross_entropy wants (N, C) as input and (N) as target, from what I gather, and the dimensions are wrong)
He said “use ‘same’ padding”, but that is currently only operable in TF/Keras as far as I know. We’ve tried to read across multiple posts, but since we’re novices, we can’t seem to figure it out. Any help is appreciated.
This is our WaveNet, which probably has some issues (particularly padding and perhaps causal convolution seems iffy?).
"""
Wavenet model
"""

from torch import nn
import torch

#TODO: Add local and global conditioning

def initialize(m):
    """
    Initialize CNN with Xavier_uniform weight and 0 bias.
    """
    if isinstance(m, torch.nn.Conv1d):
        nn.init.xavier_uniform_(m.weight)
        nn.init.constant_(m.bias, 0.0)

class CausalConv1d(torch.nn.Module):
    """
    Causal Convolution for WaveNet

    - Jakob
    """

    def __init__(self, in_channels, out_channels, kernel_size, dilation = 1, bias = True):
        super(CausalConv1d, self).__init__()
        # padding=1 for same size(length) between input and output for causal convolution
        self.dilation = dilation
        self.kernel_size = kernel_size
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.padding = padding = (kernel_size-1) * dilation # kernelsize = 2, -1 * dilation = 1, = 1. - Jakob.
        self.conv = torch.nn.Conv1d(in_channels, out_channels,
                                    kernel_size, padding=padding, dilation=dilation,
                                    bias=bias)  # Fixed for WaveNet but not sure

    def forward(self, x):
        output = self.conv(x)
        if self.padding != 0:
            output = output[:, :, :-self.padding]
        return output

class Wavenet(nn.Module):

    def __init__(self,
                 layers=3,
                 blocks=2,
                 dilation_channels=32,
                 residual_block_channels=512,
                 skip_connection_channels=512,
                 output_channels=256,
                 output_size=32,
                 kernel_size=3
                 ):

        super(Wavenet, self).__init__()

        self.layers = layers
        self.blocks = blocks
        self.dilation_channels = dilation_channels
        self.residual_block_channels = residual_block_channels
        self.skip_connection_channels = skip_connection_channels
        self.output_channels = output_channels
        self.kernel_size = kernel_size
        self.output_size = output_size

        # initialize dilation variables
        receptive_field = 1
        init_dilation = 1

        # List of layers and connections
        self.dilations = []
        self.residual_convs = nn.ModuleList()
        self.filter_conv_layers = nn.ModuleList()
        self.gate_conv_layers = nn.ModuleList()
        self.skip_convs = nn.ModuleList()

        # First convolutional layer
        self.first_conv = CausalConv1d(in_channels=self.output_channels,
                                    out_channels=residual_block_channels,
                                    kernel_size = 2)

        # Building the Modulelists for the residual blocks
        for b in range(blocks):
            additional_scope = kernel_size - 1
            new_dilation = 1
            for i in range(layers):
                # dilations of this layer
                self.dilations.append((new_dilation, init_dilation))

                # dilated convolutions
                self.filter_conv_layers.append(nn.Conv1d(in_channels=residual_block_channels, out_channels=dilation_channels, kernel_size=kernel_size, dilation=new_dilation))

                self.gate_conv_layers.append(nn.Conv1d(in_channels=residual_block_channels, out_channels=dilation_channels, kernel_size=kernel_size, dilation=new_dilation))

                # 1x1 convolution for residual connection
                self.residual_convs.append(nn.Conv1d(in_channels=dilation_channels, out_channels=residual_block_channels, kernel_size=1))

                # 1x1 convolution for skip connection
                self.skip_convs.append(nn.Conv1d(in_channels=dilation_channels,
                                                 out_channels=skip_connection_channels,
                                                 kernel_size=1))

                # Update receptive field and dilation
                receptive_field += additional_scope
                additional_scope *= 2
                init_dilation = new_dilation
                new_dilation *= 2

        # Last two convolutional layers
        self.last_conv_1 = nn.Conv1d(in_channels=skip_connection_channels,
                                  out_channels=skip_connection_channels,
                                  kernel_size=1)

        self.last_conv_2 = nn.Conv1d(in_channels=skip_connection_channels,
                                    out_channels=output_channels,
                                    kernel_size=1)

        #Calculate model receptive field and the required input size for the given output size
        self.receptive_field = receptive_field
        self.input_size = receptive_field + output_size - 1

    def forward(self, input):

        # Feed first convolutional layer with input
        x = self.first_conv(input)

        # Initialize skip connection
        skip = 0

        # Residual block
        for i in range(self.blocks * self.layers):

            (dilation, init_dilation) = self.dilations[i]

            # Residual connection bypassing dilated convolution block
            residual = x

            # input to dilated convolution block
            filter = self.filter_conv_layers[i](x)
            filter = torch.tanh(filter)
            gate = self.gate_conv_layers[i](x)
            gate = torch.sigmoid(gate)
            x = filter * gate

            # Feed into 1x1 convolution for skip connection
            s = self.skip_convs[i](x)

            #Adding skip & Match size with decreasing dimensionality of x
            if skip is not 0:
                skip = skip[:, :, -s.size(2):]
            skip = s + skip # Sum all skip connections

            # Feed into 1x1 convolution for residual connection
            x = self.residual_convs[i](x)
            #Adding Residual & Match size with decreasing dimensionality of x
            x = x + residual[:, :, dilation * (self.kernel_size - 1):]

            # print(x.shape)

        x = torch.relu(skip)
        #Last conv layers
        x = torch.relu(self.last_conv_1(x))
        x = self.last_conv_2(x)
        soft = torch.nn.Softmax(dim=1)
        x = soft(x)
        return x

The training file:
model = Wavenet(layers=3,blocks=2,output_size=32).to(device)
model.apply(initialize) # xavier_uniform_ : Does this work?
model.train()

optimizer = optim.Adam(model.parameters(), lr=0.0003)
for i, batch in tqdm(enumerate(train_loader)):
    mu_enc_my_x = encode_mu_law(x=batch, mu=256)
    input_tensor = one_hot_encoding(mu_enc_my_x)

    input_tensor = input_tensor.to(device)
    output = model(input_tensor)
    # TODO: Inspect input/output formats, maybe something wrong....
    loss = F.cross_entropy(output.T.reshape(-1, 256), input_tensor[:,:,model.input_size - model.output_size:].long().to(device)) # subtract receptive field instead of pad it, workaround for quick debugging of loss-issue.
    print("\nLoss:", loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i % 1000 == 0:
        print("\nSaving model")
        torch.save(model.state_dict(), "wavenet.pt")

Context:
Adding the padding to the conv layers is the right approach, but I’m unsure why you are slicing it in the forward:
    def forward(self, x):
        output = self.conv(x)
        if self.padding != 0:
            output = output[:, :, :-self.padding]
        return output

This would remove the “right hand side” of the padded output, so is this intended?
Also, nn.CrossEntropyLoss expects raw logits, so remove the nn.Softmax in your model and just pass the output of the last layer to the criterion.


TorchAudio : __init__() got an unexpected keyword argument ‘center’

Question:
Hello,
I was passing some mel arguments into the torchaudio.transforms.MFCC function but it doesn’t seem to recognize the argument “center”. Center is a parameter to the Mel Spectorgram so I don’t understand why it’s not working.
Code:
melkwargs = {
    "center": False,
    "n_fft": n_fft,
    "hop_length": hop_length,
    "power": 2,
    "f_min": f_min,
    "f_max": f_max
}

mfcc_module = torchaudio.transforms.MFCC(sample_rate=sample_rate, n_mfcc=20, melkwargs=melkwargs)
torch_mfcc = mfcc_module(torch.tensor(waveform))

Error:

TypeError: init() got an unexpected keyword argument ‘center’

Any help would be appreciated.

Context: You might need to update torchaudio to 0.8.0, as the center argument doesn’t seem to be available in previous versions.


Pass custom cmake build arguments to setup.py

Question:
Hi everyone!
I am trying to build torchaudio from source. In my environment, CUDA and its related libraries (e.g. cuDNN) are installed in non-default paths.
How can I pass custom build arguments to cmake via setup.py?
Thanks a lot.
Kind regards,
GT

Context: We do not have CUDA specific code yet, and so the torchaudio build doesn’t depend the nvidia compiler. What error are you encountering? If you do run into an error with this, I’d recommend opening an issue directly on github.com/pytorch/audio 1


Question:
When creating a custom dataset loader like that shown here 6. Is it advisable to do something like
class CustomDataset(Dataset):
    def __init__(self, csv_file, root_dir):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        audio_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        target_path = os.path.join(self.root_dir, self.annotations.iloc[index, 1])
        audio, _ = torchaudio.load(audio_path)
        target, _ = torchaudio.load(target_path)
        return audio, target

when both the input and expected output are waveforms? In my case, I created a csv file that contains the filenames of both audios and I just load the path-like object into torchaudio.load(). Also, how do I ensure that all the audio and target in a batch are of the same length when being loaded?

Context:
Well there are two main possibilities.
One is that you use a fix length of audio. You should make chunks of audio given a sample.
Another option is that you use variable length (typically when using transformers)
In that case you should rewrite dataloader’s collate function.


Question: Hi I’m trying to make an autoencoder for speech data. The network’s input and output are Mel spectrograms. How can I obtain the audio waveform from the generated mel spectrogram?

Context: https://librosa.org/doc/latest/generated/librosa.feature.inverse.mel_to_audio.html?highlight=mel%20spectrogram 91


Question:
Hi!
I was wondering if there’s a proper way to learn the parameters of a sinusoid by gradient descent?
Here’s an example setup:
x = torch.arange(0, 5, 1 / sr) # =5s of audio
y = torch.sin(2 * np.pi * 2.85 * x) # want to get sinusoid with frequency 2.85
f = nn.Parameter(torch.tensor(2.0)) # initialize frequency parameter as 2.85

opt = torch.optim.AdamW([f], lr=0.001)
loss = nn.MSELoss()

for i in range(100):
    out = torch.sin(2 * np.pi * f * x)
    l = loss(out, y)
    l.backward()
    opt.step()
    opt.zero_grad()

This won’t learn the correct value for f (2.85), but rather stay around 2.
Is there a way to set this up so it can learn through a periodic function?
Thanks for any input!
edit: simplified code somewhat

Context: I think maybe evaluate the loss in frequency domain via fft will be easier, because directly train on time domain, in my experiences, is generally very hard


Using pytorch vggish for audio classification tasks

Question:
I am researching on using pretrained VGGish 86 model for audio classification tasks, ideally I could have a model classifying any of the classes defined in the google audioset. I came across a nice pytorch port 68 for generating audio features. The original model generates only audio features as well. The original team suggests generally the following way to proceed:

As a feature extractor : VGGish converts audio input features into a semantically meaningful, high-level 128-D embedding which can be fed as input to a downstream classification model. The downstream model can be shallower than usual because the VGGish embedding is more semantically compact than raw audio features. So, for example, you could train a classifier for 10 of the AudioSet classes by using the released embeddings as features. Then, you could use that trained classifier with any arbitrary audio input by running the audio through the audio feature extractor and VGGish model provided here, passing the resulting embedding features as input to your trained model. vggish_inference_demo.py shows how to produce VGGish embeddings from arbitrary audio

I’m not sure how to go about using getting the released embeddings and using them for training in pytorch. I’m also not sure how to translate the embeddings into classification. Could any one kindly share some pointers? Thanks!

Context:
Based on the description you’ve posted it seems the authors call the output features embeddings.
This might be a bit confusing, as there are nn.Embedding layers, which are apparenrently not meant here.
If I understand the use case correctly, you could store each output feature of the VGGish model with its corresponding target, create a new classification model, and use these output features + targets to train this new classifier.
According to their claim, this classifier can be “shallower”, as the “embeddings” are so great.
Let me know, if that makes sense.


How can I get the Hidden Embeddings from the 2nd last fully connected layer for t-SNE visualization?

Question:
how can we do this in pytorch?
After training the model,   get the learned representations from the 2nd last FC layer for t-SNE visualization. The Model is defined below
class VisualNet(nn.Module):
    def __init__(self):
        super(VisualNet, self).__init__()
        self.conv1 = nn.Conv2d()  #Ignore the function values
        slef.conv2 = nn.Conv1d()
        slef.conv3 = nn.Conv1d()
        slef.conv4 = nn.Conv1d(512,512)

       self.fc1   = nn.Linear(512,256)
        self.fc2   = nn.Linear(256, 8)

    def forward(self, x):
        x = self.conv1(self.conv2(self.conv3(self.conv4(x)))
        x_fc1 = self.fc1(x)
        x = self.fc2(x_fc1)

        return x

model = VisualNet()

After finishing Training, How can I get x_fc1 embeddings
for t-SNE ??

Context: Assuming you are looking for the output activations of self.fc1 (unsure what embeddings would mean in this context), you could use forward hooks as described here 16.


Question:
Hello,
I am trying to train this on a Raspberry 2 which has 512MBs of RAM. I reduced the original Google Commands Dataset into 12 classes and 34000 training samples and I am using a Resnet model with 8 layers.
class ResNet(BaseModel):
    def __init__(self, config):
        super().__init__()
        self.n_layers = config["n_layers"]
        n_maps = config["n_feature_maps"]

        self.layers = nn.ModuleDict()

        self.layers["conv_0"] = nn.Conv2d(1, n_maps, (3, 3), padding=1, bias=False)

        for i in range(1, self.n_layers + 1):
            if config["use_dilation"]:
                padding_size = int(2**((i-1) // 3))
                dilation_size = int(2**((i-1) // 3))
                self.layers[f"conv_{i}"] = nn.Conv2d(n_maps, n_maps, (3, 3), padding=padding_size, dilation=dilation_size, bias=False)
            else:
                self.layers[f"conv_{i}"] = nn.Conv2d(n_maps, n_maps, (3, 3), padding=1, bias=False)
            self.layers[f"bn_{i}"] = nn.BatchNorm2d(n_maps, affine=False)

        if "pool" in config:
            self.layers["pool"] = nn.AvgPool2d(config["pool"])

        self.layers["output"] = nn.Linear(n_maps, config["n_labels"])

        self.activations = nn.ModuleDict({
            "relu": nn.ReLU()
        })

    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.layers["conv_0"](x)
        x = self.activations["relu"](x)

        if "pool" in self.layers:
            x = self.layers["pool"](x)

        prev_x = x
        for i in range(1, self.n_layers + 1):
            x = self.layers[f"conv_{i}"](x)
            x = self.activations["relu"](x)

            if i % 2 == 0:
                x = x + prev_x
                prev_x = x

            x = self.layers[f"bn_{i}"](x)

        x = x.view(x.size(0), x.size(1), -1) # shape: (batch, features, o3)
        x = x.mean(2)
        x = self.layers["output"](x)
        return x

It can be trained successfully on my laptop, however, I get MemoryError on the rpi2. I split the wav list thus they all won’t be loaded to the memory at once. (There are almost 1,5 GBs of wav files, rpi2 has only 512 MBs of RAM) I set --num_workers argument to 0. (CPU memory gradually leaks when num_workers > 0 in the DataLoader) However I can’t solve the problem.
Environment:
PyTorch Version: 1.3.0
Python version: 3.7.3
OS: Raspbian GNU/Linux 10 (buster)
CPU: armv7l
What could be the problem?
Thank you so much!

Context:
Could you reduce the batch size to a single sample and rerun the code, in case your current batch size is larger?
If that doesn’t help, could you remove the data loading as a debugging step and feed a single sample as x = torch.randn(...) to the model to check the memory usage?


Which dimensions are the real and imaginary parts of an old complex tensor in?

Question:
I’m currently trying to use torchaudio.transforms.Spectrogram with power = None to return the complex spectrogram. The forward method of this class returns a tensor with a shape of (...,2). I am aware that the last two dimensions represent the real and imaginary parts, but it is not clear from the documentation which one is which and how they represent a complex tensor.
For example, if torchaudio.transforms.Spectrogram returns x with shape (256,256,2), is x[...,0] the real part or the imaginary part? Also, if x was a complex tensor, then would it be expressed as x[...,0] + x[...,1]j or x[...,1] + x[...,0]j?

Context: So it’s x[...,0] + x[...,1]j?


PyTorch Wavenet Model loss is not decreasing (Help)

Question:
I’m a college student trying to implement Wavenet using PyTorch, this is my first time writing custom modules for a model in PyTorch and I’m having a problem with my model in that it won’t train. Essentially the project is this: I have a set of wav files that I am reading in, processing and quantizing as in the Wavenet paper, and am arranging into series of 1024 data points (the model takes a series of 1024 amplitudes from the wav as input and should output a tensor of 256 values describing the probability that the next item in the series is one of those 256 values).
I’m currently trying to train the model on a single music file, hoping to get it to overfit so that I can be sure that it actually learns from the data. Here lies the problem: the loss won’t decrease as I train. I’ve tried making the model smaller, changing the loss function, changing what kind of layer the output layer is, and making the learning rate larger but nothing seems to work.
I suspect that the problem is somewhere in the model itself, the way that it is constructed may be wrong. It’s possible that I could have linked the custom modules together with the model in a way that interferes with back propagation, at least that is my best guess. My code for the model and my training code is below. I would really appreciate some help!
#model https://github.com/Dankrushen/Wavenet-PyTorch/blob/master/wavenet/models.py
#https://github.com/ryujaehun/wavenet/blob/master/wavenet/networks.py
#https://medium.com/@satyam.kumar.iiitv/understanding-wavenet-architecture-361cc4c2d623
#https://discuss.pytorch.org/t/causal-convolution/3456/4

import torch
import torch.optim as optim
from torch import nn
from functools import reduce

#causal convolution (citation above)
class CausalConv1d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, dilation=1, **kwargs):
        super(CausalConv1d, self).__init__()
        self.pad = (kernel_size - 1) * dilation
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.kernel_size = kernel_size
        self.dilation = dilation

        self.conv = nn.Conv1d(in_channels, out_channels, kernel_size, padding=self.pad, dilation=dilation, **kwargs)

    def forward(self, x):
        return self.conv(x)

class ResidualBlock(nn.Module):
    def __init__(self, input_channels, output_channels, kernel_size, skip_size, skip_channels, dilation=1):
        super(ResidualBlock, self).__init__()
        self.dilation = dilation
        self.skip_size = skip_size
        self.conv_s = CausalConv1d(input_channels, output_channels, kernel_size, dilation)#dim
        self.sig = nn.Sigmoid()
        self.conv_t = CausalConv1d(input_channels, output_channels, kernel_size, dilation)#dim
        self.tanh = nn.Tanh()
        self.conv_1 = nn.Conv1d(output_channels, output_channels, 1)#dim -> k = 1

        self.skip_conv = nn.Conv1d(output_channels, skip_channels, 1)

    def forward(self, x):
        o = self.sig(self.conv_s(x)) * self.tanh(self.conv_t(x))
        skip = self.skip_conv(o)
        skip = skip[:,:,-self.skip_size:] #dim control for adding skips
        residual = self.conv_1(o)

        return residual, skip

class WaveNet(nn.Module):#SET SKIP SIZE default
    def __init__(self, skip_size=256, num_blocks=2, num_layers=10, num_hidden=128, kernel_size=2):
        super(WaveNet, self).__init__()
        self.layer1 = CausalConv1d(1, num_hidden, kernel_size)#dim
        self.res_stack = nn.ModuleList()

        for b in range(num_blocks):
            for i in range(num_layers):
                self.res_stack.append(ResidualBlock(num_hidden, num_hidden, kernel_size, skip_size=skip_size, skip_channels=1, dilation=2**i))#dim

        #self.hidden = nn.ModuleList(self.hidden)

        self.relu1 = nn.ReLU()
        self.conv1 = nn.Conv1d(1,1,1)#dim
        self.relu2 = nn.ReLU()
        self.conv2 = nn.Conv1d(1,1,1)#dim
        self.output = nn.Softmax()

    def forward(self, x):
        skip_vals = []
        #initial causal conv
        o = self.layer1(x)

        #run res blocks
        for i, layer in enumerate(self.res_stack):
            o, s = layer(o)
            skip_vals.append(s)

        #sum skip values and pass to last portion of network
        o = reduce((lambda a,b: a+b), skip_vals)
        o = self.relu1(o)
        o = self.conv1(o)
        o = self.relu2(o)
        o = self.conv2(o)

        return self.output(o)

#overfit model to test if it will train
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)
net = WaveNet(num_layers=1)

#send to gpu
net.to(device)

criterion = nn.CrossEntropyLoss() #preproc data, stream, train, remember to reformat y label as onehot vector for classification
optimizer = optim.Adam(net.parameters(),lr=0.001)
num_epochs = 20
losses = []

_, inp = wav_to_data(data_path+'/'+wav_files[0])
data = encode(inp)

batch_size = 32

i = 0
for epoch in range(num_epochs):
    i += 1
    for s in range(0, len(data) - 1024, batch_size):
        (x, y) = create_singular_input_stream(data, s, batch_size)
        optimizer.zero_grad()

        output = net(torch.reshape(torch.FloatTensor(x).to(device), (batch_size,1,1024)))
        print(output.shape)
        print(torch.Tensor([y]).shape)

        #find loss between distributions of amplitudes
        #https://discuss.pytorch.org/t/indexerror-target-1-is-out-of-bounds-nlloss/68656
        loss = criterion(torch.squeeze(output), torch.squeeze(torch.Tensor(y)).type(torch.LongTensor).to(device))

        loss.backward()
        optimizer.step()
        losses.append(loss.item())

        print('Epoch {}/{}, Loss: {:.6f}'.format(i, num_epochs, loss.item()))

A helpful note:
create_singular_input_stream returns x (1,1024 tensor with series) and y (next series value [0,255])
Again, any help is appreciated - I’d love to know what I’m doing wrong.

Context:
@Arham_Khan
Just some mistakes I have saw:

The padding in CausalConv1d is not causal. You’re going to pad (kernel_size - 1) * dilation on both side of the input’s last dimension. The correct way should be padding the input before the Conv1d layer in the forward call and set padding to 0 in Conv1d.

...
    self.conv = nn.Conv1d(in_channels, out_channels, kernel_size, padding=0, dilation=dilation, **kwargs)

def forward(self, x):
    x = F.pad(x, (self.pad, 0))       # only pad on the left side
    return self.conv(x)

The skip_size param doesn’t make any sense to me, especially in this line:

skip = skip[:,:,-self.skip_size:] #dim control for adding skips

You set skip_size to 256 to match the one-hot vector of y, which represent the value of the sample just next to 1024 input samples. But you take 256 values from the last axis which represent time, so you’re somehow mapping 256 different time position values to 256 different amplitude values, looks weird to me .
Those 256 hidden values should be taken from the second axis, which represent hidden channels.

You directly cast x to float type, which make the quantization completely useless. In my opinion, the shape of x should be (batch, 256, 1024) with one-hot vector along the second axis, and the shape of y should be (batch, 256, 1) with one-hot vector along the second axis.

I think you might have some misunderstanding of WaveNet, but this is normal, because the original paper didn’t write everything very clearly (to me), so I recommend you to take a look at others’ implementation to get a wider view of WaveNet.
The wavenet model I have implemented:

      github.com

    yoyololicon/wavenet-like-vocoder/blob/3591786b6507173323b7b657a98b9bdffabbee46/model/model.py#L23 16

        dilations = radix ** torch.arange(n_layers)        dilations = dilations.tolist()        if descending:            dilations = dilations[::-1]        self.dilations = dilations * n_blocks        self.cls = classes        self.r_field = sum(self.dilations) + 1        self.rdx = radixclass WaveNet(_BaseModel):    def __init__(self,                 n_blocks=4,                 n_layers=10,                 classes=256,                 radix=2,                 descending=False,                 aux_channels=80,                 dilation_channels=256,                 residual_channels=256,                 skip_channels=256,


Question:
The docs to suggest using of logarithmized probabilities for an input of CTCLoss
https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss 6
However, there is example in pytorch repo (the DeepSpeech model) where softmax func (instead of log_softmax) is used just for evaluation but not training.

      github.com

    pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L135-L140

class InferenceBatchSoftmax(nn.Module):    def forward(self, input_):        if not self.training:            return F.softmax(input_, dim=-1)        else:            return input_

So I want to clarify what should I use for training and evaluation in CTCLoss:

softmax/log_softmax for train/eval?
identity for the training and softmax/log_softmax for eval like in example I shared above?

Context: As far as I know, for training you need log_softmax. For inference you can just do argmax. But using argmax might only give you Top-1 accuracy. If you use softmax and get top 5 scores you can get Top-5 accuracy. I think DeepSpeech model does something similar.


Feeding 1d PackedSequence data to an LSTM

Question:
I am trying to train an LSTM on audio signal data. I gave used pad_sequnce() and pack_padded_sequence() to get resultant PackedSequence data.
However this data is 1 dimensional. I have checked it using x.data.shape in forward() function and it is a 1d tensor. (x here is a PackedSequence data)
And I’ve used batch_first=True parameter also when defining lstm.
I’m  getting this error:
~/.conda/envs/ml/lib/python3.8/site-packages/torch/nn/modules/rnn.py in check_input(self, input, batch_sizes)
    172         expected_input_dim = 2 if batch_sizes is not None else 3
    173         if input.dim() != expected_input_dim:
--> 174             raise RuntimeError(
    175                 'input must have {} dimensions, got {}'.format(
    176                     expected_input_dim, input.dim()))

RuntimeError: input must have 2 dimensions, got 1

Would adding a dummy dimension to this data work?
How can I do this to PackedSequence data?

Context: I solved this by adding a newaxis to sequence data before padding and packing in collate_fn() of dataloader.


Torchaudio: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Question:
Here is my code:
import sys
import torch
import torchaudio

def train(net,dataloader,loss_func,optimizer,device):

    # put in training mode

    net.train()

    # to compute training accuracy

    num_true_pred = 0

    # to compute epoch training loss

    total_loss = 0

    for i,(signals,labels) in enumerate(dataloader):

        # track progress

        sys.stdout.write('\rProgress: {:.2f}%'.format(i*dataloader.batch_size/len(dataloader)))
        sys.stdout.flush()

        # load onto GPU
        signals = signals.to(device).unsqueeze(dim=1)
        labels = labels.to(device).type_as(signals) # needed for BCE loss

        # compute log Mel spectrogram

        mel_spec = torchaudio.transforms.MelSpectrogram(sample_rate = 16000,
                                                        n_fft = 1024,
                                                        n_mels = 256,
                                                        hop_length = 63).to(device)
        to_dB = torchaudio.transforms.AmplitudeToDB().to(device)
        images = to_dB(mel_spec(signals))

        # zero the accumulated parameter gradients

        optimizer.zero_grad()

        # outputs of net for batch input

        outputs = net(images).squeeze()

        # compute (mean) loss

        loss = loss_func(outputs,labels)

        # compute loss gradients with respect to parameters

        loss.backward()

        # update parameters according to optimizer

        optimizer.step()

        # record running statistics

        # since sigmoid(0) = 0.5, then negative values correspond to class 0
        # and positive values correspond to class 1

        class_preds = outputs > 0
        num_true_pred = num_true_pred + torch.sum(class_preds == labels)

        # loss is not mean-reduced

        total_loss = total_loss + loss

    train_loss = total_loss.item() / len(dataloader.dataset)

    train_acc = num_true_pred.item() / len(dataloader.dataset)

    return net,train_loss,train_acc

I keep getting the following error:
RuntimeError                              Traceback (most recent call last)

<ipython-input-20-b175e934d6f9> in <module>()
     18                                      loss_func,
     19                                      optimizer,
---> 20                                      device)
     21
     22     print('Training Loss: {:.4f}'.format(train_loss))

6 frames

/content/drive/My Drive/Colab Notebooks/disc_baseline/net_train.py in train(net, dataloader, loss_func, optimizer, device)
     34                                                         n_mels = 256,
     35                                                         hop_length = 63).to(device)
---> 36         to_dB = torchaudio.transforms.AmplitudeToDB().to(device)
     37         images = to_dB(mel_spec(signals))
     38

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torchaudio/transforms.py in forward(self, waveform)
    424             Tensor: Mel frequency spectrogram of size (..., ``n_mels``, time).
    425         """
--> 426         specgram = self.spectrogram(waveform)
    427         mel_specgram = self.mel_scale(specgram)
    428         return mel_specgram

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torchaudio/transforms.py in forward(self, waveform)
     82         """
     83         return F.spectrogram(waveform, self.pad, self.window, self.n_fft, self.hop_length,
---> 84                              self.win_length, self.power, self.normalized)
     85
     86

/usr/local/lib/python3.6/dist-packages/torchaudio/functional.py in spectrogram(waveform, pad, window, n_fft, hop_length, win_length, power, normalized)
    160     # default values are consistent with librosa.core.spectrum._spectrogram
    161     spec_f = torch.stft(
--> 162         waveform, n_fft, hop_length, win_length, window, True, "reflect", False, True
    163     )
    164

/usr/local/lib/python3.6/dist-packages/torch/functional.py in stft(input, n_fft, hop_length, win_length, window, center, pad_mode, normalized, onesided)
    463         input = F.pad(input.view(extended_shape), (pad, pad), pad_mode)
    464         input = input.view(input.shape[-signal_dim:])
--> 465     return _VF.stft(input, n_fft, hop_length, win_length, window, normalized, onesided)
    466
    467

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

I am using Google Colab. The device variable is device(type='cuda'). I have already moved torchaudio.transforms.MelSpectrogram and torchaudio.transforms.AmplitudeToDB to the GPU but I am not sure what else to do to fix this.
Any suggestions?

Context:
I cannot reproduce. Can you provide a minimal example?
import torchaudio

device = "cuda"

waveform, sample_rate = torchaudio.load("sinewave.wav") # from audio/test/torchaudio_unittest/assets
waveform = waveform.to(device)

mel_spec = torchaudio.transforms.MelSpectrogram(sample_rate = sample_rate, n_fft = 1024, n_mels = 256, hop_length = 63).to(device)
to_dB = torchaudio.transforms.AmplitudeToDB().to(device)

images = to_dB(mel_spec(waveform))


Question:
Hi,
We’re using sox_io backend for torchaudio. New version of the AudioMetaData class returned by the info method doesn’t have all the data I need for analysis.
 info method in the deprecated sox backend used to return sox_signalinfo_t and ‘sox_encodinginfo_t’ that had much more detailed information about encoding, bit depth etc.
What should I use to get the info about audio file’s encoding, bit depth and other details?

Context: Thanks for the feedback! Could you create an issue on torchaudio’s github 1? More context about this BC-breaking change is also given here.


Audio Classification: How to change input weights when working with a pretrained model?

Question:
In an Audio Classification problem, I am firstly loading a pretrained model, then running my own data through the model.
In audio problems, I am searching for optimum parameters (hop length, window size, etc) for transforming features into Mel Spectrograms.  However, this changes the size of the original inputs.  Thus, there is an error when attemping to load_state_dict of the model.
RuntimeError: Error(s) in loading state_dict for Cnn14Extractor:
        size mismatch for spectrogram_extractor.stft.conv_real.weight: copying a param with shape torch.Size([513, 1, 1024]) from checkpoint, the shape in current model is torch.Size([552, 1, 1102]).
        size mismatch for spectrogram_extractor.stft.conv_imag.weight: copying a param with shape torch.Size([513, 1, 1024]) from checkpoint, the shape in current model is torch.Size([552, 1, 1102]).
        size mismatch for logmel_extractor.melW: copying a param with shape torch.Size([513, 64]) from checkpoint, the shape in current model is torch.Size([552, 64]).

All other discussions I have found talk about Image problems (size doesn’t change each time you change a parameter).
My main question is:
How can I change these 3 layers weights?
Can I make the model accept the new weights shape? What are my options besides:

Training from scratch (led to poorer results)
Resizing feature to fit the expected shape (loss of data?)

Context:
shaneR:

However, this changes the size of the original inputs. Thus, there is an error when attemping to load_state_dict of the model.

Changing the input shape shouldn’t change the parameter shapes and the inputs would also not be stored in the model.state_dict(). It seems you’ve manipulated the model parameters instead.
You would have to apply these manipulations to the original model before loading the state_dict.


Question:
Hello, I hope you’re all doing fine.
I’m trying to preprocess .wav files with torchaudio, when i run the instruction waveform, sample_rate = torchaudio.load(filename)
the waveform tensor  is of a shape [number_of_channels, some_number],
sometimes the number of channels is 1 and sometimes it’s 2.
I want to know whether there is a way to force the number of channels to always be one.
I tried to look on the we to understand what is the meaning of the number of channel but I did’nt find anything useful.

Context: If the two channels are identical, and even if they aren’t, you can always just use one of them.  If you have truly stereo audio, then you have to decide how you want to monoize it.  Usually just averaging the channels works OK, but it could be problematic especially if the two channels might be out of phase, which can happen due to processing or just because the microphones were far away from each other.  There are probably more sophisticated techniques than averaging that could be useful in some cases, but I never tried them.


Question:
Hi. I was preprocessing my audio file. after scaling my mel spectogram with standard scaler , minmax (I’ve tried both) It occurs a runtime error. but the input without scaling was also float type.

RuntimeError                              Traceback (most recent call last)
 in ()
5 criterion_multioutput = nn.CrossEntropyLoss()
6 optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
----> 7 model =train_model(model, criterion_multioutput, optimizer)
8
9 SAVE_PATH = os.path.join(os.getcwd(),‘models’)
11 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
418                             _pair(0), self.dilation, self.groups)
419         return F.conv2d(input, weight, self.bias, self.stride,
–> 420                         self.padding, self.dilation, self.groups)
421
422     def forward(self, input: Tensor) -> Tensor:
RuntimeError: expected scalar type Double but found Float

Context: The input tensor or the model parameters seem to be DoubleTensors while the others are FloatTensors. You could transform the data type via model.float() or tensor.float() depending which of these two objects is using the double type.


Looking for a better way to split up input data to conv1d so I don’t run out of RAM

Question:
I am using PyTorch to process audio data through conv1d layers, however, I am running out RAM (only 8GB available). I have an input .wav and a target .wav for the network to learn, and each file is 40MB (about 4 minutes of audio).
In this model, one sample of audio is learned from the previous 200 samples. (Input 200 samples and output 1 sample). In order to accomplish this, I am taking (for example) the input 8000000 samples and “unfolding” it into (8000000, 200, 1), where each audio sample becomes an array of the previous 200 samples. I then train on “unfolded_input_samples” as the training data “target_samples” as the validation data.
The problem is I quickly run out of RAM when unfolding the input data. Is there a way around creating this massive array while still telling PyTorch to use the previous 200 samples for each output data point? Can I break up the unfolded input array into chunks and train on each part without starting a new epoch? Or is there an easier way to accomplish this using some kind of built in method in Pytorch. Thanks!

Context:
I’m not sure of the motivation for this ‘unfolded’ tensor. The data is 99.5% redundant, and the 200-sample window can be created at train-time from the audio-data tensor. Just take a slice audio[i:i+200] of the 8m audio tensor, and audio[i+201] as your target.
The only extra ingredient, then, is the set of offset indices into your audio data, which you could sample randomly or sequentially.


Covert Conv-TasNet from torch to onnx successfully, but not work

Question:
I coverted Conv-TasNet from torch to onnx. Follow steps from website (Pytoch onnx). Then used $“testing.assert_almost_equal” to verify output. The orignal TasNet pre-trained model worked, however the result of onnx are zeros. Its weights seems to be all zero, and the output dimension also different.  I cann’t figure which step was wrong.
Here is my project 4.
Run demo_tasnet.py 1 onnx will be save in checkpoints.
Also write a test.py to campare result.

Context:
Hi,
I tried to figure out the problem. So I checked the code again.

torch.onnx.export(model, input,
“.path/to/conv_tasnet.onnx”,
export_params=True
)

I set up export_params=True, and test .onnx with onnxruntime the ouput are still all zeros. It seems that the weights are not be saved.


Question:
I am finetuning wav2vec2 on my own data. I am running this on k80 and as it doesn’t support fp16. When I was training with fp16 flag got loss scale reached to 0.0001
FloatingPointError: Minimum loss scale reached (0.0001). Your loss is probably exploding. Try lowering the learning rate, using gradient clipping or increasing the batch size.

Then I switched to FP32 but loss became nan this time:
getting log:
/data/fairseq/fairseq/utils.py:306: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
  "amp_C fused kernels unavailable, disabling multi_tensor_l2norm; "
/data/fairseq/fairseq/utils.py:306: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
2020-10-01 10:25:50 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:25:50 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:25:50 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:25:50 | INFO | train | {"epoch": 73, "train_loss": "nan", "train_ntokens": "15558.8", "train_nsentences": "427.098", "t
rain_nll_loss": "nan", "train_wps": "2443", "train_ups": "0.16", "train_wpb": "15558.8", "train_bsz": "427.1", "train_num_updates": "
9291", "train_lr": "1e-08", "train_gnorm": "nan", "train_loss_scale": null, "train_train_wall": "867", "train_wall": "0"}
2020-10-01 10:25:50 | INFO | fairseq.trainer | begin training epoch 74
2020-10-01 10:40:41 | INFO | fairseq_cli.train | begin validation on "valid" subset
2020-10-01 10:42:40 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:42:40 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:42:40 | INFO | valid | {"epoch": 74, "valid_loss": "nan", "valid_ntokens": "2570.42", "valid_nsentences": "71.4286", "v
alid_nll_loss": "nan", "valid_uer": "100", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2780.3", "valid_wpb": "2570.4",
"valid_bsz": "71.4", "valid_num_updates": "9454", "valid_best_wer": "100"}

Any suggestions on how to overcome such conditions.

Context:
Are you using opt_level='O0' in apex.amp? If so, could you remove apex and run the model in “pure” float32 and check if you are still seeing invalid outputs?
If so, could you check, if the inputs are containing valid values via torch.isfinite(input).all()?


Question:
Hi Everybody,
I am using the torchaudio.transforms.Spectrogram to get the Spectrogram of a sin wave which is as follows:
Fs = 400
freq = 5
sample = 400
x = np.arange(sample)
y = np.sin(2 * np.pi * freq * x / Fs)
Then, I get the Spectrogram of the mentioned sin wave as follows:
specgram = torchaudio.transforms.Spectrogram(n_fft=256,
win_length=256,
hop_length=184,
window_fn=torch.hamming_window,
power=1,
normalized=True)
output = specgram(torch.from_numpy(y))
As you see, the sin wave only has a frequency 5 Hz, so I expect the output does not change in different time bins, but I got the following figure which is so strange:
specg1514×649 8.23 KB
I believe this is wrong, so I decided to used the signal library to get Spectrogram of the mention sin wave as follows:
frequencies_samples, time_segment_sample, spectrogram_of_vector = signal.spectrogram(
x=y,
fs=fs,
nperseg=256,
noverlap=184,
window=“hamming”,
detrend=False,
mode=‘magnitude’)
And, then by using  signal.spectrogram, I get the following figure:
spec21498×649 4.26 KB
As we expected, the spectrogram should not change in different time bins because we have only one frequency.
So, what is the matter with the spectrogram when I use torch library? why it changes over the time when I have only one frequency?

Context: This probably is from the phase shift incurred when the partitioning isn’t a multiple of the wave length in samples.


Question: Is the torchaudio.transforms.MelSpectrogram 27 class in torchaudio differentiable? As in, can I backpropagate through it? If so, can someone point to me towards some documentation on how this backpropagation works?

Context:
I think so, just do requires_grad_ on your input. In the end it goes through torchaudio.transforms.functional.spectrogram and uses the torch.stft function. This calls torch.fft (I think), which has a derivative defined.
There are several texts about how the inner parts of PyTorch work, I wrote something simple a long time ago 13 and @ezyang has an awesome comprehensive tour of PyTorch internals 8.
Best regards
Thomas


Question:
Hello,
What is up with the audio tutorials?
When trying to search for pytorch audio quickstart-ish code only found https://pytorch.org/tutorials/beginner/audio_classifier_tutorial.html which seems to be old and missing code.
The one tutorial that is in pytorch/tutorials is https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html which only shows file loading and few transforms.
Is there a way one could contribute on writing those?

Context:
Contributions are always welcome!
Could you open an issue here 9 and describe what kind of tutorial you would like to write to get some feedback?


How much work should your dataset class do for you?

Question:
I’m trying to determine if I should refactor the way may dataset __getitem__ method pre-processes.
Are there any heuristics beyond the obvious time complexity and code readability I should consider?
For a given audio file that I load I want to:

Pad the length to match a given interval size.
Cut the audio into equal pre-determined interval lengths
Create several types of spectrograms of each piece of this new sequence.

It’s a lot. Is there any reason that I shouldn’t do all of these things right inside of __getitem__?

Context:
__getitem__ is the process which is spawn with multiprocess.
The workload should be there (mainly the spectrograms)
As @hash-ir mentions, any other task can be performed in the __init__ function (or even outside the dataset class)
It’s all about how many RAM do you have.
Do you need to read audio in __getitem__?
Well, if your dataset fits in your RAM you can preload it in init.
You can use any python function inside __getitem__, therefore it can still be readable.


Sound event detection model not learning

Question:
Hello everyone,
I am fine tuning a model for sound event detection, taken from https://github.com/qiuqiangkong/audioset_tagging_cnn 11, on the Urbansed dataset. In this task, the model should predict a [batch, n_classes, time_steps] matrix, with a value of one indicating the presence of an event at a certain time step.
However, my network does not seem to train. Specifically, after about the first 10 epochs, my loss stops decreasing. If i check the predictions of the model, the output is composed entirely of 0.5s.
I have tried:

Changing the amount of l2 regularization, even turning it off completely
Changing the learning rate
Doing a mock training with only 2 samples to see if the network could learn the simple problem. The result was the same matrix of 0.5.
Different optimizers (Adam and SGD so far)
BCEWithLogitsLoss with reduction = mean and sum (using this loss as in theory multiple classes can be active at a time)

My loss and optimizer:
criterion = nn.BCEWithLogitsLoss(reduction='sum')
optimizer = optim.SGD(model.parameters(), lr=0.001)

My training loop:
    for i, data in enumerate(dataloader_train):
        inputs, labels = data
        inputs = inputs.type(torch.FloatTensor)
        optimizer.zero_grad()
        outputs = model(inputs).cpu()
        loss = 0
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

I can’t figure out what’s wrong. Any thoughts?
Thanks,
Federico

Context: Try viewing ur data to see what it’s actually training with


Any new 1.60 Keyword spotting examples out there now we have torchaudio?

Question:
Just wondered if anyone has created or has any example code of a more recent pytorch using tourchaudio MFCC hopefully in a streaming model that can use Alsa sources?
Or if anybody is up for the idea of kickstarting something.
I quite like the Linto tensorflow HMG (Hotword model generator) as it allows you to create a profile of MFCC & model parameters.

      GitHub

linto-ai/linto-desktoptools-hmg 9

GUI Tool to create, manage and test Keyword Spotting models using TF 2.0 - linto-ai/linto-desktoptools-hmg

Would be great to have something similar with Pytorch with a GUI where maybe you could test GRU, CRNN, DS-CNN models with datasets.
But you always live in hope
Stuart

Context:
Hi Stuart,
the idea sounds interesting.
Would you be interested to work on a tutorial for this kind of functionality?


Training pytorch model in Google Colab vs local GPUs results in less accurate models using same hyperparams. How to fix?

Question: I recently discovered Google Colab and I uploaded my Pytorch project related to training models for processing audio. I got it training models using google’s TPUs, but I noticed that the models were less accurate than the ones I trained on my local machine. It turns out the the state dict weights and biases have about half the decimal places as the locally trained model. After some searching I read that Colab uses float16 By default instead of float32 precision to increase speed, but since the audio I’m training is in float32, it really needs to train in float32 precision. Is there a way to change this in Colab? Or is there a way to change my pytorch model to ensure float32 precision is kept? My model uses a stack of 1d convolutional layers, if that matters.

Context: PyTorch uses float32 by default on CPU and GPU. I’m not deeply familiar with TPUs, but I guess you might be using bfloat16 on them? Could you try to call float() on the model and inputs and check, if the TPU run is forcing you to use this format?


Question: Hi, I am trying to figure out how a nn.conv1d processes an input for a specific example related to audio processing in a WaveNet model. I have input data of shape (1,1,8820), which passes through an input layer (1,16,1), to output a shape of (1,16,8820). That part I understand, because you can just multiply the two matrices. The next layer is a conv1d, kernel size=3, input channels=16, output channels=16, so the state dict shows a matrix with shape (16,16,3) for the weights. When the input of (1,16,8820) goes through that layer, the result is another (1,16,8820).  What multiplication steps occur within the layer to apply the weights to the audio data? In other words, if I wanted to apply the layer(forward calculations only) using only numpy for this example, how would I do that?

Context:
Hi @Keith72, this is how pytorch conv1d actually do in your case:
x = torch.rand(1, 16, 8820)
weight = torch.rand(16, 16, 3)

# first pad zeros along the time dimension
x = torch.pad(x, [1, 1])    #shape = (1, 16, 8822)

#unfolded, so you have 8820 moving windows with size = (16, 3)
x = x.unfold(2, 3, 1)    #shape = (1, 16, 8820, 3)

# matrix multiplication, I use tensordot for simplicity
y = torch.tensordot(x,weight, dims=([1, 3], [1, 2]))     #shape = (1, 16, 8820)

In numpy you can simply replace pad and tensordot with corresponding numpy function; for unfold you can use numpy.lib.stride_tricks.as_strided.


Question:
My task is to take an episode of a TV show and its subtitles. Then make the subtitle timings more accurate (from 200ms to 20ms). So I want to learn what is speech and what is not.
I’ve now taken the audio, converted it into a spectrogram and separated each column of the spectogram to be a single data item. So now I have two arrays:
print(train_speech.size()) # torch.Size([93482, 201])
print(train_silence.size()) # torch.Size([35038, 201])

All I want to do is a simple multi-linear NN to make a difference. train_speech is FFT’s of people talking and train_silence is no talking (used subtitles for the distinction).
My question is what DataLoader can I use to take these into torch?

Context:
There is one DataLoader 1, which accepts a Dataset and provides different functionalities such as shuffling, creating batches using multiple workers, etc.
To create a custom Dataset you could have a look at this tutorial 1.


Question:
Hello!
I created a conda environment using a provided .yml file on this repo 1.
However, when installing torchaudio , torch 1.5.1 was automatically installed and I needed pytorch 1.0.0 (required for the code). I ended up having

torch 1.5.1

pytorch 1.0.0
on the same environment. Is his a problem? I think it is, because now I’m getting the following error when I execute the audio training:

File “/home/fjaviersaezm/anaconda3/envs/infomax/lib/python3.6/site-packages/torchaudio/functional.py”, line 40, in
@torch.jit.ignore

AttributeError: module ‘torch.jit’ has no attribute ‘ignore’

Can I uninstall torch 1.5.1? How can I solve the “ignore” problem?
Thanks guys!

Context: torchaudio requires more recent version of torch, see dependencies. It appears the environment is looking at pytorch 1.0 and functionalities required by torchaudio are missing.


Question:
I’m using a dataset of audio feature vectors (each sample is represented with 40 features). At the moment I create a matrix in the init_ method of my custom dataset and push it to GPU:
self.frame_array_device = torch.from_numpy(self.frame_array)
self.frame_array_device.to(device)

the dataset getitem method is defined as follows:
def __getitem__(self, idx):
        return self.frame_array_device[idx, :]

Samples are drawn by the dataloader:
train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=batch_size,shuffle=False)

in the training loop I double check that the features are on the GPU:
batch_features = batch_features.to(device)
However, training is embarassingly slow and CPU usage is at 100%…
I think that there is something wrong, could you give me some hint?
Thanks a lot

Context:
I am newer to ML so take what I say with a grain
Loading the data is what I think is the slowest part.  I notice that between epochs my CPU is always 100-200% even thought my device prints as Device: Tesla P100-PCIE-16GB. Then, during a short portion of every epoch my GPU also goes to 100%.
Subscribed to see what experts have to say.


Question:
Hello all,
I am trying to make a CNN model for Encoder Parameter Suggestion for Audio data
Input: Audio File(Length : 480000 samples)
Output: Parameter(Length : 469, eg :[0001111222001233312000] )
I am getting very low Validation and Test Accuracy
class AudioClassifier(nn.Module):
    def __init__(self):
        super(AudioClassifier, self).__init__()

        self.conv11 = nn.Conv1d(in_channels=1, out_channels=16, kernel_size=Kernel_size_conv, stride=1,
                                bias=True, padding=int((Kernel_size_conv - 1) / 2))
        self.conv12 = nn.Conv1d(in_channels=16, out_channels=16, kernel_size=Kernel_size_conv, stride=1,
                                bias=True, padding=int((Kernel_size_conv - 1) / 2))
        self.conv13 = nn.Conv1d(in_channels=16, out_channels=4, kernel_size=Kernel_size_conv,
                                stride=1024, bias=True)

    def forward(self, X):
        out1 = torch.tanh(self.conv11(X))
        out1 = torch.tanh(self.conv12(out1))
        out1 = (self.conv13(out1))
        return out1

optimizer = torch.optim.Adam(model.parameters(),lr=1e-2,  weight_decay=1e-5)
loss_fn = nn.CrossEntropyLoss()
    for epoch in range(E):
        for batch in range(B)
            pred_1= model(training_data)
            loss = loss_fn(pred_1, Target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

Can Anybody suggest any other approach to rectify it.

Context: What about you training loss and training accuracy? If it is good, then your model is overfitting (which would be surprising given that it is quite simple), if not, then your model is probably too simple for the task


Question:
I have a dumpy question I have a loss function that depends on Tensorflow and Keras backend, the question is can I use this loss function to train my PyTorch network.
Another question is there is any PyTorch function that equivalent to this TensorFlow command.
stft_true = tf.contrib.signal.stft(y_true,256,128,512,window_fn,pad_end=False)
Thanks

Context:
Hey,
Have you tried torch.stft 2 that looks like the same thing.


Question:
Hello all,
I have query regarding the placement of optimizer.zero_grad() and optimizer.step()
for idx in range(epoch):
for kdx in range(batch):
y_pred = model(X)
loss = loss_fn(y_pred,y_target)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Do we need to call them once for every epoch or for every batch iteration?

Context:
It depends on your use case.
The usual approach would be to zero out the old gradients, calculate the new gradients via backward(), and update the parameters via optimizer.step() for each batch (so once per iteration).
However, you could simulate a larger batch size by accumulating the gradients using multiple batches and call the complete update steps after a specific number of iterations.


How to create a RNN that applies *n* different recurrence relations to the input?

Question:
Hello everyone,
I am sorry for the long post but would really appreciate any help you guys can offer !
I am trying to create a custom RNN that would apply n different recurrent connections (that are in fact n biquadratic filters) on the input. Another way of thinking about it would be to have n different RNN that works on the input and to concatenate thair results afterwards, however I believe that would lead to very poor performances (please tell me if I am wrong).
For instance :
I have a mini-batch of size [32, 16000], and want to apply 128 filters on it, which means that my output size is [32,128,16000].
What I did so far is :

Expand and clone the input so I have a tensor of size : [32, 128, 16000].
Permute axes to get a size of [16000, 32, 128].
Iterate on the sequence and use matrices products to compute the input, since the filters are linears. In fact, I use this recurrence relation that works for only one sequence of size N (except the first two samples ofc) :

where the a_i and b_i are the learnable weights, x[n] is the n-th sample of the input, y[n] is the state at the time frame n, and i is for the i-th filter (or the i-th recurrence relation if you prefer).
I already tried two methods to make it work (see below). The problem is that my versions are too slow and I don’t have a good enough understanding of pytorch to optimize them.
So I would really appreciate any help you can provide on these points :

Is there a better way of implementing an RNN with n different recurrence relations ?
Do you see improvements I could make to my code (see below) that would yield to good performances ?
May computing the outputs of the different RNNs in parallel and concatenate (with torch.cat) the results yield to better results ?
May implementing in C++ (as it is the case for pytorch for the recurrence relation) become necessary to achieve good performance ?

Links for the pytorch RNN :

RNN.py
RNN.cpp

QuantizeLinear.cpp which seems to contain the function that achieve the loop on the sequence : fbgemm_linear_int8_weight_fp32_activation.

Please, tell me if there is anything unclear or if you need more info.
Thanks for reading this post, and thanks for any piece of advice you can provide !

Code :
In each of the following version, the loop on the sequence is the piece of code that is the longest to execute.

Version A :

def forward(self, X) :
    bs = X.size()[0]

    X = X.unsqueeze(1).expand(-1,self.kernels_number,-1).clone()
    B0, A1, A2 = self.filters()

     if(self.is_cuda):
      out = torch.zeros(self.points_per_sequence, bs, self.kernels_number).cuda()
    else:
      out = torch.zeros(self.points_per_sequence, bs, self.kernels_number)

    out[0] = torch.mul(X[0],B0) # [bs,1]*[1,128] = [bs,128]
    out[1] = torch.mul(X[1],B0) - torch.mul(out[0],A1)

    for n in range(2, X.size()[0]):
      out[n] = self.f_2(out[n-1], out[n-2], X[n], X[n-2], B0, A1, A2)

    out = torch.flip(out, dims=[2]).permute(1,2,0)
    return out

(Since I am using pass-band filters, I only need the three tensors B0, A1, A2 of size [1,n_channels] each, there are computed from only two weights but it does not matter here).
The function self.f_2 :
def f_2(self, y_1, y_2, x, x_2, b0, a1, a2):
    """
    Computing y[n] with y[n-1], y[n-2], x[n], x[n-1], x[n-2], b0, a1, a2
    Sizes :
       x : [bs,128]
       b0,a1,a2 : [1,128]
       y_1, y_2 : [bs, 128]
    """
    return torch.mul(x-x_2,b0) - torch.mul(y_1,a1) - torch.mul(y_2,a2)

I have not tried this version on the backward pass but the forward pass works.

Version B :

For this one, I used the function lfilter 1   from torchaudio. Since the filters are all differents, I started by looping over the filters and applying lfilter which did not work well : it took longer than the previous version and had RAM issues.
Then I modified the function lfilter so it now accepts different filters. It now behaves, performance wise, as the version A.
Here is my version of the filter :
def m_lfilter(
        waveform: torch.Tensor,
        a_coeffs: torch.Tensor,
        b_coeffs: torch.Tensor
)  -> torch.Tensor:
    r"""Perform an IIR filter by evaluating difference equation.

    NB : contrary to the original version this one does not requires normalized input and does not ouput normalized sequences.

    Args:
        waveform (Tensor): audio waveform of dimension of `(..., number_of_filters, time)`.
        a_coeffs (Tensor): denominator coefficients of difference equation of dimension of `(n_order + 1)`.
                                Lower delays coefficients are first, e.g. `number_of_filters*[a0, a1, a2, ...]`.
                                Must be same size as b_coeffs (pad with 0's as necessary).
        b_coeffs (Tensor): numerator coefficients of difference equation of dimension of `(n_order + 1)`.
                                 Lower delays coefficients are first, e.g. `number_of_filters*[b0, b1, b2, ...]`.
                                 Must be same size as a_coeffs (pad with 0's as necessary).

    Returns:
        Tensor: Waveform with dimension of `(..., number_of_filters, time)`.

    Note :
      The main difference with the original version is that we are not packing anymore  the batches (since we need to apply different filters)
    """

    shape = waveform.size() # should returns [batch_size, number_of_filters, size_of_the_sequence]

    assert (a_coeffs.size(0) == b_coeffs.size(0))
    assert (len(waveform.size()) == 3)
    assert (waveform.device == a_coeffs.device)
    assert (b_coeffs.device == a_coeffs.device)
    device = waveform.device
    dtype = waveform.dtype
    n_channel,n_filters, n_sample = waveform.size()
    n_order = a_coeffs.size(1)
    assert (a_coeffs.size(0) == n_filters) # number of filters to apply - for each filter k, the coefs are in a_coeffs[k] and b_coeffs[k]
    n_sample_padded = n_sample + n_order - 1
    assert (n_order > 0)

    # Pad the input and create output
    padded_waveform = torch.zeros(n_channel, n_filters, n_sample_padded, dtype=dtype, device=device)
    padded_waveform[:,:,(n_order - 1):] = waveform
    padded_output_waveform = torch.zeros(n_channel, n_filters, n_sample_padded, dtype=dtype, device=device) # padded_output_waveform = torch.zeros(n_channel, n_sample_padded, dtype=dtype, device=device)

    # Set up the coefficients matrix
    # Flip coefficients' order
    a_coeffs_flipped = a_coeffs.flip(1).unsqueeze(0)
    b_coeffs_flipped = b_coeffs.flip(1).t()

    # calculate windowed_input_signal in parallel
    # create indices of original with shape (n_channel, n_order, n_sample)
    window_idxs = torch.arange(n_sample, device=device).unsqueeze(0) + torch.arange(n_order, device=device).unsqueeze(1)
    window_idxs = window_idxs.repeat(n_channel, 1, 1)

    window_idxs += (torch.arange(n_channel, device=device).unsqueeze(-1).unsqueeze(-1) * n_sample_padded)
    window_idxs = window_idxs.long()

    # (n_filters, n_order) matmul (n_channel, n_order, n_sample) -> (n_channel, n_filters, n_sample)
    A = torch.take(padded_waveform, window_idxs).permute(0,2,1)  # taking the input coefs
    input_signal_windows = torch.matmul(torch.take(padded_waveform, window_idxs).permute(0,2,1),b_coeffs_flipped).permute(1,0,2)
    # input_signal_windows size : n_samples x batch_size x n_filters

    for i_sample, o0 in enumerate(input_signal_windows):
        windowed_output_signal = padded_output_waveform[:, :, i_sample:(i_sample + n_order)].clone() # added clone here for back propagation
        o0.sub_(torch.mul(windowed_output_signal,a_coeffs_flipped).sum(dim=2))
        o0.div_(a_coeffs[:,0])

        padded_output_waveform[:, : , i_sample + n_order - 1] = o0

    output = padded_output_waveform[:, :,(n_order - 1):]
    return output

As for the the forward function :
  def forward(self, X):
      # creating filters
    A, B = self.filters() # A = [[a1_0, a2_0, a3_0],...], A = [[b1_0, b2_0, b3_0],...] - size : [128, 3]

    X = X.unsqueeze(1).expand(-1,self.kernels_number,-1).clone()  # we have to expand the input to the size : [bs, n_filters, n_samples]

      # applying the filters
    X = m_lfilter(X,A,B)

    return X

This method works for the backward pass even if it takes ages to perform (I am working on implementing the TBPTT in parallel to improve these algorithms).

Context:
Pre-calculating non-recurrent term is a good approach. You can use Tensor.unfold to create (16000-2,32,3) feature tensor. You could then also apply 3x128 map to it (with conv1d or matmul), but most rnn implementations do just that with input-to-hidden matrix.
Now, recurrent part is tricky. There are RNN implementations with independent hidden-to-hidden transitions - SRU, IndRNN among others, they could almost do what you want (i.e. they act as a stack of width 1 rnns), with some tweaks. But I’m not aware of implementations that look two steps back (maybe it is possible to emulate this somehow, not sure).

 PaulC:

May implementing in C++ (as it is the case for pytorch for the recurrence relation) become necessary to achieve good performance ?

And to this I would say yes. I’m sceptical about python loops with hundreds of steps already - timestep data slices are small, invocation overheads are huge, backward graph is a chain of small ops too. Actually, in my experience, such loops with GPU tensors are slower than with cpu tensors.


Question:
I tried to import default_collate from torch.utils as shown
from torch.utils.data.dataloader import default_collate
The problem is the default_collate is not found and I found _collate_fn_t are they similar functions?

Context:
Hi @Mohamed_Nabih,
I tried the import you shown and it works for me. Which version are you using?. Also, I remember I recently imported the same function using: from torch.utils.data._utils.collate import default_collate.
To check if the import you used pointed at the same funciton, I checked with the following code:
from torch.utils.data.dataloader import default_collate as dc1
from torch.utils.data._utils.collate import default_collate as dc2

print(dc1 is dc2)
# >> True

So yes, both ways import the actual default_collate method. I couldn’t find _collate_fn_t, but I think they refer to the same method (*.pyi files seem to be related with annotations, so they’re not actual declarations).


Question:
I am trying to combine CNN and LSTM for the audio data.
Let us say the output of my CNN model is torch.Size([8, 1, 10, 10] which is [B X C_out X Frequency X Time ]
and the LSTM requires [L X B X InputSize].
My question is what is the inputSize in LSTM and how shall I feed the output of CNN to the LSTM
Please help @ptrblck

Context:
shakeel608:

My question is what is the inputSize in LSTM

The mentioned inputSize in your shape information would correspond to the “feature” dimension.
Since your CNN output is 4-dimensional, you would have to decide which dimensions are corresponding to the temporal dimensions and which to the features.
Assuming you would like to use C_out and Fequency as the features, you could use:
x = torch.randn(8, 1, 10, 10)
x = x.view(x.size(0), -1, x.size(3)) # [batch_size, features=channels*height, seq_len=width]
x = x.permute(2, 0, 1) # [seq_len, batch_size, features]

and pass it to the RNN.
PS: Please don’t tag certain people, as this might discourage others to post a solution


Question:
Hi,
I’m looking at QuartzNet in NeMo and trying to probe some of the internal tensors. I see that I can use evaluated_tensors = neural_factory.infer(tensors=[a,b,c])  to run inference and return the evaulations of a,b,c, but I can’t figure out how to get a list of the intermediate activation tensors, or to get a pointer to one.  I’m looking for a method of either the neural_factory or the individual models (like jasper_encoder) that would return a list of tensors that I can pass to infer().  Any ideas?
Thanks
Edit:  I know infer() and the neural_factory object are NeMo, so maybe out of scope for this board, but the underlying model is based on a PyTorch module, so I’m hoping a general PyTorch method for getting internal activations will be useful here.  Here’s the inheritance tree for that encoder model.
[nemo.collections.asr.jasper.JasperEncoder,
 nemo.backends.pytorch.nm.TrainableNM,
 nemo.core.neural_modules.NeuralModule,
 abc.ABC,
 torch.nn.modules.module.Module,
 object]

Context: For a general PyTorch model you could use forward hooks to get the intermediate activations as described here 7.  Since NeMo seems to be using a PyTorch model internally, you would have to access its layers to register the hooks.


Audio Dataset - Load large file into memory in background

Question:
Hi everyone,
I have a specific thing I want to achieve, and was wondering if it’s already possible using the current Dataset/DataLoader implementation. I saw a PR for a ChunkDataset API that may serve my needs, but it isn’t there yet.
Data: Audio, with lots of small (1-10s) sound files.
I want to process this audio in terms of frames, but also incorporate a hop parameter (ie. take the first 1024 samples, then the next frame will start at 256 instead of 1024)
What I want to do is concatenate all the short audio examples into long .wav files, of which two can fit into memory. I wrote code to index individual sounds and their respective frames and it works well.
The idea is to serve frames from one (long, e.g. 1GB) .wav file, and have another one loaded in the background. When all frames from the first file have been served, I replace the “current” file with the one that was loaded in the background, and load a new file.
Everything works, except for the fact that the IO on loading a new file will block the getitem call, interrupting training. I was thinking of some async io structure, but lack some experience there in getting it to interop with the Dataset/Loader classes.
How can I do a non-blocking IO call to replace the current “buffered” file while keep serving frames?

Context:
One way to manage async iterators is to use a background iterator 22 to prefetch ahead of time. But the exact setup depends on what you are trying to do of course
If you start with lots the small files (for a total of 1 GB), then you could create a dataset that reads them on __getitem__. You could either then cache the dataset in memory after loading, using a cache 8. And/or you could use a background iterator 22 to prefetch files ahead of time. The downside is lots of random disk seeks, but only on first read.
If you have a single large (1 GB) data file with offsets, you could pay the price of loading it once in memory, and then the dataset simply knows about the offsets on __getitem__. To load the 1 GB async, you would need to read per block as an iterator, and you could use the background_iterator to push that in the background. You could still use the cache to keep the data in memory. The benefit would be faster starting time at the beginning since you don’t wait for the whole file to be loaded.
Side note: You could also create a virtual RAM disk and copy the file(s) there once. Then everything after is done from RAM after that, so you could do lots of small file from there, or one big one. Reads are then fast.


Is there a way to compute Kaldi pitch features with gradients?

Question:
I want to compute kaldi pitch features at the end of a network. Is there a way to compute gradients of the kaldi pitch feature extraction? (Maybe something similar to spectrogram of torchaudio).
Thanks!

Context:
Hi,
I’m not familiar with kaldi pitch, how do you compute it? If you use pytorch construct, the autograd should work just fine


How turn a torch audio to binary wav without saving to disk

Question:
I need to send some audio I use a torch based text to speech to generate as a JSON. I cannot find a way to turn it into a base64 without the following awful code:
torchaudio.save('C:/Work/test_images/temp.wav', wav_array, sampling_rate)
wav_file = open('C:/Work/test_images/temp.wav', 'rb')

any ideas how to clean this up? there doesn’t appear to be any built-in functions with torch to do this, but I’m fairly new so would appreciate any help.

Context: I’m not deeply familiar with torchaudio, but would it be possible to use an io.BytesIO object for temporal storage instead of a file work?


Accelerate Spectrograms with GPU and PyTorch?

Question:
I am working with data that I spend a lot of time converting 1D signals to 2D spectrograms that are then fed to a CNN in Pytorch. I have a 2nd GPU and wanted to know if it was reasonable to accelerate my fft/spectrogram workflow through PyTorch on my GPU instead of what I am currently doing in my dataloader?
What I am doing now:
-In my custom dataset (using torch.utils.data dataset class) I initialize and build my dataset of time-series audio data before training begins (fast). Then inside my getitem method I have some things that get done to the time-series data before I pass the data sample to a custom fft function that builds the spectrograms in the way I need and then returns that spectrogram as the data sample.
-My spectrogram function is custom built for various things I need but the main workhorse is built around the np.fft.fft() function, which is where I need to accelerate.
-I am using the “num_workers” kwarg in the PyTorch dataloader to better use my cores.
What I would like to do:
Either in my getitem or elsewhere, I’d like to send it to my 2nd GPU (GTX 980ti) in hopes to faster generate the spectrograms then send them to my main GPU to pass the data through the model.
Is this reasonable? Can I expect to see accelerations in contrast to a Ryzen 3950x and utilizing the “num_workers” kwarg in the dataloader? How would I do this? I’m open to ideas for other libraries and tools that exist out of PyTorch, but not sure where to start and seeking insight from this community.

Context:
Hi,
I think the best way to speed this up would be to move it as preprocessing.
Have a seperate script that converts your audio data to the spectrogram and save them to disk.
Then your dataloader in the training script will just load the spectrograms directly.


Installing torchaudio with pytorch 1.5.0 lead to downgrade

Question: while installing torchaudio with conda install torchaudio -c pytorch, anaconda downgrades torchvision (0.6.0 to 0.2.2) and pytorch (1.5 to 1.4). How to avoid this?

Context:
Hi,
Thanks for the report. Which OS are you using?
It is torchaudio 0.5 that you are installing right?
If not, make sure that your conda is up to date as it sometimes prevent you from installing the latest version of libraries.


LSTM time series prediction network copies the input

Question:
I am creating a model for music generation, but my proble is that for some reason modelpredicts a current step, and not a next step as a label says, even though i compute loss between model output and labels, and labels are shifted by 1 forward:

Sequence  - model input
Label - sequence shifted 1 step forward
Output - model output
And if i feed in the label as input, it “predicts” the label, so model is basically repeating the input

Dataset code:
class h5FileDataset(Dataset):
  def __init__(self, h5dir, seq_length):
    self.h5dir = h5dir
    self.seq_length = seq_length + 1
    with h5py.File(h5dir,'r') as datafile:
      self.length = len(datafile['audio']) // self.seq_length
  def __len__(self):
    return self.length
  def __getitem__(self,idx):
    with h5py.File(self.h5dir,'r') as datafile:
      seq = datafile["audio"][idx*self.seq_length:idx*self.seq_length+self.seq_length]

    feature = seq[0:len(seq)-1].astype('float32') #from 0 to second-to last element
    label = seq[1:len(seq)].astype('float32') #from 1 to last element

    return feature,label

Model code:
class old_network(nn.Module):
  def __init__(self, input_size=1, hidden_layer_size=1, output_size=1, seq_length_ = 1, batch_size_ = 128):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size
        self.batch_size = batch_size_
        self.seq_length = seq_length_

        self.lstm = nn.LSTM(input_size, hidden_layer_size, batch_first = False, num_layers = 2)

        self.linear1 = nn.Linear(hidden_layer_size, output_size)
        self.linear2 = nn.Linear(hidden_layer_size, output_size)
        #self.tanh1 = nn.Tanh()
        self.tanh2 = nn.Tanh()

  def forward(self, input_seq):
      lstm_out, _ = self.lstm(input_seq)
      lstm_out = lstm_out.reshape(lstm_out.size(1),lstm_out.size(0),1) #reshape to batch,seq,feature
      predictions = self.linear1(lstm_out)
      #predictions2 = self.tanh1(predictions1)
      predictions = self.linear2(predictions)
      predictions = self.tanh2(predictions)
      return predictions.reshape(predictions.shape[1],predictions.shape[0],1) #reshape to seq,batch,feature to match labels shape

Training loop:
epochs = 10
batches = len(train_data_loader)
losses = [[],[]]
eval_iter = iter(eval_data_loader)
print("Starting training...")
try:
  for epoch in range(epochs):
    batch = 1
    for seq, labels in train_data_loader:
      start = time.time()
      seq = seq.reshape(seq_length,batch_size,1).to(DEVICE)
      labels = labels.reshape(seq_length,batch_size,1).to(DEVICE)
      optimizer.zero_grad()

      y_pred = model(seq)

      loss = loss_function(y_pred, labels)
      loss.backward()
      optimizer.step()

      try:
        eval_seq, eval_labels = next(eval_iter)
      except StopIteration:
        eval_iter = iter(eval_data_loader)
        eval_seq, eval_labels = next(eval_iter)
      eval_seq = eval_seq.reshape(seq_length,batch_size,1).to(DEVICE)
      eval_labels = eval_labels.reshape(seq_length,batch_size,1).to(DEVICE)

      eval_y_pred = model(eval_seq)

      eval_loss = loss_function(eval_y_pred, eval_labels)
      losses[1].append(eval_loss.item())
      losses[0].append(loss.item())

      print_inline("Batch {}/{} Time/batch: {:.4f}, Loss: {:.4f} Loss_eval: {:.4f}".format(batch,batches,time.time()-start, loss.item(), eval_loss.item()))
      batch += 1

      if batch%50 == 0:
            print("\n Epoch: {}/{} Batch:{} Loss_train:{:.4f} Loss_eval: {:.4f}".format(epoch,epochs,batch,loss.item(),eval_loss.item()))

            plt.close()
            plt.plot(range(0,len(losses[0])),losses[0], label = "Learning dataset")
            plt.plot(range(0,len(losses[1])),losses[1], label = "Evaluation dataset")
            plt.legend()
            plt.show()
            torch.save({'model_state_dict':model.state_dict(), 'optimizer_state_dict' : optimizer.state_dict()},save_dir)

except KeyboardInterrupt:
  plt.close()
  plt.plot(range(0,len(losses[0])),losses[0], label = "Learning dataset")
  plt.plot(range(0,len(losses[1])),losses[1], label = "Evaluation dataset")
  plt.legend()
  plt.show()

I am kinda running out of ideas by this point, not sure what is wrong

Context:
No real ideas, just some comments:
(1) I would never use reshape() or view() to adjust the tensor shape. I’ve seen to many cases where this was used incorrectly and broke the tensor. Just because the shape is correct in the network doesn’t throw an error doesn’t mean the tensor is correct. If possible, I always use transpose() or permute() since in almost all cases I only need to swap dimensions. For exanole, instead of
seq = seq.reshape(seq_length,batch_size,1).to(DEVICE)

I would do
seq = seq.tranpose(1,0).to(DEVICE)

or
seq = seq.permute(1,0,2).to(DEVICE)

This ensures that dimensions are only swapped but never “torn apart” which can happen with reshape() or view(). The latter are mostly needed to maybe (un-)flatten tensors, but that’s not needed here.
(2) I’m also not quite sure about
predictions = self.linear1(lstm_out)

since the shape of lstm_out is (batch_size, seq_len, features). I know that nn.Linear takes as input (N,∗,H_in) but I’m not sure if you really want go that way. Usually the last hidden state is used for prediction. So I would try:
lstm_out, (h, c) = self.lstm(input_seq)
predictions = self.linear1(h[-1])

h[-1] is the last layer of the last hidden state.


Save the output of the network in .ark format

Question:
I trained a PyTorch model for speech recognition and I want to save the output of the model as .ark file
Can anyone give me a help?
Thanks

Context:
Not really pytroch related it seems ?
Are you referring to this ? https://fileinfo.com/extension/ark 2
If so… it’s a pretty arcane format, I’m pretty sure people switched to .zip and .gzip a few decade ago, if you need it for some reason you can use arc (https://sourceforge.net/projects/arc/ 1), at least on linux (but should work on OSX and windows as well I’d assume)
Just output whatever you have to a text file and then use the arc tool to compress it. There’s also a python library for it seemingly: https://pypi.org/project/arc/ 2, but I’m not sure if it actually works.


Question: I am currently trying to download torchaudio for Conda to train an RNN on audio, but I can’t download it. I used the command conda install -c pytorch torchaudio , and also downloaded all of the required libraries, but when I try to download it, it says PackagesNotFoundError: The following packages are not available from current channels: torchaudio. Why does it happen, and how can I download torchaudio succesfully?

Context:
Hi,
Which python version are you using?
Also have you tried using pip or installing the nightly builds?


RuntimeError: CUDA error: out of memory in Pytorch

Question:
I am getting this error in the last batches while i am training a large speech dataset. I tried reducing batch sizes (16,8,4,2) but every time i got this error at the end of the epoch.
can someone give me the solution?
99%|█████████▉| 11706/11812 [29:32<01:08,  1.55it/s, avg_loss=tensor(76.1413, device=‘cuda:0’), iter=11705, loss=tensor(117.4730, device=‘cuda:0’)]
RuntimeError                              Traceback (most recent call last)
 in
3     start = time.time()
4
----> 5     run_state = run_epoch(model, optimizer, train_ldr, *run_state)
6
7     msg = “Epoch {} completed in {:.2f} (s).”
~/Hasan/Project/SpeechRNNT/speech-master/train.py in run_epoch(model, optimizer, train_ldr, it, avg_loss)
28         optimizer.zero_grad()
29         loss = model.loss(batch)
—> 30         loss.backward()
31
32         grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 200)
~/miniconda3/envs/ariyan/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
91                 products. Defaults to False.
92         “”"
—> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
94
95     def register_hook(self, hook):
~/miniconda3/envs/ariyan/lib/python3.6/site-packages/torch/autograd/init.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
88     Variable._execution_engine.run_backward(
89         tensors, grad_tensors, retain_graph, create_graph,
—> 90         allow_unreachable=True)  # allow_unreachable flag
91
92
~/Hasan/Project/SpeechRNNT/speech-master/transducer/functions/transducer.py in backward(self, *args)
78         grads = parent.backward(*args)[0]
79         if self.size_average:
—> 80             grads = grads / grads.shape[0]
81         return grads, None, None, None
82
RuntimeError: CUDA error: out of memory

Context:
Hi,
I’m wondering would you mind to share your implementation of transducer loss? Is it a python native code or did you used c/c++ extensions.
Best,


Question: Dear experts, when will the torchaudio be compatible with win10? I really need it. Thank you very much!

Context: You can track the progress here 15.


Question:
I have a problem with multi-classification I built this Model, but the test accuracy is constant at 4
input_size = 13
hidden1_size = 1024
hidden2_size = 1024
hidden3_size = 1024
hidden4_size = 1024
hidden5_size = 1024
output_size = 1976
class DNN(nn.Module):
def __init__(self, input_size, hidden1_size, hidden2_size, hidden3_size, hidden4_size, hidden5_size, output_size):
    super(DNN, self).__init__()
    self.fc1 = nn.Linear(input_size, hidden1_size)

    self.sig1 = nn.Sigmoid()
    self.fc2 = nn.Linear(hidden1_size, hidden2_size)

    self.sig2 = nn.Sigmoid()
    self.fc3 = nn.Linear(hidden2_size, hidden3_size)

    self.sig3 = nn.Sigmoid()
    self.fc4 = nn.Linear(hidden3_size, hidden4_size)

    self.sig4 = nn.Sigmoid()
    self.fc5 = nn.Linear(hidden4_size, hidden5_size)

    self.sig5 = nn.Sigmoid()
    self.fc6 = nn.Linear(hidden5_size, output_size)

def forward(self, x):
    out = self.fc1(x)

    out = self.sig1(out)
    out = self.fc2(out)

    out = self.sig2(out)
    out = self.fc3(out)

    out = self.sig3(out)
    out = self.fc4(out)

    out = self.sig4(out)
    out = self.fc5(out)

    out = self.sig5(out)
    out = self.fc6(out)

    return out

model = DNN(input_size, hidden1_size, hidden2_size, hidden3_size, hidden4_size, hidden5_size,
output_size)
criterion = nn.CrossEntropyLoss()
learning_rate = 0.008
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
for epoch in range(1, 50):
for i, (X_train, y_train) in enumerate(train_loader):

    model.train()

    optimizer.zero_grad()

    outputs = model(Variable(X_train))

    loss = criterion(outputs, Variable(y_train))

    print('Iter %d/%d --> loss %f' % (i, len(train_loader), loss.item()))

    loss.backward()
    optimizer.step()
correct = 0
total = 0
print('test')
for X_test, y_test in test_loader:
    model.eval()
    out = model(Variable(X_test)).detach()
    pred = out.max(dim=1)[1]#.argmax(dim=1, keepdim=True)
    total += y_test.size(0)
    correct += (pred.squeeze() == y_test).sum() # pred.eq(y_test.view_as(pre  d)).sum().item()
accuracy = 100 * correct / total
print('epoch: {}.  Accuracy: {}'.format(epoch, accuracy))

Context:
Try to overfit a small data sample (e.g. just 10 samples) to make sure you don’t have any hidden bugs in your code and that your model architecture works for this problem.
From my past experience I would claim, that relu activation functions might work better than sigmoids, so you could play around with the architecture and some hyperparameters.