Accuracy of signal prediction model stuck at 51% after few epochs

Henry_Chibueze: <p>So, I’m training a neural network architecture on a particular wave signal detection.<br>
the raw data are in .npy files and contains the time domain signal.<br>
I did some feature extraction on the time domain signal like:<br>
Discrete Fourier Transform to convert the time domain signal to frequency domain signal<br>
I also got the magnitude of the complex numbers returned by the transform to get the amplitude spectrum, and the square of that to get the power spectrum (which were the two features I used)</p>
<p>My raw data shape is (N, 3, 4096), where N is number of samples, 3 corresponds to the 3 different detectors in the detector network used, and 4096 is the sample frequency.</p>
<p>My processed data shape is (N, 3, 2, 2049), where N is number of samples, 3 corresponds to the 3 different detectors in the detector network used, 2 for the amplitude spectrum and power spectrum features, and 2049 corresponding to the length of each feature</p>
<p>The extracted features are scaled to be within range of 0 and 1 with sklean MinMaxScaler.</p>
<p><strong>The training and testing accuracy never exceeds 51%, however the loss seems to be reducing fine and I know that the loss and accuracy of classification problem that uses Cross Entropy are unrelated</strong></p>
<p>Any suggestions?</p>
<p><strong>Here’s code for my feature extraction class:</strong></p>
<pre><code class="lang-auto">#Feature Extraction Class
class FeatureExtractor():
    def __init__(self, input, sample_frequency=4096):
        self.fs = sample_frequency
        self.input = np.array(input.reshape(-1, 3, self.fs))
        self.min_max_scaler = MinMaxScaler()
        self.fourier_transform = self.__discrete_fourier_transform()
        self.power_spectrum = self.__power_spectrum()
    
    def __discrete_fourier_transform(self):
        transform = np.fft.rfft(self.input)
        transform = np.abs(transform)
        return transform
    
    def __power_spectrum(self):
        PS = self.fourier_transform**2
        return PS
    
    def extract_features(self):
        required_shape = (self.fourier_transform.shape[0], self.fourier_transform.shape[1], self.fourier_transform.shape[2])
        Dim_1_transform = self.fourier_transform.reshape(-1, required_shape[2]).get()
        Dim_1_power_spectrum = self.power_spectrum.reshape(-1, required_shape[2]).get()
        scaled_fourier_transform = self.min_max_scaler.fit_transform(Dim_1_transform)
        scaled_PSD = self.min_max_scaler.fit_transform(Dim_1_power_spectrum)
        scaled_fourier_transform, scaled_PSD = scaled_fourier_transform.reshape(required_shape[0], required_shape[1], required_shape[2]), scaled_PSD.reshape(required_shape[0], required_shape[1], required_shape[2])
        features = np.concatenate((np.array(scaled_fourier_transform), np.array(scaled_PSD)))
        feature = features.transpose(1, 0, 2)
        return features.reshape(-1, 3, 2, scaled_PSD.shape[-1])
</code></pre>
<p><strong>My neural network consists of a CNN with 5 conv2d layers and a Feed Forward Network with 4 Linear layers like so:</strong></p>
<pre><code class="lang-auto">#convolutional neural network class
class ConvolutionalNework(nn.Module):
    def __init__(self, in_channels, out_channels, hidden_channels):
        super(ConvolutionalNework, self).__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.hidden_channels = hidden_channels
                
        self.conv_layer_1 = nn.Sequential(
            nn.Conv2d(self.in_channels, self.hidden_channels, kernel_size=1, stride=1),
            nn.BatchNorm2d(self.hidden_channels),
            nn.ELU()
        )
        
        self.conv_layer_2 = nn.Sequential(
            nn.Conv2d(self.hidden_channels, self.hidden_channels*3, kernel_size=1, stride=2),
            nn.LeakyReLU(0.3),
        )
        
        self.conv_layer_3 = nn.Sequential(
            nn.Conv2d(self.hidden_channels*3, self.hidden_channels*6, kernel_size=1, stride=2),
            nn.LeakyReLU(0.3),
        )
        
        self.conv_layer_4 = nn.Sequential(
            nn.Conv2d(self.hidden_channels*6, self.hidden_channels*9, kernel_size=1, stride=2),
            nn.BatchNorm2d(self.hidden_channels*9),
            nn.LeakyReLU(0.3),
        )
        
        self.conv_layer_5 = nn.Sequential(
            nn.Conv2d(self.hidden_channels*9, self.out_channels, kernel_size=1, stride=1),
            nn.LeakyReLU(0.3),
        )
        
    def forward(self, input):
        output = self.conv_layer_1(input)
        output = self.conv_layer_2(output)
        output = self.conv_layer_3(output)
        output = self.conv_layer_4(output)
        output = self.conv_layer_5(output)
        #output shape: torch.Size([10, 1, 508])
        return (output, output.shape)


#define Feed Forward Neural network structure
class FeedForwardNetwork(nn.Module):
    def __init__(self, in_features, out_features, hidden_size):
        super(FeedForwardNetwork, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.hidden_size = hidden_size
        
        self.FC_layer_1 = nn.Sequential(
            nn.Linear(self.in_features, self.hidden_size*4),
            nn.BatchNorm1d(self.hidden_size*4),
            nn.LeakyReLU(0.3),
        )
        self.FC_layer_2 = nn.Sequential(
            nn.Linear(self.hidden_size*4, self.hidden_size*8),
            nn.LeakyReLU(0.3),
        )
        self.FC_layer_3 = nn.Sequential(
            nn.Linear(self.hidden_size*8, self.hidden_size*12),
            nn.BatchNorm1d(self.hidden_size*12),
            nn.LeakyReLU(0.3),
        )
        self.FC_layer_4 = nn.Sequential(
            nn.Linear(self.hidden_size*12, self.out_features),
        )
        
    def forward(self, X):
        output = self.FC_layer_1(X)
        output = self.FC_layer_2(output)
        output = self.FC_layer_3(output)
        output = self.FC_layer_4(output)
        return output

#Networks combine
class ModelNetwork(nn.Module):
    def __init__(self, in_channels, out_channels, 
                 out_features, hidden_channels, 
                 hidden_features, dropout_rate):
        super(ModelNetwork, self).__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.out_features = out_features
        self.hidden_channels = hidden_channels
        self.hidden_features = hidden_features
        self.dropout_rate = dropout_rate
        self.ff_network_in_shape = None
        
        self.conv_network = ConvolutionalNework(self.in_channels, self.out_channels, self.hidden_channels)
        self.__get_conv_shape()
        self.ff_network = FeedForwardNetwork(self.ff_network_in_shape, self.out_features, self.hidden_features)
        self.dropout_layer = nn.Dropout(self.dropout_rate)
        
    def __get_conv_shape(self):
        rand_sample_data = torch.randn(1, 3, 2, 2049)
        _, shape = self.conv_network(rand_sample_data)
        if(self.ff_network_in_shape == None):
            self.ff_network_in_shape = shape[1]*shape[2]*shape[3]
            
    def forward(self, input):
        output, _ = self.conv_network(input)
        output = self.dropout_layer(output)
        output = output.reshape(-1, self.ff_network_in_shape)
        output = self.ff_network(output)
        output = self.dropout_layer(output)
        return output
</code></pre>
<p><strong>hyper parameters:</strong></p>
<pre><code class="lang-auto">### set device for tensor computing
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

#hyper_parameters
in_channels = 3
out_channels = 1
out_features = 2
hidden_channels_size = 6
hidden_feature_size = 200
dropout_rate = 0.2
EPOCHS = 100
batch_size = 600
learning_rate = 1e-3

#model utils
NNModel = ModelNetwork(
        in_channels, out_channels, out_features,
        hidden_channels_size, hidden_feature_size, dropout_rate
    ).to(device)
lossFunc = nn.CrossEntropyLoss()
optimizer = optim.Adam(NNModel.parameters(), lr = learning_rate)
#learning rate scheduler
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, verbose=True)
</code></pre>
<p><strong>My training and testing phase is like so:</strong><br>
The total data sample for training is 560,000, however I’ve used only 5000 samples for training and 500 for testing. The targets numbers are evenly distributed for both real and fake</p>
<pre><code class="lang-auto">#model processes class
class ModelProcess():
    def __init__(self, data_frame, train_size=0.98):
        self.df = data_frame
        self.train_size = train_size
        self.df_dictionaries = self.__data_splice()
        self.train_df, self.test_df = self.df_dictionaries['train_df'].head(5000), self.df_dictionaries['test_df'].head(500)
        #scale by multiplying input vetcors by 10e+19
        self.scaler = lambda vector:vector*10e+19
        self.train_losses = list()
        self.test_loss = None
        self.saved_model_path = f'model_state-[{today.strftime("%b-%d-%Y")}].pth.tar'
        
    #load data in batches specified
    def __data_loader(self, df_to_be_loaded, start, stop):
        df = df_to_be_loaded.iloc[start:stop]
        df = df.sample(frac=1, random_state=40)
        X_paths, y = df['data_path'].to_array(), df['data_targets'].to_array()
        X = list()
        for i in X_paths:
            X.append(np.load(i))
        X, y = np.array(X).get(), y.reshape(-1)
        feature_extractor = FeatureExtractor(X)
        X = feature_extractor.extract_features()
        X, y = self.__vector_dtype_converter(X, y)
        return self.scaler(X), y
    
    #convert from numpy to tensor and vise versa
    def __vector_dtype_converter(self, *data, to='tensor'):
        if(to == 'tensor'):
            tensor = tuple(torch.Tensor(i) for i in data)
            return tensor
        elif(to == 'numpy'):
            numpy_array = tuple(np.array(i.detach()) for i in data)
            return numpy_array
    
    #train + test model
    def run(self, epochs, train_batch_size):
        NNModel.train()
        correct_predictions, total_targets = 0, 0
        for epoch in range(epochs):
            print(f'epoch: {epoch}')
            batch_losses = list()
            batch_accuracies = list()
            for idx in tqdm(range(0, len(self.train_df), train_batch_size)):
                X, y = self.__data_loader(self.train_df, start=idx, stop=idx+train_batch_size)
                NNModel.zero_grad()
                pred = NNModel(X.to(device))
                loss = lossFunc(pred, y.long().to(device))
                batch_losses.append(loss.item())
                loss.backward()
                optimizer.step()
                _, pred = torch.max(pred, 1)
                correct_predictions += (pred.to('cpu') == y).sum().item()
                total_targets += len(y)
                batch_accuracy = (correct_predictions/total_targets)*100
                batch_accuracies.append(batch_accuracy)
            lr_scheduler.step()
            mean_batch_loss = np.mean(np.array(batch_losses))
            mean_batch_accuracies = np.mean(np.array(batch_accuracies))
            self.train_losses.append(mean_batch_loss)
            print(f'mean_batch_error: {mean_batch_loss} \n mean_batch_accuracy: {mean_batch_accuracies}%')
            correct_predictions, total_targets = 0, 0
            if(epoch%10 == 0):
                self.__test(train_batch_size)
        self.__model_save()
        print('model saved successfully...')
    
    #test model
    def __test(self, test_batch_size):
        NNModel.eval()
        test_losses = list()
        correct_predictions, total_targets = 0, 0
        print('testing...')
        with torch.no_grad():
            for idx in tqdm(range(0, len(self.test_df), batch_size)):
                X, y = self.__data_loader(self.test_df, start=idx, stop=idx+test_batch_size)
                pred = NNModel(X.to(device))
                loss = lossFunc(pred, y.long().to(device))
                test_losses.append(loss.item())
                _, pred = torch.max(pred, 1)
                correct_predictions += (pred.to('cpu') == y).sum().item()
                total_targets += len(y)
                test_accuracy = (correct_predictions/total_targets)*100
            print(f'test_error:{np.mean(np.array(test_losses))} test_accuracy: {test_accuracy}%')
    
    #save model 
    def __model_save(self):
        model_state = dict({
            'model_state':NNModel.state_dict(),
            'optimizer_state':optimizer.state_dict(),
        })
        torch.save(model_state, self.saved_model_path)
    
    # splice data into training and testing set
    def __data_splice(self):
        data_size = len(self.df)
        train_size = int(self.train_size*data_size)
        train_df, test_df = self.df.iloc[:train_size+1], self.df.iloc[train_size+1:]
        return {'train_df':train_df, 'test_df':test_df}


model_process = ModelProcess(data_frame=data_idx_df)
model_process.run(EPOCHS, batch_size)```</code></pre>

gphilip: <p>Try the following: apply the <code>nn.ELU()</code> and <code>nn.LeakyReLU()</code> <strong>before</strong> the respective <code>nn.BatchNorm2d()</code> layers, not after them as you do now.</p>
<p>Do tell us if this makes a difference!</p>

Henry_Chibueze: <p>Thanks for the reply<br>
I’ve tried that and still no difference…</p>

gphilip: <p>A couple of other things to try:</p>
<ol>
<li>Reduce the learning rate by one or more orders of magnitude.</li>
<li>Set the dropout rate to zero (That is: don’t have any dropout).</li>
</ol>
<p>From what I understand, dropout becomes relevant when the training accuracy grows significantly larger than the test accuracy. If you have trouble increasing the training accuracy, then having a dropout layer may not help.</p>

Henry_Chibueze: <p>After setting the drop out rate to 0 and changing the learning rate, the accuracy after 20 epochs is stuck at 50.47013227513227%<br>
The test accuracy after 10 epochs increased to 52.2%, but that’s it<br>
no further changes</p>

Henry_Chibueze: <p>I’ve also noticed something<br>
The model predicts only one class per batch<br>
some batches it predict only 0s and some only 1s</p>
<p>but the targets per batch are an equally mix of 0s and 1s</p>

gphilip: <p>I would remove the line <code>output = self.dropout_layer(output)</code> from the very end of the <code>forward</code> method of <code>ModelNetwork</code>. It doesn’t make any sense to me to apply dropout to the output layer. Also, while you are debugging this, try commenting out all calls to dropout layers.</p>

gphilip: <p>Try printing out <code>pred</code> after the line <code>pred = NNModel(X.to(device))</code>, and see if it makes sense.</p>
<p>In fact, go through the <code>run</code> code, describe each line in words, try printing intermediate values to see if they match what you expect, and see if the real values match your expectations. Something in this code looks off to me, but I am too tired now to debug it in my mind.</p>

Henry_Chibueze: <p>Yes I’ve done that<br>
The dropout is originally commented out</p>

gphilip: <p>Change</p>
<p><code>NNModel.zero_grad()</code></p>
<p>to</p>
<p><code>optimizer.zero_grad()</code></p>
<p>and tell us if you get better results.</p>

Henry_Chibueze: <p>I also tried this<br>
It was initially what I did before changing to <code> NNModel.zero_grad()</code></p>

Henry_Chibueze: <p>So, I tried print the <code>pred</code> before and after the <code>_, pred = torch.max(pred, 1)</code><br>
<strong>before</strong> it, the <code>pred</code> seems to always have negative values at index 0 and positive values at index 1 per batch and it’s not a deterministic behavior coz it happens vise versa.<br>
These values repeat themselves a lot and they barely change</p>
<p><strong>after</strong> it, the <code>pred</code> is only one class per batch (either 0s or 1s) as opposed to the target which is an almost even mixture of both classes, that’s why it still gives an accuracy within the range of 49% - 51%</p>
<p>The loss still reduces by the way, it’s just the accuracy that’s stuck</p>
<p>I’ve also tried a different data normalization method that removed a good degree of skewedness from the extracted features, and I also tried to manually initialize weights across the linear layers, conv2d layers, and both with:</p>
<pre><code class="lang-auto">def init_xavier_weights(m):
    if isinstance(m, nn.Linear) or isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform_(m.weight)
        m.bias.data.fill_(0.01)
</code></pre>
<p>and then applying it like so: <code>NNModel.apply(init_xavier_weights)</code></p>
<p>Still no improvement</p>
<p><strong>Edit:</strong><br>
I tried reducing the batch size from 600 to 50 and something strange and interesting happened.<br>
The <code>mean_batch_loss</code> increased to hundreds.<br>
I’ve checked to see if I’m computing the <code>mean_batch _loss</code> correctly and it seems correct to me.</p>

gphilip: <aside class="quote no-group" data-username="Henry_Chibueze" data-post="12" data-topic="130082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_chibueze/48/28458_2.png" class="avatar"> Henry_Chibueze:</div>
<blockquote>
<p>I tried print the <code>pred</code> before and after the <code>_, pred = torch.max(pred, 1)</code><br>
<strong>before</strong> it, the <code>pred</code> seems to always have negative values at index 0 and positive values at index 1 per batch and it’s not a deterministic behavior coz it happens vise versa.<br>
These values repeat themselves a lot and they barely change</p>
<p><strong>after</strong> it, the <code>pred</code> is only one class per batch (either 0s or 1s)</p>
</blockquote>
</aside>
<p>Before you do <code>_, pred = torch.max(pred, 1)</code>,  <code>pred</code>  is an Nx2 array where, within any one batch, either the 0th column is all negative and the 1st column is all positive, or vice versa.</p>
<p>When you do  <code>_, pred = torch.max(pred, 1)</code> you update the value of <code>pred</code> to be an Nx1 array with the following property: <code>pred[i]</code> is the <strong>index</strong> of the <em>column</em> which had the maximum value in row <code>i</code> of the original <code>pred</code>. Since the original <code>pred</code> is as you described above, obviously this results in a new <code>pred</code> which is all <code>0</code>s (if the original <code>pred</code> had positive values at index 0) or all <code>1</code>s (if the original <code>pred</code> had positive values at index 1). See the documentation for <a href="https://pytorch.org/docs/stable/generated/torch.max.html?highlight=max#torch.max" rel="noopener nofollow ugc"><code>torch.max</code></a>. Note that there are <em>two</em> variants of <code>torch.max</code> described in that page; your usage is the second one.</p>
<p>If this processing of <code>pred</code> is as you expect it to be, you have to investigate why the original <code>pred</code> has the same sign for each column, within any one batch. If this processing of <code>pred</code> is <strong>not</strong> as per your expectation (but the original value of <code>pred</code> is fine), then you may want to check the other parameters of <code>torch.max</code> to see if you can use them to get what you want.</p>

gphilip: <aside class="quote no-group" data-username="Henry_Chibueze" data-post="12" data-topic="130082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_chibueze/48/28458_2.png" class="avatar"> Henry_Chibueze:</div>
<blockquote>
<p>I tried reducing the batch size from 600 to 50 and something strange and interesting happened.<br>
The <code>mean_batch_loss</code> increased to hundreds.</p>
</blockquote>
</aside>
<p>Does 600 have some meaning with respect to your input? E.g.: there are 60 samples per second, and the waveform switches signs every 10 seconds, or some such? This could explain a lot of the observed behaviour.</p>

Henry_Chibueze: <p>yes the 1st pred is an Nx2 array.<br>
what <code>torch.max()</code> does is to return the maximum value and it’s corresponding index, and I’ve used it as it should be.</p>

Henry_Chibueze: <p>I tried 2 different approaches to solving this signal detection problem<br>
training with a normalized raw data (no feature extraction), and training with the extracted feature which is the one I’m currently talking about</p>
<p><strong>The 1st one</strong> with 6000 samples from the total 560,000 samples for training achieves an accuracy of 100% after about 95 epochs, but the testing accuracy was 50% and after a dropout rate of 0.8, the testing accuracy barely crossed 56%</p>
<p><strong>The second one</strong> is this one I’m currently facing…</p>

Henry_Chibueze: <p>By the way, I tried a different normalization method and seems that the model is nor more predicting one class per batch<br>
However, still no significant improvement in accuracy and it seems the loss stops reducing after it has reached 0.69</p>

gphilip: <p>Is it possible that normalization is removing information from the input? Have you tried running your network on non-normalized input data?</p>

Henry_Chibueze: <p>Yes I’ve tried that<br>
I did it in my 1st approach to solving the problem with no feature extraction<br>
It was doing poorly, this was understandable because the data points are really small, sth like: 0.33 x 10^-20 or so<br>
So all I just did was multiply all input by 10^19, and I started to get training accuracy of 95 to 100% but testing accuracy of 50 to 56%.<br>
And this scaling  also affects the testing data</p>
<p>However, the normalization I tested in this approach today was <code>torch.exp(n)/100</code> and it made the data value distribution more balanced than before</p>

gphilip: <p>Since your input has temporal information (i.e, “before” and “after” matters), perhaps CNNs are not the best networks to learn their structure? Have you tried using Recurrent Neural Networks on your data?</p>