CNN Model Keeps Restarting Kernel

tired_grad_student: <p>Hi there. I built a CNN that I am training on data I generated. It is a version of VGG-16 that uses 1d convolutions since each data sample is a row of numbers rather than a picture. I am running into a strange issue where the program uploads the data I generated into RAM and after a few minutes, I get a message stating that the kernel has restarted. The odd thing is that I am getting this issue after I cut down the amount of memory I am using for the total amount of data. Nothing changed in the model structure-wise between it working and not working. The only change was that the number of samples increased, but the amount of data per sample decreased such that I am using about 1/8th of the amount of data I used previously. Currently, I am uploading only about 1Gb of data into RAM for training/testing.</p>
<p>Any assistance would be greatly appreciated. Please see my model below.</p>
<p>Thank you.</p>
<p>import time<br>
from numpy import savetxt<br>
import torch<br>
from torch.utils.data import DataLoader<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torchvision<br>
import torchvision.transforms as transforms<br>
import matplotlib.pyplot as plt<br>
import numpy as np<br>
import csv<br>
from numpy import genfromtxt</p>
<p>#%%</p>
<p>start = time.time()  # Begin Timer</p>
<h1><a name="device-configuration-1" class="anchor" href="#device-configuration-1"></a>Device configuration</h1>
<p><span class="hashtag-raw">#device</span> = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
device = torch.device(‘cuda:0’ if torch.cuda.is_available() else ‘cpu’)</p>
<h1><a name="hyperparameters-2" class="anchor" href="#hyperparameters-2"></a>Hyperparameters</h1>
<p>ngrid = 2592;<br>
num_epochs = 500;<br>
batch_size = 1;<br>
learning_rate = 0.0003 # originally 0.001<br>
<span class="hashtag-raw">#momentum</span> = 0.5;<br>
neg_slope = 0.4   ; # Neg slope for a Leaky Relu<br>
alpha = 1; # ELU constant</p>
<h1><a name="store-the-data-in-a-numpy-variable-3" class="anchor" href="#store-the-data-in-a-numpy-variable-3"></a>Store the data in a Numpy variable</h1>
<p>with open(‘Ngrid2592_train_data.xlsx’, ‘r’) as f1:<br>
<span class="hashtag-raw">#with</span> open(‘train_data.xlsx’, ‘r’) as f1:</p>
<pre><code>reader = csv.reader(f1) # store excel file in variable 'reader'
training_data_np_list = list(reader) # convert 'reader' to a list
training_data_np = np.array(training_data_np_list, dtype = np.float32) # convert list into a np array
training_data_np = training_data_np[:, 0:ngrid]; # only keep N_grid data (get rid of labels)
</code></pre>
<p>with open(‘Ngrid2592_train_data_labels.xlsx’, ‘r’) as f2:<br>
<span class="hashtag-raw">#with</span> open(‘train_data_labels.xlsx’, ‘r’) as f2:</p>
<pre><code>reader = csv.reader(f2)
training_data_labels_list = list(reader)
training_data_labels_np = np.array(training_data_labels_list, dtype = np.float32)
</code></pre>
<p>with open(‘Ngrid2592_test_data.xlsx’, ‘r’) as f3:<br>
<span class="hashtag-raw">#with</span> open(‘test_data.xlsx’, ‘r’) as f3:</p>
<pre><code>reader = csv.reader(f3)
test_data_np_list = list(reader)
test_data_np = np.array(test_data_np_list, dtype = np.float32)
test_data_np = test_data_np[:, 0:ngrid]; # only keep n_grid data
</code></pre>
<p>with open(‘Ngrid2592_test_data_labels.xlsx’, ‘r’) as f4:<br>
<span class="hashtag-raw">#with</span> open(‘test_data_labels.xlsx’, ‘r’) as f4:</p>
<pre><code>reader = csv.reader(f4)
test_data_labels_list = list(reader)
test_data_labels_np = np.array(test_data_labels_list, dtype = np.float32)
</code></pre>
<h1><a name="convert-numpy-data-into-pytorch-tensors-the-data-includes-the-labels-4" class="anchor" href="#convert-numpy-data-into-pytorch-tensors-the-data-includes-the-labels-4"></a>Convert Numpy data into Pytorch Tensors - The data includes the labels</h1>
<p><span class="hashtag-raw">#training_data_pth</span> = torch.from_numpy(training_data_np, requires_grad=True);<br>
<span class="hashtag-raw">#training_data_labels_pth</span> = torch.from_numpy(training_data_labels_np, requires_grad=True);<br>
<span class="hashtag-raw">#test_data_pth</span> = torch.from_numpy(test_data_np, requires_grad=True);<br>
<span class="hashtag-raw">#test_data_labels_pth</span> = torch.from_numpy(test_data_labels_np, requires_grad=True);</p>
<h1><a name="convert-numpy-data-into-pytorch-tensor-original-42124-5" class="anchor" href="#convert-numpy-data-into-pytorch-tensor-original-42124-5"></a>Convert Numpy data into Pytorch Tensor - Original 4.21.24</h1>
<p>training_data_pth = torch.from_numpy(training_data_np);<br>
training_data_labels_pth = torch.from_numpy(training_data_labels_np);<br>
test_data_pth = torch.from_numpy(test_data_np);<br>
test_data_labels_pth = torch.from_numpy(test_data_labels_np);</p>
<p>print(‘Training Data Type:’, training_data_pth.dtype)<br>
print(‘Training Data Labels Type:’, training_data_labels_pth.dtype)<br>
print(‘Test Data Type:’, test_data_pth.dtype)<br>
print(‘Test Data Labels Type:’, test_data_labels_pth.dtype)</p>
<h1><a name="transform-the-tensor-into-a-dataset-6" class="anchor" href="#transform-the-tensor-into-a-dataset-6"></a>Transform the tensor into a dataset</h1>
<p>training_data = torch.utils.data.TensorDataset(training_data_pth, training_data_labels_pth);<br>
test_data = torch.utils.data.TensorDataset(test_data_pth, test_data_labels_pth);</p>
<h1><a name="wrap-the-dataset-with-a-data-loader-this-makes-it-an-iterable-object-7" class="anchor" href="#wrap-the-dataset-with-a-data-loader-this-makes-it-an-iterable-object-7"></a>Wrap the dataset with a data loader - this makes it an iterable object</h1>
<p>train_loader = DataLoader(training_data, batch_size = batch_size, shuffle = False)<br>
test_loader = DataLoader(test_data, batch_size = batch_size, shuffle = False)</p>
<h1><a name="one-batch-of-random-training-data-samples-8" class="anchor" href="#one-batch-of-random-training-data-samples-8"></a>one batch of random training data samples</h1>
<p><span class="hashtag-raw">#data_iter</span> = iter(train_loader)<br>
<span class="hashtag-raw">#label_iter</span> = iter(train_loader_labels)<br>
<span class="hashtag-raw">#MUSIC_data</span> = data_iter.next()<br>
<span class="hashtag-raw">#labels</span> = label_iter.next()</p>
<h1><a name="one-batch-of-random-training-data-samples-9" class="anchor" href="#one-batch-of-random-training-data-samples-9"></a>one batch of random training data samples</h1>
<p>dataiter = iter(train_loader)<br>
MUSIC_data, labels = next(dataiter)<br>
print(‘Size of MUSIC_data’, MUSIC_data.size())<br>
print(‘Size of labels’, labels.size())</p>
<h1><a name="cnn-model-10" class="anchor" href="#cnn-model-10"></a>CNN Model</h1>
<p>class ConvNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>    # First attempt - input sizes not correct bc they should be 3D unbatched
    # Should there be 3 input channels? One for azimuth, one for colat, one for MUSIC output? Also consider channels as number of freq bins
    
    #self.bn1 = nn.BatchNorm1d(1); # input to batch size is number of features (channels)
    self.conv1 = nn.Conv1d( batch_size , 64 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) 
    self.bn1 = nn.BatchNorm1d(64);
    self.conv2 = nn.Conv1d( 64 , 64 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn2 = nn.BatchNorm1d(batch_size);
    self.pool1 = nn.MaxPool1d(2, 2) 
    
    self.conv3 = nn.Conv1d( 64 , 128 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output is 1 x 900
    self.bn3 = nn.BatchNorm1d(batch_size);
    self.conv4 = nn.Conv1d( 128 , 128 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn4 = nn.BatchNorm1d(128)
    self.pool2 = nn.MaxPool1d(3, 3)
    
    self.conv5 = nn.Conv1d( 128 , 256 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output is 1 x 900
    self.bn5 = nn.BatchNorm1d(256);
    self.conv6 = nn.Conv1d( 256 , 256 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn6 = nn.BatchNorm1d(256);
    self.conv7 = nn.Conv1d( 256 , 256 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn7 = nn.BatchNorm1d( 256 );
    self.pool3 = nn.MaxPool1d(2, 2)
    
    self.conv8 = nn.Conv1d( 256 , 512 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output is 1 x 900
    self.bn8 = nn.BatchNorm1d( 512 );
    self.conv9 = nn.Conv1d( 512 , 512 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn9 = nn.BatchNorm1d( 512 );
    self.conv10 = nn.Conv1d( 512 , 512 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn10 = nn.BatchNorm1d( 512 );
    self.pool4 = nn.MaxPool1d( 2 , 2 ) 
    
    self.conv11 = nn.Conv1d( 512 , 512 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output is 1 x 900
    self.bn11 = nn.BatchNorm1d( 512 );
    self.conv12 = nn.Conv1d( 512 , 512 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn12 = nn.BatchNorm1d( 512 );
    self.conv13 = nn.Conv1d( 512 , 1 , kernel_size = 3, stride = 1, dilation = 1, groups = 1, bias = True) # output size = [(W−K+2P)/S]+1, W = flattened image length, K = kernel size, P = padding, S = Stride
    self.bn13 = nn.BatchNorm1d( 512 );
    self.pool5 = nn.MaxPool1d( 3 , 2 ) 
    
    self.fc1 = nn.Linear( 48 , 48 , bias = False)
    self.bn14 = nn.BatchNorm1d( 48 );
    self.fc2 = nn.Linear( 48 , 48 , bias = False) # originally 1000, 225
    self.bn15 = nn.BatchNorm1d( 48 );
    self.fc3 = nn.Linear( 48, 3 )   # originally 225, 3
    
def forward(self, x):
   
   # This was an attempt to reshape the input for use with batch sizes other than 1 
   # Transpose data to correct shape -
   # Current shape = [embedded dim, batch size, length] = [1, 50, 16207]
   # New Shape = [batch size, embedded dim, length] = [50, 1, 16207]
   # swap 2nd and 3rd dims
   #print('Size of x before transpose', x.size())
   #x = torch.transpose(x, 1, 2);
   #print('Size of x before resizing into CNN', x.size())
   #x_temp = torch.empty(( 1 , batch_size , x.size( dim=1 ) ) ) # empty tensor
   #x_temp[ 0 , : , : ] = x;                                    # assign input 'x' values to empty tensor
   ##x = torch.transpose( x_temp, 0, 1 );                       # transpose 'x_temp' and assign it to input 'x' 
   #x = x_temp;                                                 # assign x_temp to input 'x'
   #x = x.to(device);                                           # push data to device
   
   # BLOCK 1
   #x = self.bn1(x)
   #print('x dtype after batchnorm:', x.dtype)
   #print('Size of x after batchnorm', x.size())
   
   # define leaky relu Activation Fcn, then apply as shown below
   m = nn.LeakyReLU(neg_slope) 
  
   x = self.conv1(x)
   x = m(x) # apply LeakyReLU       
   #x = torch.permute(x, (1, 0))
   #x = torch.permute(x, (0, 1))
   x = self.conv2(x)
   x = m(x) # apply LeakyReLU
   x = self.pool1(x)            
   
   # BLOCK 2
   x = self.conv3(x)            
   x = m(x) # apply LeakyReLU
   x = self.conv4(x)
   x = m(x) # apply LeakyReLU
   x = self.pool2(x)            
   
   # BLOCK 3
   x = self.conv5(x)           
   x = m(x) # apply LeakyReLU
   x = self.conv6(x)  
   x = m(x) # apply LeakyReLU
   x = self.conv7(x)            
   x = m(x) # apply LeakyReLU
   x = self.pool3(x)
   
   # BLOCK 4
   x = self.conv8(x)            
   x = m(x) # apply LeakyReLU
   x = self.conv9(x)
   x = m(x) # apply LeakyReLU
   x = self.conv10(x)      
   x = m(x) # apply LeakyReLU
   x = self.pool4(x)  
   
   # BLOCK 5
   x = self.conv11(x)            
   x = m(x) # apply LeakyReLU       
   x = self.conv12(x) 
   x = m(x) # apply LeakyReLU    
   x = self.conv13(x)            
   x = m(x) # apply LeakyReLU   
   x = self.pool5(x)  
   
   # BLOCK 6
   x = self.fc1(x)  
   x = m(x) # apply LeakyReLU
   x = self.fc2(x)  
   x = m(x) # apply LeakyReLU
   x = self.fc3(x)              
   
   #print("I made it through!")
   
   return x
</code></pre>
<p>model = ConvNet().to(device) # Send model to GPU</p>
<p><span class="hashtag-raw">#if</span> torch.cuda.is_available(): # send model to GPU</p>
<h1><a name="modelcuda-11" class="anchor" href="#modelcuda-11"></a>model.cuda()</h1>
<p>criterion = nn.MSELoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)<br>
<span class="hashtag-raw">#optimizer</span> = torch.optim.SGD(model.parameters(), lr = learning_rate, momentum = momentum)</p>
<p>training_loss_np_array = np.zeros([ 1 , num_epochs ]);<br>
loss_np_array_index = 0;</p>
<p>n_total_steps = len(train_loader)<br>
for epoch in range(num_epochs):</p>
<pre><code>n_train_samples = len(train_loader)

training_comparison_data = torch.zeros([n_train_samples, 6]) # empty tensor for comparing  samples
training_index = 0; # iterable index variable    

running_loss = 0.0

for i, (MUSIC_data, labels) in enumerate(train_loader):
    MUSIC_data = MUSIC_data.to(device) # send to GPU
    labels = labels.to(device) # send to GPU

    # Forward pass
    outputs = model(MUSIC_data)
    #loss = criterion(outputs, labels) # use this for non-augmented losses
    loss = torch.sqrt(criterion(outputs, labels)) # changes MSEloss to euclidean distance since MSEloss is (X**2 + Y**2 + Z**2)
    
    # Output Comparison and Predicted vs Actual Sound Source distance calculation
    training_predicted = outputs ;
    training_comparison_data[training_index : training_index + batch_size , :3 ] = training_predicted ; 
    training_comparison_data[training_index : training_index + batch_size , 3:6 ] = labels ; 
    
    training_index += 1; 
    
    # Backward and optimize
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    running_loss += loss.item()
#training_loss_np_array[ 0 : loss_np_array_index]  = loss.detach().numpy();
#loss_np_array_index += 1; # index to next column in training loss array
#print(f'[{epoch + 1}] loss: {running_loss:.3f}')
print(f'[{epoch + 1}] loss: {running_loss :.3f}')
</code></pre>
<p>training_comparison_data_np = training_comparison_data.detach().numpy(); <span class="hashtag-raw">#convert</span> to numpy array<br>
print(‘Finished Training’)<br>
<span class="hashtag-raw">#PATH</span> = ‘./VGG16_rev_DOE3_Trial3.pth’<br>
<span class="hashtag-raw">#torch</span>.save(model.state_dict(), PATH)</p>
<p>end = time.time()  #  End Timer<br>
print(‘Total Time to Execute Training[Seconds]’)<br>
print(end - start)  #  Print Time to Execute Program<br>
print(‘Total Time to Execute Training[minutes]’)<br>
print((end - start)/60)  #  Print Time to Execute Program<br>
print(‘Total Time to Execute Training[hours]’)<br>
print((end - start)/3600)  #  Print Time to Execute Program</p>
<p>#%%</p>
<h1><a name="evaluation-section-12" class="anchor" href="#evaluation-section-12"></a>Evaluation Section</h1>
<p>dataiter = iter(test_loader)<br>
MUSIC_data, labels = next(dataiter)</p>
<p>testing_running_loss = 0.0</p>
<p>start = time.time()  # Begin Timer<br>
with torch.no_grad():</p>
<pre><code>n_samples = len(test_loader)

comparison_data = torch.zeros([n_samples, 6]) # empty tensor for comparing  samples
index = 0; # iterable index variable

for MUSIC_data, labels in test_loader:
    MUSIC_data = MUSIC_data.to(device)
    labels = labels.to(device)
    outputs = model(MUSIC_data)
    testing_loss = torch.sqrt(criterion(outputs, labels)) # changes MSEloss to euclidean distance since MSEloss is (X**2 + Y**2 + Z**2)

    
    # Output Comparison and Predicted vs Actual Sound Source distance calculation
    testing_predicted = outputs ;
    #distance = torch.sqrt( (outputs[0] - labels[0] )**2 + (outputs[1] - labels[1] )**2 + ( outputs[2] - labels[2] )**2 );
    comparison_data[index, :3 ] = testing_predicted ; 
    comparison_data[index, 3:6 ] = labels ; 
    #comparison_data[index, 6] = distance 
    
    index += 1; 
   
    testing_running_loss += testing_loss.item()
print(' Testing loss: {testing_running_loss :.3f}')
</code></pre>
<h1><a name="convert-data-to-numpy-13" class="anchor" href="#convert-data-to-numpy-13"></a>Convert data to numpy</h1>
<p>comparison_data_np_array = comparison_data.numpy();</p>
<p>comparison_data_np_array = comparison_data.numpy() # convert tensor to np array<br>
<span class="hashtag-raw">#savetxt</span>(‘predicted_SSL_vs_real_SSL_CNN3_ngrid_16200_LR0.0005_LeakyRelu0.4_epochs150_5.15.24.csv’, comparison_data_np_array, delimiter=‘,’, header = ‘Predicted XYZ, Real XYZ, Euclidean Distance’)<br>
<span class="hashtag-raw">#savetxt</span>(‘DOE_Trial_16_Test_data_Loss_CNN3_ngrid_16200_LR0.0005_LeakyRelu0.4_epochs150_5.15.24.csv’, comparison_data_np_array, delimiter=‘,’, header = ‘Predicted XYZ, Real XYZ, Euclidean Distance’)</p>
<p>end = time.time()  #  End Timer<br>
print(‘Total Time to Execute Testing[Seconds]’)<br>
print(end - start)  #  Print Time to Execute Program<br>
print(‘Total Time to Execute Testing[minutes]’)<br>
print((end - start)/60)  #  Print Time to Execute Program<br>
print(‘Total Time to Execute Testing[hours]’)<br>
print((end - start)/3600)  #  Print Time to Execute Program</p>
<pre><code>#acc = 100.0 * n_correct2 / n_samples
#print(f'Accuracy of the loaded model: {acc} %')
</code></pre>