Do I need to manually reset weights for each fold in K-fold CV?

username: Avash
cooked: <p>I’m trying to perform k-fold cross validation. I’ve split my dataset into training and test set. The training set will be used to create validation set and actual training set for each fold. I am not sure if I’ve to manually reset my weights for the code I’ve written here.</p>
<h1>
<a name="creating-data-loader-train-data-loader-will-be-created-later-for-cross-validation-1" class="anchor" href="#creating-data-loader-train-data-loader-will-be-created-later-for-cross-validation-1"></a>Creating Data Loader (Train Data Loader will be created later for cross validation)</h1>
<p>test_data_loader = DataLoader(test_dataset, batch_size= batch_size, shuffle = False)</p>
<h1>
<a name="building-neural-network-2" class="anchor" href="#building-neural-network-2"></a>Building Neural Network</h1>
<p>class OurNN(nn.Module):</p>
<pre><code>def __init__(self, input_size = 12, output_size = 1):
    super(OurNN, self).__init__()
    #print(input_size, output_size)
    hidden_units1 = 32
    hidden_units2 = 32
    hidden_units3 = 32
    
    
    #Layers of our NN
    self.hidden_layer1 = nn.Linear(input_size, hidden_units1)
    self.hidden_layer2 = nn.Linear(hidden_units1, hidden_units2)
    self.hidden_layer3 = nn.Linear(hidden_units2, hidden_units3)
    self.output_layer = nn.Linear(hidden_units3, output_size)
    
    #Activation Functions
    self.relu = nn.ReLU()
    self.sigmoid = nn.Sigmoid()
    print('Instance Created')
    
def forward(self, X):
    
    #First Hidden Layer
    z = self.hidden_layer1(X)
    z = self.relu(z)
    
    #Second Hidden Layer
    z = self.hidden_layer2(z)
    z = self.relu(z)
    
    #Third Hidden Layer
    z = self.hidden_layer3(z)
    z = self.relu(z)
    
    #Output Layer
    z = self.output_layer(z)
    result = self.sigmoid(z)
    return result
</code></pre>
<p>model = OurNN().to(device)</p>
<p>loss_function = nn.BCELoss() <span class="hashtag">#Binary</span> Cross Entropy Loss<br>
optimizer = torch.optim.AdamW(model.parameters(), lr = learning_rate) <span class="hashtag">#AdamW</span> Optimizer</p>
<h1>
<a name="k-fold-cross-validation-3" class="anchor" href="#k-fold-cross-validation-3"></a>K-fold cross validation</h1>
<p>k_num = 10<br>
k_fold = KFold(n_splits=k_num , shuffle = True)</p>
<p><span class="hashtag">#Model</span> Training</p>
<p>loss_train_list = []<br>
loss_validation_list = []<br>
accuracy_train_list = []<br>
accuracy_validation_list = []<br>
loss_test_list = []<br>
accuracy_test_list = []<br>
start_time = time.time()<br>
for k, (train_index, val_index) in enumerate(k_fold.split(np.arange(len(train_dataset)))):<br>
print(train_index, val_index)<br>
print(len(train_index), len(val_index))</p>
<pre><code>print(f"\n Fold-{k+1}:\n") 

train_set_sampler = SubsetRandomSampler(train_index)
validation_sampler = SubsetRandomSampler(val_index)
train_data_loader = DataLoader(train_dataset, batch_size=batch_size, sampler = train_set_sampler)
validation_data_loader = DataLoader(train_dataset, batch_size=batch_size, sampler=validation_sampler)

  
for epoch in range(epochs):
    correct_prediction_train = 0
    per_epoch_loss_train = 0
    correct_prediction_validation = 0
    per_epoch_loss_validation = 0 
    correct_prediction_test = 0
    per_epoch_loss_test = 0

    #Training Set
    for i, (X_data_train, y_data_train) in enumerate(train_data_loader): #For each batch
        X_data_train = X_data_train.to(device)
        y_data_train = y_data_train.to(device)

        #Forward pass
        result_train = model(X_data_train)

        #Results label and correct predictions
        result_label_train = (result_train&gt;0.5).float()
        correct_prediction_train += (result_label_train == y_data_train).sum()  

        #Loss
        loss_train = loss_function(result_train,y_data_train)
        per_epoch_loss_train += loss_train.item()

        #Backward pass
        optimizer.zero_grad()
        loss_train.backward()
        optimizer.step()


    # Appending loss_train_list list for each epoch averaged over batch_size 
    avg_loss_train = round((per_epoch_loss_train/len(train_data_loader)),5)
    loss_train_list.append(avg_loss_train)
    
    
    #Train Set Accuracy
    accuracy_train = ((correct_prediction_train/len(train_data_loader.dataset)).item()) * 100
    accuracy_train_list.append(accuracy_train)
    print(f"End of epoch {epoch+1}: Train Loss = {avg_loss_train} &amp; Train Accuracy = {round(accuracy_train,5)} ")
    
    
    
    #Validation Test
    
    with torch.no_grad(): 
        for i, (X_data_validation, y_data_validation) in enumerate(validation_data_loader):
            X_data_validation = X_data_validation.to(device)
            y_data_validation = y_data_validation.to(device)

            #Forward pass
            result_validation = model(X_data_validation)
            result_label_validation = (result_validation&gt;0.5).float()
            correct_prediction_validation += (result_label_validation == y_data_validation).sum()  

            #Loss
            loss_validation = loss_function(result_validation,y_data_validation)
            per_epoch_loss_validation += loss_validation.item()

    # Appending train_loss list for each epoch averaged over batch_size 
    avg_loss_validation = round((per_epoch_loss_validation/len(validation_data_loader)),5)
    loss_validation_list.append(avg_loss_validation)

    #Accuracy
    accuracy_validation = round((((correct_prediction_validation/len(validation_data_loader.dataset)).item()) * 100),2)
    accuracy_validation_list.append(accuracy_validation)
    
    
    # Test Set
    with torch.no_grad(): 
        for i, (X_data_test, y_data_test) in enumerate(test_data_loader): #For each batch
            X_data_test = X_data_test.to(device)
            y_data_test = y_data_test.to(device)

            #Forward pass
            result_test = model(X_data_test)
            result_label_test = (result_test&gt;0.5).float()
            correct_prediction_test += (result_label_test == y_data_test).sum()  

            #Loss
            loss_test = loss_function(result_test,y_data_test)
            per_epoch_loss_test += loss_test.item()

    # Appending train_loss list for each epoch averaged over batch_size 
    avg_loss_test = round((per_epoch_loss_test/len(test_data_loader)),5)
    loss_test_list.append(avg_loss_test)

    #Accuracy
    accuracy_test = round((((correct_prediction_test/len(test_data_loader.dataset)).item()) * 100),2)
    accuracy_test_list.append(accuracy_test)
    #print(f"Test Accuracy:{accuracy_test}")
</code></pre>
<p>stop_time = time.time()</p>
<h1>
<a name="averaging-train-and-test-accuracy-based-on-k_num-4" class="anchor" href="#averaging-train-and-test-accuracy-based-on-k_num-4"></a>Averaging Train and Test Accuracy based on k_num</h1>
<p>average_accuracy_train_list = []<br>
average_accuracy_test_list = []<br>
average_loss_test_list = []<br>
average_loss_train_list = []</p>
<p>for i in range(len(accuracy_train_list)):<br>
train_acc_avg = sum(accuracy_train_list[i:i+epochs])/len(accuracy_train_list[i:i+epochs])<br>
test_acc_avg = sum(accuracy_test_list[i:i+epochs])/len(accuracy_test_list[i:i+epochs])<br>
train_loss_avg = sum(loss_train_list[i:i+epochs])/len(loss_train_list[i:i+epochs])<br>
test_loss_avg =  sum(loss_test_list[i:i+epochs])/len(loss_test_list[i:i+epochs])</p>
<pre><code>if i % k_num == 0:
    average_accuracy_train_list.append(train_acc_avg)
    average_accuracy_test_list.append(test_acc_avg)
    average_loss_test_list.append(train_loss_avg)
    average_loss_train_list.append(test_loss_avg)
</code></pre>
<p>print(f"average_accuracy_train_list: {average_accuracy_train_list}")<br>
print(f"average_accuracy_test_list: {average_accuracy_test_list}")<br>
print(f"average_loss_test_list:{average_loss_test_list}")<br>
print(f"average_loss_test_list:{average_loss_test_list}")</p>

username: mxahan
cooked: <p>Yes. Each time you may have to take a new instance of the model class.</p>

username: Avash
cooked: <p>In my code, how can I call a new instance from training loop or where and how can I actually reset weights?</p>

username: mxahan
cooked: <p>Well, you can create an additional loop using the number of folds we want to test and run the whole setup for the split data. Hope this <a href="https://medium.com/dataseries/k-fold-cross-validation-with-pytorch-and-sklearn-d094aa00105f" rel="noopener nofollow ugc">post</a> helps.</p>