Let’s Build a Fashion-MNIST CNN, PyTorch Style
By: Michael Li
Tags: Machine Learning, Pytorch, Artificial Intelligence, Education, Deep Learning

Hyperparameters

Normally, we can just handpick one set of hyperparameters and do some experiments with them. In this example, we want to do a bit more by introducing some structuring. We’ll build a system to generate different hyperparameter combinations and use them to carry out training ‘runs’. Each ‘run’ uses one set of hyperparameter combinations. Export the training data/results of each run to Tensor Board so we can directly compare and see which hyperparameters set performs the best.

We store all our hyperparameters in an OrderedDict:

# put all hyper params into a OrderedDict, easily expandable

params = OrderedDict(

lr = [.01, .001],

batch_size = [100, 1000],

shuffle = [True, False]

)

epochs = 3

lr : Learning Rate. We want to try 0.01 and 0.001 for our models.

batch_size : Batch Size to speed up the training process. We’ll use 100 and 1000.

shuffle : Shuffle toggle, whether we shuffle the batch before training.

Once the parameters are down. We use two helper classes: RunBuilder and RunManager to manage our hyperparameters and training process.

RunBuilder

The main purpose of the class RunBuilder is to offer a static method get_runs . It takes the OrderedDict (with all hyperparameters stored in it) as a parameter and generates a named tuple Run , each element of run represent one possible combination of the hyperparameters. This named tuple is later consumed by the training loop. The code is easy to understand.

# import modules to build RunBuilder and RunManager helper classes

from collections import OrderedDict

from collections import namedtuple

from itertools import product



# Read in the hyper-parameters and return a Run namedtuple containing all the

# combinations of hyper-parameters

class RunBuilder():

@staticmethod

def get_runs(params):



Run = namedtuple('Run', params.keys())



runs = []

for v in product(*params.values()):

runs.append(Run(*v))



return runs

RunManager

There are four main purposes of the RunManager class.

Calculate and record the duration of each epoch and run. Calculate the training loss and accuracy of each epoch and run. Record the training data (e.g. loss, accuracy, weights, gradients, computational graph, etc.) for each epoch and run, then export them into Tensor Board for further analysis. Save all training results in csv and json for future reference or API extraction.

As you can see, it helps us take care of the logistics which is also important for our success in training the model. Let’s look at the code. It’s a bit long so bear with me:

# Helper class, help track loss, accuracy, epoch time, run time,

# hyper-parameters etc. Also record to TensorBoard and write into csv, json

class RunManager():

def __init__(self):



# tracking every epoch count, loss, accuracy, time

self.epoch_count = 0

self.epoch_loss = 0

self.epoch_num_correct = 0

self.epoch_start_time = None



# tracking every run count, run data, hyper-params used, time

self.run_params = None

self.run_count = 0

self.run_data = []

self.run_start_time = None



# record model, loader and TensorBoard

self.network = None

self.loader = None

self.tb = None



# record the count, hyper-param, model, loader of each run

# record sample images and network graph to TensorBoard

def begin_run(self, run, network, loader):



self.run_start_time = time.time()



self.run_params = run

self.run_count += 1



self.network = network

self.loader = loader

self.tb = SummaryWriter(comment=f'-{run}')



images, labels = next(iter(self.loader))

grid = torchvision.utils.make_grid(images)



self.tb.add_image('images', grid)

self.tb.add_graph(self.network, images)



# when run ends, close TensorBoard, zero epoch count

def end_run(self):

self.tb.close()

self.epoch_count = 0



# zero epoch count, loss, accuracy,

def begin_epoch(self):

self.epoch_start_time = time.time()



self.epoch_count += 1

self.epoch_loss = 0

self.epoch_num_correct = 0



#

def end_epoch(self):

# calculate epoch duration and run duration(accumulate)

epoch_duration = time.time() - self.epoch_start_time

run_duration = time.time() - self.run_start_time



# record epoch loss and accuracy

loss = self.epoch_loss / len(self.loader.dataset)

accuracy = self.epoch_num_correct / len(self.loader.dataset)



# Record epoch loss and accuracy to TensorBoard

self.tb.add_scalar('Loss', loss, self.epoch_count)

self.tb.add_scalar('Accuracy', accuracy, self.epoch_count)



# Record params to TensorBoard

for name, param in self.network.named_parameters():

self.tb.add_histogram(name, param, self.epoch_count)

self.tb.add_histogram(f'{name}.grad', param.grad, self.epoch_count)



# Write into 'results' (OrderedDict) for all run related data

results = OrderedDict()

results["run"] = self.run_count

results["epoch"] = self.epoch_count

results["loss"] = loss

results["accuracy"] = accuracy

results["epoch duration"] = epoch_duration

results["run duration"] = run_duration



# Record hyper-params into 'results'

for k,v in self.run_params._asdict().items(): results[k] = v

self.run_data.append(results)

df = pd.DataFrame.from_dict(self.run_data, orient = 'columns')



# display epoch information and show progress

clear_output(wait=True)

display(df)



# accumulate loss of batch into entire epoch loss

def track_loss(self, loss):

# multiply batch size so variety of batch sizes can be compared

self.epoch_loss += loss.item() * self.loader.batch_size



# accumulate number of corrects of batch into entire epoch num_correct

def track_num_correct(self, preds, labels):

self.epoch_num_correct += self._get_num_correct(preds, labels)



@torch.no_grad()

def _get_num_correct(self, preds, labels):

return preds.argmax(dim=1).eq(labels).sum().item()



# save end results of all runs into csv, json for further analysis

def save(self, fileName):



pd.DataFrame.from_dict(

self.run_data,

orient = 'columns',

).to_csv(f'{fileName}.csv')



with open(f'{fileName}.json', 'w', encoding='utf-8') as f:

json.dump(self.run_data, f, ensure_ascii=False, indent=4)

__init__ : Initialize necessary attributes like count, loss, number of correct predictions, start time, etc.

begin_run : Record run start time so when a run is finished, the duration of the run can be calculated. Create a SummaryWriter object to store everything we want to export into Tensor Board during the run. Write the network graph and sample images into the SummaryWriter object.

end_run : When run is finished, close the SummaryWriter object and reset the epoch count to 0 (getting ready for next run).

begin_epoch : Record epoch start time so epoch duration can be calculated when epoch ends. Reset epoch_loss and epoch_num_correct .

end_epoch : This function is where most things happen. When an epoch ends, we’ll calculate the epoch duration and the run duration(up to this epoch, not the final run duration unless for the last epoch of the run). We’ll calculate the total loss and accuracy for this epoch, then export the loss, accuracy, weights/biases, gradients we recorded into Tensor Board. For ease of tracking within the Jupyter Notebook, we also created an OrderedDict object results and put all our run data(loss, accuracy, run count, epoch count, run duration, epoch duration, all hyperparameters) into it. Then we’ll use Pandas to read it in and display it in a neat table format.

track_loss , track_num_correct , _get_num_correct : These are utility functions to accumulate the loss, number of correct predictions of each batch so the epoch loss and accuracy can be calculated later.

save : Save all run data (a list of results OrderedDict objects for all runs) into csv and json format for further analysis or API access.