Finetune a Facial Recognition Classifier to Recognize your Face using PyTorch | by Mike Chaykowsky | Towards Data Science

This is part of a series I am writing on tricking facial recognition systems using adversarial attacks with GANs.
However, before we trick a facial recognition classifier we need to build one to trick. I personally want to build one that can recognize my own face. Instead of training a neural network from scratch I can start with a pre-trained network and then finetune it to recognize my face. Finetuning is greatly beneficial as we can start with the model weights already trained on a large-scale face database and then update some of them to reflect the new tasks we want it to perform. These weights already understand how to recognize faces, but the only difference is it does not know my face. So to have this pretrained model learn my own face is much easier to train as the model weights already contain much of the needed information to perform the task.
project│   README.md│   AGN.ipynb  │└───data│   │   files_sample.csv│   └───eyeglasses│   ││   └───test_me│       └───train|           └───Adrien_Brody|           ...|           └───Michael_Chaykowsky|           ...|           └───Venus_Williams│       └───val|           └───Adrien_Brody|           ...|           └───Michael_Chaykowsky|           ...|           └───Venus_Williams│   └───models│   │   inception_resnet_v1.py│   │   mtcnn.py│   └───utils
models directory is from the PyTorch facenet implementation based on the Tensorflow implementation linked above.
└───models│   │   inception_resnet_v1.py│   │   mtcnn.py│   └───utils
This inception_resnet_v1.py file is where we will pull in the pretrained model. The Inception Resnet V1 model is pretrained on VGGFace2 where VGGFace2 is a large-scale face recognition dataset developed from Google image searches and “have large variations in pose, age, illumination, ethnicity and profession.”
Each layer’s weights in the model have an attribute called requires_grad that can be set to True or False . When you run loss.backward() in the training loop these weights are updated and this is what contains all of the information needed to perform the predictions. When finetuning the network we freeze all of the layers up through the last convolutional block by setting the requires_grad attributes to False and then only update the weights on the remaining layers — which intuitively you can imagine the earlier layers as containing the base-level information needed to recognize face attributes and base level characteristics so we keep all of that performance while updating the final layers to include another face (mine).
All train directories have 11 or 12 images of each individual and all val directories have 4 or 5 images of each individual. Michael_Chaykowsky is a directory of my face where I used various poses, lighting, and angles. To collect these images I took videos with a standard iPhone in various spaces and then transformed these videos to image and used MTCNN on each to perform face-alignment and cropping to appropriate size (160 x 160 pixels).
from torch import nn, optim, as_tensorfrom torch.utils.data import Dataset, DataLoaderimport torch.nn.functional as Ffrom torch.optim import lr_schedulerfrom torch.nn.init import *from torchvision import transforms, utils, datasets, modelsfrom models.inception_resnet_v1 import InceptionResnetV1import cv2from PIL import Imagefrom pdb import set_traceimport timeimport copyfrom pathlib import Pathimport osimport sysimport matplotlib.pyplot as pltimport matplotlib.animation as animationfrom skimage import io, transformfrom tqdm import trange, tqdmimport csvimport globimport dlibimport pandas as pdimport numpy as np
from IPython.display import VideoVideo("data/IMG_2411.MOV", width=200, height=350)
Capture the frames of the video as .png files and rotate/crop/align.
vidcap = cv2.VideoCapture('IMG_2411.MOV')success,image = vidcap.read()count = 0success = Truewhile success:    cv2.imwrite(f"./Michael_Chaykowsky/Michael_Chaykowsky_{                  format(count, '04d') }.png", image)    success,image = vidcap.read()    print('Read a new frame: ', success)    count += 1
The images come in rotated so I use imagemagick to make them right-side up. Make sure to brew install imagemagick first. I think there’s another way to install the library but if I recall it was a nightmare — so definitely suggest brew install .
%%!for szFile in ./Michael_Chaykowsky/*.pngdo     magick mogrify -rotate 90 ./Michael_Chaykowsky/"$(basename "$szFile")" ; done
! pip install autocrop
Autocrop has a nice feature where they do some resizing of face images and you can specify the face percentage. You can forgo this if you are doing the full MTCNN method (prefferred), but if not you can do this which is quite a bit easier.
! autocrop -i ./me_orig/Michael_Chaykowsky -o ./me/Michael_Chaykowsky160 -w 720 -H 720 --facePercent 80
! pip install tensorflow==1.13.0rc1
! pip install scipy==1.1.0
Now using the align_dataset_mtcnn.py script from David Sandberg’s Tensorflow implementation of facenet we can apply this to the directory of face images.
%%!for N in {1..4}; do \python ~/Adversarial/data/align/align_dataset_mtcnn.py \ # tensorflow script~/Adversarial/data/me/ \ # current directory~/Adversarial/data/me160/ \ # new directory--image_size 160 \--margin 32 \--random_order \--gpu_memory_fraction 0.25 \& done
Now you have a directory with all of your faces aligned and cropped appropriately for modeling.
When we load in the data we will perform some random transforms to the images to improve training. Different transforms can be attempted and you may try various ones, like
Here I use random horizontal flip. All of these transforms make the model more generalizable and prevent overfitting.
data_transforms = {    'train': transforms.Compose([        transforms.RandomHorizontalFlip(),        transforms.ToTensor(),        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])    ]),    'val': transforms.Compose([        transforms.ToTensor(),        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])    ]),}data_dir = 'data/test_me'image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),                                          data_transforms[x])                  for x in ['train', 'val']}dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x],                                              batch_size=8,                                              shuffle=True)              for x in ['train', 'val']}dataset_sizes = {x: len(image_datasets[x]) for x in ['train','val']}class_names = image_datasets['train'].classesclass_names
Out[1]:
['Adrien_Brody','Alejandro_Toledo','Angelina_Jolie','Arnold_Schwarzenegger','Carlos_Moya','Charles_Moose','James_Blake','Jennifer_Lopez','Michael_Chaykowsky','Roh_Moo-hyun','Venus_Williams']
def imshow(inp, title=None):    """Imshow for Tensor."""    inp = inp.numpy().transpose((1, 2, 0))    mean = np.array([0.485, 0.456, 0.406])    std = np.array([0.229, 0.224, 0.225])    inp = std * inp + mean    inp = np.clip(inp, 0, 1)    plt.imshow(inp)    if title is not None:        plt.title(title)    plt.pause(0.001)  # pause a bit so that plots are updated# Get a batch of training datainputs, classes = next(iter(dataloaders['train']))# Make a grid from batchout = utils.make_grid(inputs)imshow(out, title=[class_names[x] for x in classes])
from models.inception_resnet_v1 import InceptionResnetV1print('Running on device: {}'.format(device))model_ft = InceptionResnetV1(pretrained='vggface2', classify=False, num_classes = len(class_names))
Recall earlier I mentioned that we will freeze up through the last conv block. To find where that is we can iterate through this list using -n, -n-1, ... until we find the block.
list(model_ft.children())[-6:]
Out[2]:
[Block8(   (branch0): BasicConv2d(     (conv): Conv2d(1792, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)     (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)     (relu): ReLU()   )   (branch1): Sequential(     (0): BasicConv2d(       (conv): Conv2d(1792, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)       (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)       (relu): ReLU()     )     (1): BasicConv2d(       (conv): Conv2d(192, 192, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), bias=False)       (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)       (relu): ReLU()     )     (2): BasicConv2d(       (conv): Conv2d(192, 192, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)       (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)       (relu): ReLU()     )   )   (conv2d): Conv2d(384, 1792, kernel_size=(1, 1), stride=(1, 1)) ), AdaptiveAvgPool2d(output_size=1), Linear(in_features=1792, out_features=512, bias=False), BatchNorm1d(512, eps=0.001, momentum=0.1, affine=True, track_running_stats=True), Linear(in_features=512, out_features=8631, bias=True), Softmax(dim=1)]
Remove the last layers after conv block and place in layer_list .
layer_list = list(model_ft.children())[-5:] # all final layerslayer_list
Out[3]:
[AdaptiveAvgPool2d(output_size=1), Linear(in_features=1792, out_features=512, bias=False), BatchNorm1d(512, eps=0.001, momentum=0.1, affine=True, track_running_stats=True), Linear(in_features=512, out_features=8631, bias=True), Softmax(dim=1)]
Put all beginning layers in an nn.Sequential . model_ft is now a torch model but without the final linear, pooling, batchnorm, and sigmoid layers.
model_ft = nn.Sequential(*list(model_ft.children())[:-5])
If training just final layers:
for param in model_ft.parameters():    param.requires_grad = False
Re-attach the last 5 layers which automatically sets requires_grad = True .
This linear layer Linear(in_features=1792, out_features=512, bias=False) actually requires writing two custom classes which is not entirely obvious by looking at it, but if you look at the data input/output you can see that there is a Flatten and normalize class within the layer. Check resnet implementation for reason of reshaping in last_linear layer.
class Flatten(nn.Module):    def __init__(self):        super(Flatten, self).__init__()            def forward(self, x):        x = x.view(x.size(0), -1)        return xclass normalize(nn.Module):    def __init__(self):        super(normalize, self).__init__()            def forward(self, x):        x = F.normalize(x, p=2, dim=1)        return x
Then you can apply the final layers back to the new Sequential model.
model_ft.avgpool_1a = nn.AdaptiveAvgPool2d(output_size=1)model_ft.last_linear = nn.Sequential(    Flatten(),    nn.Linear(in_features=1792, out_features=512, bias=False),    normalize())model_ft.logits = nn.Linear(layer_list[3].in_features, len(class_names))model_ft.softmax = nn.Softmax(dim=1)model_ft = model_ft.to(device)criterion = nn.CrossEntropyLoss()# Observe that all parameters are being optimizedoptimizer_ft = optim.SGD(model_ft.parameters(), lr=1e-2, momentum=0.9)# Decay LR by a factor of *gamma* every *step_size* epochsexp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
def train_model(model, criterion, optimizer, scheduler,                num_epochs=25):    since = time.time()    FT_losses = []    best_model_wts = copy.deepcopy(model.state_dict())    best_acc = 0.0    for epoch in range(num_epochs):        print('Epoch {}/{}'.format(epoch, num_epochs - 1))        print('-' * 10)    # Each epoch has a training and validation phase        for phase in ['train', 'val']:            if phase == 'train':                model.train()  # Set model to training mode            else:                model.eval()   # Set model to evaluate mode            running_loss = 0.0            running_corrects = 0            # Iterate over data.            for inputs, labels in dataloaders[phase]:                inputs = inputs.to(device)                labels = labels.to(device)                # zero the parameter gradients                optimizer.zero_grad()                # forward                # track history if only in train                with torch.set_grad_enabled(phase == 'train'):                    outputs = model(inputs)                    _, preds = torch.max(outputs, 1)                    loss = criterion(outputs, labels)                    # backward + optimize only if in training phase                    if phase == 'train':                        loss.backward()                        optimizer.step()                        scheduler.step()                                FT_losses.append(loss.item())                # statistics                running_loss += loss.item() * inputs.size(0)                running_corrects += torch.sum(preds == labels.data)            epoch_loss = running_loss / dataset_sizes[phase]            epoch_acc = running_corrects.double() /                         dataset_sizes[phase]            print('{} Loss: {:.4f} Acc: {:.4f}'.format(                phase, epoch_loss, epoch_acc))            # deep copy the model            if phase == 'val' and epoch_acc > best_acc:                best_acc = epoch_acc                best_model_wts = copy.deepcopy(model.state_dict())    time_elapsed = time.time() - since    print('Training complete in {:.0f}m {:.0f}s'.format(        time_elapsed // 60, time_elapsed % 60))    print('Best val Acc: {:4f}'.format(best_acc))    # load best model weights    model.load_state_dict(best_model_wts)    return model, FT_losses
model_ft, FT_losses = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=500)plt.figure(figsize=(10,5))plt.title("FRT Loss During Training")plt.plot(FT_losses, label="FT loss")plt.xlabel("iterations")plt.ylabel("Loss")plt.legend()plt.show()
Keep an eye out for more from this series where I will describe how to trick this classifier using adversarial attacks with GANs.