Finetune a Facial Recognition Classifier to Recognize your Face using PyTorch
By: Mike Chaykowsky
Tags: Machine Learning, Artificial Intelligence, Deep Learning, Facial Recognition, Pytorch

Finetune a Facial Recognition Classifier to Recognize your Face using PyTorch

Tricking facial recognition systems using adversarial attacks with GANs.

This is part of a series I am writing on tricking facial recognition systems using adversarial attacks with GANs.

However, before we trick a facial recognition classifier we need to build one to trick. I personally want to build one that can recognize my own face. Instead of training a neural network from scratch I can start with a pre-trained network and then finetune it to recognize my face. Finetuning is greatly beneficial as we can start with the model weights already trained on a large-scale face database and then update some of them to reflect the new tasks we want it to perform. These weights already understand how to recognize faces, but the only difference is it does not know my face. So to have this pretrained model learn my own face is much easier to train as the model weights already contain much of the needed information to perform the task.

Directory Structure

project

│ README.md

│ AGN.ipynb

│

└───data

│ │ files_sample.csv

│ └───eyeglasses

│ │

│ └───test_me

│ └───train

| └───Adrien_Brody

| ...

| └───Michael_Chaykowsky

| ...

| └───Venus_Williams

│ └───val

| └───Adrien_Brody

| ...

| └───Michael_Chaykowsky

| ...

| └───Venus_Williams

│

└───models

│ │ inception_resnet_v1.py

│ │ mtcnn.py

│ └───utils

models directory is from the PyTorch facenet implementation based on the Tensorflow implementation linked above.

└───models

│ │ inception_resnet_v1.py

│ │ mtcnn.py

│ └───utils

This inception_resnet_v1.py file is where we will pull in the pretrained model. The Inception Resnet V1 model is pretrained on VGGFace2 where VGGFace2 is a large-scale face recognition dataset developed from Google image searches and “have large variations in pose, age, illumination, ethnicity and profession.”

Each layer’s weights in the model have an attribute called requires_grad that can be set to True or False . When you run loss.backward() in the training loop these weights are updated and this is what contains all of the information needed to perform the predictions. When finetuning the network we freeze all of the layers up through the last convolutional block by setting the requires_grad attributes to False and then only update the weights on the remaining layers — which intuitively you can imagine the earlier layers as containing the base-level information needed to recognize face attributes and base level characteristics so we keep all of that performance while updating the final layers to include another face (mine).

All train directories have 11 or 12 images of each individual and all val directories have 4 or 5 images of each individual. Michael_Chaykowsky is a directory of my face where I used various poses, lighting, and angles. To collect these images I took videos with a standard iPhone in various spaces and then transformed these videos to image and used MTCNN on each to perform face-alignment and cropping to appropriate size (160 x 160 pixels).

Imports

from torch import nn, optim, as_tensor

from torch.utils.data import Dataset, DataLoader

import torch.nn.functional as F

from torch.optim import lr_scheduler

from torch.nn.init import *

from torchvision import transforms, utils, datasets, models

from models.inception_resnet_v1 import InceptionResnetV1 import cv2

from PIL import Image

from pdb import set_trace

import time

import copy from pathlib import Path

import os

import sys

import matplotlib.pyplot as plt

import matplotlib.animation as animation

from skimage import io, transform

from tqdm import trange, tqdm

import csv

import glob

import dlib import pandas as pd

import numpy as np

Multi-task Cascaded Convolutional Neural Network (MTCNN) for face alignment

from IPython.display import Video Video("data/IMG_2411.MOV", width=200, height=350)

Video of me rotating my face

Capture the frames of the video as .png files and rotate/crop/align.

vidcap = cv2.VideoCapture('IMG_2411.MOV')

success,image = vidcap.read()

count = 0

success = True

while success:

cv2.imwrite(f"./Michael_Chaykowsky/Michael_Chaykowsky_{

format(count, '04d') }.png", image)

success,image = vidcap.read()

print('Read a new frame: ', success)

count += 1

The images come in rotated so I use imagemagick to make them right-side up. Make sure to brew install imagemagick first. I think there’s another way to install the library but if I recall it was a nightmare — so definitely suggest brew install .

%%!

for szFile in ./Michael_Chaykowsky/*.png

do

magick mogrify -rotate 90 ./Michael_Chaykowsky/"$(basename "$szFile")" ;

done

! pip install autocrop

Autocrop has a nice feature where they do some resizing of face images and you can specify the face percentage. You can forgo this if you are doing the full MTCNN method (prefferred), but if not you can do this which is quite a bit easier.

! autocrop -i ./me_orig/Michael_Chaykowsky -o ./me/Michael_Chaykowsky160 -w 720 -H 720 --facePercent 80

! pip install tensorflow==1.13.0rc1

! pip install scipy==1.1.0

Now using the align_dataset_mtcnn.py script from David Sandberg’s Tensorflow implementation of facenet we can apply this to the directory of face images.

%%!

for N in {1..4}; do \

python ~/Adversarial/data/align/align_dataset_mtcnn.py \ # tensorflow script

~/Adversarial/data/me/ \ # current directory

~/Adversarial/data/me160/ \ # new directory

--image_size 160 \

--margin 32 \

--random_order \

--gpu_memory_fraction 0.25 \

& done

Now you have a directory with all of your faces aligned and cropped appropriately for modeling.

Load Data

When we load in the data we will perform some random transforms to the images to improve training. Different transforms can be attempted and you may try various ones, like

Random Color Jitter

Random Rotation +/- 5 degrees

Random Horizontal Flip

Here I use random horizontal flip. All of these transforms make the model more generalizable and prevent overfitting.

data_transforms = {

'train': transforms.Compose([

transforms.RandomHorizontalFlip(),

transforms.ToTensor(),

transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

]),

'val': transforms.Compose([

transforms.ToTensor(),

transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

]),

} data_dir = 'data/test_me'

image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),

data_transforms[x])

for x in ['train', 'val']}

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x],

batch_size=8,

shuffle=True)

for x in ['train', 'val']}

dataset_sizes = {x: len(image_datasets[x]) for x in ['train','val']}

class_names = image_datasets['train'].classes

class_names

Out[1]:

['Adrien_Brody','Alejandro_Toledo','Angelina_Jolie','Arnold_Schwarzenegger','Carlos_Moya','Charles_Moose','James_Blake','Jennifer_Lopez','Michael_Chaykowsky','Roh_Moo-hyun','Venus_Williams']

def imshow(inp, title=None):

"""Imshow for Tensor."""

inp = inp.numpy().transpose((1, 2, 0))

mean = np.array([0.485, 0.456, 0.406])

std = np.array([0.229, 0.224, 0.225])

inp = std * inp + mean

inp = np.clip(inp, 0, 1)

plt.imshow(inp)

if title is not None:

plt.title(title)

plt.pause(0.001) # pause a bit so that plots are updated # Get a batch of training data

inputs, classes = next(iter(dataloaders['train'])) # Make a grid from batch

out = utils.make_grid(inputs) imshow(out, title=[class_names[x] for x in classes])

Get pretrained ResNet on VGGFace2 dataset

from models.inception_resnet_v1 import InceptionResnetV1 print('Running on device: {}'.format(device)) model_ft = InceptionResnetV1(pretrained='vggface2', classify=False, num_classes = len(class_names))

Freeze early layers

Recall earlier I mentioned that we will freeze up through the last conv block. To find where that is we can iterate through this list using -n, -n-1, ... until we find the block.

list(model_ft.children())[-6:]

Out[2]:

[Block8(

(branch0): BasicConv2d(

(conv): Conv2d(1792, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)

(bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)

(relu): ReLU()

)

(branch1): Sequential(

(0): BasicConv2d(

(conv): Conv2d(1792, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)

(bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)

(relu): ReLU()

)

(1): BasicConv2d(

(conv): Conv2d(192, 192, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), bias=False)

(bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)

(relu): ReLU()

)

(2): BasicConv2d(

(conv): Conv2d(192, 192, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)

(bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)

(relu): ReLU()

)

)

(conv2d): Conv2d(384, 1792, kernel_size=(1, 1), stride=(1, 1))

),

AdaptiveAvgPool2d(output_size=1),

Linear(in_features=1792, out_features=512, bias=False),

BatchNorm1d(512, eps=0.001, momentum=0.1, affine=True, track_running_stats=True),

Linear(in_features=512, out_features=8631, bias=True),

Softmax(dim=1)]

Remove the last layers after conv block and place in layer_list .

layer_list = list(model_ft.children())[-5:] # all final layers

layer_list

Out[3]:

[AdaptiveAvgPool2d(output_size=1),

Linear(in_features=1792, out_features=512, bias=False),

BatchNorm1d(512, eps=0.001, momentum=0.1, affine=True, track_running_stats=True),

Linear(in_features=512, out_features=8631, bias=True),

Softmax(dim=1)]

Put all beginning layers in an nn.Sequential . model_ft is now a torch model but without the final linear, pooling, batchnorm, and sigmoid layers.

model_ft = nn.Sequential(*list(model_ft.children())[:-5])

If training just final layers:

for param in model_ft.parameters():

param.requires_grad = False

Re-attach the last 5 layers which automatically sets requires_grad = True .

This linear layer Linear(in_features=1792, out_features=512, bias=False) actually requires writing two custom classes which is not entirely obvious by looking at it, but if you look at the data input/output you can see that there is a Flatten and normalize class within the layer. Check resnet implementation for reason of reshaping in last_linear layer.

class Flatten(nn.Module):

def __init__(self):

super(Flatten, self).__init__()



def forward(self, x):

x = x.view(x.size(0), -1)

return x class normalize(nn.Module):

def __init__(self):

super(normalize, self).__init__()



def forward(self, x):

x = F.normalize(x, p=2, dim=1)

return x

Then you can apply the final layers back to the new Sequential model.

model_ft.avgpool_1a = nn.AdaptiveAvgPool2d(output_size=1)

model_ft.last_linear = nn.Sequential(

Flatten(),

nn.Linear(in_features=1792, out_features=512, bias=False),

normalize()

)

model_ft.logits = nn.Linear(layer_list[3].in_features, len(class_names))

model_ft.softmax = nn.Softmax(dim=1) model_ft = model_ft.to(device) criterion = nn.CrossEntropyLoss() # Observe that all parameters are being optimized

optimizer_ft = optim.SGD(model_ft.parameters(), lr=1e-2, momentum=0.9) # Decay LR by a factor of *gamma* every *step_size* epochs

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

Train

def train_model(model, criterion, optimizer, scheduler,

num_epochs=25):

since = time.time()

FT_losses = []

best_model_wts = copy.deepcopy(model.state_dict())

best_acc = 0.0 for epoch in range(num_epochs):

print('Epoch {}/{}'.format(epoch, num_epochs - 1))

print('-' * 10) # Each epoch has a training and validation phase

for phase in ['train', 'val']:

if phase == 'train':

model.train() # Set model to training mode

else:

model.eval() # Set model to evaluate mode running_loss = 0.0

running_corrects = 0 # Iterate over data.

for inputs, labels in dataloaders[phase]:

inputs = inputs.to(device)

labels = labels.to(device) # zero the parameter gradients

optimizer.zero_grad() # forward

# track history if only in train

with torch.set_grad_enabled(phase == 'train'):

outputs = model(inputs)

_, preds = torch.max(outputs, 1)

loss = criterion(outputs, labels) # backward + optimize only if in training phase

if phase == 'train':

loss.backward()

optimizer.step()

scheduler.step()



FT_losses.append(loss.item())

# statistics

running_loss += loss.item() * inputs.size(0)

running_corrects += torch.sum(preds == labels.data) epoch_loss = running_loss / dataset_sizes[phase]

epoch_acc = running_corrects.double() /

dataset_sizes[phase] print('{} Loss: {:.4f} Acc: {:.4f}'.format(

phase, epoch_loss, epoch_acc)) # deep copy the model

if phase == 'val' and epoch_acc > best_acc:

best_acc = epoch_acc

best_model_wts = copy.deepcopy(model.state_dict()) time_elapsed = time.time() - since

print('Training complete in {:.0f}m {:.0f}s'.format(

time_elapsed // 60, time_elapsed % 60))

print('Best val Acc: {:4f}'.format(best_acc)) # load best model weights

model.load_state_dict(best_model_wts)

return model, FT_losses

Evaluate

model_ft, FT_losses = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=500) plt.figure(figsize=(10,5))

plt.title("FRT Loss During Training")

plt.plot(FT_losses, label="FT loss")

plt.xlabel("iterations")

plt.ylabel("Loss")

plt.legend()

plt.show()

More to come

Keep an eye out for more from this series where I will describe how to trick this classifier using adversarial attacks with GANs.