‘Simpsonize’ Yourself using CycleGAN and PyTorch | by Neel Iyer | Towards Data Science

Cyclegan is a framework that is capable of unpaired image to image translation. It’s been applied in some really interesting cases. Such as converting horses to zebras (and back again) and converting photos of the winter to photos of the summer.
I thought this could be potentially applied to The Simpsons. I was inspired by sites like turnedyellow and makemeyellow. The idea is that you upload a photo of your face. Cyclegan would translate that into a Simpsons Character.
It’s worth noting that the paper explicitly mentions that large geometric changes are usually unsuccessful. So this isn’t likely to turn out well.
But I’m going to attempt this anyway.
First we need to install CycleGAN.
!git clone https://github.com/junyanz/pytorch-CycleGAN-and-pix2piximport osos.chdir('pytorch-CycleGAN-and-pix2pix/')!pip install -r requirements.txt
Creating the dataset is harder than you would initially think.
To create this dataset I needed to find close up shots of Simpsons characters and close up shots of regular people.
Initially my thinking was to scrape images from google images. Unfortunately to do that it looks like you need a developer key from google console.
So instead I scraped images from Bing.
This worked to an extent. But it took so long to download all the images. And after I looked at the images I noticed that some of them didn’t include any faces at all.
Thankfully, I stumbled across a dataset on kaggle that had everything I needed. It contains Simpsons faces extracted from a few seasons. Each image is 200x200 pixels and contains one face.
This dataset will be stored in the trainA folder.
def create_training_dataset_simpson_faces(download_dir):  %cd $download_dir  # download dataset and unzip  !kaggle datasets download kostastokis/simpsons-faces --force  !unzip \*.zip  !rm *.zip  !cp -a $download_dir/cropped/. $download_dir  # remove unnecessary folders  !rm -Rf $download_dir/cropped  !rm -Rf $download_dir/simplified  # go back to orig directory  %cd /content/pytorch-CycleGAN-and-pix2pixcreate_training_dataset_simpson_faces(TRAIN_A)
To create the dataset of real faces I got a little bit experimental.
Will Kwan recently using stylegan2 to generate a dataset in one of his recent videos. It seemed to work fairly well for him. So I thought I could do the same thing.
Here’s some example faces taken from Nvidia’s stylegan2 github repository. As you can see the output from this GAN is fairly photorealistic.
This could turn out far better than scraping the internet for faces. I could create as many faces as I needed for my model. Plus I wouldn’t have to download cumbersome large files.
This dataset will be stored in the trainB folder
import matplotlib.pyplot as pltfrom tqdm.notebook import tqdm# see Github for full code: https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/cyclegan_simpsonify.ipynbdef create_training_dataset_real_faces_stylegan(download_dir):  # create in batches of 100  # reduces RAM requirements  counter = 0  pbar = tqdm(total = LIMIT)  while counter < LIMIT:    seeds = np.random.randint(10000000, size=100)    imgs = generate_images_from_seeds(seeds, 0.7)    for img in imgs:      img.save(download_dir/'real_face_{}.jpg'.format(counter), 'JPEG', quality=100)      counter+=1      pbar.update(1)    del imgscreate_training_dataset_real_faces_stylegan(TRAIN_B)
Next we’ll need to split the data into training and testing. testB will contain real faces that we want to convert into Simpsons characters. testA will contain simpsons characters that we want to convert into real people.
# move images to a new folder# `images` is the existing image directory: # `new_dir` is the path that the images will be moved to# `files_limit` is the limit of files that will be moveddef move_all_images_to_new_folder(images, new_dir, files_limit = None):  files = glob.glob(str(images/'*.*g'))  if(files_limit is not None):    files = files[:files_limit]  for file in files: shutil.move(file, new_dir/os.path.basename(file))move_all_images_to_new_folder(TRAIN_A, new_dir = TEST_A, files_limit = int(min(LIMIT*0.1, 25)))move_all_images_to_new_folder(TRAIN_B, new_dir = TEST_B, files_limit = int(min(LIMIT*0.1, 25)))
Let’s see the images we’re working with.
import PILimport randomdef plot_from_image_path(path, title):  all_imgs = glob.glob(str(path/'*.*g'))  print(f'{len(all_imgs)} imgs in {title} directory')  img_path = random.choice(all_imgs)  img = PIL.Image.open(img_path)  plt.imshow(img)  plt.title(title)  plt.show()plot_from_image_path(TRAIN_A, 'TRAIN_A')plot_from_image_path(TRAIN_B, 'TRAIN_B')plot_from_image_path(TEST_A, 'TEST_A')plot_from_image_path(TEST_B, 'TEST_B')
Everything looks pretty good!
Now we can create the model. I’ve made a few adjustments to the existing script.
Let’s create a few helper functions.
These functions help me copy the saved models to my google drive. It also helps test the model and store the output images to google drive.
import osfrom pathlib import Pathfrom distutils.dir_util import copy_treeimport matplotlib.pyplot as pltimport randomdef copy_to_drive(folder = 'cyclegan_simpsonify'):  drive_folder = Path('/content/drive/My Drive/')/folder  if(drive_folder.exists()):    shutil.rmtree(drive_folder)  shutil.copytree('/content/pytorch-CycleGAN-and-pix2pix/checkpoints/'+NAME+'/', str(drive_folder))def get_corresponding_photo(file_path):  return file_path.replace('fake', 'real')def plot_results(number):  for i in range(number):    img_path = random.choice(glob.glob('./results/'+NAME+'/test_latest/images/*fake.*g'))    print(img_path)    img = plt.imread(img_path)    plt.imshow(img)    plt.title('fake')    plt.show()    print(get_corresponding_photo(img_path))    img = plt.imread(get_corresponding_photo(img_path))    plt.imshow(img)    plt.title('real')    plt.show()def get_model(src, dst):  # copy across model  try:    os.remove(dst)  except:    pass  shutil.copyfile(src, dst)def copy_from_drive(folder = 'cyclegan_simpsonify'):  drive_folder = Path('/content/drive/My Drive/')/folder  if(not Path('/content/pytorch-CycleGAN-and-pix2pix/checkpoints/').exists()):     os.mkdir('/content/pytorch-CycleGAN-and-pix2pix/checkpoints/')  if(Path('/content/pytorch-CycleGAN-and-pix2pix/checkpoints/'+NAME+'/').exists()):     shutil.rmtree('/content/pytorch-CycleGAN-and-pix2pix/checkpoints/'+NAME+'/')  shutil.copytree(str(drive_folder), '/content/pytorch-CycleGAN-and-pix2pix/checkpoints/'+NAME+'/')def test_model (number_results = 5, direction = 'BtoA', src = None, dst = None):  # delete results folder and recrete  shutil.rmtree('./results')  os.mkdir('./results')  # get appropriate model  if (src is None): src = './checkpoints/'+NAME+'/latest_net_G_'+direction.split('to')[-1]+'.pth'  if (dst is None): dst = './checkpoints/'+NAME+'/latest_net_G.pth'  get_model(src, dst)  if (direction == 'BtoA'):    test = TEST_B  else:    test = TEST_A    cmd = 'python test.py --dataroot '+str(test)+' --name '+str(NAME)+' --model test --no_dropout'  os.system(cmd)  plot_results(number_results)
Let’s create the options for training.
import timefrom options.train_options import TrainOptionsfrom data import create_datasetfrom models import create_modelfrom util.visualizer import Visualizerimport shutilimport osfrom pathlib import Pathfrom tqdm.notebook import tqdmoptions_list = ['--name', NAME,\		'--dataroot', TRAIN_A.parent,\		'--batch_size', BATCH_SIZE,\		'--checkpoints_dir', './checkpoints',\		'--lr', 2e-4,\		'--n_epochs', EPOCHS,\		'--n_epochs_decay', EPOCHS//2,\		'--name', NAME]opt = TrainOptions().parse(options_list)
First we create the dataset using the options we specified earlier.
dataset = create_dataset(opt)dataset_size = len(dataset)print('The number of training images = %d' % dataset_size)
Then we create the model and run the setup call.
model = create_model(opt)model.setup(opt)visualizer = Visualizer(opt)total_iters = 0
Let’s see the generator from A to B. The naming convention is slightly different from the paper. In the paper the generator was called G.
In the code they refer to this mapping function as G_A. The meaning is still the same.
This generator function maps from A to B.
In our case it maps from Simpsons to Real Life.
model.netG_A
We can see here that the model uses Resnets.
We have Conv2d, Batchnorm, ReLU, InstanceNorm2d and ReflectionPad2d. InstanceNorm2d and ReflectionPad2d are new to me.
InstanceNorm2d: This is very similar to batch norm but it is applied to one image at a time.
ReflectionPad2d: This will pad the tensor using the reflection of the input boundary.
Now we can look at the discriminator as well.
model.netD_A
The discriminator uses LeakyReLU, Conv2d and InstanceNorm2d.
LeakyReLU is interesting. ReLU is an activation that adds non-linearity to the network. But what is LeakyReLU?
ReLU converts all negative values to 0. Since, the gradient of 0 is 0 neurons that reach large negative values effectively neuron cancel out to 0. They effectively 'die'. This means that your network eventually stops learning.
This effect is known as the dying ReLU problem.
LeakyReLU aims to fix this problem. The function is as follows:
​ This function essentially translates to: if a value is negative multiply it by negative_slope otherwise do nothing. negative_slope is usually 0.01, but you can vary it.
So LeakyReLU significantly reduces the magnitude of negative values rather than sending them to 0. But the jury is still out on whether this really works well.
Now we can train the model over a number of epochs. I’ve specified 10 epochs here.
Here’s the code for training:
model.optimize_parameters() is where all the magic happens. It runs the loss functions, gets the gradients and updates the weights. By doing so it optimises the generators and the discriminators.
Let’s train it!
I let the model train overnight on google colab and copied the .pth model to my google drive.
Let’s see the output.
test_model(10, 'BtoA')
It’s a good start. I particularly like this image:
It could use some improvement, to be honest.
Let’s try running the AtoB cycle. So we'll convert simpsons characters into human faces.
test_model(10, 'AtoB')
The authors of Cyclegan noted that tasks requiring large geometric changes aren’t likely to be successful. I’ve just confirmed this.
The network seems to struggle with the large geometric shifts required to convert a simpsons character to a real person (and vice-versa). I’m unsure if more training would rectify this issue. The tricky thing with GANs is figuring out when to stop training. Visual inspection seems to be the answer for Cyclegan. It might be worthwhile to train for another few days and see what happens.
The full jupyter notebook can be found on Github
Originally published at https://spiyer99.github.io on August 30, 2020.