GANfolk: Using AI to Create Portraits for NFTs | Towards Data Science

I have previously written on Medium about using AI to create visual art, like abstract paintings and landscapes. During my research, I noticed that Generative Adversarial Networks (GANs) seem to have a hard time creating images of people. I decided to take this challenge [ahem] head-on, so I trained two GANs using paintings and photographs of people. I then put the images for sale as NFTs in a collection on OpenSea called GANfolk.
Note that I am releasing the dataset of training images, the source code, and the trained models under the Creative Commons Attribution Share-alike license. This means you can use my code to create your own digital paintings of people and sell them, as long as you give me attribution. See the Source Code section below for details.
Here is an overview of the GANfolk system. You can find details of the components and processes in the sections further below.
I started by writing a script to collect paintings of people that are in the public domain on WikiArt and open-source images of people from Google’s Open Images dataset. I pre-processed the images by aligning facial features and filling in any blank spots using the LaMa system for inpainting [1]. I then trained two GANs, StyleGAN 2 ADA [2] and VQGAN [3], with the GANfolk training set of 5,400 images. I collected 2,700 old paintings of people and 2,700 new photos of people.
For the first step in the creation process, the trained StyleGAN 2 system generated 1,000 images as a baseline set. I used GPT-3 from OpenAI [4] to generate text prompts for the pictures, like “drawing of a thoughtful Brazilian girl.” I then used the CLIP system [5], also from OpenAI, to find the best images that match the prompt. I chose the best picture and fed it into the trained VQGAN system for further modification to get the image to more closely match the text prompt.
I went back to GPT-3 and asked it to write a name and a brief backstory for each portrait. As a post-processing step, I added a vignette effect and resized the image up by four times (from 512x512 to 2048x2048). After a mild editing pass, I uploaded the pictures and backstories to OpenSea for sale as the GANfolk NFTs.
Before I get into the details of GANfolk, here is a brief section on what other people have done to generate portraits of people.
In conjunction with developing their StyleGAN series of generative networks, NVidia released a dataset of photographs of people called Flickr-Faces-HQ Dataset (FFHQ). According to NVidia...
... [the] dataset consists of 70,000 high-quality PNG images at 1024×1024 resolution and contains considerable variation in terms of age, ethnicity and image background.
Although the quality and variety of the FFHQ images are excellent, NVidia released the dataset under non-commercial terms. Also, I found that the faces seem too “tightly cropped” to make good portraits.
NVidia also released the MetFaces dataset of faces from paintings in the Metropolitan Museum of Art. They write that...
[the] dataset consists of 1,336 high-quality PNG images at 1024×1024 resolution. The images were downloaded via the Metropolitan Museum of Art Collection API, and automatically aligned and cropped using dlib. Various automatic filters were used to prune the set.
Again, NVidia released the dataset under non-commercial terms, and they used similarly tight cropping for the faces.
Here’s what newly generated images look like with StyleGAN 2 ADA trained on the FFHQ dataset and fine-tuned with the MetFaces dataset.
Although the results are impressive, not surprisingly, the results seem to be too tightly cropped. In addition to the datasets, NVidia released the official source code under non-commercial terms, so these faces cannot be sold as NFTs. Also, there seems to be a distinct lack of cultural diversity in the generated faces.
I will discuss the details of the components and processes used in the GANfolk system in the following sections.
I wrote two scripts to gather the source images for GANfolk. The first gathers public domain paintings on WikiArt from the 19th and early 20th centuries. The second collects portraits from Google’s Open Images dataset. The dataset consists of photos on Flickr released under the CC-BY-SA license, which allows for commercial use.
To find and orient the faces in the images, I used a face-finding algorithm from a package called DLIB. I modified the face-finding code to crop the faces more loosely. Here are some of the results from the paintings from WikiArt.
Although there is some more negative space around the portraits, it did pick up a lot of empty black areas around most of the images. This is because the original pictures didn’t have enough background area to account for the rotation and scaling for the face orientation.
To compensate for this, I used an inpainting AI system called Large Mask (LaMa) inpainting [7] by Roman Suvorov et al. The system will automatically “fill in” portions of an image specified by a second mask image. Here is the code I wrote to do the inpainting with LaMa. And below are the results of inpainting the sample portraits from WikiArt.
And here is a sample of photos from Open Images after the same pre-processing steps.
There seems to be a good variety of styles, ages, and ethnicities in the photos. And, for the most part, the effects of the inpainting are not visible.
I used two GANs for this project, an independent implementation of StyleGAN2 by Kim Seonghyeon (aka rosinality) and VQGAN by Patrick Esser et al. I trained both GANs for three weeks using the 5,400 training images in the GANfolk dataset.
In my prior research, I found that StyleGAN2 does an excellent job creating a global structure in the generated images that roughly resembles the types in the training data. However, the image details are often hazy or missing. But VQGAN is entirely complementary. It doesn’t know how to create a global structure, but it does a good job filling in realistic image details. Using both GANs is the best of both worlds.
To kick off the creation process, I used GPT-3 to suggest prompts to make the images. I literally asked the GPT-3 davinci-instruct model to do this:
"Create prompts to render fictional people at different ages and nationalities."
The source code is here. And typical results are below.
drawing of a thoughtful Brazilian girlacrylic painting of a sassy Mexican girlcharcoal sketch of an inquisitive Turkish boypencil drawing of a determined Indian womanink drawing of a playful Japanese girlacrylic painting of an optimistic Spanish boy
I used these prompts for two purposes: (1) as input to CLIP to direct the GANs to generate the corresponding images and (2) as the titles of the NFTs.
After generating the prompts, I ran StyleGAN2 to generate 1,000 random paintings of people. I then used the CLIP system from OpenAI to find the images that most closely match the prompt. As I described in an earlier article, the CLIP system has a text encoder and an image encoder for determining the similarities between phrases and images.
Here are the top 14 images generated by StyleGAN 2 that best match the prompt, “drawing of a thoughtful Brazilian girl.”
You can see a variety of styles in the results. Some look like drawings; some look like photos. Although some may appear thoughtful, it would be difficult to say that they look like girls from Brazil. So I picked the first one in the series to see what VQGAN could do with it.
To home in on an image that matches the prompt, I took a resultant image from StyleGAN2 and refined it iteratively using VQGAN. Thanks to work done by Katherine Crowson with further modifications by Justin John, I used CLIP again to analyze each iteration of the image and steer VQGAN to modify the image to better match the prompt. Here are the results.
The images above are iterations number 0, 25, and 50 using VQGAN and CLIP to match the prompt, “drawing of a thoughtful Brazilian girl.” You can see how the woman in the picture became younger and perhaps “more Brazilian.” Also, the portrait style changed to be more like a drawing than a painting.
I performed several post-processing steps on each of the images to give them a finished, unifying look:
Added a vignette to shade down the corners of the image;Used unsharp mask to make the details a bit more clear (optional);Performed super-resolution resize to increase the size of the image from 512x512 to 2048x2048;
Added a vignette to shade down the corners of the image;
Used unsharp mask to make the details a bit more clear (optional);
Performed super-resolution resize to increase the size of the image from 512x512 to 2048x2048;
It’s a bit subtle, but you can see how the vignette effect emphasized the subject in the center, and the unsharp mask effect sharpened the details. Here is the source code for the vignette and the unsharp mask effect.
The images coming out of StyleGAN2 and VQGAN have a resolution of 512x512 pixels. I used a super-resolution resize system from a German company called Idealo to increase the resolution. Before the resize-up, I added a little random noise to the image to create a painterly effect. The source code is here. Below is the sample drawing resized up to 2048x2048. You can click on the image to zoom in and see the details.
Before posting the images to OpenSea marketplace, I decided to “add interest” to the NFTs by using the GPT-3 davinci-instruct system to create names and backstories for the fictional people.
The prompt I used for creating a name is below.
Create a name for a thoughtful Brazilian girl.First name:
And the GPT-3 system responded with:
Sophia  Last name:  Santos
I noticed that the system would sometimes come up with the name of a famous person, so I wrote some code to check if the person has a Wikipedia page or not. I then created a loop to keep generating names until it created one not on Wikipedia. The code is here.
I then used a similar technique to create a backstory for each fictional person. The prompt I used is below.
Create a brief backstory for Sophia Santos, a thoughtful Brazilian girl.
And GPT-3 wrote:
Sofia Santos was born in the heart of Brazil to a Brazilian father and an American mother. Sofia grew up in a bilingual household and learned to appreciate both of her cultures. Sofia is a thoughtful and intelligent young woman who always looks for ways to help others.
The results were pretty good! My collaborator and reviewer, Jennifer, took the time to read all 100 backstories for GANfolk, and only a few of them needed some editing.
Minting GANfolk on the OpenSea NFT Marketplace
For my earlier NFT project, GANshare, I chose to use the Polygon blockchain because it is better for the environment than Etherium. And I decided to put the GANfolk collection on the Polygon chain for the same reason. Once I had created all 100 images with names and backstories, it was pretty straightforward to upload and mint them as NFTs. You can now see the entire GANfolk collection on OpenSea.
Here are the first 5 GANfolk, showing the output images from StyleGAN2 and VQGAN with post-processing.
GANfolk #1 — Painting of an Enigmatic French Woman, Mathilde DuboisMathilde Dubois was born in the small town of Saint-Jean-de-Luz in the south of France. She was the only child of a wealthy shipping magnate and an opera singer. From a young age, Mathilde showed a passion for the arts.Listing on OpenSea
GANfolk #2 — Painting of a Focused Portuguese Man, João SilvaJoão Silva was born in Lisbon, Portugal, in 1984. He was interested in learning about the world around him and different cultures from a young age. When he was 16, he and his family moved to the United States, and he finished high school in California.Listing on OpenSea
GANfolk #3 — Photo of a Mischievous English Teenager, Nigel LarkspurNigel Larkspur was born to wealthy English parents who spoiled him rotten. He quickly became a troublesome teenager, always getting into trouble at school and with the law. His parents tried everything to get him to behave, but nothing worked. Nigel loved to cause trouble and loved to see the look of disappointment on his parents’ faces.Listing on OpenSea
GANfolk #4 — Photo of a Stoic Bulgarian Teenager, Stefan StoichkovStefan Stoichkov was born in a small town in the heart of Bulgaria. His parents were hard-working farmers who taught him the importance of honesty, integrity, and self-reliance. Stefan was always a quiet, introspective child who preferred to spend his time reading, studying, and exploring the natural world around him.Listing on OpenSea
GANfolk #5 — Painting of a Concerned Korean Woman, Choon-Hee KimChoon-Hee Kim grew up in a suburb of Seoul. From a young age, she was taught the importance of family and traditional values. Her parents instilled in her a strong work ethic, and she quickly developed a reputation as a hard worker.Listing on OpenSea
You can see all 100 GANfolk here, https://opensea.io/collection/ganfolk
I learned a lot about GANs with their weaknesses and strengths while working on this project. As I mentioned above, StyleGAN2 generated decent images that have good overall form, although they often lack fine details. VQGAN is complementary in that it doesn’t know how to create images with global form, but if it started with a picture with decent form, it did a good job adding details, primarily when directed with the CLIP system.
I also noticed a bias towards European people while working on this project. StyleGAN2 seemed to struggle when creating images of people from diverse nationalities. This is probably due to a lack of diversity in the training images, especially the paintings from WikiArt. But CLIP seemed to know what people from around the world look like, and VQGAN was up to the task of modifying the images appropriately.
The 5,400 images I collected can be found on Kaggle. The source code for this project is available on GitHub. I am releasing the training images, the source code, and the trained models under the CC BY-SA license.
If you use these resources to create new images, please give attribution like this: This image was created with GANfolk by Robert A. Gonsalves.
I want to thank Jennifer Lim and Oliver Strimpel for their help with this article.
[1] LaMa by R.Suvorov et al., Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021)
[2] StyleGAN2 ADA by T. Karras et al., Training Generative Adversarial Networks with Limited Data (2020)
[3] VQGAN by P. Esser, R. Rombach, and B. Ommer, Taming Transformers for High-Resolution Image Synthesis (2020)
[4] GPT-3 by Tom B. Brown et al., Language Models are Few-Shot Learners (2020)
[5] CLIP by A. Radford et al., Learning Transferable Visual Models From Natural Language Supervision (2021)
To get unlimited access to all articles on Medium, become a member for $5/month. Non-members can only read three locked stories each month.
Some rights reserved