Title: CycleGAN: Unpaired Image-to-Image Translation (Part 1) - PyImageSearch

URL Source: https://pyimagesearch.com/2022/09/12/cyclegan-unpaired-image-to-image-translation-part-1/

Published Time: 2022-09-12T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [CycleGAN: Unpaired Image-to-Image Translation (Part 1)](#h2BPTitle)

*   [Introduction](#h3Introduction)
*   [Unpaired Image Translation](#h3Unpaired)
*   [CycleGAN Pipeline and Training](#h3Pipeline)

*   [Loss Formulation](#h4Formulation)
*   [Adversarial Loss](#h4Adversarial)
*   [Cycle Consistency](#h4Consistency)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, you will learn about image-to-image translation and how we can achieve it in case we have unpaired image data. Further, we will see how CycleGAN, one of the most famous efforts toward unpaired image translation, works and take an in-depth dive into the mechanism it uses to seamlessly translate images from one domain to another without needing paired image data.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/Cycle-GAN-Featured-1.png)

Specifically, in detail, we will discuss the following in this tutorial.

*   The paradigm of Unpaired Image Translation
*   The idea and intuition behind CycleGAN (one of the most successful methods for unpaired image translation)
*   The details and intuition behind the loss functions used by CycleGAN, especially cyclic consistency loss
*   Details about the pipeline and training process of CycleGAN for unpaired image translation

This lesson is the first in a 3-part series on **GANs 301**:

1.  [**_CycleGAN: Unpaired Image-to-Image Translation (Part 1)_**](https://pyimg.co/7vh0s) **(this tutorial)**
2.  _[CycleGAN: Unpaired Image-to-Image Translation (Part 2)](https://pyimg.co/jnael)_
3.  _CycleGAN: Unpaired Image-to-Image Translation (Part 3)_

In the first part of this series (this tutorial), we will learn about the idea behind CycleGAN and understand the mechanism it uses to perform image translation. Further, we will understand the concept of cycle consistency that allows CycleGAN to perform image translation without needing paired data.

Next, in Part 2 of this series, we will start implementing the CycleGAN model using TensorFlow and Keras and dive into the details of the model architecture and the [Apples-to-Oranges Dataset](https://universe.roboflow.com/pyimagesearch-v8dx3/apples2oranges/?ref=pyimagesearch), which we will use for our unpaired image translation task.

Finally, in the last part, we will look into the training details and generate images to see our real-time CycleGAN model in action.

**To learn how CycleGAN performs image-to-image translation from unpaired images,** **_just keep reading._**

* * *

[**CycleGAN: Unpaired Image-to-Image Translation (Part 1)**](#TOC)
------------------------------------------------------------------

* * *

### [**Introduction**](#TOC)

A pivotal point in the journey toward high-quality image generation was the inception of Generative Adversarial Networks in the year 2014. We have seen in a [previous tutorial](https://pyimagesearch.com/2021/10/25/training-a-dcgan-in-pytorch/) how GAN can be used to generate images from arbitrary distributions. Further, in [another tutorial](https://pyimagesearch.com/2022/01/10/torch-hub-series-4-pgan-model-on-gan/), we have seen how we can transform the training paradigm of GANs and generate high-resolution images suited for practical applications.

This is mainly possible due to the elegant adversarial training paradigm that forms the basis for GAN training. It is worth noting that the underlying 2-player-based adversarial paradigm can be used to match arbitrary distributions by formulating it as a competition between the generator and discriminator.

This raises the question of whether such a powerful GAN framework can be used for more than just image generation. In one of our [previous tutorials](https://pyimagesearch.com/2022/07/27/image-translation-with-pix2pix/), we took a detailed dive into how these networks can be repurposed for translating images from one domain to another: a task called Image Translation.

However, the pix-to-pix model relies on the presence of paired data in the two domains across which image translation has to be performed. **Figure 1** shows examples of paired and unpaired data.

[![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20424'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/Figure_1.png)

**Figure 1:** Example depicting the case where data in the two domains is paired (_left_) and data is unpaired (_right_) (source: [Zhu et al., 2020](https://arxiv.org/pdf/1703.10593.pdf)).

This creates a bottleneck and hinders utilization in practical scenarios where such tailored paired image datasets might not be present due to availability, practical feasibility constraints, or high annotation costs. Thus, it is natural to ask if we can still harness the capabilities of GANs for image translation without the need for paired images.

* * *

### [**Unpaired Image Translation**](#TOC)

The CycleGAN model marks one of the first and most elegant solutions to tackle the problem of unpaired image-to-image translations. Instead of using the conventional supervised paradigm, which requires one-to-one mapping in dataset images, it proposes to use cycle consistency loss to achieve image translation from unpaired datasets.

Formally, given a source domain _X_ and a target domain _Y_, CycleGAN aims to learn a mapping _G_: _X_ → _Y_ such that the _G_(_X_) is the translation of the image from domain _X_ to domain _Y_. Additionally, it also aims to learn a reverse mapping _F_: _Y_ → _X_ such that the _F_(_Y_) is the translation of the image from domain _Y_ to domain _X_. **Figure 2** shows the overview of the components of CycleGAN.

For example, this could mean translating zebra images into corresponding horse images (changing class semantics), transferring the stylistic difference from a summer view to a winter view, or predicting segmentation masks of an image scene, as shown in **Figure 3**.

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20395%20500'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/dS9-NpOYmHZ_h8GISe52Qrzo3x3aT9sCz4sesxK-FiVqj-gidqhmDorl-KB-uMfNwV_pKksqlRQIUXrB3sl8KSjahvuA14ABOkoauAhvcVDHZ7rLrkSK8jRcLMW2dNRTd5mjcchMCI-MlYfXwZhLcRU)

**Figure 3:** Example images depicting image translation from winter to summer (_top_), zebra to a horse (_middle_), apple to orange (_bottom_) (source: [Zhu et al., 2020](https://arxiv.org/pdf/1703.10593.pdf)).

However, it is worth noting that adversarial loss only ensures correspondence at the distribution level. Thus it can ensure that an image that belongs to the distribution of images from Domain A is translated into an image from the distribution of images from Domain B. It does not guarantee correspondence of images at the sample level.

Let us understand this with an example where we want to translate a zebra (Domain A) to a horse (Domain B). As shown in **Figure 4**, we see that the first is an image that belongs to the distribution of zebra images, and the second and third are images that belong to the distribution of horse images.

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20192'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/0WBkByPjo9gexxNnREeJeTdsQ_JMY5Qg9uOdJJF3hLa6m2kDFpXHh6ZjNgaGvBPropydaVIsDXS0lJUlTXMDzJK1jKFNjnaRczMZ-KxqjOfqiOAnujK4U8wJ_NmB7_MqFD2iAPdBYPpY6F05V-5z-Cg)

**Figure 4:** Example images depicting image translation from Domain A (zebra) to Domain B (horse) (source: [Zhu et al., 2020](https://arxiv.org/pdf/1703.10593.pdf)).

However, note that only the second image is the corresponding sample to the first image, and the third image (even though an image from Domain B) is not a corresponding sample for the first image. Applying adversarial loss can ensure that the translated image belongs to the distribution of images from Domain B but cannot guarantee that we get the exact corresponding sample in Domain B.

In the case of pix-to-pix, a translation of an image from Domain A to a corresponding image from Domain B was easily possible since we had paired samples from both domains. Thus, a simple supervised loss like the mean absolute error on the output of the generator and the ground-truth image in Domain B was enough to ensure that the translation of an image from Domain A results in the corresponding image in Domain B.

However, since we do not have access to corresponding paired samples in the task of unpaired image translation, we cannot use supervised loss, which makes it difficult to ensure correspondence at the sample level in the two domains.

In order to tackle this issue, the CycleGAN framework proposes to use cyclic consistency loss, which we discuss in detail later. The cyclic consistency loss ensures that the two functions learned are inverse functions of each other. Since the requirement for functions to have an inverse is that they have to be bijective (i.e., one-one and onto), it implicitly ensures a one-to-one correspondence at the sample level in the case of unpaired image translation.

* * *

### [**CycleGAN Pipeline and Training**](#TOC)

Now that we have understood the intuition behind CycleGAN, let us take a deeper dive into its mechanism for performing unpaired image translation.

Since we will be using the [Apples-to-Oranges Dataset](https://universe.roboflow.com/pyimagesearch-v8dx3/apples2oranges/?ref=pyimagesearch) for our implementation in later tutorials, let us consider a case where we have to translate an image _x_ in Domain A, that is, apples, to an image _y_ in Domain B, that is, Oranges.

**Figure 5** shows the overall pipeline of a CycleGAN.

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20298'%3E%3C/svg%3E)](https://lh3.googleusercontent.com/0_NSF60-Gj3ff7EQp_ThbI7Ug-4W0N3Z36M5Y6G0mgFANV2woDpsGrKkpdycYV4nP9oikkdSQa3whBt46-F3yAH2ai0TAn9lOcNdv29vCRFyBC-oBZdQoZ88sb5ntk8LLSzN74B3sYMkfcJ_5eEdCC8)

**Figure 5:** The end-to-end training pipeline of CycleGAN to achieve unpaired image translation (image by the author).

First, we take image _x_ from Domain A, which belongs to the distribution of images that depict apples (_top_). This image is passed through Generator _G_ (as shown), which tries to output an image that belongs to the distribution of images in Domain B.

Discriminator _D_ is an adversary against which differentiates between the samples generated by Generator _G_ (i.e., _y_′) and actual samples from Domain B (i.e., _y_). The generator is trained against this adversary using an adversarial training paradigm. This allows the generator to generate images at the output that belong to Domain B’s distribution (i.e., oranges).

Similarly, we take an image _y_ from Domain B, which belongs to the distribution of images that depict oranges (_bottom_). This image is passed through a Generator _F_ (as shown), which tries to output an image that belongs to the distribution of images in Domain A.

Discriminator _D_ is an adversary against which differentiates between the samples generated by Generator _F_ (i.e., _x_′) and actual samples from Domain A (i.e., _x_). The generator is trained against this adversary using an adversarial training paradigm. This allows the generator to generate images at the output that belong to the distribution of Domain A (i.e., apples).

Finally, we notice that in addition to the two adversarial losses, we also have the forward and backward cyclic consistency losses. This ensures that:

*   for each image _x_ from Domain A, the image translation cycle should be able to bring _x_ back to the original image
*   for each image _y_ from Domain B, the image translation cycle should be able to bring _y_ back to the original image

* * *

#### [**Loss Formulation**](#TOC)

Finally, to complete our understanding of the training process, let us look at the mathematical formulation of the loss functions employed by CysleGAN during training.

As discussed earlier, CycleGAN basically uses two loss functions (i.e., adversarial loss and cyclic consistency loss). Let us now take a look at the formulation of these losses.

* * *

#### [**Adversarial Loss**](#TOC)

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20103'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/WLI1b-odvUJRUXXw0dQZ77zJb2zv9TQUrfxBLfR7z9n5TNDjNB1azEkxRuHFpQMd75YePY7MCCxI2gC_kjRmYfEg_ud4mkhzC3bhXRtI-2IouKj5IRta2LuCd9j01AXRZ0DXClS1-YkHqvbenO1SsCY)

**Equation 1**

The mathematical formulation of adversarial loss is defined by **Equation 1**.

Here, the generator tries to generate images that look like they belong to the distribution of images from Domain B. On the other hand, the adversary or the discriminator tries to distinguish between the output of the generator (i.e., _G_(_x_)) and real samples from Domain B (i.e., _Y_).

[![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%2061'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/dDWajx_ZPDH3spyQFuXUMuoZaPcwRAemXU4nMoLDYF2Rd8rhTwXbdy8wkx3GoDNUS4BnvLHZTMZx_VWDJxX-xVJZV7mV6P0LTBjCjoWOSR66tr3hEBfTLX3A3UlWHqJJhXNjZHQBZkVsw0cASyGkRLk)

**Equation 2**

Thus, at each training iteration:

*   the discriminator is updated to maximize the loss (with the generator frozen)
*   the generator is updated to minimize the loss (with the discriminator frozen)

Following the adversarial training paradigm, eventually, the generator learns and can generate images from the distribution of Domain B (which are indistinguishable from original samples _y_).

[![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%2064'%3E%3C/svg%3E)](https://lh4.googleusercontent.com/lcoZsP9Kqe8AUPfggafXzSvKYp1ij-tGWr_hw3kzWbX7CHqln0O-N5t703ty8N_Yf6SxJ2zA1mqmWzhAneWStbSJN3kzCwof6S9ELREbtpJHN3NlpbavkjE1pTuts5BLc9XJeu_13UpglBf5HbWIwOo)

**Equation 3**

Similarly, as discussed earlier, another adversarial loss is imposed for the reverse mapping from domain _Y_ to domain _X_. Here, Generator _F_ tries to generate images that resemble the distribution of Domain A. Also, Discriminator _D__X_ tries to distinguish between the output of the generator (i.e., _F_(_x_)) and the real samples from Domain A (i.e., _X_).

* * *

#### [**Cycle Consistency**](#TOC)

As we have seen earlier, the adversarial loss only ensures consistency at the distribution level. However, to ensure that corresponding samples are generated when image translation is performed, CycleGAN utilizes cyclic consistency loss at the sample level.

[![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20144'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/uDd_Iad3eNise26sExJJM5qY8mUfKPgHdtlWvBC9ySiAbUGSbg-8XeWIoQY8VZeyDkgf_MfDA8N1YA82mFT7At7MtLxHPg2MTZSj-m0Dk6rpTzqm6_RqV9P7nh0o8ooseUESkSPiPtuKs4Z1SZ4RZgc)

**Equation 4**

As shown in **Equation 4**, the cycle consistency loss consists of two terms:

*   forward cyclic consistency
*   backward cycle consistency

Specifically, the first term ensures that when a sample in Domain A (say _x_) is passed through Generator _G_ (that transforms from Domain A to B) and then through Generator _F_ (when it transforms it back to Domain A), the output that is _F_(_G_(_x_)) is the same as the original input sample _x._ (_x_ → _G_(_x_) → _F_(_G_(_x_)) ![Image 10: \approx](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%209'%3E%3C/svg%3E) _x_).

Similarly, the second term ensures that when a sample in Domain B (say _y_) is passed through Generator _F_ (that transforms from Domain B to A) and then through Generator _G_ (when it transforms it back to Domain B), the output that is _G_(_F_(_x_)) is the same as the original input sample _y_. (_y_ → _F_(_y_) → _G_(_F_(_y_)) ![Image 11: \approx](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%209'%3E%3C/svg%3E) _y_).

In other words, both terms together aim to learn functions _F_() and _G_() such that they are inverses of each other. Thus, the cyclic consistency loss ensures correspondence at the sample level between sample _x_ from Domain A and sample _y_ from Domain B.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

In this tutorial, we learned about the task of image translation and how it can be achieved when paired input images in the two domains are unavailable.

Specifically, we discussed in detail the idea behind CycleGAN and understood the paradigm of cyclic consistency, which allows CycleGAN to perform image translation from unpaired data seamlessly.

In addition, we discussed the various losses that CycleGAN used during training and understood their mathematical formulation and the role they play in the task of unpaired image translation.

Finally, we discussed the end-to-end pipeline of CycleGAN to decode the process of image translation from unpaired data.

* * *

### [**Citation Information**](#TOC)

**Chandhok, S.** “CycleGAN: Unpaired Image-to-Image Translation (Part 1),” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, R. Raha, and A. Thanki, eds., 2022, [https://pyimg.co/7vh0s](https://pyimg.co/7vh0s)

@incollection{Chandhok\_2022\_CycleGAN,
  author = {Shivam Chandhok},
  title = {{CycleGAN}: Unpaired Image-to-Image Translation (Part 1)},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/7vh0s},
}

* * *

![Image 12: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.