Unaligned Image-to-Sequence Transformation with Loop Consistency
Authors: Siyang Wang
Justin Lazarow
Kwonjoon Lee
Zhuowen Tu
Venue:  (2020)

Abstract: We tackle the problem of modeling sequential visual phenomena. Given examples of a phenomena that can be divided into discrete time steps, we aim to take an input from any such time and realize this input at all other time steps in the sequence. Furthermore, we aim to do this \textit{without} ground-truth aligned sequences --- avoiding the difficulties needed for gathering aligned data. This generalizes the unpaired image-to-image problem from generating pairs to generating sequences. We extend cycle consistency to \textit{loop consistency} and alleviate difficulties associated with learning in the resulting long chains of computation. We show competitive results compared to existing image-to-image techniques when modeling several different data sets including the Earth's seasons and aging of human faces.

Under review as a conference paper at ICLR 2020
UNALIGNED INAGE-TO-SEQUENCE TRANSFORMATION WITH LOOP CONSISTENCY
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle the problem of modeling sequential visual phenomena. Given exam- ples of phenomena that can be divided into discrete time steps, we aim to take input from any such time and realize this input at all other time steps in the sequence. Furthermore, we aim to do this without ground-truth aligned sequences -avoiding the difficulties needed for gathering aligned data. This generalizes the unpaired image-to-ing to-gaee problem from generating pairs to generating sequences. We extend cycle consistency loop consistency and alleviate difficulties associ- ated with learning in the resulting long chains of computation. We show com- petitive results compared to existing image-to tomie techniques when modeling several different data sets including the Earth's ' seasons and aging of human faces.
INTRODUCTION
Image-to- tori-aggemee translation has gained tremendous attention in recent years. A pioneering work by (Isola et al. 2017) shows that it is possible to realize real image from one domain as a highly real- istic and semantically meaningful image in another when paired data between the domains are avail- able. Furthermore, CycleGAN (Zhu et al.) (017) extended the image-to-image translation frame- work in an unpaired manner by relying on the ability to build a strong prior in each domain based off generative adversarial networks (GANs, (Goodfellow al.] (0144) and enforcing consistency on the cyclic transformation from and to a domain. Methods (Kim et al. 2017 Liu et al.) 2017) similar CycleGAN have also been developed roughly around the same time. Since its birth, Cy- cleGAN (Zhu et al. (017) has become a widely adopted technique with applications even beyond computer vision (Fu et al. 2018). However, CycleGAN family models are still somewhat limited since they only handle the translation problem (directly) between two domains. Modeling more than two domains would require separate instantiations of CycleGAN between any two pairs of domains resulting in a quadratic model complexity. A major recent work, StarGAN (Choi et al. 2018), graph, allowing transformation
addresses this by facilitating a fully connected comainntranslatte between arbitrary domains with a single model. This flexibility, however, appears restricted to domains corresponding to specific attribute changes such as emotions and appearance.
Within nature, multitude of settings exist where neither of pairs fully-connected graph are the most natural representations of how one might proceed from one domain to another. In particular, many natural processes are sequentialand therefore the translation process should reflect this. A common phenomena modeled as an image-to-image task is the visual change of natural scenes between two seasons (Zhu al. 2017), e.g., Winter and Summer. This neglects the fact that nature first proceeds to Spring after Winter and Fall after Summer and therefore the pairing induces a very discontinuous reflection of the underlying process. Instead, we hope that by modeling higher resolution discretization of this process, the model can more realistically approach the true model while reducing the necessary complexity of the model.
It is difficult to obtain paired data for many image-to-image problems. Aligned sequential are even more difficult to come by. Thus, it is more plausible to gather a large number examples from each step (domain) in a sequence without correspondences between the content of the examples. Therefore, we consider a setting similar to unpaired image-to- taggget---mae transformation where we only have access to unaligned examples from each time step of the sequence being modeled. Given an example from an arbitrary point in the sequence, we then generate an aligned sequence over all other time steps - expecting faithful realization of the image at each step. The key condition that

---

Under review as a conference paper at ICLR 2020
required is that after generating entire loop (returning from the last domain to the input domain), one should expect to return to the original input. This is quite a weak condition and promotes model flexibility. We denote this extension to the cycle consistency of (Zhu et al. 2017) as loop consistency and therefore name our approach as Kpop-Consistent Generative Adversarial Networks ( (LoopGAN), This is departure from many image-to-image approaches that have very short (usually length 2) paths of computation defining what it means to have gone "there and back", e.g. the ability to enforce reconstruction or consistency, Since we do not have aligned sequences, the lengths of these paths for LoopGAN are as large the number of domains being modeled and require different approaches make learning feasible. These are not entirely different from the problems that often arise in recurrent neural networks and we can draw similarities to our model as memory-less recurrent structure with applied images.
We apply our method to the sequential phenomena of human aging (Zhang & Qi 2017) and the seasons of the Alps (Anoosheh et al. 2018) with extensive comparisons with baseline methods for image-to-image translation. We also present additional results gradually changing azimuth angle of chairs and gradual change of face attributes to showcased the flexibility of our model. We show favorable results against baseline methods for image-to-image translation in spite of allowing for them to have substantially larger model complexity.
Gen
Gen B
Gen
Gen
G
DB
Dc
Real
Real
Real
Real
A
B
C
D
Loop-consistency
GAN Loss
Figure 1: LoopGAN framework: for simplicity, only single loop starting one real domain in four-domain
problem is illustrated here. All four steps share single generator G. parameterized by the step variable. When training G, our ohjective function actually consists of four loops including - + + + A, B + C + D + A + B, C' + D > A B + C. and D + how CycleGAN is trained where two cycles are included.
C + D. This is consistent with
2 RELATED WORK
Generative Adversarial Networks Generative adversarial networks (GANs, (Goodfellow et al., 2014)) implicitly model a distribution through two components, a generator G that transforms sample from simple prior noise distribution into sample from the learned distribution over ob- servable data. An additional component known as the discrimintor D, usually a classifier, attempts to distinguish the generations of G with samples from the data distribution. This forms minimax game from which both G and D adapt to one another until some equilibrium is reached.
Unpaired Image-to-Image Transformation As extension to the image-to image-to-image translation framework (pix2pix, (Isola et al. 2017)), (Zhu et al. 2017) proposed CycleGAN which has a sim- ilar architecture in (Isola al.) (017) but able learn transformation between two domains without paired training data. To achieve this, CycleGAN simultaneously train two generators, one for each direction between the two domains. Besides the GAN loss enforeed upon by domain-wise discriminators, the authors proposed to add cycle-consistency loss which forces the two gener- ators to be reversible. Similar to pix2pix, this model aims at learning a transformation between two domains and cannot be directly applied in multi-domain setting that involves more than two domains. Concurrent to CycleGAN, (Liu et al. 2017) proposed a method named UNIT that implic- itly achieves alignment between two domains using like structure where both domains share a common latent space. Furthermore, StarGAN (CChoi et al. 2018)) proposed an image-to-in to-image translation model for multiple domains. A single network takes inputs defining the source image and desired domain transformation, however, it has been mainly shown to be successful for the domains consisting of facial attributes and expressions.
2

---

Under review as a conference paper at ICLR 2020
Multi-Modal Transformation The problem of learning non-deterministic multi-modal transfor- mation between two image domains has made progress in recent years ((Huang et al.) 2018 Liu et al. 2018). The common approach that achieves good performance is to embed the images for both domains into a shared latent space. At test time, an input image in the source domain is first embedded into the shared latent space and decoded into the target domain conditioned on a random noise vector. These models avoid one-to-one deterministic mapping problem and are able to learn different transformations given the same input image. However, these models are developed exclu- sively for two-domain transformation and cannot be directly applied to problems with more than two domains.
Style Transfer A specific task in image-to-image transformation called style transfer is broadly defined as the task of transforming a photo into an artistic style while preserving its content (Gatys jet al. 2015 Johnson et al., 2016). Common approaches use a pre-trained CNN as feature extrac- tor and optimize the output image match low-level features with that of style image and match high-level features with that of content image (Gatys et al.] 2015; Johnson et al.] (2016) A net- work architecture innovation made popular by this field known AdaIn (Huang & Belongie 2017 Dumoulin et al. 2017) combines instance normalization with learned affine parameters. Tt needs just small set of parameters compared the main network weights achieve different style trans- fers within the same network. It also shows great potential in improving image quality for image generation (Karras al. 2019) and mmage-to-image transformation (Huang al., 2018).
Face Aging Generating a series of faces in different ages given a single face image has been widely studied in computer vision. Sfate-of-the-art methods (Zhang & Qi 2017; Palsson et al. 2018) use a combination of pre-trained age estimator and GAN to learn to transform the given image different ages that are both age-accurate and preserve original facial structure. They rely heavily on a domain-specific age estimator and thus have limited application to the more general sequential image generation tasks that we try tackle here.
Video Prediction Video prediction attempts predict some number of future frames of video based on set of input frames (Shi et al. 2015 Vondrick et al., 2016). Full videos with annotated input frames and target frames are often required for training these models. A combination of RNN and CNN models has seen success in this task (Srivastava et al. 2015: Shi et al.] (015).. Predictive vision techniques (Vondrick et al.] 2016; Vondrick & Torralba, 2017 Wang et al. 2019)) that use CNN or RNN to generate future videos also require aligned video clips in training. A recent work Gupta et al. 2018) added a GAN as an extra layer of supervision for learning human trajectories. At high level, video prediction can be seen supervised setting of our unsupervised task. Moreover, video prediction mostly aims at predicting movement of objections rather than transformation of a still object or scene which is the focus of our task.
3 METHOD
We formulate our method and objectives. Consider a setting of n domains, XI,..., Xa where i < j implies that X; occurs temporally before Xj. This defines sequence of domains. To make this independent of the starting domain, we additionally expect that can translate from X, to X1 something a priori when the sequence represents periodic phenomena. We define single generator G(r, i) where i E [I,.... n and x X, Then, a translation between two domains X, and Xj of an input I; E X; is given by repeated applications of G in the form of Gll,- i) (allowing for incrementing the second argument modulo n 1 after each application of G). By applying G to an input times, we have formed direct loop of translations where the source and target domains are equal. While we use a single generator, we make use of n discriminators {D.)" where D; is tasked with discriminating between translation from any source domain X,- Since we are given N; only samples from each domain X,, we refer to each domain X, {x; 1-=1 consisting of N; examples from the domain X; with data distribution Pdata (x:). (
3

---

Under review as a conference paper at ICLR 2020
3.1 ADVERSARIAL LOss
Suppose I; ~ Pdata (Fi) Then we expect that for all other domains j. Glli-il i) should be indistinguishable under Dj from (true) examples drawn from (1)))()))) Additionally, each Dj should aim to minimize the ability for G to generate examples that it cannot identify as fake. This forms the adversarial objective for a specific domain as:
LGAN (G,
[log Di(x;)] + Σ
E iti pd
[log(1 (x(G"(z,)))
where G denotes iteratively applying G until zj is transformed into domain X,, i.e. || - ill times. Taking this over all possible source domains, we get an overall adversarial objective as:
LGAN Di....,
D.) = E
E Prataaa
log Da(a:)) +
Σ fot a)
E
[log(1 (xy)))
is)(()
where q(i) is a prior on the set of domains, eg. uniform.
3.2 LOOP CONSISTENCY Loss
Within Zhuu et al. 2017), adversarial loss was supplemented with cycle consistency loss that ensured applying the generator from domain domain followed by applying a separate gener- ator from B to A acts like an identity function. However, LoopGAN only has a single generator and supports an arbitrary number of domains. Instead, we build loop of computations by applying the generator G' to source image times (equal to the number of domains being modeled). This stitutes loop consistency and allows us to reduce the set of possible transformations learned to those that adhere to the consistency condition. Loop consistency takes the form of an L₃ reconstruction objective for a domain X; as:
LLoo (G, (G, Xi)=
||=;- - G G2a,, 1)
2imp(20)
3.3 FULL OBJECTIVE
The combined loss of LoopGAN both adversarial and loop-consistency losses can be written as:
[(G, DI,..., Dny XI,...,,AA, Xn) LGAN(C, Dn) XBeng(t) [Croop (G, X;)Il
Eingli) (eeeppaaaate) (2t) log (a()))
+
Σ E Epprpanc(()) [lg Dz (xz))))
1#1
+ (Beespeacc(a)) 1oa - Gth Gx:))ll
(1)
where À weighs the trade-off between adversarial and loop consistency losses.
An example instantiation of our framework for one loop in four-domain problem is shown in Figure
IMPLEMENTATION
4.1 NETWORK ARCHITECTURE
We adopt the network architecture for style transfer proposed in (Johnson al. 2016) as gen- erator. This architecture has three main components: a down-sampling module Enc( x), a sequence of residual blocks [h,, i), and an up-sampling module Dec(h). The generator G therefore is the composition G(æ, i) = Dec(T(()) ((nn(((())) where the dependence of T on only relates to the step- specific AdaIN parameters (Huang & Belongie 2017) while all other parameters independent of
4

---

Under review as a conference paper at ICLR 2020
i. Following the notations from (Johnson al. 2016 Zhu et al. 2017), let c7-k denote 7 x 7 Conv- ReLU layer with k filters and stride dk denote a 3 x Conv-ReLU layer with k filters and stride layers with filters each, uk de- Rel layer with k filters and stride 2. The layer compositions of modules down-sampling: c7-32, d64, d128; residual blocks: R128 x 6; up- sampling: u128, u64, c7-3. We use the PatchGAN discriminator architecture as (Zhu et al. (0117): c4-64, c4-128, c4-256, c4-1, where c4-k denotes 4 www.--sttaneNommmLaaakkRRRRl with k filters and stride 2.
2, Rk denote residual block with two 3 x 3 -onv-Adaln-Relle
notes a 3 x 3
layer
4.2 RECURRENT TRANSFORMATION
Suppose we wish to translate some E X; to another domain Xj. A naive approach would formulate this repeated application of j i| times. However, referencing our definition of G, we can unroll this to find that we must apply Enc and Dec j i times throughout the computation. However, Enc and Dec are only responsible for bringing an observation into and out of the space of T. This is not only a waste of computation when we only require output at Xj. but it has serious implications for the ability of gradients to propagate through the computation. Therefore, we implement Gl(z,, i) single application of Ene(z;). j applications of T(h,, and single application of Dec(h). T is applied recurrently and the entire generator of the form:
G( i) = Dec(TTi-i (Enc(z;))
We show in our ablation studies that this re-formulation is critical to the learning process and the resulting quality of the transformations learned. Additionally, T(h, i) given a of separate, learnable normalization (AdaIN (Huang & Belongie 2017)) parameters that it selects based off of i with all other parameters of T being stationary across time steps. The overall architecture is shown in Figure
Input A
Output
Output
Adain Params
Adaln Params
A
A
C
Up
sampling
sampling
Residual Blocks
Residual Blocks
Feature maps A
Feature maps B
Feature maps C
Figure 2: LoopGAN network. All modules share parameters.
4.3 TRAINING
For all datasets, the loop-consistency loss coefficient À set to 10. We use Adam optimizer (Kingma & Ba 2014)) with the initial learning rate of 0.0002, 33, 0.5, and 3r 0.999. We train the face aging dataset and Alps seasons dataset for 50 epochs and 70 epochs respectively with initial learning rate and linearly decay learning rate for 10 epochs for both datasets.
EXPERIMENTS
We apply LoopGAN to two very different sequential image generation tasks: face aging and chaging seasons of scenery pictures. Baselines are built with two bi-domain models, CycleGAN (Zhu al. 2017 and UNIT (Liu et al. 2017) and also a general-purpose multi- doltin model StarGAN (Choi al. 2018). We interested in the sequential transformation capabilities of separately trained bi-domains compared to LoopGAN. Therefore, for each of the two bi-domains models, we train separate model between every pair of sequential domains, i.e. X; and X and additionally
5

---

Under review as a conference paper at ICLR 2020
MIEM
CHANSEI 3O0 J00
ADB30 BOG J0
CH4--0 DISB-100
ASSecrAl DiSeeenn
CiFaB
DWWatee
ASriaa Surme
Ciean
DiKisie!
CydldA
CydGGAN
UNIT
UNIT
SarGAN
=
LegGAN
CydeGaN
Chared- Oyd45AN
UNIT
Chised UNIT
StuCAN
Figure 3: Face Aging and Alps seasons change with LoopGAN compared baselines. Input real images highlighted with rectangles (viewed in color).
train model between every pair (not necessarily sequential) domains X, and X3 (i * j). The first approach allows us to build a baseline for sequential generation by chaining the (separately learned) models in the necessary order. For instance, if we have four domains: A, B, C, D, then we can train four separate CycleGAN (or UNIT) models: GAB. GBC. GCD, GDA and correctly compose them replicate the desired sequential transformation. Additionally, we can train direct versions e.g. GAc of CycleGAN (or UNIT) for more complete comparison against LoopGAN, We refer to composed versions of separately trained models as Shained-Cycllede depending on the base translation method used. Since StarGAN ((Choi et al., (008)) inherently allows transformation between any two domains, we can apply this in a chained or direct manner without any additional models needing to be trained.
and Chained- UNIT
FACE AGING
We adopt the UTKFace dataset (Zhang & Qi 2017) for modeling the face aging task. It consists of over 20,000 face-only images of different ages. We divide the dataset into four groups in order of increasing age according to the ground truth age given in the dataset as A consisting of ages from 10-20, containing ages 25-35, C containing ages 40-50, and containing ages 50- 100. The number of images for each group are 1531, 5000, 2245, 4957, respectively, where a 95/5 train/test split is made. The results of LoopGAN generation are shown in on the left side in Figure
LoopGAN shows advantage over baseline models in two aspects. The overall facial structure is preserved which we believe is due to the enforced loop consistency loss. Moreover, LoopGAN able to make more apparent age changes compared to the rest of bascline models.
In order to quantitatively compare the amount of age change between models, we obtain an age distribution of generated images by running pre-trained age estimator DEX (Rothe et al. 2015). The estimated age distributions of generated images (from input test images) are compared against those of the train images in Figure 4 The age distribution of LoopGAN generated images is closer to that of the train images across all four age groups when compared to the baseline models suggesting that it more faithfully learns the sequential age distribution changes of the training data.

---

Under review as a conference paper at ICLR 2020
A/10-20)
B(25-35)
C(40-50)
D(50-100)
0.08
Charnd OydeCAh
UMT
ChardduNd
0.02
0.00
40
20
40
20
60
20
40
50
80
Figure 4: Comparing estimated age distribution between model generated images and train images.
5.2 CHANGING SEASONS
We use the collected scenery photos of Alps mountains of four seasons from (Anoosheh et al., 2018). They are ordered into sequence starting from Spring (A), to Summer (B), Fall (C), and Winter (D). They each have approximately 1700 images and are divided into 95/5% training and test set.
Overall, LoopGAN is able to make drastic season change while maintaining the overall structure of the input scenery images. To further quantify the generation results, we conducted a user study with Amazon Mechanical Turk (AMT) Table which shows that LoopGAN generations preferred by human users.
We show the results in Figure 3
Table 1: AMT user study results. 20 given each set in Figure 3 above and asked choose the best generation sequence. LoopGAN generations are preferred by human users over baseline models.
Model
Face Aging Season Changel
Overall
CycleGAN
7.50%
11.25%
9.375%
Chained-Cycledee
17.50%
13.75%
15.625%
UNIT
12.50%
16.25%
14.375%
Chained- UNIT
27.50%
20.00%
23.750%
StarGAN
11.25%
10.00%
10.625%
LoopGAN (ours)
23.75%
28.75%
%
5.3 ADDITIONAL DATASETS
To showcase the universality of our model, apply LoopGAN two additional datasets in four different sequential transformation tasks: chairs with different azimuth angles, and gradual change of face attributes in degree of smiling, gender feature, and hair color. The chairs dataset (Aubry et al. 2014) comes with ground truth azimuth angle and is divided into four sets each containing chairs facing distinct direction. To obtain linear manifolds for the face attributes, we train binary classifier with 0/1 labels available for each attribute (Liu et al.) and use the predicted probability to determine the position an image on an attribute manifold. The results are shown in Figure 5
Figure 5: Additional datasets. Top row: (left) changing azimuth of chair. (right) increasing degree of smile.
Bottom row: (left) gradually changing gender features, (right) gradually changing hair color.

---

Under review as a conference paper at ICLR 2020
6 ABLATION STUDIES
We experiment with several network architecture variations and investigate their effect on generation quality. First, attention mechanisms have proven to be useful in GAN image generation (Zhagg cct al. 2018). We added attention mechanim in both space and time dimension (Wang et al. 2018). however we found that the network struggles to generate high quality image after adding this type of attention mechanism. We also noticed that (Huang et al.] (018) mentioned that for sampling, is better is better to use layer-normalization for faster training and higher quality. We applied these changes and found that they indeed help the network produce better results. The results under these variations are shown in Figure 6 (first three rows).
no normalization preserve information from input image, and for up-sampling
wio attention
down- sampling
In-norm up sompling
attention
Model
Parameter Count
No norm tn domp-ianpling
Lay er up sampling
CycleGAN
94.056 M
Chained- Chained-CycleGAN
62.70 M
wlo attentionn
UNIT
133.680 M
No norm in down-sampling
Chained-UNIT
89.120 M
Layen up sampling (purs)
StarGAN
8.427 M
LoopGAN(ourr))
11.008 M
Same neworrk ours Up- and down- www--saplling between consecutive steps
(a)
(b)
Figure 6: (a). Ablation study for the architecture changes. (b) Model size comparison. * Note that parameter count for vanilla and chained versions of bi-domain models (CycleGAN, Chained-CycleGANN UNIT, and Chained-UNTTT are totals of separate pair-wise generators that together facilitate sequence generation.
Moreover, show the importance of the recurrent form of T(h) discussed in Section 4.2 We
compare the choice to invoke Enc and Dec at each time step versus applying them once with some number of recurrent applications of T in Figure ( (last row) and show the poor quality observed when performing the loop naively.
Lastly, we calculate the parameter count of generator networks compared in the face aging and sea- son change experiments above and show that our final generator network architecture is parameter- efficient compared baseline models in Figure b.b.
For completeness, we also include a selection of failure cases in table in Figure 7
Ar10-20)
B(25-35)
C(40-50)
((501100)
A(10-20)
B(25-35)
C(40-50)
D(50-100)
(a) Failure case
(b) Failure case 2.
Figure 7: Two failure cases. Input images and failure generations are highlighted respectively in red and purple (viewed in color). In both cases, the highlighted generated images (the first column in (a) and the last column in (b)) bear some semantic dissimilarity the input images. It seems that sometimes the network overfit more drastic transformations that only preserve overall facial structure and orientation but neglects all other features.
CONCLUSION
We proposed extension to the family of image-to-image translation methods when the set of do- mains corresponds to a sequence domains. We require that the translation task can be modeled as consistent loop. This allows to shared generator across all time steps leading to significant efficiency gains over nave chaining of bi-domiin image translation architectures. Despite this, our architecture shows favorable results when compared with the classic CycleGAN family algorithms.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Asha Anoosheh, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Combogan: Unrestrained scalability for image domain translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 783-790, 2018
Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3762-3769, 2014.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 8789-
8797, 2018.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. Proc. of ICIR, 2, 2017.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Explo- ration and evaluation. In Shirty-Second AAAI Conference on Artificial Intelligence, 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:/558.006577 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- 2014.
mation processing systems, pp.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2264-2264, 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance malization. In Proceedings of the IFEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Xun Huang, Ming-Yo Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 172-189, 2018.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imes -mage translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1125-11344 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-61977111 Springer, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, 2019.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th Interna- tional Conference on Machine Cetrning-Volumee 70, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: htt:::412.6800 69900, 2014.
Alexander H Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. In Advances in Neural Infor- mation Processing Systems, pp. 2590-2599, 2018
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to to to-i translation networks. In Advances in Neural Information Processing Systems, pp. 700-708, 2017.

---

Under review as a conference paper at ICLR 2020
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset.
Sveinn Palsson, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Generative adversarial style transfer networks for face aging. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2084-2092, 2018.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Dex: Deep expectation of apparent age from a sin- gle image. Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 10-15, 2015.
Xingjian Shi, Zhourong Chen, Hao Wang. Dit- Yan Yeung. Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, pp. 802-810, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms. arXiv preprint arXiv: 04040011, 2015.
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattrnn Recognition, pp. 1020-1028, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, pp. 613-621, 2016.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794- 7803, 2018.
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. ICLR, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: 2018.
Song Yang Zhang, Zhifei and Hairong Qi. Age progression/regresionn by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
10


Reviews:
[{"review_id": "SyxKW99S5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes loop consistency on the base of cycle consistency and alleviate difficulties associated with learning in the resulting long chains of computation. Like CycleGAN, the method proposed in this paper does not require data to be specifically matched.\n\nThis paper still has the following problems：\n1）For the time series images, the reason for processing with a single generator is not clearly stated. In CycleGAN, the two transformations use two different generators, respectively, and use the cycle consistency loss to force the results to converge. In the examples presented in this paper, such as Face Aging, it isn't a process that can be reversed. This paper does not explain why the use of a single generator has been effective.\n2）To be more convincing, this article needs to be tested on more baselines. The indicators provided in this article are not objective enough.\n3）This article also does not give the training computational complexity and testing time cost of the proposed method.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Skl5hr_6FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the LoopGan, which aims to enforce consistency in a sequential set of images but without aligned pairs.  The prototypical example used in the paper is a network to transform seasonal images, where Summer -> Fall -> Winter -> Spring -> Summer -> etc.\n\nI suggest rejecting the paper.\n\nWhile the idea has some merit, it is an interesting premise to train a recurrent generator to create images in multiple domains, I feel as though the experiments are quite lacking.  The primary evidence the method works is Figure 3, for which we see only 4 examples of the method on two tasks.  Looking at the results they don't look all that great and the proposed method is hard to identify as the best. \n\nThe idea doesn't strike me as particular innovative, feeling like a natural extension of the prior work listed.\n\nTable 1 shows the results of a user study with 20 users.  With only 20 users rating 4 examples of each method we get very little power to resolve the best method, and the papers proposed method is selected in aggregate only 26% of the time out of 6 methods, this is hardly a clear winner.  \n\nFigure 4 showing the imputed ages for the generated images doesn't seem that strong of evidence the proposed method is that much better either.  The histograms are difficult to compare by eye, perhaps the paper should report computed estimates of the kl divergence between the two?  \n\nThe ablation studies of section 6 are out of place.  The paper does not compare against the incremental removal of its' proposed loss, it simply reports results of alternative architectures.  ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlpRXDAtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to extend CycleGAN by replacing a bidirectional cycle with a \"loop\" or a sequence of transformations across gradually changing image domains (sequential transformation). This is achieved with a simple modification to CycleGAN training where the training involves additional steps to create a loop. A number of experimental results are shown on specific sequential datasets, comparing to a number of other baselines. On the chosen datasets and tasks, the proposed architecture provides good results over the baselines. However, I would have liked to see ablation studies showing that they achieve the same results as CycleGAN when training on 2 domains. The paper's technical novelty is limited to this simple loop extension and generator sharing. As such, while the experimental results are reasonable, I find this lack of any technical novelty a negative factor. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main concern raised by the reviewers is limited experimental work, and there is no rebuttal.
Decision: Reject