Unaligned Image-to-Sequence Transformation with Loop Consistency

Abstract: We tackle the problem of modeling sequential visual phenomena. Given examples of a phenomena that can be divided into discrete time steps, we aim to take an input from any such time and realize this input at all other time steps in the sequence. Furthermore, we aim to do this \textit{without} ground-truth aligned sequences --- avoiding the difficulties needed for gathering aligned data. This generalizes the unpaired image-to-image problem from generating pairs to generating sequences. We extend cycle consistency to \textit{loop consistency} and alleviate difficulties associated with learning in the resulting long chains of computation. We show competitive results compared to existing image-to-image techniques when modeling several different data sets including the Earth's seasons and aging of human faces.

Reviews and author responses:

This paper introduces the LoopGan, which aims to enforce consistency in a sequential set of images but without aligned pairs.  The prototypical example used in the paper is a network to transform seasonal images, where Summer -> Fall -> Winter -> Spring -> Summer -> etc.

I suggest rejecting the paper.

While the idea has some merit, it is an interesting premise to train a recurrent generator to create images in multiple domains, I feel as though the experiments are quite lacking.  The primary evidence the method works is Figure 3, for which we see only 4 examples of the method on two tasks.  Looking at the results they don't look all that great and the proposed method is hard to identify as the best. 

The idea doesn't strike me as particular innovative, feeling like a natural extension of the prior work listed.

Table 1 shows the results of a user study with 20 users.  With only 20 users rating 4 examples of each method we get very little power to resolve the best method, and the papers proposed method is selected in aggregate only 26% of the time out of 6 methods, this is hardly a clear winner.  

Figure 4 showing the imputed ages for the generated images doesn't seem that strong of evidence the proposed method is that much better either.  The histograms are difficult to compare by eye, perhaps the paper should report computed estimates of the kl divergence between the two?  

The ablation studies of section 6 are out of place.  The paper does not compare against the incremental removal of its' proposed loss, it simply reports results of alternative architectures.  

This paper proposes to extend CycleGAN by replacing a bidirectional cycle with a "loop" or a sequence of transformations across gradually changing image domains (sequential transformation). This is achieved with a simple modification to CycleGAN training where the training involves additional steps to create a loop. A number of experimental results are shown on specific sequential datasets, comparing to a number of other baselines. On the chosen datasets and tasks, the proposed architecture provides good results over the baselines. However, I would have liked to see ablation studies showing that they achieve the same results as CycleGAN when training on 2 domains. The paper's technical novelty is limited to this simple loop extension and generator sharing. As such, while the experimental results are reasonable, I find this lack of any technical novelty a negative factor. 

This paper proposes loop consistency on the base of cycle consistency and alleviate difficulties associated with learning in the resulting long chains of computation. Like CycleGAN, the method proposed in this paper does not require data to be specifically matched.

This paper still has the following problems：
1）For the time series images, the reason for processing with a single generator is not clearly stated. In CycleGAN, the two transformations use two different generators, respectively, and use the cycle consistency loss to force the results to converge. In the examples presented in this paper, such as Face Aging, it isn't a process that can be reversed. This paper does not explain why the use of a single generator has been effective.
2）To be more convincing, this article needs to be tested on more baselines. The indicators provided in this article are not objective enough.
3）This article also does not give the training computational complexity and testing time cost of the proposed method.


Meta-review: The main concern raised by the reviewers is limited experimental work, and there is no rebuttal.

Decision: reject