Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The submission proposes a method for representation learning in a setting where data is annotated by set membership (i.e. grouped data). More specifically, the authors aim to extract representations of the factors of variations that are shared across groups.
To do so, the proposed method embeds different sets into a shared latent space using a learning objective called Affinity Cycle Consistency (ACC). ACC imposes a soft version of the cycle consistency on the nearest neighbourg relationship between the two learned sets of embeddings.
Using this method, the authors show in experiments on 3DShapes and MNIST that the inactive features (features that fixed in each set) are removed from the embeddings that, in turn, better encodes for the active features (features that vary inside each set).
They also show how their methods can be used to accurately estimate the pose of objects (cars and chairs) in unlabeled real pictures by aligning them with sets of synthetic images of cars grouped by pose.


################################################

Strong points:

-Representation learning for grouped data is a relevant topic, for which the submission proposes a simple and effective method. 

-The authors present a practical use case in section 4.3, showing how the method can help tackle the domain gap problem. Also, experiments presented in Figure 3 uncovers some very intriguing properties that might be worth exploring further.

-The paper is well organized, formalism is clear and related work is informative.


Weaknesses:

-ACC, as noted by the authors, is very similar to the previously introduced Temporal Cycle Consistency. The main contribution of this work is to provide empirical results when applied to a more general context.

-The paper lacks crucial discussions about why ACC allows learning a good alignment between sets in the general context. The choice of ACC seems arbitrary, as I find unclear how ACC relates to latent spaces alignment.

-While the experimental results are intriguing, they aren't very convincing in terms of the usefulness of the proposed method. Results in section 4.3 are obtained using a 2-dimensional latent space, which is useful for visualization but is a very odd choice for practical uses. Section 4.5 presents an interesting application but the fact that the authors had to use a complex pipeline for these relatively simple images raises concerns about how well the method can generalize to other problems or scale to more difficult images.


################################################

Rating motivation:

The paper advocates for the use of ACC for representation learning but doesn't provide an explanation about why the method should work. This makes it difficult to assess how much of the results can be attributed to the use of ACC and how much is due to other inductive biases. Also, while the car experiments are interesting, the results by themselves aren't convincing enough.

I might change my evaluation if this point is clarified (either via discussion or by additional control experiments).


################################################

Questions and minor remarks:

-Can the authors comment on the importance of the size of the embeddings? Does the capacity bottleneck explain that geometric features are less prominent in figure 3c? What happens when using larger embedding?

-Can the author confirm that phi:A->L and phi:B->M are implemented as two different convolutional networks? The paper is a little ambiguous.

-The method is simple but reproducing would probably need some additional implementation details (architectures, optimization method, hyper-parameters...)

-While the active/inactive features formalism is clear, it still might help to illustrate with examples how some classical tasks (for instance in domain transfer) fit in.

review_title: A feature learning method for grouped data. The method seems not supported enough by motivation nor results.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an approach to isolate factors of variation using weak supervision in the form of group labels. The proposed method Affinity Cycle Consistency (ACC)  claims to work with these group labels, which are weaker than the more common, one factor per group type labeling.  An important aspect of this approach is that it does not attempt to disentangle the factors of variation, but only capture (or isolate) them in the latent space. 

Following are the strengths of the paper: 
+ It uses a very simple strategy of combining soft nearest neighbors with cycle consistency in the latent space to achieve the ability of isolating factors of variation. The training of the network while imposing cycle consistency is done by simply minimizing the cross-entropy to predict the nearest neigbhor of an input point in the embedding space.  
+ Some of the empirical results are interesting, as the paper reports increased mutual information (MI) between the embeddings and the factors of variation of interest. 

While I appreciate the simplicity of the approach, there are some important concerns which the paper fails to address adequately. 
- The analysis of empirical results is missing, which raises many questions. 
 - Experiments of Fig. 3 indicate that with one inactive factor, two of the remining five active factors are isolated, and with two inactive factors, one of the remaining four active ones is isolated. This behavior is not clear as to why it happens? Why are the other factors not isolated? What if we have other factors (the not so easy ones like scale, shape or pose) inactive, will still the background objects' factors will be isolated? This behavior should ideally be explained through further analysis experiments. Similarly for other datasets. 
 - It is not clear how the temperature value (as used in Definition 2) is set. How sensitive is the performance to this parameter. 
 - How is the dimensionality of the embedding space determined? How does it effect the isolation / disentangling performance? (Jha et al. 2018) seem to show that the latent space dimensionality impact disentangling performance. Could this be a reason for the other factors not being isolated in Fig. 3? The embedding dimensionality is only 2. 

- Comparisons with group level supervision work 
 - (Bouchacourt et al., 2018) used group level supervision for disentanglement, and would have been more appropriate for comparisons than e.g., (Jha et al. 2018). It is not clear why no comparisons were made with this work. 

- There are other mistakes, possibly typographical in nature. 
 - From Definition 2, it appears that the soft nearest neigbhor of l_i has larger \alpha_j for points m_j that are farther away from l_i. 
 - Para before Definition 1, defines B, such that |B|=m, but it seems that it is meant to be n. 

My recommendation is to reject this paper, primarily because of the lack of analysis and ablative experiments, as well as comparisons to the related approach of (Bouchacourt et al. 2018). It is unclear why certain factors get isolated while others do not, based on the active and inactive sets of factors. Well-designed analysis experiments will perhaps bring about some additional insights. 

I believe the paper will make a far more compelling case if there are analysis experiments presenting the strengths of the approach that provides insights into why certain factors are easily isolated and others are not. 


review_title: Leveraging affinity cycle consistency to isolate factors of variation in learned representations

final_score: rating: 3: Clear rejection
review_content: This paper applies a weakly-supervised learning approach to identify factors of object postures in an image dataset. The core idea is to introduce two sets of images. The first set is the reference data set with grouped objects of different active/inactive posture constraints. This set is used to provide weak supervision information in posture identification. The second set is the probe set. It does not necessarily require posture grouping of objects. Affinity Cycle Consistency loss is set up to automatically map objects of similar active postures between the two image sets (objects of similar postures are supposed to be the nearest neighbors in the learned embedding space). The experimental study verifies the validity of the proposed factor isolation algorithm. 

Generally this paper is well written and clearly explains the motivation/problem definition. However, we have the following concerns on the innovative contribution of this work.

The innovation of this paper is very limited. The core technology applied in this work was originally proposed by Dwibedi et al in the paper "Temporal Cycle-Consistency Learning".  As cited in Section.1 of this paper, this work employs directly the Cycle-Consistency learning mechanism (while in a different application scenario). The only difference is: this work is built based on affinity relation between pairs of images with similar postures, while the original idea was applied for temporal sequence alignment in video processing. Not significant algorithmic innovation is introduced, compared to the previous work.  It is clearly below the threshold for a high-quality venue like ICLR. 
review_title: This paper applies a weakly-supervised Affinity Cycle Consistency loss to recognize factors of variation in image data sets and learn the image embedding representation of each identified factor. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper uses affinity cycle consistency to isolate factors of variation with only weak supervision on set membership. It has extensive experiments with both synthetic and real data.
 
The strength is that the problem setting is reasonable and important. The algorithm is sounding, and the evaluation is valid. The weakness is that the experiments and discussions do not cover enough cases for extracting and suppressing factors.
 
I recommend that this paper is marginally above the border.
 
The positive reasons are the following. Isolating factors is a difficult and important problem, so the progress is good. Using weak supervision is practically convenient. Also, the paper shows that it helps a synthetic-to-real transfer problem. The following are some concerns.
1. Extracting all active factors may need more samples in set A.
In the experiment of Figure 3c, second from the top, the learned representation does not isolate active geometric factors, indicating it does not include the information. This means set A does not contain enough variety of elements so that the object and floor hues are enough to distinguish the elements. To isolate all the active factors, set A should contain more elements than used in the experiment. It would be more helpful to find how many elements are necessary from theoretical and / or empirical perspectives.
2. Suppression of complicated factors.
The paper does not tell why the inactive factors can be suppressed for unseen values, especially when it is inactive in both set A and B. When inactive, a factor takes a fixed value. So when it is inactive both in A and B, the factor has at most two values in training. However, in the test, it is suppressed for all (or most) possible values. The generalization capability from two values to many values requires more explanations, especially when the factor is complicated (e.g. non-linear) to extract. From experiment perspective, Figure 3c covers the cases of inactive ‘easy’ ’hue factors. It would be helpful to see what happens when the three hue factors and a geometric factor (e.g. orientation) are all inactive.
 
Additional feedback:
The last section is named “Discussion”, but it looks like “Conclusion” from its contents and position.
The paper does not provide code to reproduce the results.
Some figures and letters on them are very small.
Grammars:
“a affinity cycle consistency loss”
“for a each generative factor”

review_title: Review

Meta-review: This paper proposes to employ affinity cycle consistency(ACC) for extracting active (or shared) factors of variation across groups. Experiments shows how ACC works in various scenarios.

Pros:
- The problem is important and relevant.
- The paper is well written.
- The proposed method is simple and effective.

Cons:
- The experimental section is weak:
 It lacks an ablation to validate the contribution of ACC and discussion on
  why the method works and the scalability of the proposed method to more complex cases.
- The novelty is limited because the proposed ACC is similar to previous work temporal cycle consistency(TCC).
- The paper missed some implementation details and could be difficult to reproduce without code
 provided.

Reviewers raised the concerns listed in Cons. The authors conducted additional experiments and added more discussions on the experimental results in the revised paper. The authors also explained that ACC is more general than TCC. However, the reviewers were not convinced by the rebuttal and kept their original ratings.

Due to the two main weaknesses -- limited novelty and weak experimental analysis, I recommend reject.
Decision: Reject