Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel idea of disentangling the content and style of structured data by treating style as permutation invariant information. It adopts VQ network for content encoding, and Cross-Attention for Style and Linking Attention at decoder. It is shown to be domain agonistic - worked well in image and audio domain.
main_review: Interesting idea of content and style disentanglement; validated on both image and audio domain.
# Strength:
1.	The paper proposes a new way to define style (for a given dataset) using permutation invariant (P.I) network. This is a good attempt to have a common definition of style.
2.	The network architecture is novel although using existing components such as VQ network and attention network. The idea of disentangling and link attention are what make the paper interesting.
3.	The model is domain-agonistic – demonstrated to work well in audio and image domain
4.	The paper demonstrates various downstream task in each domain. The part style transfer is impressive although there are some artifacts. Nice that the authors provided audio sample to play in supplementary file.

# Weakness/Suggestion/Queries 

The paper proposes to separate out style representations. But it lacks visualization of how actually each individual style visually look like. It may be that these styles are actually relates to attribute of the dataset which for some datasets (e.g. DeepFashion, CUB birds) already defined. It would be interesting to cluster these style features and see the discovered styles from a given dataset.

Overall, there seems to a lot of hyperparameters to tune and may be that one must have good understanding of dataset for proper hyperparameter settings. Details as follows:

1.	How is $r_{\theta}$ implemented? It seems this is the linking attention network. Could you please include some discussion with guiding equations at least in the supmat.
2.	Content Encoder: It is stated  the spatially adjacent tokens are forced to share the same VQ code. If so, how does this work when there are two different objects (parts) in the adjacent tokens (adjacent patches in images)?
3.	$\mathcal{Z}_o \in \mathbb{R}^{m \times d_s} $: What is a suitable value for m? Does the value of m relates to the number of styles the network can represent?
4. The total loss is sum of three loss terms with $\lambda$ weighting parameters. How sensitive is the model (training/convergence) to the hyper-parameters $\lambda$ s? If the choice of these parameters slightly off, do the network perform still reasonably?
Any training strategy used to deal with discrepancies brought by these parameters? How do you train the network in different stages e.g., adding more loss terms in subsequent stage?

5.	How do you come up with G=2 and V=100 VQ is sufficient to capture all content information for audio dataset? Any heuristics for this, or perform rigorous parameter search?
6.	Do you do manual observation to associate content codes with parts in image domain? It is stated that V is dependent on image datasets. Does this mean that you set V to the number of parts you want to segment out from images?
7. It is not clear how the authors generate: (a) Codebooks for content (b) Prototype vectors for style?


summary_of_the_review: The idea is interesting and seems novel, and the method is domain agonistic that may be applied to other domains. Experiement conducted are convincing, I am leaning to accept the paper, provide the minor concerns raised are resolved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for learning disentangled representations of content and style in an unsupervised way. The paper presents a slightly uncommon definition of content and style, which serves as a foundation for the proposed methodology and experiments. Experiments are conducted on speech and image datasets. 
main_review: *Strengths*
- S1. The proposed method is multimodal and can serve multiple modalities with a broad definition of style and content. The paper provides results on speech and vision data.
- S2. The proposed method is unsupervised, so it does not need annotation on style/content for the training data. This is very useful, as the boundaries on style and content are not straightforward and challenging to annotate.
- S3. The paper is clear and easy to read. 
- S4. Experiments are technically sounds.

*Weaknesses*
- W1. The definition of style as "the information that is not affected by the permutation of tokens" is a bit counterintuitive — no justification or examples of why the paper chooses this, and no other definitions are provided. 
- W2. The definition used in the paper does not correspond to the traditional definition of style and content in computer vision, which has been (and still is) a widely studied problem. This creates some confusion, especially in the image domain section.
- W3. After multiple examples referring to text as a potential target modality for this framework, no experiments are conducted on NLP datasets.
- W4. Additionally, for the image domain experiments, it would have been interesting to see results on art datasets, a field in which the problem of style and content disentanglement has been severally studied.
- W5. In the image domain experiments, there is no comparison to other methods other than for the proxy task of landmark regression. As visual results on style transfer are provided, it is expected to compare against other methods to see the potential of the proposed model against current technology.
summary_of_the_review: The idea of the paper is interesting and potentially useful for the community. However, the experimental results seem limited for now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an unsupervised framework to learn content and style representations of structured data, such as texts, images and speeches. The task is interesting to me and decomposition representation learning is important. In addition, a bipartite graph is proposed to model the link between content tokens and style tokens, making the model more interpretable. The proposed model performs relatively well on both speech and image domains.
main_review: Strengths:

S1: This paper addresses an important problem -- content and style decomposition representation learning and proposes an unsupervised framework to learn content-style representations.

S2: The proposed model performs relatively well on both speech and image domains.

Weaknesses:

W1: It is difficult to follow the paper some notations lack details. For example, what are the prototypes and link keys in Figure 3 and how to obtain them? For speech and image domains, how to obtain the input of the model?

W2: Lack of ablation study on hyper-parameters in Eq. (1). What are the roles of the three items in Eq(1)?

W3: It is unclear how the model avoids or mitigates information leaking. It seems that a perfect model should avoid the style leak while achieving perfect reconstruction (shown in Figure 2 (c)).

summary_of_the_review: This paper addresses an important problem and proposed a framework to learn content and style representations. The experiments show that the performance of the proposed model is relatively well. But some details are missing, so it is difficult to follow the paper and it is unclear the roles of the loss functions and how the model avoids information leaking.
review_title: null

Meta-review: This paper proposes a framework for learning disentangled representations of content and style in an unsupervised way, using a permutation invariant network. It adopts VQ network for content encoding, and Cross-Attention for Style and Linking Attention at decoder. It is shown to be domain agonistic, working well in image and audio domain. Experiments are conducted on speech and image datasets.

The paper is recommended as an accept (weak) to ICLR. The reviewers have given detailed feedback and suggestions -- please address them in the next revision of the paper.
Decision: Accept (Poster)