Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the disentangled representation learning. In particular, authors provide the evidence on the improved generalization when the disentangled representations  coupled with sparse base-predictors are learned. The paper then presents a theoretical result on identifiability condition wonder with maximally sparse base-predictors that lead to the desired disentangled representations. Relying on the presented theory, representation that maximizes sparsity is sought, in a hope of eventual disentanglement, in the context of the multi-task learning. Such sparsity maximization however not being trivial, authors propose a sparsity-promoting bi-level optimization paradigm for learning. Later, a connection between bi-lavel optimization and the meta-learning problems is established.  

strength_and_weaknesses: ++The paper is well written and motivated. The addressed problem is a problem of sufficient interest. 

++The theoretical consideration of the paper is praiseworthy. Specifically, identifiability result  of Theorem 1 is particularly interesting, intuitive, and convincing. 

++The paper also proposes bi-level optimization and meta-learning based approaches, which are empirically shown to be effective in Figure 1,  with fewer samples for the Disentangled-Lasso method.

++Supplementary material with additional theoretical and experimental analysis is helpful.

--The reported experiments are rather toy-like settings. It is not clear when the proposed method is expected to be useful. For example, results reported in Table.1 (supplementary material) shows that the proposed method does not compete well against MetaOptNet (although not being directly comparable).

--The meta-learning formulation of Section 3.2 is not easy to follow. The section lacks intuitive explanation and the jump form equation (7) to Proposition (3) is not well explained.

--The paper relies a lot on the supplementary material. This in itself is not a big problem, given the nature of the addressed problem. However, I believe that the paper can be made more intuitive with the help of graphic illustrations and by bringing the B.2 Discussion of Assumption in the main paper.

clarity,_quality,_novelty_and_reproducibility: Please, refer to the strengths above.
summary_of_the_review: Overall, the paper makes a good case for the addressed problems and provides convincing and theoretical insights. The paper can benefit from more intuitive explanations (considering a broader audience) and the experiments on more realistic datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study how disentangled representations can be beneficial for downstream tasks under certain assumptions. The authors come up with Theorem 1, which states that leveraging multi-task learning can naturally learn disentangled representation when regularizing the base-predictors to achieve maximal-level sparsity. The authors propose to solve the problem of spare multi-task learning via solving its relaxed problem. The authors show empirical evidence on few-shot learning to support their theorem and to validate the effectiveness of their method. 
strength_and_weaknesses: Strength:

-- The paper formalizes the sparse task assumption (I.e., most tasks do not require all features), and introduces a novel identifiability result. The authors validate that disentanglement can be achieved by combining with sparse base-predictors, and can improve generalization, via both empirical evidence and the theories (under certain assumptions).

-- The theoretical analysis shed light on better understanding disentangled representations.


Overall, The authors could consider conducting experiments on more tasks and more datasets. I did realize some potential weaknesses on motivation and experiments:

-- Do the authors want to show disentanglement helps with generalization, or sparsity regularization helps with generalization? The authors can consider comparing with the disentanglement methods that do not combine with sparse base-predictors. 

-- The authors propose Theorem 1 (and other theoretical results) under certain assumptions. I’m not sure how realistic those assumptions are. According to Figure one, the disentangled methods would work well under two assumptions — The ground-truth coefficient matrix (See equation 2) should be very sparse, and there should be very few training samples. Without the two conditions, combining sparsity and disentanglement do not show clear benefits.

-- It seems most theorems and propositions are based on MLE, while the meta learning part switches to SVM. 
I don’t know how to understand the significance of the benefits of the group-sparse SVM method (when compared with SVM). For example, lambda/lambda_max = 0.01 yields the best performance (according to Figure 3 and Table 1 in the Appendix). In most configurations, the group-sparse solutions do not seem to outperform the vanilla SVM. In the configuration of 0.01, the benefit in terms of Meta-test is also not so big.
clarity,_quality,_novelty_and_reproducibility: I don’t have specific concerns on the clarity. The authors wrote a well-organized draft, and reminded readers via using a lot of hyper-links. 
summary_of_the_review: Overall, I think this is an interesting paper which aims to thoroughly understand the relationship between disentanglement, sparsity and generalization. The authors tried to formalize a few intuitions, propose a few theoretical results and prove them under the dome of formalized assumptions. I think the paper shed some light on better understanding disentangled representations. It’s unclear to me how strong the theoretical results are, how good the methods would be on large realistic datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Based on this theoretical result, they propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. The authors also explore a meta-learning version of the algorithm based on group Lasso multi-class SVM base predictors. 
strength_and_weaknesses: Strength:
The authors developed theoretical studies and results with extensive mathematical proof. The theory in the paper is relatively novel. 

Weakness:
1. The experiments on disentanglement are insufficient to support the claims.  The authors need to visually show the disentanglement effects regarding different latent factors.  Otherwise, we cannot say the learned representations are meaningful. 

2. Only one disentanglement metric was used in the paper, the author could use multiple disentanglement scores to validate the results. 

3. The author could present additional experimental results on how disentanglement benefits prediction or other tasks, and it could make the paper stronger. 

4. The notations of different variables are not easy to follow. The author could list them in a table to improve readability. 

clarity,_quality,_novelty_and_reproducibility: The notations and the technical sections are not easy to follow.  
summary_of_the_review: The theory developed in the paper is relatively novel. The paper could be stronger with additional experimental results on disentanglement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present technical results showing that disentangled representations improve generalization if base predictors are sparse (they only use a limited number of features from the representation). Building on this, the authors propose an algorithm that learns disentangled representations (by encouraging base-predictor sparsity) and show its effectiveness on semi-synthetic data and on meta-learning. 
strength_and_weaknesses: The series of technical results is definitely interesting: the authors start from the assumption that only few features of the representation are useful for prediction. Building on the assumption, they provide insight on why learning disentangled representations is beneficial to downstream performance. Finally, an algorithm is proposed to capitalize on the gained insight (and is made practical, by operating in the dual space, for the meta-learning case).
The results would appear to be adequately backed by relevant semi-synthetic experiments and on meta-learning.

It would be nice, as an add-on, if the authors could provide evidence that the proposed approach is beneficial in a larger variety of contexts. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, as it makes the content accessible even for someone who is not deeply familiar with the topic, as myself. Technical results and assumptions are adequately motivated and explained, and relevant experiments are presented to support the claims.
The authors provide code for reproducibility.
summary_of_the_review: The authors present a series of technical and empirical results that explain the practical benefit of disentangled representations if the base predictors are sparse. I am not an expert of the research area, but this appears to be an interesting and insightful paper. It would be nice to see the usefulness of the proposed approach on additional benchmarks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies disentangled representations and to what degree disentanglement affect generalization in downstream tasks. Both theoretical and experimental analysis are provided. 

All reviewers - despite having reservations regarding clarity of some part and the somewhat toy types setting of experiments - liked the paper and recommends acceptance. However, the reviewers have some reservations and the average score is below the typical cutoff of accepted papers so rejection is recommended. The reviewers have given plenty of constructive input that can be used to improve the paper for the next conference.  
justification_for_why_not_higher_score: Reviewers are lukewarm so hard to argue for higher.
justification_for_why_not_lower_score: Good solid work.
Decision: Reject