Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper formulates the problems of learning in invariant representations as a min-max game, exploring tradeoffs between accuracy and invariance of these representations via a geometric plane analysis. Specifically. the paper considers both classification (cross entropy loss) and regressions settings (squared loss).The related minimax problem is separable in the sense that for any fixed feature transformation, the optimization for the min and max agent are independent of each other, resulting is a simple, concise representation of the resulting optimization problem.  The symmetric nature of this description allows for a geometric description of the feasible set in regards to the actions of both min-max agents and the paper goes on to provide some characterizations of external points and other properties (e.g. convexity) of this region/set. The paper also derives a tight lower bound that for the Lagrangian form of accuracy and invariance.

This is an interesting theoretical take on the issue of exploring tradeoffs between accuracy and invariance of representations, but I am not sure to what extend this theoretical analysis provides actionable insights about real world problems (e.g. what doe convexity of the feasible set imply for algorithmic fairness?). The paper would be much more convincing if it offered some concrete use of these ideas in examples/applications. 

Along these lines, I am not sure about whether some modelling assumptions are always satisfied in practice. For example, it seems to me that in the analysis both Y/A tasks are considered to be either jointly classification or jointly regression tasks but in many of the examples put forward it could be the Y is a continuous variable e.g. salary where A could correspond to a discrete one e.g. gender. What does this framework imply in this case?  How sensitive is it on the choice of the loss functions?

Overall, this geometric approach seems like a cute idea but at least in its current form maybe it is not overly insightful for the concrete applications that it aims to model. 
review_title: A geometric approach to invariant representations

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work studies a fundamental and important problem in representation learning, the tradeoff between accuracy and invariance of the learned representations. For classification and regression problems, the paper analyzed the inherent tradeoffs by providing a geometric characterization of the feasible region in the information plane, where it connects the geometric properties of this feasible region to the fundamental limitations of the tradeoff problem. The work is well written and well presented overall, but the implication of the result is not very clear and the work lacks experimental validations.
Below are some major comments/concerns from the reviewer:

1. One of the major questions the reviewer has is what is the implication of the result. In the other words, given the characterization of the feasible region in the information plane, including its boundedness, convexity, and extremal vertices, how can the result in this paper improve representation learning? What is the right balance of classification and invariance? It seems quite vague in the current form of the work.

2. Does Assumption 4.1 and Assumption 5.1 hold in general? Does the mapping f always exist, or under what condition does the mapping f exist? More discussion about the assumptions is needed. It is not how stringent the assumptions are in practice.

3. Another question the paper does not address, is how to construct g(X) to achieve certain points on the information plane.

4. Although the result is theoretical, it could be better if the authors provide some experimental justifications, which is completely missing.
review_title: Studies a very important problem in representation learning but the implication is vague

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors extend the information bottleneck by Tishby to also include an additional auxiliary variable that represents either domain index or private information. Using this generalined IB the authors study tradeoffs/fundamental limtis on achievalbe accuracy in predicting labels while maintaining invariance against the auxialiary variable. 

The proposed approach seems promising but much more work is needed: 
- it is unclear how the proposed frameworl relates precisely to plain vanilla IB. I feeld that convential IB could be obtained somehow when using the auxiliary variable to identify individual traning samples instead of larger domains. 

- how can the obtained characterization be used ? can one use these characterizations to verify sub-optimiality of existing methods for privacy-preserving ML or algorithmic fairness 

- how can the proposed model be turned into practical algorithms that (nearly) achieve the fundamental limits/tradeoffs. 

minor issues: 
- pls indicate the optimization domain for every optimization problem (e.g. (1)) 
- unclear what is meant by "could be the identity of domain index in domain adaptation," 
- "..Lagrangian has the following lower bound:.."
- "Evidently, the key quantity in the lower bound..." why is this evident ? 
review_title: Very promising approach!

Meta-review: This paper studies an interesting information-theoretic trade-off between accuracy and invariance by posing it as a minimax problem. The results are of theoretical nature. However, the implications of the results are not clear. Also, the model/assumptions authors consider are not completely justified. Therefore, the paper at this stage is not recommended for acceptance. However, I highly encourage the authors to improve upon their existing work and resubmit to the next ML conference. 
Decision: Reject