Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: When facing an ML problem with multiple tasks, and few datapoints per task, meta learning is used as a mean to provide good results simultaneously on all tasks. Typical approaches to meta learning include 1) fine-tuning a large model for each task at hand, and 2) learning a low dimensional representation to be shared by all tasks and used for predictions. The first approach provides good results in practice, however, it is not always feasible due to memory and computational constraints. In contrast, the second approach is efficient, however, provides lower performance. 

The authors of this paper propose a new solution that aims to combine the goods of both worlds. In particular, the authors propose learning 1) a shared low dimensional representation, and 2) a task specific fine-tuning of this low dimensional representation To this end, the authors employ a technique based on low rank and sparse decomposition of the original large weights. 

The authors then analyze theoretically the proposed method, called AMHT-LRS, in terms of computational bounds, when considering a Gaussian setup and a linear projection scheme. Furthermore, they propose a small tweak to the proposed method in order to satisfy differential privacy for each of the tasks.

The authors finally empirically verify their results in a toy setup, and on a real world dataset, showing the advantages of their proposed method. 
strength_and_weaknesses: Strengths:
- the problem addressed in the paper is of direct real world implications, where companies have access to user data from multiple services, and need to learn a good model for all of them.
- the authors considered the privacy implications of their scheme and proposed a modification to solve it.
- the authors analyzed theoretically their approach, and provided computational bounds for their method.
- the results achieved by the proposed methods are promising and validate the intuition behind the method.

Weaknesses:
- the authors evaluate the proposed method, but do not indicate whether the employed method is differentially private or not.
- the authors do not provide actual values for the DP parameters for their proposed method.
- the authors test their method in one real-world dataset (MovieLens). While this step shows some validation of the results, I think more evaluation with real datasets is beneficial (e.g. meta-dataset)
- the authors do not provide runtime values for their proposed method.
- the evaluation lacks intuition about the choice of the model's parameters, e.g. low dimensional embedding size.
- I am slightly confused about the results. In the synthetic experiment, the proposed method is much better than all methods by a big margin. This however is not the case in the MovieLens dataset, where the proposed method achieves on average similar results to full fine-tuning. An intuition about the discrepancy source would be nice.
clarity,_quality,_novelty_and_reproducibility: Quality: 
The authors studied extensively the theoretical framework proposed, and provided computational bounds. The experimental evaluation needs a bit more work.

Clarity: 
The paper is well-written, and the general idea is clear. When delving deeper into the theoretical part, the message becomes a bit more messy, and the authors rely heavily on mathematical formulas rather than plain english to explain the assumptions and the theory.

Originality: 
The paper idea of combining two common approaches of meta learning is new to the best of my knowledge. One related the work that would be beneficial for the paper is: _"SPARK: co-exploring model SPArsity and low-RanKness for compact neural networks"_. That paper explores model compression for a single task using low rank and sparse decomposition.
summary_of_the_review: The paper provides a new approach for meta learning that combines two existing approaches. The proposed approach is intuitive and makes sense, and the authors study their method theoretically. The authors also validate their method using a toy experiment and a literature dataset. The authors discuss theoretically the privacy implications of their method, however, it is unclear whether this aspect has been validated empirically.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies a linear meta-learning model comprising task-specific parameters: a low-rank weight and a sparse bias. The authors propose an alternating optimization algorithm for it. The generalization bound is also derived. The empirical performance is evaluated on Movielens 1M and synthetic datasets. 
strength_and_weaknesses: Strengths
- Rigorous theoretical analysis is provided.

Weaknesses
- The paper is densely written and hard to follow.
- The model looks too simple to solve complex real-world data.
- The empirical evaluation is limited. 
clarity,_quality,_novelty_and_reproducibility: Overall the paper is not clearly written. It contains many contents in 9 pages, resulting in the paper being dense and hard to read. 

The paper provides theoretical results of high quality. In contrast, the empirical evaluation is not sufficient. First, the Movielens dataset is usually used for recommender systems. I'm not sure it is reasonable for meta-learning methods.  Second, Movielens is the only real dataset used in the experiments. In addition, only linear models are compared in the main results. Nonlinear models such as NNs would achieve better performance, and they should be compared. I found such experiments in Appendix A, where a baseline method (full fine-tuning) is better than the proposed one. It is explained that the memory usage of the baseline is high, but it was not empirically measured and less convincing. 

Although the proposed model is simple, the derived algorithm has a theoretical guarantee and is novel. 

The reproducibility is low (no code is provided). 
summary_of_the_review: This paper proposes a simple linear model with a theoretically-guaranteed algorithm. However, the empirical evaluation is limited and whether the model is really applicable to real-world tasks remains mostly unknown.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies meta-learning approaches for representation learning in a linear setting with a low-rank and a sparse assumption. The authors proposed an alternating minimization method to solve the resulting problem. The sample complexity bound is also provided. 
strength_and_weaknesses: Strengths:

1: The theoretical analysis is rigorous and solid.

2: The studied problem is interesting and could have a high impact on the community.

Weaknesses:

1: The motivation/intuition for adding such a low-rank and a sparse assumption is unclear. 

2: Comparisons with [1] are not clear. What are the contributions/advantages beyond [1]?

[1] Differentially private model personalization. NeurIPS 2021
clarity,_quality,_novelty_and_reproducibility: Novelty: This paper is beyond [1] by adding a low-rank and a sparse assumption. It is something new in this area, but not very interesting. 

[1] Differentially private model personalization. NeurIPS 2021

Quality: The theoretical analysis is technically sound. 

Clarity: The paper is well-organized. However, the motivation for adding such a low-rank and a sparse assumption is unclear.

Reproducibility: The code needed to reproduce the experimental results is not provided.
summary_of_the_review: This paper studies meta-learning approaches for representation learning in a linear setting with a low-rank and a sparse assumption. The motivation for adding such a low-rank and a sparse assumption is unclear. Comparisons with the existing work [1] are not clear/enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is motivated by a question in meta-learning. Ultimately this is a very theoretical paper that boils down to estimating the sum of a rank-r and a k-column sparse matrix. The question itself is neat, and even well-motivated. The entire proof of correctness is rather long. The authors efforts in responding to the reviewers, including providing additional experiments is greatly appreciated. Ultimately, I'm not convinced that there would be sufficiently wide interest in this work at ICLR, the results will be better appreciated in venues such as COLT or ALT, and the authors should consider submitting to those conferences.
justification_for_why_not_higher_score: The theoretical proofs in the paper are long, and while there is some experimental work presented, the paper may be better suited to a more (explicitly) theory-oriented venue.
justification_for_why_not_lower_score: NA
Decision: Reject