Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors propose Matryoshka Representations, a flexible coarse-to-fine representation that can be applied to multiple downstream tasks based on the computational budget. The authors demonstrate that utilizing the proposed Matryoshka Representations leads to a substantial reduction in embedding size for large-scale retrieval tasks. Finally, the authors have shown that the proposed representations can be incorporated into web-scale dataset experiments.     
strengths_and_weaknesses: Strengths:

- The proposed solution is simple and intuitive.
- The learned low-dimensional representations are as good as the models trained independently with that particular dimension without any additional computational overhead.
- Significantly improves the efficiency of large-scale classification and retrieval tasks.
- Allows for adaptive classification with variable embedding size.
- Achieves promising improvement on long-tail continual learning experiments.
- Matryoshka Representations can be easily extended to different modalities and even to web-scale datasets.

Weaknesses:
- No major weakness. The idea is simple and the results are compelling.


questions: - Have the authors experimented with arbitrary granularities? For instance, m random numbers can be generated up to the highest embedding size and these random numbers can be used for nesting dimensions. It will be interesting to see how the embeddings interpolate if the features are learned using such an arbitrary granularity.  

- Have the authors tried lower than 8 dimensions?

- Any insight into why tuning the relative importance weight of dimension 16 did not lead to improvement of dimension 32 whereas only increasing the weight of dimension 8 led to a noticeable improvement of dimension 16.
limitations: The limitations of the proposed work have been adequately addressed. However, no discussion about the potential negative soceital impact has been provided in the main text and supplementary materials.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces "Matryoshka Representations" which are a way to encode dimensions over multiple scales such that for a chosen embedding dimension d, there are log(d) embeddings of increasingly smaller size that can encode the same information as the original d-dimensional vector, with minimal loss in performance of the downstream task. The paper thoroughly validates MRs on an impressively large variety of tasks spanning over challenging and large scale datasets, and different modalities (vision+language, vision, language).
strengths_and_weaknesses: Strengths:
- the main strength of the paper is that it proposes a simple and practical solution for the problem, which can be used in many downstream tasks
- The paper is written very clearly.
- 14 x faster retrieval is a very significant improvement
- The experiments are exhaustive and satisfying, for an approach that offers a lot of generality. 

Weaknesses:
- Outside of increasing the speed for retrieval tasks, I am not sure what other practical applications of MRs are there. 
- I would like to see the authors discuss the research done on anytime neural networks[1]. They are not similar in methodology, but try to solve a very similar problem.
- The improvement over FF is impressive, albeit unclear to me, why there is still improvements are higher dimensions (~64+). I would think/hope that at around 64 dim, the amount of information starts to become noisy. I would also 
- I would be very interested in these two baselines on any task: 
    - VIB [2] reformulated for this task, to see if its possible that there are any direct or indirect connections if the low-dim representations learned using VIB can be useful. 
    - Taking the full d-dim embedding and projecting it to a lower dim space using a random projection and seeing if that works. According to Gram-Schmidt, the low-dim representations should be usefully informative, and it was tried in [3] to some success. It can be a useful baseline for the paper as well. 


[1] https://arxiv.org/abs/1708.06832
[2] https://arxiv.org/abs/1612.00410
[3] https://arxiv.org/abs/1910.13540
questions: Please see strengths/ weaknesses.
limitations: The authors do not explicitly discuss societal impacts of their work, but its unclear if the work has direct impacts. It would be useful for the community if the authors commented/showed if the lower dimensional representations are more or less susceptible of encoding biases in their representations, as its important to discuss such details. For practical purposes, is there a tradeoff between the dimensionality of the representations and the tendency to encode biases.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes the so-called matryoshka representation learning, which can learns coarse-to-fine representations by modifying the regular learning pipeline. The matryoshka representation can encode information at different granularities, which achieves the same level of performance while being of much smaller sizes. The authors show the effectiveness of the matryoshka representations for image classification and retrieval tasks. Extensive ablation studies demonstrates the robustness and the ability of learning with limited data (few shots) and long-tailed data settings.
strengths_and_weaknesses: Strengths

1. it is an interesting idea to learn representations that can encode information of different granularities. 

2. The learned representations are deployed to different downstream tasks, e.g., classification and retrieval.

3. The experiments and ablation studies are thorough and comprehensive.

Weakness

1. The idea is simple and straightforward. The major weakness of this work is the technical novelty, which makes the contribution a bit weak.

2. The proposed algorithm does not show performance advantages when the representation size is large although it performs much better when size is very small. It seems the proposed algorithm also relies on a large representations to produce SOTA performance. 
questions: 1. It is unclear to me why the proposed algorithm can deal with out-of-distribution data and long-tailed data.
limitations: I did not see any discussion about the limitations of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a flexible representation learning approach MRL for an adaptive deployment on downstream tasks, where a single embedding can encode information at multiple granularities. An efficient variant MRL-E is also introduced with a set of common weights for nested dimensions. The coarse -to-fine representations are then applied to adaptive classification and retrieval tasks with various settings, where the method matches the performances of fixed-feature baselines. They provide as well further analysis in terms of robustness, few-shot learning and different dimensions.
strengths_and_weaknesses: 1)    The paper is well-written and quite easy for readers to follow the principal ideas, though switching between appendix and main paper from time to time troubles a bit.
2)    The author provides a surprising number of experiments, covering different aspects of modalities, datasets and applications, which is quite impressive and complete.
3)    Coming to the classification task, in Tables 1,2,4, for the high dimensions, it’s interesting to see that MRL sometimes cannot achieve the same results as FF. I am wondering if some explanations could be provided. 
questions: I am wondering if there exist any difficulties in training by introducing the nested loss.
limitations: I think the limitations mentioned in Section 6 are quite complete.
review_title: null

Meta-review: metareview: This paper proposes a Matryoshka Representation Learning paradigm to learn representations at multiple granularities, which can adapt to downstream tasks with different computational budgets. All the reviewers find the idea simple and interesting, and acknowledge that the experiments are thorough and impressive. The authors were successful at addressing the reviewers' concerns. Overall, the meta-reviewer recommends acceptance of the paper.
confidence: Certain
Decision: Accept