Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new distillation technique for learning Information Retrieval (IR) models whereby not only the scores of the documents are aligned with the teacher, but also the representations. The paper exhibits two propositions showing (1) a relationship involving the empirical distillation loss and the representations (2) the bias between the empirical and expected distillation loss. Experiments show that the regularization is based on embeddings. The authors show that this improves the result with two types of distillation (dense to dense, and cross-encoder to dense).

strength_and_weaknesses: This paper shows that aligning embeddings (with an ad-hoc procedure in the case of cross-encoders where queries and documents do not have a dense representation) helps improve the performance of student models when distilling. 

The various experiments conducted clearly show that the proposed methodology do help in a variety of settings (dense to dense, cross-encoder to dense, dense to lighter dense) with two representative datasets (MS Marco and Natural Questions), showing the robustness of the approach. 

The paper provides two propositions justifying the approach (I did not check the proofs in the appendix, but they seem to be sensible), but I failed to understand how they help showing in theory how aligning the embeddings does help improving the student:

1) Proposition 1 shows that the difference between the distillation loss and the teacher loss is bounded by the expectation of embeddings distance and a dense teacher loss: how this shows that minimizing the proposed loss (distillation loss + embeddings loss) will help improving generalization?

2) Proposition 2 states that the difference between the empirical and theoretical loss is bounded by some quantity that is lower when one part of the network is frozen (i.e. the document embeddings part in this paper). If this is the case, first, it seems that just proving the bound would be enough (which should be doable since the Rademacher complexity of the sum of two classes of functions is the sum of their complexities, by decomposing the query representation as $q + \delta_q$ and the same for the document); after that, I wonder what is the practical or theoretical value of such bounds in this paper's context.

I might add that from a theoretical point of view, it would have been interesting to understand (1) what is the last term of the equation of proposition 1 and how it relate to the performance of the model (2) to study the difference between the "[CLS]" classification and the one based on the dense embeddings obtained through pooling.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, and most details for reproducibility are given in the appendix. The novelty of the paper is to apply embedding alignment on top of the distillation loss and to show that this works in IR. Note that embeddings alignment has already been studied (as discussed by the authors), so the novelty is on how to apply it to an IR setting where there are small differences (two representations instead of one, and the fact that cross-encoders do not have a separate representation for documents and queries).


There are just a few unclear/to correct parts:
- It is not clear how the teacher and student performance are computed: is it on the same re-reranking task (which is not the primary use case of dense retrievers) or on different tasks (re-ranking for the teacher and full ranking for the student). 
- the discussion about the fact that the cross-encoder representation cannot be used for aligning document/query representation (sec 4.2) is quite obvious since the CLS token is only trained on separating relevant from non-relevant (query, document) pairs.


summary_of_the_review: This paper proposes to regularize the distillation loss by aligning the query and document representations (or a substitute when they are not readily available, e.g. for cross-encoders), and clearly shows its benefit on two standard IR datasets. The theoretical section fails to show anything (IMHO) that could really be leveraged or provide insight on what this distillation is doing, and this lessen the interest of the paper (a qualitative or more focused theoretical analysis would have been more valuable).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a distillation approach that supports both retrieval and re-ranking stages and crucially leverages the relative geometry among queries and documents learned by the large teacher model. 
strength_and_weaknesses: ## Strength
1. This paper provides a theoretical justification of the proposed method.
1. This paper includes a good analysis of the embeddings, which empirically justifies their approach.
1. The ablation study (Table 2) is informative.

## Weaknesses
1. The mathematical expressions seem to be overused. There are a lot of unnecessary formulas in the paper, e.g., Eq. 1,2,3
1. The proposed method is not new as there have been many works on distillation for retrieval, both CE to DE and DE to DE.
1. There are mentions of prior studies on text retrieval + knowledge distillation. However, they are not compared in the experiments. 
1. I hope to see evaluation on more datasets, e.g., BEIR benchmark. 
1. The performance of the student model is somewhat disappointing. It's maybe because of the weak teacher. Why not use a SOTA model as the teacher?

#### Minor issues:
1. Figure 1 is too small.
clarity,_quality,_novelty_and_reproducibility: The paper is overall clear while lacking novelty. It seems no code is available at this moment and I recommend the authors consider open-sourcing the code.
summary_of_the_review: This paper explores two ways of distilling into dual-encoder text retrieval models. However, it lacks novelty and comparison with other approaches.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel embedding-based knowledge distillation method for Transformer in QA, including DE2DE and CE2DE. Moreover, they give a theory that proves the effectiveness of the distillation method on the generalization of student networks. 
strength_and_weaknesses: Strength:

     -- The theory proof makes the method more convincing.
     -- Try to distill the knowledge based on the embeddings(or, say, outputs) from Transformer.


Weakness:

    -- Poor writing and too many long sentences in the Introduction Section.
    -- Lack of comparison. The experiments seem poor, and they only compare the proposed embedDistill with naïve logit-based distillation, and they do not compare with other methods mentioned in related work. 
clarity,_quality,_novelty_and_reproducibility: Clarify: can be improved.

Quality: good

Novelty: fair, The embedDistill is similar to the feature-based distillation. There are many previous works.
Reproducibility: good
summary_of_the_review: It is a valuable topic to perform feature-based distillation for transformer in QA tasks. But more comparisons are needed to prove the effectiveness, and the writing can be improved. Moreover, embedDistill brings an extra aligned projection matrix, which may take more training resources.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new distillation approach that supports both retrieval and re-ranking stages and crucially leverages the relative geometry among queries and documents learned by the large teacher model. The paper provides stronger signals about local geometry via embedding matching and attains better coverage of data manifold globally via query generation.
strength_and_weaknesses: Strength:
1. The paper is generally well-written and easy to follow.
2. The paper proposes a new idea of distillation approach that goes beyond score matching and aligns the embedding spaces of the teacher and student models.
3. The paper has solid theoretical analysis on the proposed method

Weakness:
1. The question generation method is not well described. As a main contribution emphasized by the authors, more details are required.
2. Both the CE and DE models are based on the RoBERTa-base or RoBERTa-mini, where the model size is no larger than 124M. The authors can try some larger models to show the distillation performance on these larger models.
3. The idea that aligns the embedding space between teacher and student seems not novel.
4. No distillation baselines are compared in the paper. 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and has sold theoretical analysis. The idea that aligns the embedding space between teacher and student seems not novel. 
summary_of_the_review: Overall, the paper is well-written and easy to follow. But it can be greatly improved after the concern about the technical details on question generation and the experimental settings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The proposed distillation technique with embedding matching for learning neural IR models in different scenarios (DE to DE and CE to DE) and the theoretical justification are interesting. However, the reviewers were not convinced even after the rebuttal on the: (a) novelty, (b) usefulness of the theoretical proof for distillation, and (c) completeness of experiments/evaluation (e.g., not using a strong teacher, other baselines not compared). I am in favour of rejecting the paper in its current form.   


justification_for_why_not_higher_score: Not good enough
Decision: Reject