Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces FiD-light, a more efficient variant of the fusion-in-decoder model that maintains/outperforms state-of-the-art performances on the KILT dataset, while drastically increasing the model's efficiency. To achieve this, FiD light compresses the length of input vectors and uses re-ranking to improve the top-ranked provenance precision.


strength_and_weaknesses: Strength
+ simple and effective solution to improve efficiency
+ clear improvements across datasets and tasks

Weaknesses
- some parts of the paper should be better clarified (see my comments below)

clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written. The appendix contains information for reproducibility.
summary_of_the_review: Overall I found the paper clear. I do have some concerns regarding some choices made regarding the architecture and a couple of suggestions/questions.

1) Intro RQ2 - what's unexpected in distribution learned by FiD-light?
2) Sec 3 Decoder efficiency - what's the fk function used to compress vectors?
3) In some sections of the paper (e.g., sec 4.1) you are referring to "common practices" without citing papers that follow such approaches. For example, Sec 4.1 mentions that the community compares results to your second oracle scenario. I suggest adding citations to relevant works that do this. 
4) Other works (e.g., KG-FiD) applies re-ranking between the encoder and the decoder of the T5 model. In this work, instead, you focus on the source pointing. I might have missed this part, but it is unclear why you are taking that direction. What's the reason for including source pointing?
5) Sec 4.3 mentions that the model lower the latency by 2x. It would be good to specify with respect to which of the models in table3
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper modifies two aspects of the FID model (retrieval-augmented text generation) in section 3: (1) the authors truncate the passages to speed up the model (2) they modify the explainability component by using a ranking task. Results on KILT show a substantial improvement over the FID model.

strength_and_weaknesses: Strengths: simple modification of the FID model that improves its performance a lot.

Weaknesses: lack of clarity in the description of the model (still not quite sure what the source pointing method is, but the answers clarified y doubts)

The proposed modifications are well-motivated and sensible, showing an improvement for the KILT tasks. Both modifications are quite simple but do work in practice, the paper has some experimental value.


clarity,_quality,_novelty_and_reproducibility: The novelty relies mostly on the use of a new ranking loss, but unfortunately this is the less well explained part of the paper (and this hinders reproducibility a lot, as well as the interest of the paper):

- in section 3: Eq. 6 is not understandable: $\hat o$ is supposed to ber the representation of the encoded passages (Eq. 5) - what does $r \in \hat o$ mean? Also, how is used when training the model? In the experimental section, 
- How figure 3 shows that the model struggles to output 3 passages as often as it should? What does "we are filling up holes" means? How can we "observe a stark reduction when the number of input passages is reduced" on Figure 4?

In section 4.4, table 3 should be replaced by the one in appendix (non aggregated results) since the picture is less clear.

Other parts that could be improved are the related works section (no positioning of the paper with respect to related works).

summary_of_the_review: This paper proposed a modification of an existing model (FID) that improves its efficiency with a simple truncation of inputs -- and a modification of a loss (but which is not well explained in the paper). Overall, the novelty is quite limited nevertheless and the paper lacks of clarity in many parts.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an extension of Fusion-in-decoder for improving both the effectiveness and efficiency. To improve the efficiency, noting that the decoding step occupies most of the time complexity, the method proposes a “selection-based compression” of the encoded representations, denoted as FiD-Light, by using only first-k vectors in the sequnce of the encoded token representations for each passage before directly fusing them in the decoder. To enhance the effectiveness, the reranking is performed based on the modified application of the “source pointing” method, denoted as FiD^{SP} by FiD-Ex by explicitly moving the source-pointed passages (i.e. highly relevant passages) forward in the list of the passages in the encoder, such that the distribution of the encoder’s input becomes more similar between training and test samples. Experiment results show that the modified “source pointing” method of Fid-Ex improves the performance both on the setting of FiD-Ex and FiD-Light. In addition, FiD-Light^{SP}, the method of applying both FiD-Light and FiD^{SP}, substantially reduces the time complexity, demonstrating that FiD-Light^{SP} achieves significantly improved performance at the fixed latency time budget. 

Overall, the key contribution of the paper is the simple but novel extension of FiD, which has widely been used for retrieval-augmented LM and its empirical usefulness on several standard datasets. 


strength_and_weaknesses: Strengths
- This proposed idea is quite simple and well-motivated, making an interesting extension of FiD, which would be helpful to the FiD-based literature. 
- The experiment results are solidly done, showing that the proposed FiD-Light is quite efficient, as well as keeping the effectiveness on various datasets. 

Weaknesses:
- The proposed method mainly focuses on the efficiency. While single query latency is used for the efficiency metric in the paper, FiD could be designed efficiently under the batch-style GPU-based parallel processing. Thus, a stricter GPU-aware time complexity needs to be used for the efficiency measure of FiD. The concern is that the time complexity of FiD may be exaggerated in the experiments. Assuming the realistic situation that the input is processed at the maximum length limit of the encoder using the nicely designed GPU-based implementation, the authors need to compare the time complexities between the proposed method and FiD.
- In the proposed method, the chosen options need to be further explored or discussed. Why the first-k tokens in FiD-Light are selected? Why the source pointed passages are moved forward? (not moved backward)? How performances are changed when varying k? 
- The naïve combination of reranking and FiD can reduce the time complexity, in the way that after reranking, the ranked top m passages can be selected as input for FiD while discarding other passages. But, it is not clear whether these reranking-and-selecting experiments are performed in the comparison of the paper. 


clarity,_quality,_novelty_and_reproducibility: The paper is often not easy to read, due to somehow dense manner. A substantial revision for readability may be required for better clarity. The proposed idea is an incremental work of FiD, not being very novel (not so innovative), although it is helpful to the litererature. The experiments of the tradeoff on the efficiency and effectiveness are solidly done, demonstrating the interesting behaviors and results of FiD and FiD-Light.


summary_of_the_review: 
Overall, the paper is well-motivated with an interesting extension of the widely-used FiD, drastically improving the efficiency. The detailed experiments comparing with other recent SOTA works are also helpful to the literature. 
While the major parts are helpful and quite interesting, however, major concerns are as follows:
1. the efficiency measure may not be fairly designed. GPU-based parallel processing needs be well-considered for designing FiD. 
2. the baseline method of reranking needs be evaluated, in the way that the reranked passages are only selected as input for the encoder. 

Also, the paper is often not very readable for some parts, so requiring a substantial revision for bettter clarity.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a lighter version of FiD by compressing the length of encoded vectors and using a decoding-time re-ranking to improve the top-ranked provenance precision. While the method shows significant boosts in efficiency, reviewers were not very convinced about the technical novel on both aspects (compression is to select simply top-k tokens and re-ranked has been done before although via an encoder).   
justification_for_why_not_higher_score: Not novel enough 
justification_for_why_not_lower_score: NA
Decision: Reject