Eclipsing LLMs: Gaussian-Enhanced Neural Information for Multi-Query Vector Expansion (GENIE)
Authors: Arthur SARMINI DET SATOUF, Pablo Piantanida, Habiboulaye Amadou Boubacar


Reviewer paper summaries: This study incorporates the addition of Gaussian noises in training a small VAE to generate multiple related query vectors for the original query, thereby enriching the retrieved documents.
Summary of strengths: The concept of multi-query vector expansion for the original query proves to be beneficial for dense retrieval. The experimental results demonstrate that this method outperforms other baselines.
Summary of weaknesses: 1. Lack of novelty: Small neural networks have been commonly utilized in query expansion tasks for quite some time, as shown by the QE-GAN [1] and other traditional sequence to sequence models. This paper introduces a text-to-vector approach that specifically targets the extraction of decoder vectors as query representations, limiting its usefulness to dense retrieval. In contrast, text-to-text expansion methods like QE-GAN offer more versatility, allowing the expanded query to be directly used in lexical retrieval or first encoded and then utilized in dense retrieval.

[1] Lee M C, Gao B, Zhang R. Rare query expansion through generative adversarial networks in search advertising[C]//Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining. 2018: 500-508.

2. Unfair comparison: GENIE is a query expansion method, yet the baseline used is the original BM25, as opposed to BM25+PRF, BM25+RM3 or other query expansion methods. Additionally, considering the paper's focus on "Eclipsing LLMs", the baselines should include methods that LLMs utilize in query expansion, as the expectation is for the model in this paper to outperform LLMs.

3. Untrustworthy results: The experiments conducted across eight BEIR benchmarks raise concerns due to disparities between the reported results in this paper and those presented by BEIR. For instance, the nDCG@10 score for BM25 in the ArguAna benchmark is recorded as 0.315 in the BEIR benchmarks but reported as 24.7 in this paper. Similar discrepancies are evident in results across the other six datasets, where the BM25 results in this paper are notably lower than those originally reported by the BEIR team in their own work. Furthermore, while the experiment suggests that GENIE combined with dense retrieval is faster than BM25, it's worth noting that my experience suggests the time required for computing dense vectors for each query using BERT, as well as the subsequent process of BERT sequence encoding and feeding into an Auto-Encoder for obtaining multi-query vectors, can be substantial. This leads to the question of whether the experiment used an offline query expansion approach rather than an online inference method.
Comments, suggestions and typos: The writing in this paper requires improvement.
1. Arguana->ArguAna
2. line 177 should have straight quotation marks instead of Chinese quotation marks.
Overall assessment scores: 1.5