Multivariate Dense Retrieval: A Reproducibility Study under a Memory-limited Setup
Authors: Georgios Sidiropoulos
Samarth Bhargav
Panagiotis Eustratiadis
Evangelos Kanoulas
Venue: Accepted by TMLR ()

Abstract: The current paradigm in dense retrieval is to represent queries and passages as low-dimensional real-valued vectors using neural language models, and then compute query-passage similarity as the dot product of these vector representations. A limitation of this approach is that these learned representations cannot capture or express uncertainty. At the same time, information retrieval over large corpora contains several sources of uncertainty, such as misspelled or ambiguous text. Consequently, retrieval methods that incorporate uncertainty estimation are more likely to generalize well to such data distribution shifts. The multivariate representation learning (MRL) framework proposed by Zamani & Bendersky (2023) is the first method that works in the direction of modeling uncertainty in dense retrieval. This framework represents queries and passages as multivariate normal distributions and computes query-passage similarity as the negative Kullback-Leibler (KL) divergence between these distributions. Furthermore, MRL formulates KL divergence as a dot product, allowing for efficient first-stage retrieval using standard maximum inner product search.

In this paper, we attempt to reproduce MRL under memory constraints (e.g., an academic computational budget). In particular, we focus on a memory-limited, single GPU setup. We find that the original work (i) introduces a typographical/mathematical error early in the formulation of the method that propagates to the rest of the original paper’s mathematical formulations, and (ii) does not fully specify certain important design choices that can strongly influence performance. In light of the aforementioned, we address the mathematical error and make some reasonable design choices when important details are unspecified. Additionally, we expand on the results from the original paper with a thorough ablation study which provides more insight into the impact of the framework’s different components. While we confirm that MRL can have state-of-the-art performance, we could not reproduce the results reported in the original paper or uncover the reported
trends against the baselines under a memory-limited setup that facilitates fair comparisons of MRL against its baselines. Our analysis offers insights as to why that is the case. Most importantly, our empirical results suggest that the variance definition in MRL does not
consistently capture uncertainty. The source code for our reproducibility study is available at: https://github.com/samarthbhargav/multivariate_ir/.

Published in Transactions on Machine Learning Research (01/2025)
Multivariate Dense Retrieval: A Reproducibility Study under a Memory-limited Setup
Georgios Sidiropoulos"
g. @.siiroopoulsshwo..t
IRLab, University of Amsterdam, Amsterdam, The Netherlands
Samarth Bhargav
b.bargan&uuva.r
IRIab, Universiby of Amsterdam, Amsterdam, The Netherlands
Panagiotis Eustratiadis
D.efstratiadis@uwv..l
IRLab, University of Amsterdam, Amsterdam, The Netherlands
Evangelos Kanoulas
kenoulas@@va.. ml
IRLab, University of Amsterdam, Amsterdam, The Netherlands
Reviewed on OpenReview: https: // openreview. net/ forum? id= #F22t80007
Abstract
The current paradigm in dense retrieval is to represent queries and passages as low- dimensional real-valued vectors using neural language models, and then compute query- passage similarity as the dot. product of these vector representations. A limitation of this approach is that these learned representations cannot capture or express uncertainty. At the same time, information retrieval over large corpora contains several sources of uncertainty, such as misspelled or ambiguous text. Consequently, retrieval methods that incorporate un- certainty cstimation are more likely to generalize well to such data distribution shifts. The multivariate representation learning (MRL) framework proposed by Zamanii & Bendersky (023) the first method that works the direction modeling uncertainty in dense re- trieval. This framework represents queries and passages as multivariate normal distributions and computes anery-passage similarity as the negative Kullback-Leible (KL) divergence be- tween these distributions. Furthermore, MRL formulates KT. divergence dot product, allowing for efficient first-stage retrieval using standard maximum inner product search.
In this paper, we attempt to reproduce MRL under memory constraints (e.g., an aca- demic computational budget). In particular, we focus on a memory-limited, single GPU setup. We find that the original work (i) introduces a snperapliallmmaheeaatiellllll error early in the formulation of the method that propagates to the rest of the original paper's mathematical formulations, and (ii) does not fully specify certain important design choices that can strongly influence performance. In light of the aforementioned, we address the mathematical error and make some reasonable design choices when important details are unspecified. Additionally, we expand on the results from the original paper with a thor- ough ablation study which provides more insight into the impact of the framework's dif- ferent components. While we confirm that MRL can have state-of-the-art performance, we could not reproduce the results reported the original paper or uncover the reported trends against the baselines under memory-limited setup that facilitates fair comparisons of MRL against its baselines. Our analysis offers insights to why that is the case. Most importantly, our empirical results suggest that the variance definition in MRL does not consistently capture uncertainty. The source code for our reproducibility study is available at: https://gitthu: comm wtsssamartttaa wawaaarbhaaaaaaalliaaaa........
mathematical
"Equal Contribution

---

Published in Transactions on Machine Learning Research (01/2025)
1 Introduction
Dense retrieval has become the new paradigm in first-stage retrieval, largely replacing lexical methods which cannot model semantic information well as neural models. Dense retrievers following the dual-encoder ar- chitecture (Karpukhin et al. 2020) are popular first-stage retrievers due to their performance and scalability. This paradigm uses pre-trained neural language models to encode queries and passages as low-dinensional real-valued dense vectors, with relevance defined as their dot product. Passages are encoded offline and stored in a dense index. At query time, retrieval can be done efficiently using maximum inner product search (MIPS). However, representing queries and passages as single vectors has an important limitation that has influenced the research landscape: these representations do not model, capture, or express pre- dictive uncertainty (Vilmis & McCallum 2015). At the same time, there are various sources of uncertainty arising both from the data and the neural retrieval models:
Query uncertainty. User queries may include misspellings, ambiguity, and incomplete or inaccurate in- formation (e.g., false memorics). Furthermore, in a realistic setting, the retrieval system has minimal to no prior knowledge about the distribution of the queries, except for possibly a few assumptions (e.g., the language they are in, or common user mistakes).
Passage uncertainty. Passages may present similar uncertainty- artifacts to queries, such misspellings and ambiguity. Unlike queries, the retrieval model has little prior knowledge of the passage collection as a whole, i.c., only of the documents used during training.
Relevance uncertainty. Relevance, or ranking uncertainty refers to the confidence of the model in the estimated query-passage relevance. Such an estimator may be anything between a deterministic function of query and passage uncertainty. c.g., the model reproduced in this paper, to a stochastic function of deterministic query and passage representations, e.g., Monte-Carlo dropout. Bayesian estimator (Cohen et al. 2021
Uncertainty estimation remains largely unexplored for the case of first-stage dense retricval, despite it having received increased allention from the community in the case of re-ranking (Wang & Zhu 2009 Zhu et al.
2009 Feng et al. 2020 Cohen et al. 2021 Heuss et al. 2023). Recently, Zamani & Bendersky (2023) proposed the multivariate representation learning (MRL) framework, the first approach that models uncertainty in the context of dense retrieval. MRL uses predictive variance as a proxy for uncertainty. Each query and passage is mapped to multivariate Gaussian distríbution parameterized by a mean vector and (diagonal) covariance matrix, where the mean represents the predicted query passage embodding, and the variance represents the uncertainly of said embedding. However, different from existing approaches to modelling uncertainly in IR that use Bayesian inference (Coben et al. 2021), variance in the MRL framework does not express statistical variance, i.c., deviation from the mean, as much as it expresses predicted risk. In essence, this is a tradc-off between being theoretically principled and computationally elficient. Admittedly, computational elficiency is of the utmost importance in first-stage retrieval, where one has to manage collections of potentially billions of passages.
Having represented queries and passages as multivariate normal distributions, the authors of the origi- nal paper proceed to formulate a query-passage relevance scoring function based on a simplified version of the Kullback Leibler (KL) divergence. Further, they express this function as a dot product between query and passage representations, thereby allowing for efficient retrieval by means of standard MIPS (e.g., FAISS (Johnson et al. 2021)). . Finally, they report state-of-hhe-at retrieval performance and show that the predicted covariance matrix can be used as a pro-retricval query performance prodictor.
Even though the results reported in the original study showcase the effectiveness of the proposed method, our study is motivated by several important questions that still need to be explored. Many of these questions have the character of reproducibility question. First and foremost, the unavailability of the source code and model checkpoints makes it difficult to verify the paper's substantial claims. Second, in the original work, a batch of 512 was used to train MRL, indicating that substantial computational resources were available in contrast, some of the basclines (c.g., CLDRD) were trained with a batch size of 8 on a single GPU. Third, even though the model consists of various components and several stages of knowledge distillation, the original work does not include an extensive ablation study that explores the impact of each component
2

---

Published in Transactions on Machine Learning Research (01/2025)
downstream performance. Therefore, it is unclear to what degree the performance gains of the proposed method stem from the multivariate representations of queries and passages. To this extent, it is important to understand the representations learned by MRL. In this reprodueibility work, we aim to answer the following research questions:
RQ1 To what extent can the results and findings of the original paper be reproduced under a memory- limited setup?
RQ2 Can the multivariate query and passage representations express uncertainty?
RQ3 What is the contribution of cach MRL component to the downstream retricval performance?
RQ4 What is the impact of batch size in training MRL?
Furthermore, we summarize our contributions to the original work:
error. We correct a mathematical error the original work, made early in the formulation of the method, in an attempt by the authors to simplify the computation of KL divergence between two multivariate normal distributions. This error propagated to the rest of the original paper's mathematical formulations. We suspect that this error is typographical and that it did not leak into the technical implementation of the original authors' experiments. In fact, we provide experimental evidence that if the incorrect similarily scoring function is used instead of our corrected version, il harms retrieval performance.
Correction of syporaphhial/mmathematii
Reproducing retricval and QPP experiments. We reproduce the experimental sctup of the original paper, for the lasks of dense retrieval and pre-retrieval query performance prediction (QPP). The original MRL model is trained with batch size of 512, while its primary competing approach of 8. We explore to what extent the original work's findings are reproducible when MRL is trained with significantly smaller batch size under a memory-limited setup that facilitates fair comparisons of MRL against its baselines (i.e., both MRT and its baselines are trained with the same batch size). We showcase that even though MRL can still achieve state-of-the-ar retrieval results, retrieval performance significantly drops under a memory-limited setup. Moreover, we show that MRL does not outperform the baselines fair comparison. Finally, MRL yields inconsistent results for the QPP experiments, with our analysis revealing that the variance vectors do not consistently capture notions of uncertainty.
batch size
Ablation study. The MRL framework is composed of multiple components, including multivariate represen- tations, knowledge distillation, and model initialization from an already effective pre-trained dense retriever. We conduct a thorough ablation study on the model's components to unveil their importance in training an effective retriever. We find that multivariate represent ations do not boost the model's performance, and the high effectiveness stems from the model initialization and knowledge distillation from the re-ranker. We further conduct thorough ablation study on the effect of different batch sizes on the retrieval performance of MRL. We believe that the impressive results reported in the original paper due to training with large batch size for many training steps.
Proposed improvements upon the original MRL model. We propose simple alteration to the original model, which results in a reduced hyperparameter search space. In short, instead of a parametric softplus activation which ensures positive semi-definiteness of the covariance matrix, we propose predicting the log-variance instcad, which obviates scarching for the hyperparameter of the softplus function. We show that the log-variance model either matches or outperforms the original softplus model.
2 Related Work
Uncertainty-aware retrieval. Uncertainty estimation in neural information retrieval (IR) has been ex- plored in the past, although not in the context of dense (first-stage) passage retrieval, which is the main novelty aspect of the MRL method. The work of Cohen et al. (2021) and Heuss et al. (2023) focuses on risk-aware (second-stage) re-ranking. Both approaches attempt to approximate Bayesian models that predict
3

---

Published in Transactions on Machine Learning Research (01/2025)
distribution of relevance scores rather than point estimates; the former uses Monte-Carlo dropout (Gal & Ghahramani 20160, while the latter leverages Taplace approximation. The common denominator across these Bayesian methods is that the predictive distribution p(y|0, D) is approximated by performing forward inference using multiple samples of 0. This is the main difference between prior work and MRL: In MRL, predictive uncertainly is not framed as weight. uncertainly, and variance does not represent deviation from the mean prediction. Rather, variance in MRL is a predicted value of a deterministic estimator.
While variety of efforts exist in the area of stochastic representations in image retrieval (Warburg et al., 2021, Chun et al 2021 recent work Warbbur et al. (2023) showed that Bayesian image retrieval with Laplace approximation can achieve some desirable properties. They show that the uncertainly of prediction increases (almost monotonically) with the amount of corruption in the input. The model's predictive uncertainty further behaves as expected when making out-of-domain predictions. These insights are valuable in text retrieval as well, where these desirable properties have not yct been achieved effoctively.
Uncertainty for detecting anttoof-distributionncoocee
Knowledge distillation. The MRL framework is surrounded by multiple layers of wwwwledge/parametee distillation, which we summarize in this section. First, the selected neural architecture employed in MRL is DistilBERT (Sanh et al.) 2019) ; distilled version of BERT with 40% fewer parameters for 97% of its original performance. Furthermore, the architecture has been distilled with balanced topic-aware sampling (TAS- B) (Hofstätter et al. 2021,, , which uses two teacher models to construct better training batches. Finally, MRL itself uses a knowledge distillation loss inspired by CLDRD (Zeng et al., 2022). While the original paper does not discuss how these sources of distilled knowledge affect downstream performance, in Section of this paper we perform thorough ablation study that examines them one-by-one.
3 Methodology
The proposed MRL framework represents queries and passages as multivariate Gaussian distributions. It does so by computing a mean vector μ and diagonal covariance matrix Σ for a query q and passage d, using query and passage encoders fo and fe, parameterized by 0 and & respectively,
(pQ) ff(()) (MD: fd(d).
(1)
(2)
It is also possible to have 0 = o. i.c., weight sharing, which the authors of the original paper opt for. Dense retrieval models (e.g., DPR, Karpukhin et al., 2020) typically utilize the embedding of a special token, the [CLS] token as the low-dimensional representation of queries and documents. Given a piece of text, for instance, "Hello world", pre-processing appends the special [CLS] token to the start of the text, producing "[CLS] Hello world", and the output of the transformer model for the [CLS] token is used. Relevance is a function of query and document representations, typically a dot product or cosine similarity. MRT, however, produces two vectors per input, which motivates the choice in the original study to use an additional special token, termed the [VAR] token, appended after the [CLS] token, but before the text. For instance, "Hello world" is pre-processed to " [CLS] [VAR] Hello world". The output representation of the [CLS] token is used to compute the mean, and the output representation of the [VAR] token is used to compute the variance.
The relevance score between querics and passages is then defined as the negative KL divergence between their distributional embeddings: Q ~ ~N(HQ Σa) and D ~ NN(p Ep),
rel(g, d) KTD(QI)))
(3)
The minus sign is there to implement "higher is better" type of scoring. To simplify matters, we will disregard it in the upcoming derivations and re-introduce it at the very end. In this section, we detail the reproducibility study of the above framework. First, we direct atlention to a small mathematical error that was made early in the formulation of the relevance scoring in the original paper, that propagated through the rest of the mathematical derivations. We then discuss matters of model training. Whenever we make a strong assumption due to the lack of implementation detail in the original paper, or the lack of shared source code, it is explicitly mentioned.
4

---

Published in Transactions on Machine Learning Research (01/2025)
3.1 KL divergence- ereegennce-based relevance scoring
In Eq. we start by repeating the slandard definition of KL divergence, as written in Eq. 9 of the original paper:
1 [ det Ed k tr m2) + ~D) ED' (ivee--) -D)]
KID(QI 2 det Σ
where & denotes the dimensionality of the multivariate Gaussian embeddings. For the purpose of relevance scoring, the authors proceed to further simplify Eq. and reformulate it as document ranking function. They do so by eliminating document- iimmentt-ideeeedeent terms and constants, and by taking advantage of the fact that the covariance matrices are diagonal. Let us follow their simplification steps by considoring cach term separately. For the first term we have,
log det ED log det En log det Eg = logdet det. E. log det Eq
d =
log di
(5)
constant w.r.t. doc. ranking
The subsequent steps in the original paper contain error in the simplification of the second term. We include the original formulation in Appendix for completeness. We note that using the original formulation leads to drastically lower performance, which makes it likely that this error is typographical i.e., did not propagate to the implementation (sce Section 5.1 for more details). We provide the correct dcrivation in Eq. as follows:
tr(Ep Eq]
Σ
10
(6)
0
Finally, for the third term have,
(Uia Wip)
2pig
Pip
Mip
(rq
ED (nQ (D)
(7)
0/10
o
ip
i=1
Combining Eq.
5
1 and 7
into
Eq. and removing constants, we arrive at the intended derivation of the
ranking function:
k
k c diD
2 Pip
KLD(QII = _ log 0
(8)
Note that unlike Eq. Eq. is no longer the KT. divergence. After all the simplifications, is a KT. divergence-based relevance scoring function for ranking documents given a query. From this point forward, continue with the work described in the original paper, but base it Eq. g which is the derivation of the relevance scoring function that includes our correction.
The next step of this reproducibility study is to express Eq. S as a dot product between query and passage vectors i.., DL = qT d, with the purpose of reusing standard efficient inner product similarity search (Johnson et al. 2021). To do so, we isolate the document-specific terms of Eq. that can pre- computed:
log of +
ND
In the original paper, the term TD is referred to "document prior". Now express the relevance score dot product between query and passage vector representations:
- =
1,00 1,0ig
Tho
..... k
Mlq....
Pka
(10)
d
7D,
oi
1
kn
2
dip
1
2pin
of
2
o7
(11)
where q. d. E (3+++) At this point, we remind the reader that following Eq. negative of the KL divergence.
3
the relevance score is the
5

---

Published in Transactions on Machine Learning Research (01/2025)
3.2 Listwise knowledge distillation
Knowledge distillation has become of great importance in boosting the effectiveness of dense retrievers (Ho-- stätter et al., 2020). In detail, highly effective cross-encoder re-ranker is used teacher to transfer knowledge to a less effective but efficient first-stage dense retriever student model. Consequently, the effec- tiveness of the dense retriever is increased while it retains its efficiency. In the original work by Zamani & Bendersky (2023), the authors employ a listwise distillation loss function ŞZeng et al.] 2022) to train their dense retriever (i.e., student model). For each query q and a set of passages Dg (later in this section we provide details on how this set is constructed), the loss is compuled as:
Σ d,d'ED,
1 Hvy > w(ddd)) [d(d)
TT(d') log(1 + -Molgdd"
(12)
where (d) denotes the position of passage d in the ranked list produced by the dense retrieval student model Ma and (d) denoles the relevance judgment produced by the teacher model for the pair of query ? and passage d; (d) can be either a score or a label. In the original work of Zamani & Bendersky (2023) Vg (d) is the raw score from the teacher model for query-passago pair, and Dg is constructed as follows. Given a query 4, the passage set D2 constructed with positive passages provided by the dataset's 's official relevance judgments. On the other hand, the negative passages are sampled from the top-k passages retrieved with BM25 and the top-k passages retrieved by the student dense retrieval model itself. Finally, the original work uses in-batch negative training to reuse passages from other queries that are already in the batch.
4 Experimental Setup
4.1 Datasets and metrics
Our evaluation is performed both in-domain (ID) and out-of-domain (OOD) data. In the OOD setting, perform zero-shot evaluation. All tasks are ad-hoc retrieval, with fixed set of documents. Statistics of the datasets are reported in Appendix D We summarize the datasets and evaluation methodology below.
In Domain (ID). We train all models on the MS-MARCO (Nguyen et al.) 2016) training set. Note that we split the full training set into a training and validation set for hyperparameter tuning as described in Section 4.4 There are three in-domain evaluation sets, all of which are based on the MS-MARCO corpus. This includes the MS-MARCO Dev set, the TREC-DL 2019 (Craswell
al.) (0200) and TREC-DL 2020 Craswell et al. 2021) datasots. Both TREC datasets are densely labeled by humans. The evaluation metric for the Dev set is the mean reciprocal rank (MRR) with a cut-off of 10, denoted as MRR@10. For the TREC subsets, we use the standard evaluation metrics of normalized discounted cumulative gain at 10 (nDCG@10), and mcan average precision (MAP).
Out of Domain (OOD). We evaluate the retrieval models' generalization ability in different domains via zero-shot passage retrieval experimentation. All retrieval models are trained on the MS- MARC training set and tested on previously unseen queries and underlying corpus. We replicate the evaluation setup outlined in Zamani & Bendersky (2023), with nDCG@10 10 as the primary metric. We evaluate the following OOD datasets
in zero-shot setting: (i) SciFact (Wadden et al. 2020): scientific claim verification dataset where the task involves retrieving abstracts that either refute or support a claim, (ii) FIQA (Maia et al.) 2018): a dataset that involves retrieval of documents in the financial domain using natural language questions, (iii) TREC- COVID (Voorhees et al. 2021): a biomedical dataset of scientific articles about COVID-19, with questions as the aopics/queriae and (iv) CQADupStack (Hoogeveen al. 2015):: a community question answering (CQA) dataset, with the task of retrieving duplicate questions in community website (StackOverllow).
4.2 Baselines
We compare MRL against the following single-vector dense retrieval models:
DPR.
Karpukhin et al. 2020 is a traditional dense retriever that is trained with softmax cross-entropy.
6

---

Published in Transactions on Machine Learning Research (01/2025)
TAS-B [Hofstätter et al. 2021 ): is effective dense retriever that is trained by combining (i) knowledge distillation from re-ranker teacher model (i.e., cross-encoder) with (ii) a balanced topic-aware sampling method. This method alternates the creation process of the training batches by composing batches based on queries clustered in the same topic. Furthermore, it selects passage w.r.t. the pairwise margin between positive and negative passages the batch so that the margin of positive-negalive pairs balanced the margin range.
CLDRD Zeng et al. 2022 :: is a state-of-the-art dense retriever that uses TAS-B as initialization and is trained by combining curriculum learning with knowledge distillation; in particular, it uses the listwise loss of Eq. 12 The student dense retriever trained via an iterative training process in which the difficulty of the training data, produced by the re-ranking teacher model, increases with each iteration. Additional information regarding the training setup of CLDRD can be found in the Appendix
The motivation behind selecting these baselines is twofold: First, their inclusion in the original study, and second, to enable fair comparisons in our subsequent ablation study. For instance, MRL can be compared with CLDRD to assess the impact of the multivariate representations, and a similar assessment can be made when MRL without distillation is compared against DPR.
4.3 Query performance prediction
The QPP task (He & Ounis 2004
2006 Carmel & Yom-Tov 2010 involves inferring the difficulty of a given query for a search system without using relevance judgments. We replicate the pre-retrieval QPP setup in Zamani & Bendersky (2023 evaluating on the TREC-DL'19 and TREC-DL'20 datasets. That is, we retrieve documents for given query using a search system and evaluate it using nDCG@10 obtain ground-truth asscssment of performance. Then, we use a QPP mothod to prodict the performance and evaluate it against the ground-Gruth assessment using three correlation measures, Spearman's correlation, Pearson correlation, and Kendall's Tau.
The effectiveness of a QPP method is a function of the underlying retrieval system. Since it was unclear from the original study which system was used to compute the ground truth performance, we experiment with multiple search systems.
In addition to the model itsclf, we experiment with three retricval models independent of the MRL model, measure how well the QPP method generalizes. We include traditional lexical retriever (BM25), simple dense retriever (DPR), and an effective dense retriever (TAS-B). We use the following baselines used in the original study:
SCQ (Zhao et al.] 2008): computes the similarity between query and the corpus for each query term based on the frequency of occurrence of the term in the corpus.
considers the variance or dtavaand of the term weights of each query term, based on the documents in which the term occurs.
VAR (Carmel & Yom-Tov 2010):
IDF CCammel & Yom-Tov 2010 based on the inverse document frequency of cach query term.
PMI (Hauff 2010 is a predictor that computes the pointwise mutual information, assigning high scores for frequently co-occurring query terms. Given all possible query term pairs, cither the average or the maximum can be used as the predictor.
For SCQ, VAR, and IDF, the scores computed at the query term level and then aggregated using either summing, averaging, or taking the maximum of each score. We report each of these aggregations in the results.
QPP for MRL. Zamani & Bendersky (2023) mention that the norm of the variance Σg/ is used to compute the predicted performance. We interpret this statement as using a function of the covariance, and use the negative norm -I Eg I, because the predicted variance should increase for queries that are difficult, rather than decrease. For instance, typographical error in the query makes it more difficult to address than
7

---

Published in Transactions on Machine Learning Research (01/2025)
2024.. leading to lower performance compared to clean query. Similarly, an OOD query could also result in poorer performance on average compared to an ID query. From the QPP perspective, model should therefore assign lower predicted performance for these we show that this intuition holds empirically.
"clean" query (Sidiropoulos & Kanoulas 2022
types of queries, motivating our choice. In Section 5.2.1
4.4 Implementation details
We train MRL for 200K steps. In each step, we optimize the distillation loss (Eq. 12 using a batch of queries, one positive passage per query, and 30 negative passages per query; of the negative passages are mined with BM25, and 25 are mined with the student model. With that setup we use batch size of 15 queries - the maximum that can fit in a 40GB A100 GPU, given the size of our model. We set the maximum length for queries and passages to 32 and 256 tokens, respectively. We initialize the dense retriever student model with the official TAS-B checkpoint, and we set as the leacher model the ma-marco-MinnLM------ cross-encoder that is publicly available on HuggingFace. We use an Adam optimizer with a learning rate of 10-6, and linear learning rate schoduling with warm-up for 10% of the training steps. The / parameter for softplus is set to 2.5. For MRL, the mean and variance are obtained by passing the [CLS] token and [VAR] token respectively through fully connected projection layers. The MRL models reported use means and variances projected down to 383 (= -1.
Since MARMO does not include a validation set, we split the training set into a validation (6890 queries) and a training set. The parameters above were selected after a hyperparameter search with the validation set performance used to pick the best model. Refer to Appendix L for the full set of hyperparameters. We use the Tevatron toolkit (Gao et al., 2023 to train the models and the pytrec_ _evall library (Van Gysel & de Rijke 2018) to evaluate the retrieval performance. Finally, our QPP baselines are based on an existing implementation by Meng al. 2023
5 Discussion
We organize the discussion section around retricval experiments in Section 5.1 variance vectors in Section 5.2 and the results of the ablation study in Section 5
the investigation of the
5.1 Reproducing the retrieval results
In this study, we work under a memory-limited setup an academic computational budget with constrained access to a single GPU. We start by testing whether we can obtain the results reported in the original study under these memory constraints. We report the results in Table [ where DPR, CLDRD, and MRL are our implementations of the original methods. We report results for TAS-B by using the official pre-trained checkpoint, which also serves CLDRD and MRL initialization. This way, between the different methods. We include the original numbers in the lower group in Table [ At this point, we want to underline that for MRL we use the corrected KT. formulation we presented in Section 3 for our experiments. Our decision to do so is grounded in the belief that the original study's authors also used this formulation in their implementation and that the formulation with the mathematical error is a typographical mistake their paper. We arrived at this conclusion based on our preliminary experiments, which yielded a dramatically low retricval performance (i.c., MRR@10 was 0.229 for MS-MARCO) when following the wrong formulation.
ensure fair comparison
We first focus testing whether are able to replicate the results for CLDRD, the main competitor of MRL. As shown in Table D our experimental results affirm the state-of-tho-art retrieval performance (for single-vector dense retrievers) of CLDRD. Furthermore, our findings validate the original study regarding its ability to enhance the performance of TAS-B, both in the ID and OOD scenarios. We consider this a successful replication despite the slight discrepancy the results. The reason for not obtaining the exact same results can be attributed to different development toolkits, hardware, implementation details not present in the original work, etc.
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Reproduction results of MRL on our nemory-limited setup. The Reproduced section contains the reproduction results, while the Reported section contains the results reported in the original study. The lower half of the table reports the results of contemporary dense retrieval methods. Note: MRL remains, to date, the only work with uncertainty estimation capabilities in dense retrieval, and the methods reported in the lower half are simply for reference.
Model
MS MARCO
TREC-DL'19
TREC-DL'
SeiFact NDCG010
FIQA NDCG010
TREC-COVID
CQADupStack
MR
MAP
NDCG610
MAP
NDCG010
NDCG010
NDCG610
Reported Regroduowd
DPR TAS-B CLDRD
.312
.319
.649
.345
.625
.356
.474
.231
.500
.266
344
.31
.721
.396
.685
.30
.643
.301
.481
.333
.378
.383
.727
.118
.670
.116
.627
.308
.608
.327
MRL
3333
378
.681
TAS-B
344
331
.717
.447
.085
.455
.643
.300
.881
3344
CLDRD
382
386
.725
.453
.687
.465
.637
348
571
327
MRL
.333
402
.738
.122
.701
.179
.683
.31
668
311
Recent Work Dense Retrieval (For Reference)
SimLM Wang
2025
.411
.714
.097
ADAM Tao et m.
222
.11
.311
.594
.315
.300
Llama2 Voc 0c
2024
.311
.734
.729
.748
.485
.869
.432
Regarding the reproduction of MRL, as shown in Table I we were unable to obtain the same results as the original study. We observe a drop performance across all reported metrics for both the ID and OOD datasots. For instance, in the case of FiQA, our implementation yiclded an NDCG@10 of 0.291, which is lower than the original study's reported value of 0.371. Furthermore, we could not find the trends that were reported in the original study. Specifically, the original study showed that MRL outperformed both TAS-B and CLDRD in ID and OOD scenarios. In contrast, in our work, MRL achieves similar performance to CLDRD in the ID datasets. A similar trend holds for the OOD dataset, except for TREC-COVID, where CLDRD outperforms MRI. with a substantially higher score of 0.608 compared to 0.497 for MRL. Upon comparing MRL with TAS-B, we have mixed results, with MRL matching or outperforming TAS-B for some datasets and metrics, but faring worse for others. For the OOD datasets, MRL surpasses TAS-B only for TREC-COVID and CQADupStack.
We stress that our goal in this work is not to replicate MRL; exact replication is not possible since the original manuscript does not specify several design choices - the best hyperparameters (i.e., learning rate, softplus 3, number of training steps), the cross-encoder model that was used as a teacher, as well as the number of negative passages per query in a batch. We aim to reproduce MRL under our memory-limited setup, which facilitates fair comparisons against the baselines. From our experimental results, we conclude that although MRL is a competitive approach in our setup, the multivariate representations do not boost the retrieval performance. Furthermore, unveil that MRL cannot consistently outperform its competitors when evaluated under fair comparisons. However, different from CLDRD, MRL produces variance that be used in downst ream tasks. We investigate the utility of this predicted variance in the following section.
5.2
Analyzing the variance
We analyze the predicted variance in three experiments: query performance prediction experiments (Section 5.2.1
experiments with typos, and retrieval experiments with alternative encoding schemes in Section 5.2.2 The first is a replication of the QPP experiments included in the original paper, while the latler two are additional analyses.
5.2.1 Query performance prediction
The results for the QPP experiments are plotted in Figures 1 and 2
We also report the MRL results separately in Table which also includes the reported numbers in Zamani Bendersky (023). As mentioned previously, we used four reference models because the original study did not report which model was used, and also to see if MRL generalizes to different reference models. We note that the original paper does not mention how the norm is used in computing the predicted performance in our experiments, we use the
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: QPP results for MRL for four reference models. While MRL does perform well for TREC-DL'20 for BM25, it fails to do for TREC-DL'19, The opposite is true for DPR. For both TAS-B and MRL reference models, MRL is more consistent but fails to reach the reported performance (bottom row). Furthermore, MRL is outperformed by simple baselines in most comparisons.
is evident from Figures and 2
TREC-DL 19
TREC-DL 20
S-p
P-p
K-r
S-p
P-p
K-T
BM25
-0029
-.034
-.014
.275
.296
.190
DPR
.233
.171
.155
-.051
-.013
-004
MRL
TAS-B
.204
.196
.138
.177
.166
.117
MRL
.171
.182
.134
.157
.185
.106
MRL (reported)
.271
.259
.272
.298
H076
106.4
155
178
UP1
4.2
6.24
124
0:11
-004
6:21
1110
6130
0.14
0.55
637
2333
0344
6.30
030
0.26
114
1101
430
0.35
540
0388
8:30
313
0.44
6:44
0383
6:00
115
6.0
630
M6
0.0
100
610
379
63m
6:04
8.0
110
6:94
110
310
630
2005
81
110
0144
8:00
6:16
11
6/64
300
090
0,24
335
084
0.00
030
3.271
6105
IDCed
0373
035
3038
6.00
323
M0
6:6
301
6:00
010
6:30
110
677h
8:00
3/11
6.0
DF-ma
0327
0.66
303
0.41
6/21
0353
4.96
1234
642
6.60
368
0.00
0.26
3.24
0346
6.66
0202
6.26
3:20
6.11
1156
0.00
503
4:08
311
3.08
6.96
-111
311
4:955
MCC
Figure 1: QPP results for TREC-DL'19 for four reference models (x-axis) and several methods (y-axis). Each subplot corresponds to different correlation metric: Spcarman's 's (S-p), Pearson's (P-p), and Kendall's Tau (K-T) correlations. MRL is outperformed by simple baselines regardless of the reference model metric.
negalive norm (using the norm flips the signs of the correlation) intuitively, higher uncertainty should result in lower performance (see Section 4.3
From Table 2 were unable to reproduce the numbers for MRL reported in the original paper, with any of the reference models. While MRL achieves higher than reported numbers for TREC-DI.22 with BM25, this result does not hold for TREC-DL'19, where there appears to be a random correlation. This trend is flipped with DPR as the reference and DPR, MRL is more consistent. However, the correlation obtained lower than the original study. But, how does MRL compare with the baselines?
MRL performs well for TREC-DL'19 but not TREC-DL'20 '20. For TAS-B
Figure I contains results for TREC-DL'19 19, with each subplot corresponding to the three metrics we used. Comparing MRL (top row) with the other methods, notice that at least one baseline outperforms MRL for each metric regardless of the reference model. In particular, at least one variant of the PMI bascline outperforms MRL. We remind the reader that these baselines are simple, non-parametric methods that use statistics derived from the corpora to compute the query difficulty.
For TREC-DI. 20, the results are more encouraging. If the reference model is MRL itself, we observe that MRL beats every other baseline for all three metrics. However, this result does not generalize the other three reference models, where similar trends to TREC-DI.'1 observed, with at least one baseline (among
10

---

Published in Transactions on Machine Learning Research (01/2025)
BMG
TAG-0
D20
MRL
DMS
X133
0.17
4.15
6185
6100
8100
-04
-4
13
348
0.30
4.91
353
4.80
6.43
6.36
6166
4.30
-32
P0
3135
0.75
6.91
8:90
8.30
334
0.34
615
613
115
066
080
PM
3198
0.13
6:01
1.00
013
3200
0.33
(111
C11
1114
0.64
0.0
6:6
609
6.81
114
0303
6:31
300
310
3338
0.71
4.009
4.03
(90
622
8114
3488
669
4.45
4123
633
k00
4:30
080
035
3188
9:4
6:066
0346
300
0221
6:00
3127
Figure 2: QPP results for TREC-DL'20. Simple lexical baselines outperform MRL when the reference is BM25, TAS-B, or DPR. With MRL itself as the reference, MRL achieves better performance, often by large margins.
PMI, VAR, IDF) beating MRL. We observe similar trends for three reference models BM25, TAS-B, and DPR, with MRL outperformed by at least one baseline.
In these experiments, we investigate the degree to which the MRL framework captures the notion of uncer- lainty by measuring proxy query difficulty. Intuitively, this suggests that for some datasets and reference models, higher uncertainty was indeed assigned to difficult queries. However, the positive correlation is not consistent across datasets. MRL fails to generalize to different reference models and datasets, achieving ran- dom correlation in many settings. Furthermore, MRL is perforrreed by simple non-parametric baselines in most comparisons. The lack of consistent and strong correlations suggests that MRL is unlikely to be a strong and consistent predictor of query difficulty in our experimental setup. Motivated by these results, we explore what the predicted variance captures the next section.
5.2.2 Does MRL capture uncertainty?
We outline two additional experiments investigating the predicted variance beyond the original paper: (a) contrasting the predicted variance of corrupted and clean data and (b) experimenting with alternate encoding schemes.
Experiments with typographical errors. Here, we consider an analog to the QPP experiments above, but instead of retrieval difficulty, examine if the model is sensitive to data distribution shifts instead. Inspired by works in the vision domain (e.g., Warburg et al. 2021 2023 we argue that a model should assign higher variance to corrupted or O0D data compared to clean or II) data.
We experiment with the DL-Typo (Zhungg & Zuccon 2022 dataset that contains 60 query pairs accom- panied by relevance assessments. Fach pair consists of real user query with typographical errors and its corresponding version where these errors have been corrected. For instance, the corrupted query "what is acid reflex" and its typo-free version "what is acid reflux". Given these data, we compute the norms of the predicted variance and plot their distributions. Suppose the model indeed models uncertainty accurately. In that case, expect (a) clean data should be assigned lower variance compared to corrupted data, (b) the distributions of the clean and corrupted data are well separated, and (c) the differences between the clean Ecorrppeod |, should be negative.
and corrupted data, i.e., Erlean
We plot these distributions Figure 3 While we expected more variance to be assigned to the corrupted data, the opposite is true. There is some separation observed between the two distributions the distribution of the corrupted data ("typo") is to the left. of the clean data. The right plot underscores this result, as the
11

---

Published in Transactions on Machine Learning Research (01/2025)
clean typo
clean typo
15.0
Count
10.0
Density
7.5
5.0
2.5
0.0
6.0
6.6
7.0
0.0
0.4
0.6
Figure 3: Uncertainty of clean and corrupted data: We plot the distributions of the norm of the predicted variance of the clean and corrupted data on the left. MRL assigns lower uncertainty to the corrupted data compared to the clean data. On the right, we plot the distribution of |elean - Ecoruptted which mostly positive (i.e., right of the red-dotted line). Contrary to expectations, MRL fails assign higher uncertainly to corrupted data.
Table 3: Performance of different encoding schemes for the MRL model. The first row uses the original equations proposed in the original paper, whereas the second row includes our corrections.
Encoding
MARCO
TREC-DL'19
TREC-DL
SciFact
FIQA NDeGa10
TREC-COVID
CQADupšlack ND0Ga10
MRR310
MAP
NDOGR10
NDCGR10
MAP
NDOGG00
NDCGR10
Zamani & Bendersky 2022
.229
236
.525
.255
.186
.259
.096
.183
[Pq.
Ours (Eq.
10
.333
378
.707
462
.681
.661
.591
.291
.497
.37
Mean
.279
-285
.624
3344
.557
.311
.382
.168
.145
.187
Eelean Eerrupted distribution is mostly positive. This suggests that contrary to expectation, the model predicts higher variance for clean data. We expand this analysis by examining the role of the predicted variance in retrieval relevance.
Encoding. MRL produces both a mean and variance for querics and documents. In Table D we reported retrieval results using the encoding scheme which enables retrieval using the KL divergence, Lee,, documents are encoded and indexed with Eq. T and queries with Eq. 10 the variance only captures uncertainty, we argue that the difference retrieval performance when only the mean used should be much lower than the performance obtained with this encoding scheme. However, Table 3 shows, this is not true. Comparing the encoding scheme (second row) with using just the mean (third row), we see that performance drops sharply across all datasets, especially for the OOD datasets.
The performance drop could be explained partly due to the way the models were trained. Since the full KT, loss (Fq. 4 was used in training the model, it may not be equipped to perform retrieval using just the mean. However, intuitively, we expect the mean to model relevance and variance to model uncertainty, which means that the retrieval performance should not be drastically different when only the mean used for retrieval. The drastic drop suggests that the model may be instead using the predicted variance vectors as signal for relevance. This is the core difference between a method such as MRI and Bayesian method: variance in MRL is not statistical variance, ie., it does not express deviation from the mean prediction. Instead, variance in MRL is a deterministically estimated quantity that minimizes distance objective.
In this section, we examined the MRL model consistently predicts notion of variance that reflects a notion of uncertainty defined by either query difficulty or sensitivity to data distribution shifts. We find that the QPP results inconsistent, and against initial expectations, the model assigns higher uncertainty
12

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: Ablation study on the different components of the MRL framework. The first part of the table examines the influence of initialization (i.c., TAS-B), loss function (i.c., Listwise KD), and multivariate representations for queries and passages MRL performance. The second part explores the impact of training MRL using the curriculum learning framework of CLDRD on performance.
MS MARCO MRRAID
TREC-DL'19 TREC-DL'20
SeiFact NDCG010
FIQA
TREC-COVID NDCG&10
CQADupStack NDCG810
Variant
NDOG010
NDCG@10
NDC0610
MRL
Multivariate reprosentation
TAS-B
.227
Listoiie
Teacher raw scores
Qrels Negative Minlig
2
Multivariate repremntation
.355
.719
.686
.637
.306
.618
.334
Vector represeniation
3 TAS B
.356
.693
.677
.567
.264
.536
-320
DissiBERT
.328
.629
.644
.498
.245
.473
.268
Chees-nntroyy
MRL-CLDRD
Teacherr zaw scores
Quela negetive mining
.375
.721
.667
.605
.293
.510
.320
Teacher constructe the batch
Tracher panurilabbll
CLDRD
Tooscherr zow acords
+Quels & negutive mining
Multivariate representations
.378
.727
.670
.627
.008
.608
.327
Teacher constracts the batch
Teaacher pseudalabels
Voctor representations
to corrupted data. In addition, experiments with encoding using only the mean suggest that the variance seems to model relevance since performance drops drastically when only the mean is used for retrieval.
5.3 Ablation study
Even though the MRL performance reported in the original paper was not reproduced under our experimental sctup, MRL remains framework that can produce highly effective dense retriever even under memory constraints. MRL consists of several components: multivariate representations, knowledge distillation, model initialization from pre-trained dense retriever, and batch construction strategy. Thus, it needs to be made clear how much each component impacts effectiveness. We expand on the original paper's findings by studying each component's contribution to retrieval performance through experimentation with various MRL variants.
Multivariate representations. MRL represents queries and passages as multivariate distributions and uses negative multivariate KL divergence to compute similarity. First and foremost, we want to understand how much the multivariate representations contribute to the overall effectiveness of the retriever. In order to test this, we conduct an experiment where we substitute multivariate representations with vector representations and compute similarity via the dot product. We find that multivariate representations do not lead to a higher retrieval performance; in contrast, we observe a small decrease in performance when used (see rows 1 and 2 in Table
Model initialization. Since MRL is initialized with TAS-B, pre-trained, highly effective dense retriever, it is important to examine the impact of its initialization its downstream performance. To test this, we use DistilBERT as the initialization instead. When comparing row against row 3 in Table g we see that it is possible to train a competitive model with a DistilBERT initialization; however, it does not achieve state-oftthe-art performance.
Loss function. MRL can be trained using supervisod contrastive learning by minimizing a softmax cross- entropy loss Karpukhin et al. 2020), instead of training with knowledge distillation. In row 4 of Table we test this alternative and find that following knowledge distillation training scheme is crucial for training
13

---

Published in Transactions on Machine Learning Research (01/2025)
005
0.6
No
No 1Bh
TRECI00 TO (M00GE00)
0.6
No IBN
0.3
150
AMBARRC (MAP))
(0977)
0.4
TeCCOL
203
400
200 423
200
490
200
300
203
80
290
402
Figure 4: Impact of batch size on the MRL performance for TD datasets. IBN and No IBN indicate training MRL with in-batch and no in-batch negatives, respectively.
IB%
0.4
08
Siled (NDCG891
0.5
T124 (NOCORI8)
CGn (ND000800
0:0
0/2
0.3
003
000
0.0
0.5
Figure 5: Impact of batch size on the MRL performance for OOD datasets. IBN and No IBN indicate training MRL with in-batch and no in- in-batch negatives, respectively.
effective retriever. This result is in line with recent work in dense retrieval that have shown the efficacy of knowledge distillation training over supervised contrastive learning (Hofstätter et al. 2021 Lin et al., 2021]
Negatives from the student model. The MRL framework uses the student model sample hard negatives (sec Section 3.2 This can be achieved by using recent checkpoint from the student model and updating ANN index of the corpus representation used for negative sampling. The encodings of the documents in the corpus can be updated multiple times during training, allowing us to sample new hard negatives from the ANN index more than once. In our reproducibility work, we update the ANN index only once. However, we also experiment with updating the ANN index multiple times (i.c., every 75k steps). In our preliminary experiments, updating and sampling from the ANN index more than once does not significantly increase performance: MRR@10- .374 for MS MARCO, NDCG@10 .714 for TREC-DL'19 and IDCG@10 .685 for TREC-DL'2 '20 (these scores can be compared against the first row in Table 4.. We believe this can be attributed to the availability of high-quality hard negatives from the first update itself, since the student model is initialized with TAS-B.
Training strategy. CLDRD, which is the primary competing approach of MRL, uses a different training setup when training with listwise knowledge distillation. In particular, CLDRD constructs the training batch using the teacher model to produce the data, the teacher's relevance judgments have the form of pseudo-labels (w.r.t. controlling term 1 Hyy (d) vy (d') in Eq. 12h, does not use in-batch negatives, and it uses curriculum learning (see Section 4.2 and Appendix C for more details). In contrast, MRL constructs the training batch via ground-truth query relevance judgments (i.e., qrel file) and negative mining, and the teacher's relevance judgments are the raw scores from the model. We procced with using CLDRD's training setup to train MRL; we refer to this model variant as MRI-CLDRD. This approach ensures a fair comparison with CLDRD and allows us to attribute any performance increase solely to MRL's multivariate roprosentations. When we compare rows and Table 4 we observe that MRL can slightly benefit from incorporating the training setup of CLDRD. Furthermore, by comparing rows 5 and 6, we notice that the multivariate representations cause slight decrease in performance, consistent with the previous result.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Retrieval results for MRL when trained with 512 batch size and for different numbers of training steps.
MS MARCO
TREC-DL'19 NDCG@10
TREC-DL'20
SciPact NDCG810
FIQA NDCGa10
TREC-COVID
CQADupStack
Steps
Training time (hours)
MInI110
NDCGa10
NDCG310
NDCG310
5880
.312
.680
.641 .656
.365 .517
.226
.407
.278
11760
162
.330
.693
.249
.437
.293
5.4 The Effect of Batch Size MRL Performance
In this section; explore whether increasing the batch size can achieve the reported retrieval results of the original paper, with the following goals: (i) we study the impact of batch size on the performance of MRL, and (ii) we unveil to what extent the reported results of the original work are due to the use of large batch size. Note that the original study uses a batch size of 512. In contrast, we use a batch size of 15, owing to our memory-limited, single GPU sctup (i.c., an academic computation budget). In our implementation, a batch size of 512 would require around 34 A100 GPUs, which is unfeasible given our academic computation budget.
We adopt memory reduction, gradient caching method to resolve this limitation. Introduced by [ao et al.) (2021), GradCache allows us to train models using batch-wise contrastive loss with large batch on single CPU The original study empirically demonstrates that GradCache can reproduce state-of-the-art results of dense retriever originally trained on multiple GPUs with a large batch size on a single consumer- grade GPU, albeit with a significant increase in training time. We experiment with the following batch sizes: 15, 32, 64, 128, 256, and 512. Since there is signíficant additional training time due to GradCache's computational overhead, we train all models for the same number of epochs. We experiment with MRL trained with in-batch negatives (as proposed in the original paper) and without in-batch negatives.
As Figures and show, changes in batch size have smaller impact MRL when trained without in-batch negatives rather than with in-batch negatives. We hypothesize that the negative trend as the batch size increases is linked to an increase in the number of <hard-negative, easy-negative> passage pairs contributing to the loss where negatives are passages sampled from BM25 and the student model while easy-negatives are the in-batch negatives. As the batch size increases, due to the use of in-batch negatives, there is an increase in <positive, casy-negative> passage pairs and an even greater increase in <hard-negative, easy-negative> pairs. The latter demands the dense retriever to recover finer-grained distinctions between the passages. Therefore, it is possible that as the batch increases, the number of such pairs also incrcases; consoquently, the training steps should also increase so that the model can learn these fine-grained differences.
To validate this, we train MRL with a batch size of 512 for more training steps to examine whether that could lead to better performance. As we can see in Table 5 training for more steps can increase retrieval performance on all datasets. Thus, we hypothesize that the reported results in the original paper might be outcome of training with 512 for more training steps. Testing this hypothesis on our memory constraint environment, which consists of a single GPU and employs GradCache to simulate training with a large batch, is not feasible; as we show in Table 5] training MRL with batch size 512 on a single GPU with GradCache requires ~8 days for 11K steps. At this point, we need to underline that such a result would suggest that the trends presented in the original paper may have been inffuenced by the fact that MRL was training with a significantly larger batch size compared to its baselines (i.e., in the original work, MRL is trained with a 512 batch, while CLDRD with a batch size of 8).
5.5
A simple extension to reduce the hyperparameter search space
The MRL model produces a mean (Mo PD) and a diagonal co-variance matrix/variann vector (ED, Eq / ao, aD) given text input. Using the raw co-variance without ensuring that it is positive (and semi-definite)
1 Even though we can avoid encoding extra negatives with bathatch negatives, the loss of each sample conditioned every sample in the batch and therefore it is required to fit the entire large batch into GPC memory
15

---

Published in Transactions on Machine Learning Research (01/2025)
Table 6: MRL Variant: LogVar vs Softplus: LogVar performs similarly to Softplus, but with no hyperpa- rameter tuning required. Based two-tailed paired t-test (p 0.05); there are no statistically significant differences between the models results.
Variant
MS MARCO
TREC-DL'19 NDCG@10
TREC-DL'20 NDCG@10
SciFact NDCG@10
FIQA
TREC-COVID NDCG010
CQADupStack NDCG@10
MRR
NDCG@10
Softplus
.373
.707
.681
.591
.291
.497
.327
LogVar .373
.709
.679
.590
.296
.471
.327
would make the model produce invalid values for the variance c.g., negative variance. To ensure positivity, Zamani & Bendersky (2023) pass the raw values through a softplus activation function. If is the output of the encoder (i.e., prior to passing through the softplus), then the variance is obtained as follows:
1 log(1 + exp(3 a)):
= Softplus(a)
(13)
which ensures that the predicted variance is positive. The original study finds that retrieval performance is robust to the hyperparameter B in the softplus function. An alternative to using the softplus function (that predicts variance), is to assume that the output is the log-variance, without passing it through any activation function. Therefore, Equations 1 and become:
(Mq: log ΣQ) = fo(g).
(14)
(HD, Ep) = fo(d).
(15)
As such, these values are exponentiated element-wise prior to use e.g., when plugging it into Eq. 11 and Eq. I0 This alternate method renders tuning the 3 hyperparameter unnecessary. We term this variant "Log Var", and validate this approach through experimentation.
This experiment compares the MRL model trained using the softplus activation with the LogVar variant. We stress that we do not expect improved performance; this a variant that functions similarly to the original model. The results, reported in Table show that the Log model achieves similar performance across datasets and metrics. We also conducted an ablation with different initializations as well without distillation and observed the same trend. In essence, the LogVar model provides results similar to those of the Softplus model without requiring an additional hyperparameter.
5.6 Limitations of our study
Lack of details. As mentioned previously, our reproducibility effort. was hampered by lack of crucial details in the original paper, such as the composition of the batches used while training the model, or which underlying cross-encoder was used as a teacher. Therefore, it is unclear whether the impressive performance improvements reported in the original paper are the result of some crucial implementation detail that was omitted. However, we tried to mitigate this limitation through ezhaustive experimentation. The most promising part of that experimentation presented in our ablation study, but it still constitutes fraction of all model configurations that we attempted in order to get closer to the reported performance.
Batch size. As mentioned in Section 5.4 we rely on GradCache (Gao et al. 2021 to support training MRL with larger batches. As a result, training time when training with a larger batch size does not decrease training time since we are still limited to a single GPU. In contrast, training time increases significantly due GradCache's overhead of calculating and storing gradients of representations. Due to this limitation, we cannot fully explore training MRL with batch of 512 for longer (i.e., more training steps).
6 Conclusion
In this paper, we reproduce the multivariate representation learning framework by Zamani & Bendersky 2023) in memory-constrained environment. After addressing likely typographical error in the original
16

---

Published in Transactions on Machine Learning Research (01/2025)
paper's derivations, show that in a fair comparison, MRL achieves similar performance to bascline models. Crucially, the main claim that MRL significantly outperforms baseline models in out-of-domain datasets does not hold in our experimental setup. While MRL does not outperform baselines, we maintain that it remains a competitive retrieval model. We also conduct an extensive analysis of the predicted variance. Against our expectations, our analysis reveals that. the variance vectors not consistently express uncer- tainty. To add to the results of the original study, we conduct a thorough ablation study, investigating the impact of the different components of the MRL framework: (i) multivariate representations, (ii) distillation, conclude that multivariate representations do not
and (iii) model initialization. Through this study, improve or harm performance significantly, and knowledge distillation is the primary source of improvement. In addition, we show that models trained with an increased batch size using gradient caching methods and without increasing the number of training epochs are impacted negatively by the presence of in-batch negatives.
While we are unable reproduce the results, we maintain that the idcas the original paper are very valuable to the community. Prior this paper, uncertainty was only utilized in ranking, not first-stage retrieval. The decomposition of the KL divergence first-stage retrieval. This implies that any model that produces distribution for queries prrries/pace passaggs can be used in this framework, even if it was not trained with the objective function outlined in the paper this is a promising direction for future research. Future work could further consider incorporating document uncertainty to the framework, e.g. for post-retrieval QPP.
dot-product enables incorporating uncertainty in
17

---

Published in Transactions on Machine Learning Research (01/2025)
References
David Carmel and Elad Yom-Tov. Estimating the Query Difficulty for Information Retrieval. Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool Publishers, 2010.
Sanghyuk Chun, Scong Joon Oh, Rafael Sampaio de Rezende, Yannis Kalantidis, and Diane Larlus. Prob- abilistic embeddings for eross-modal retrieval. In IEEE Conference on Computer Vision and Pallern Recognition, (CVPR), 2021.
Daniel Cohen, Bhaskar Mitra, Oleg Lesota, Navid Rekabsaz, and Carsten Eickhoff. Not all relevance scores are equal: Efficient uncertainty and calibration modeling for deep retricval models. In ACM STGIR Conference on Research and Development in Information Retrieval, 2021.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. arXiv preprint arXiw:2003.0782) 2020.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. XrXiv preprinl ar Xrv::2102. 076622 2021.
Yulan Feng, Shikib Mehri, Maxine Eskénazi, and Tiancheng Zhao. 'none of the above': Measure uncertainly in dialog response retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, (ICML), 2016.
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size un- der memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP, RepLANLP@ACL-UCNNL 2021.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Tevatron: An efficient and flexible toolkit for neural retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Relrieval, (SIGIR), 2023.
Claudia Hauff. Predicting the effectiveness of queries and retrieval systems. SIGIR Forum, 44, 2010.
Ben He and Iadh Ounis. Inferring query performance using pre-retrieval predictors. In String Processing and Information Retrieval, SPIRE, 2004.
Ben He and Iadh Ounis. Query performance prediction. Information Systems, 31(():585-55) 594, 2006.
Maria Heuss, Daniel Cohen, Masoud Mansoury, Maarten de Rijke, and Carsten Eickhoff. Predictive uncertainty- bias mitigation in ranking. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, (CIKM), 2023.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Tm- proving efficient noural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2010. 02060002666..6 2020.
Sebastian Hofstätter, Sheng- Chich Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. Cqadupstack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium (ADCS), ADCS '15, 2015.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535 547, 2021.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, (EMNLP), 2020.
Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao, and Defu Lian. Llama2vec: Unsupervised adaptation of large language models for dense retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3490-3500, 2024.
Sheng-Chich Lin, Jheng-Hong Yang, and Jimmy Lin. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP, RepLANLP@ACL-LUCNH
2021, Online, August 6, 2021, 2021.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www 18 open challenge: Financial opinion mining and question answering. In Com- panion Proceedings of the The Web Conference 2018, 2018
Chuan Meng, Negar Arabzadeh, Mohammad Aliannejadi, and Maarten de Rijke. Query performance pre- diction: From ad-hoc to conversational search. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2583 2593, 2023.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generaled machine reading comprehension dataset. In NewrIPS Workshop on Cognitive Computation: Integrating newral and symbolic approackes, 2016.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108, 2019.
Georgios Sidiropoulos and Evangelos Kanoulas. Analysing the robustness of dual encoders for dense retrieval against misspellings. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.
Georgios Sidiropoulos and Evangelos Kanoulas. Improving the robustness of dense retrievers against typos via multi- contrastive learning. In Advances in Information Retrieval 46th European Conference on Information Retrieval, (ECIR), 2024.
Chongyang Tao, Chang Liu, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, and Daxin Jiang. Adam: Dense retricval distillation with adaptive dark examples. In Findings of the Association for Computational Linguisties ACL 2024, pp. 11639 11651, 2024.
Christophe Van Gysel and Maarten de Rijke. Pytrec eval: An extremely fast python interface to trec eval. In SIGIR. ACM, 2018
Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. In 3rd International Conference Learning Representations (ICLR '15), 2015.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushnan William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. Trec-covid: constructing a pandemic information retricval test collection. In ACM SIGIR Forum, volume 54, 2021.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Han- naneh Hajishirzi. Fact or fiction: Verilying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Jun Wang and Jianhan Zhu. Portfolio theory of information retrieval. In ACM SIGIR Conference Research and Development in Information Retrieval, 2009.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Frederik Warburg, Martin Jørgensen, Javier Civera, and Seren Hauberg. Bayesian triplet loss: Uncertainty quantification in image retrieval. In IEEE/CVE International Conference on Computer Vision, (ICCV), 2021.
Frederik Warburg, Marco Miani, Silas Brack, and Søren Hauberg. Bayesian metric learning for uncertainty quantification in image retricval. In Advances in Neural Information Processing Systems (NeurIPS). 2023.
Hamed Zamani and Michael Bendersky. Multivariate representation learning for information retrieval. ACM SIGIR Conference on Research and Development Information Retrieval, 2023.
Hansi Zeng, Hamed Zamani, and Vishwa Vinay, Curriculum learning for dense retrieval distillation. In SIGIR 22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.
Ying Zhao, Falk Scholer, and Yohannes Tsegay. Effective pre-retrieval query performance prediction using similarity and variability evidence. In European Conference on IR Research, ECIR, 2008.
Jianhan Zhu, Jun Wang, Ingemar J. Cox, and Michael Taylor. Risky business: modeling and exploit- ing uncertainty in information retricval. In ACM SIGIR Conference Research and Development in Information Retrieval, 2009.
Shengyao Zhuang and Guido Zuccon. Characterbert and self-teaching for improving the robustness of dense retrievers on queries with typos. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retricval, 2022.
20

---

Published in Transactions on Machine Learning Research (01/2025)
A Dot product formulation in the original paper
We include the original dot product formulation below of the KL divergence for completeness. Consider the term tr{Ep) Σe), the source of the error. Zamani & Bendersky
2023 formulate this as:
II oig tr E Σa}
(16)
IT of
This differs from our version in Eq. formulation becomes:
6
This error is propagated throughout the next steps.
The KL
& KLD(Q| Σ
== io
2
k
Pig
2 k
2piglip
Pip
log oi in
(17)
2
2
ip
2
iD
==1
The equivalent dot product formulation, after taking the negative of the equation above gives the original formulation (Eq. (14) in Zamani & Bendersky (2023), with signs flipped for d):
q
ii Mig ................................ Pko Mig..... ..... Mko
2
2
(18)
(=1
201
2
d =
YD
(19)
oip
oko
oip
0kp
1==
where TD is equivalent in our formulation i.e., Eq. Here, 7, d E RIx(2++1) and qJ is equal to Eq. 17 Note that the final vector for d, the signs are flipped because the similarity function is the negative KL divergence.
B Dot product formulation of full KL divergence
We start from the unsimplisied version
Eq. that includes constants:
KLD(QIID) 2
Σ log o Σ logo of k
2uiq
(20)
ip
To formulate it as dot product, we the definition of the document prior, 7p, from Eq. define 7Q as:
but further
TQ log of
(21)
Then we can extend the vector representations in Eq. 10 and Eq.
11 to include 72:
- I, ,Q, of . .....
ono 2 ... Uko Pie..... ,
(22)
ā= =
7D, 1,
1
1
1
2/h1p
2/4kp
(23)
dip
k
0
kp
2
where 7.d' (36+22, Then,
DL = ( - k).
(24)
should precisely yield the KL divergence between the distributions of and D, defined in Eq.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Table 7: Statistics and Description of Evaluation Datasets. Number of tokens for average query/documn
lengths were computed basod on the bestilbert-bas--ueeeee tokenizer.
Name
Domain
#
#
length
avg. p length
MS MARCO Dev
Miscellancous
6,890
8,841,823
9.01
76.97
TREC-DI DL 19
Miscellaneous
43
8,841,823
9.02
76.97
TREC-DL 20
Miscellancous
54
8,841,823
9.22
76.97
Scifact
Scientific Document Retrieval
300
5,183
22.84
315.65
FIQA
Financial QA
648
57,638
15.59
177.11
TREC-COVID
Biomedical document retrieval
50
171,332
18.04
224.78
CQADupStack
Community QA retrieval
13,145
457,199
13.55
248.73
C
Training setup for CLDRD
One of the perks of training with knowledge distillation that it can leverage the incomplete relevance judgments that most. large-scale retrieval datasets suffer from. Hence, it rely only the teacher super- vision signal, ie., training data is generated and scored by the teacher model without human assessments. Furthermore, combining knowledge distillation with in-batch negative training can be impractical when the teacher is a computationally expensive cross-enae coss-enccddrr (Lin et al., 2021j. On the other end of the spectrum, even though the exact relevance scores produced by the teacher model do not impact the loss value (since the loss only considers the order of passages) they still play an important role in controlling which query-passage, pairs will contribute the loss (term (y,(() v (d") in the loss). When the raw scores from the teacher model used, all pairs contribute the loss, even when contrasting two irrelevant passages.
(see Section 3.2), the work by Zeng et al. (2022) work that initially proposed the listwise distillation loss used by Zamani & Bendersky (2023) and the primary competing approach of MRL uses pseudo-labeling to create the batch and does not rely on in-batch negatives. The authors use the following approach to compute the listwise distillation loss:
In contrast to the training setup used by Zamani & Bendersky 2023
Given query q, the passage set D, is constructed with respect to the top-k passages in the ranked list returned by the teacher model (reranking order). In particular, the first K passages in the ranked list returned by the teacher model are considered positive, the next K' are considered hard negatives, and the remaining K" soft negatives.
(d) is relevance label according to the group, passage d belongs to:
iff d is positive
(d) = O
iff d is hard-ncgative
iff d is soft-negative
where ry is the ranking position of the document given the query q in the teacher ranked list.
Datasets
In Table 7
we present the statistics of the datasets. Here, "p" and "q" indicate questions and passages, respectively. The length is tokens.
E
Hyperparameters
In Table g we detail our hyperparameter scarch space used throughout our experimental sctup. For MRL we search w.r.t. learning rate, 3, and negatives. The best parameters were: S = 2.5, Ir 5 x 10-6, negative passages from BM25, and 25 negative passages from the student. model. For MRL-CIDRD search w.r.t.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: Hyperparameter search space for the models we experiment with. Negatives are presented in the following format: ANN negatives, # BM25 negatives]
Parameter
Values
10 -, 1x 10-5
1x 10-6,
Learning rate
3 10 x 10-6 7x 10-6
8
0.5, 1, 2.5, 7.5
Negatives
[5, 25), [10, 20), [15, 15), [20, 10), [25, 5]
learning rate and B. The best parameters were: / = 2.5, and Ir = [5 x 10-6; 1 x 10-9, 1 x 10-9] for the three curriculum iterations. For CLDRD the best learning rates for each the three curriculum iterations were [7 x 10 x 10-6,3 x x 10- while for DPR the best learning rale was 7 10-6
Our academic computational budget allowed Table and the model using cross-cntropy without distillation in Table However, note that the model performance is quile robust to the choice of hyperparameters in our initial experiments miligaling this limitation to an extent.
perform a hyperparameter search only on the models in
23


Reviews:
[{"review_id": "3f8sJdNXbT", "reviewer": "Reviewer_2tbi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n1. The paper rigorously reviews the Multivariate Representation Learning (MRL) framework and corrects the error in the existing work, demonstrating that MRL does not outperform baselines, which contrasts the results they reported.\n2. They further spot that multivariate representation itself is not the main source of the quality improvement, but the knowledge distillation is.\n\nWeakness:\n1. It is not the critical weakness since this paper is mainly about reproducing MRL, but it could improve more by including the reproduced results of more recent works [1].\n\n[1] Zhou, Xixi, Gao, Yang, Jie, Xin, Cai, Xiaoxu, Bu, Jiajun, and Wang, Haishuai. \"EASE-DR: Enhanced Sentence Embeddings for Dense Retrieval.\" In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '24), Washington DC, USA, 2024, pp. 2374–2378. Association for Computing Machinery. doi:10.1145/3626772.3657925", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper rigorously walk through the Multivariate Representation Learning (MRL) framework and corrects the error in the existing work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness part.", "broader_impact_concerns": "No concerns regarding ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u7h2invTtJ", "reviewer": "Reviewer_A5fF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper corrects a mathematical error in the original MRL formulation and makes reasonable design choices where the original paper was unspecified.\n2) The paper provides a comprehensive ablation study that dissects the impact of different components of the MRL framework, offering valuable insights into what contributes to the effectiveness of the model.\n3) The paper openly shares the source code for the reproducibility study.\nWeaknesses:\n1) While the paper attempts to address memory limitations, the scalability of the MRL framework to even larger datasets or more complex models is unclear. The framework's performance might degrade or become computationally infeasible as the scale increases.\n2) Comparison methodology is slightly outdated, earlier than 2022, and it is suggested that it be compared with newer state-of-the-art methods.\n3) Despite the authors' best efforts, they were unable to fully reproduce the results reported in the original paper. This could be due to omitted details or unaccounted-for factors in the original work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper thoroughly investigates the reproducibility and performance of the multivariate representation learning (MRL) framework proposed by Zamani & Bendersky (2023) for dense retrieval tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Conduct a thorough analysis of the MRL framework's scalability. Provide a detailed discussion on scalability, including potential bottlenecks and strategies for optimization, to address the unclear scalability issue.\n2) Updated Comparison Method. Include comparisons with the latest state-of-the-art methods from 2022 and beyond. Justify the choice of new comparison methods and discuss how they differ from the older ones, highlighting any advantages or disadvantages.", "broader_impact_concerns": "There is no concern about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EbFR05hl9Z", "reviewer": "Reviewer_LzJ1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- very well written and easy to follow\n\n- rigorous and extensive empirical analyses of MRL\n\n**Weaknesses:**\n\n- this paper aims to reproduce results from a previous paper. Thus, it seems that this paper lacks novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to reproduce multivariate representation learning (MRL) under memory constraints. It has spotted an error in the original MRL paper and reproduced some results in a memory-limited setting, even though the original paper has not clearly specified some key algorithm parameters. This paper has also done a thorough ablation study, and open-sourced their code.\n\nThis paper has confirmed that MRL can have the state-of-the-art performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please explain the novelty of this paper.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xB91uO79mw", "reviewer": "Reviewer_FS2P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The authors have addressed the concerns of the previous reviewers in regard to the language used to describe the work of Zamani & Bendersky (2023). The tone of the paper has become more neutral and reserved. This is good, as there are many aspects of the reproducibility study that cannot be exactly the same as the original paper, as already discussed by the authors. \n2. The authors have introduced a new experiment section discussing the impact of large batch sizes. The study indicates that the performance of MRL heavily depends on large batch sizes and extended training times. \n3. The paper rectifies errors in Zamani & Bendersky (2023), likely to be unknown to the authors, that should be highlighted. While this is a potentially risky endeavour, as highlighted by reviewer iL1v, the impact will be a net-positive for the progress in the community. \n4. The writing, presentation, and experiments are of very high quality. \n\n### Weaknesses\n1. The study does not fully replicate the results of Zamani & Bendersky (2023) due to resource constraints, notably the inability to complete experiments at batch size 512. Therefore, some of the results (e.g. Table 1) should be interpreted with caution and the conclusions are left open-ended. \n2. There are errors in the discussion when referring to Table 1 in Section 5.1. See details below. \n\n### Minor errors\n1. Sec. 5.1 (page 9): \"Regarding MRL…. For instance, in the case of FiQA, our implementation yielded an NDCG@10 of **0.308**\". \n- Table 1 shows **0.293**, not 0.308 for MRL.\n2. Sec. 5.1 (page 9): \"A similar trend holds for the OOD dataset, except for TREC-COVID, where CLDRD outperforms MRL with a substantially higher score of 0.608 compared to **0.481** for MRL.\"  \n- Table 1 shows **0.497**, not 0.481 for MRL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper's contributions are as follows:\n1. The authors identified and corrected a key mathematical error in the original Multivariate Representation Learning (MRL) framework. This error occurred early in the formulation of KL divergence and may have propagated throughout the model’s mathematical foundations. The correction leads to more accurate retrieval results compared to the original formulation in the original work.\n\n2. The authors attempted to reproduce the MRL results using a memory-limited setup (1xA100 40GB). They found that the results could not be reproduced under these constrained settings, demonstrating that the performance of MRL heavily depends on large batch sizes and extended training times.\n\n3. Through an ablation study (Sec 5.3), the authors concluded that the performance gains attributed to the original MRL paper are largely due to knowledge distillation and model initialization, rather than the multivariate representations themselves. The multivariate aspects of MRL, which were intended to model uncertainty, did not significantly contribute to performance improvements.\n\n4. The analysis (Sec 5.2.2) showed that the variance component of MRL **does not capture uncertainty**, as initially proposed. For example, in Figure 3, the model assigns higher uncertainty to clean queries rather than noisy ones, contradicting the assumptions of the MRL paper. \n\n5. The authors propose an improvement to the MRL framework that reduces the hyperparameter search space by replacing the Softplus activation with a log-variance. This adjustment maintains competitive performance without the need for tuning specific parameters, thus enhancing the practicality of the model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see \"Minor errors\" in section above. I may not have captured all of these, so please have a careful read again to make sure the references to the updated Tables are correct.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to reproduce the results of Multivariate Representation Learning (MRL)  [1] and identify and correct an issue in the mathematical derivations in the original paper which occurred in the expression of the KL divergence. In addition, they show that under a limited batch size setting, MRL doesn't consistently outperform the baselines. The authors of the present paper credibly indicate that reproducing [1] under the original setting (with a batch size of 512) would be prohibitively expensive in terms of compute time. Ablation studies are performed to evaluate most of the components of the original model and a striking conclusion is the fact that, as reviewer FS2P mentions,  *the variance component of MRL does not capture uncertainty* and at best merely acts as additional modelling parameters. The true gains are instead derived from the knowledge distillation and initialisation. 


In addition, the authors introduce a novel training component, LogVar, which eliminates the need for the Softplus function in MLR variants by interpreting the model based $\Sigma$ as the component-wise logarithm of the variance. It is shown that Logvar is competitive with softplus despite not requiring an additional hyperparameter. 

The paper makes a solid contribution, has greatly improved during the reviewing process, and clearly deserves to be accepted.





[1] Hamed Zamani, Michael Bendersky. “Multivariate Representation Learning for Information Retrieval “. SIGIR 2023.
Decision: Accept (poster)