Retrieval-based Disentangled Representation Learning with Natural Language Supervision
Authors: Jiawei Zhou
Xiaoguang Li
Lifeng Shang
Xin Jiang
Qun Liu
Lei Chen
Venue: ICLR 2024 spotlight (2024)
Keywords: Disentangled representation learning
information retriever
sparse retriever
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We extend the functionality of lexical retriever to handle multi-modal data and harness this enhancement to facilitate disentangled representation learning.

Abstract: Disentangled representation learning remains challenging as the underlying factors of variation in the data do not naturally exist. The inherent complexity of real-world data makes it unfeasible to exhaustively enumerate and encapsulate all its variations within a finite set of factors. However, it is worth noting that most real-world data have linguistic equivalents, typically in the form of textual descriptions. These linguistic counterparts can represent the data and effortlessly decomposed into distinct tokens. In light of this, we present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employ a bi-encoder model to represent both data and natural language in a vocabulary space, enabling the model to distinguish dimensions that capture intrinsic characteristics within data through its natural language counterpart, thus facilitating disentanglement. We extensively assess the performance of VDR across 15 retrieval benchmark datasets, covering text-to-text and cross-modal retrieval scenarios, as well as human evaluation. Our experimental results compellingly demonstrate the superiority of VDR over previous bi-encoder retrievers with comparable model size and training costs, achieving an impressive 8.7% improvement in NDCG@10 on the BEIR benchmark, a 5.3\% increase on MS COCO, and a 6.0% increase on Flickr30k in terms of mean recall in the zero-shot setting. Moreover, The results from human evaluation indicate that interpretability of our method is on par with SOTA captioning models.

Published as a conference paper at ICLR 2024
RETRIEVAL-BASED DISENTANGLED REPRESENTATION LEARNING WITH NATURAL LANGUAGE SUPERVISION
Jiawei Zhou! Xiaoguang Li
Lifeng Shang
Xin Jiang"
Qun Liu
Lei Chen
The Hong Kong University of Science and Technology*
Huawei Noah's Ark Lab²
[jzhoubu, lei chen @est. hk [1 [lixiaoguang1] Shang. Li Lifeng, Jiang Xin, qun wwwwwhuwwwi..e..mm com
ABSTRACT
Disentangled representation learning remains challenging the underlying fac- tors variation the data do not naturally exist. The inherent complexity real-world data makes it unfeasible to exhaustively enumerate and encapsulate all its variations within a finite set of factors. In light of this, we present Vocabu- lary Disentangled Retrieval (VDR), retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employs a bi-encoder model represent both data and natural language in a vocabulary space, enabling the model to dis- tinguish dimensions that capture intrinsic characteristics within data through its natural language counterpart, thus facilitating disentanglement. We extensively assess the performance of VDR across 15 retrieval benchmark datasets, covering text-to-text and cross-modal retrieval scenarios, as well as human evaluation. Our experimental results compellingly demonstrate superiority of VDR over previ- ous bi-encoder retrievers with comparable model size and training costs, achieving an impressive 8.7% improvement in NDCG@ 10 on the BEIR benchmark, 5.3% increase on MS COCO, and 6.0% increase Flickr30k in terms of mean recall in the zero-shot setting. Moreover, the results from human evaluation indicate that interpretability of method is par with SOTA captioning models.
INTRODUCTION
Disentangled representation learning (Bengio et al. 2009; 2013 Higgins et al. 2017 Chen al. 2018) aims to identify the underlying factors of variations within data and correlate them to distinct units of the learned representation. Essentially, a well-disentangled representation independently captures underlying factors that explain the data, thereby facilitating explainability. controllability, and debugability machine learning. However, as pointed out by [Bnngio et al. (2013), a funda- mental challenge lies in defining set of factors that are sufficiently informative to represent the data while remaining independent of the tasks at hand. Despite extensive research efforts spanning several years, the journey to effectively address this challenge remains ongoing endeavor.
Given the absence an universal set of factors variations behind data, supervising disentangle- ment remains challenging. Early works (Higgins et al. 2017 Chen et al. 2016; Kim & Mnih 2018) employ autoencoder framework without supervision. aiming to impose constraints on VAE (Kingmal & Welling 2013) enhance the independence among latent factors, without explicitly defining their meaning. These methods are later challenged by Locatello et al. (2019a), revealing that unsuper- vised disentanglement is highly reliant inductive biases, randomness, and parameter choices. Subsequent research directions have shifted toward incorporating varying degrees of supervision. For instance, certain research (Locatello al. 2019b Gabbay al. (222I) explored supervising disentangiement using a small subset of labeled data from synthetic toy datasets. These efforts often revolve around tasks of controllable image generation, typically focusing on synthetic images (Reed) et al. 2015 Matthey et al. 2017) with limited explanatory factors such as object shape, color, and
'The code is available http://gthhuu.com//hoob////

---

Published as a conference paper at ICLR 2024
category. Nevertheless, the feasibility of applying these task-specific explanatory factors to real- world scenarios remains constrained. Another line of research employ relational information among data samples, such group-wise (Locatello et al., 2020; Shakerinava et al.) (2022), pair-wise (Chen & Batmanghelich, 2020), sequence (Bai et al. 2021 Li et al. 2022b Miyato et al. (022) and graph- based (Ma et al., 2019 Mercatali et al. 2022 Wang all 2022c) information, to weakly supervise disentanglement learning. However, acquiring annotations or structured data for these methods can pose challenges, potentially limiting their generalizability within real-world scenarios.
It is important to recognize that real-world data inherently complex and cannot be fully en- capsulated by a limited set of attributes. In practice, individuals often resort to natural lan- guage to describe and identify real-world ob- jects, as offers a versatile way to represent their diverse characteristics. These natural lan- guage counterparts can be tokenized into fi- nite vocabulary space, effectively serving proxy for capturing the inherent variation in the data.
sconble
fury
kitten
cute
cat
scoolt
This a cute cat.
Text Enceder
Figure 1: Illustration of retrieval-based frame- work. The color intensity reflects the higher val- ues along the dimension.
Motivated by this, we present Vocabulary Dis- entangled Retrieval (VDR), a simple yet effec- tive retrieval-based approach to drive disentan-
gled representation learning. In essence, VDR represents both data and their linguistic counterparts within a V Vl-dimensional lexical representation space, where each dimension corresponds to a token from the vocabulary V. The value along each dimension quantifies the semantic relevance between the input data and the corresponding tokens. By aligning the sparse representations of the data to those of their linguistic counterparts, VDR encourages the dimension-wing disentanglement on the lexical representations. Our methodology is thoroughly evaluated across 15 datasets, covering both text-to-text and cross-modal retrieval, as well as human evaluations. Experimental results demon- strate the consistent superiority of VDR over existing retrieval baselines with similar complexity, in terms both model size and training configurations. Furthermore, VDR achieves competitive results when compared more sophisticated retrieval methods that employ intricate techniques to reach state-of the-ar art performance.
In summary, our work technically improves the current lexical retriever and extends its functionality to handle multi-modal data. We utilize this progress to drive disentangled representation learning in the context of multi-modal data, with primary focus contributing to the broader field of explainable machine learning research. Our contributions can be categorized into two aspects. From disentangled representation learning aspects:
1. We showcase the fcasibility of harnessing natural language as a source of supervision for disentangled representation learning. Our methodology considers task-agnostic vocabu- lary as explanatory variables, highlighting its wide-ranging applicability and adaptability.
2. Through both human evaluations and retrieval benchmarks, we establish that the repre- sentations learned by our methodology exhibit rational dimensional values to capture the intrinsic characteristics of input data.
From sparse lexical retrieval aspects:
1. We pioneer the viability of embedding non-textual data into lexical representations, ex- panding the scope of lexical retrievers encompass multi-modal data from diverse datasets and tasks. This extension presents distinctive challenges, which we have identified and ad- dressed through innovative solutions.
2. Our approach demonstrates notable improvements in both text-to-text retrieval and cross- modal retrieval. It significantly surpasses existing dense and sparse retrievers with com- parable training cost while maintaining competitiveness with more sophisticated and ad- vanced retrieval baselines.
3. Notably, our retriever supports nonparametrie inference, eliminating the need of neural network forwarding during inference. This enhancement results in over 10x improvement in efficiency while maintaining strong effectiveness, making particularly valuable for low-resource applications.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
2.1 INFORMATION RETRIEVAL
Information retrieval Maaninng 2009 Mitra et al. 2018 Zhao et al. 2022) aims to find specific
targets p that fulfill certain information needs from a corpus based on given query q. This is typically achieved by utilizing the bi bi-encoder framework, which employs two independent encoders encode the query and target into vectors and measures their relevance as the inner product their representations:
sim(g. p) Eg(g) Ep(p)*
where sim(g, p) R is the similarity between and p, and E.(.), Ey()) are query encoder and target encoder, respectively.
Notation
We use q to indicate the textual query, while indicates the target, which could be either text or image. The variable encompasses both query and target data. Whenever we refer to input data v, we imply that both the query and target sides undergo the same processing pipeline.
Dense retrieval (Karpukhin et al. 2020) stands as a prominent technique in re- trieval applications, wherein the encoder embed the input into a dddimensional latent representa- tion, i.e, Fetens((:) (x) E Rdd.
Dense Retrieval
Lexical Retrieval Lexical retrievers (Bai al. 2020 Formal et al. 2021b) encode the data into a sparse lexical representation denoted as Etezical (x) V(x) C G(r) where @ is element-wiss wise multiplication. Here, V - R is weighting function that encodes input data into /V/- dimensional vector with positive values. These values signify the semantic relevance between tokens in the vocabulary and the input æ. On the other hand, G : - {0,1) is a gating function that generates binary vector for sparsification.
2.2 DIMENSION- WISE SUPERVISION ON SPARSE REPRESENTATION
The primary goal lexical retriever is learn weighted distribution on vocabulary represent the input data. While there is no ground truth for such distributions in nature, the dimension wisse supervision is indirectly derived from a combination of contrastive learning and the gating function.
Consider batch B (gs: P 1 comprising N instances. Each instance consists of query % a positive target p,, and negative target p The retriever training objective is based on contrastive learning (Jaiswal et al. 2020). which aims to maximize the similarity of positive pairs, denoted as sim (ga, P. for all instances i in the batch, while minimize the similarity of all negative pairs, denoted as sim (gs,) p) for all other target in the batch except the positive one p," The similarity between sparse representations can be elaborated as follows:
sim(q. p) = Va(g) o G.(2) (V,(p) o (n)
Σ VoOl -.(a.))))
a.(g.p) 1,
where [i] and G(z) refers to the i-th dimensional value within V(x) and G(x), respectively. The notation a; (; p) indicates whether both Gg(4) and Gp(p) have activated the i-th dimensions. In the rest of thr paper, we refer to a,(g, = as the "co-activation" on dimension i.
In essence, the co-activation ay(g. p) determines which dimensions should contribute to increasing sim(g.P) p) whereas the contrastive objective governs whether the similarity sim (q. p) should be max- imized or minimized. When (g p) 1, the i-th dimension adds a positive term Vg(g)( i) (p) to sém(q. p). Depending the objective, the retriever may either increase or decrease the values of (1( and [i] accordingly for optimization. When ae (q p) = O, the mechanism fails to pro- vide direct supervision the i-th dimension the sparsification prevents the dimensional values from contributing to the contrastive objective. To summarize, the key to induce valid dimension- wise supervision hinges the gating function G. It should activate dimensions that effectively represent the data, thereby enabling the oo-activation a;(g. p) to accurately signify the relevance between q and p.
3

---

Published as a conference paper at ICLR 2024
3
VOCABULARY DISENTANGLED RETRIEVER
3.1 MODEL ARCHITECTURE
In summary, VDR a sparse lexical bi-encoder framework that encode the input data æ into sparse lexical representations:
E(x) = o G(z)
o
(2)
The weighting function V Ebase o fast involves conventional transformer-basee encoder to encode input into sequence of hidden states, denote as Ebase I + R and a disentanglement (DST) head transform these latent states into a lexical representation, denoted as fdst RR R L indicates the number of patches for images or number tokens for text.
Vocabulary We adopt the vocabulary from the BERT (Devlin et al. 2019) tokenizer and discard the unused tokens, resulting vocabulary V with size of - Upon analysis employing the PyEnchant Ilibrary, we find that more than 20,000 of these tokens are accurately spelled. This observation indicates that most the tokens can serve as interpretable I-gram concepts.
d transform input data into sequences of d- dimensional hidden states. For text input, we employ pre-trained BERT based model as the base encoder. For image input, use ViT-B/32 architecture (Dosovitskiy et al. 2020) and train from scratch.
Base Encoder
The base encoders Ebase
Disentanglement (DST) Head The disentanglement head process, fast R - R involves several steps. To begin, a layer normalization is applied to the output hidden states. Then, these hid- den states are projected into Vi-dimensional representations using linear projection layer denoted W Rd + RM Subsequently, elulp activation is employed to transform the representation into positive values. The elulp activation is defined as:
[r + 60
x=
elulp(x)
otherwise
Last, following Formal al. (2021a) max pooling is employed aggregate the fine-grained (token or patch) representations into global representation, transforming the representation from R to R
Gating Function Our gating function G activates the dimensions corresponding to the tokens that exist in x along with the top-k dimensions of V(z). These top-k dimensions encompass tokens that present in x, which be viewed expansion of x based on the learned weighting distribution.
{
if (opx 1 or bow [x) =1
G(
else
Here, top& (V)x(V(x) is a binary vector that indicates dimensions possessing the largest values within (xx), and bow(x) is bag-of-words representation of x. Specifically for non-textual data x, bow(x) ( = 0 for all i.
3.2 MODEL TRAINING
The main component our loss function symmetric cross-entropy (SCE) loss, defined as fol- lows:
exp(sim(u-PP77/
log (pp ..)/T) EN exp(sim(p] 44))7))
log Ey exp(sim(a. p//1) enm exp(esmmqq. )/5)
(5)
where T temperature parameter.
The final loss comprises two term. The first term applies SCE loss to Vg(g) @ Gg(a) and Vy(p). The second term involves the SCE loss between the nonparametric query representation bow(g) and (p). The final loss is computed as the sum of these two terms. Further information can be found Figure The upcoming section will delve into the rationale behind this loss design.
4

---

Published as a conference paper at ICLR 2024
Training
wtathatch suertes (texts)
sinnbauch targets (Sexts Lmages)
projection vecab
16
Hore
representation
In
Layer Hore for p representation
bow
bog of word representaciion
Text
Encoder Q
Head
topt.
bopk sparssiication
symeoric cross entropy
Contrattie
CTS
V: vocahulary stre
batch size
To in batch token ids cuery
Data
Base Encadar
DST
Desctiore
cts 20ros([N, VII
ETS, mase:
range(k): For rongett):
if 1 & 1:
ita wankEs, 5]
cts, Frt_naak(:) C. T]
Inference
neturn cta mask
escoder basefnoudeQ(C)) [M,
Tops Spariis
Embedking
Data
Encoder
Hlld
Emiealdeed Carpus
1.0 basetrieder((p) [M,
P
mxx/eluip(lo_ee00u H.o),
saxtclupp(h__00_))
Optionl: Inference
Base Fnceder
DST
Top-k Spardh
v.4.1 = sow(e) ctsuresk box sporsiff
Sparstff
Query
v_n_1 x_n (t-rts_rass) rass]
K.4.2
book(v_4) tops oparsiry
Option2:
8 PSnal Loss
Norparametric
Query
Tokenizer
loss Dow SCEtosstv. V_P_3) rorporanetric entry loss topk SCBLnestV_ K.F_2) loss 1oss_bow bow 1oss_Topk
Infereree
Q
Figure 2: Left: training and inference pipeline of VDR. Right: pseudo code for training VDR.
3.3 EXTENDING TO CROSS-MODAL SCENARIOS
Lexical retrievers largely rely on pre-trained masked language models (MLM) (Devlin et al. 2019 Sanh et al., 2019) which undergo pre-training tasks to output probability distributions over vocabu- lary predict masked tokens within a given context. This process establishes robust basis for the initial weighting and gating distributions of lexical retriever. As a result, the co-activation a;(g, p) effectively capture the relevance between q and p, and the learning process will gradually shape more rational distributions. However, challenges arise in mrosl scenarios, where the image encoder and its projection layer need to be trained from scratch. In such cases, random weighting and gating distributions are introduced on the image side, leading to biased co-activation. Unfor- tunately, this bias tends persist and even amplify during the training process, weighting and gating distributions are often mutually dependent.
In Appendix C we empirically measure the dependency of lexical retriever on MLM by initializ- ing its projection layer within the DST head. The results demonstrate that once this is done, our model struggles converge and becomes non-functional. Below delve into a comprehensive exploration of these challenges and propose effective solutions to address them.
(a) Activation amount. Previous lexical retrievers utilize ReLU activation on V(x) to introduce sparsity. While the V(x) image side is randomly initialized, the ReLU based gating produce uncontrollable amount of activation which will bias the learning. Specifically, an excessive number of activations could lead co-activation on irrelevant dimensions while inadequate activation results in less effective learning, ultimately leading to suboptimal learning outcomes.
elulp activation with top-k sparsification. Instead of ReLU, we adopt a combination elulp activation with top-A sparsification ensure deterministic number of activations. Moreover, dur- ing training, we fully activate (p) while sparsify (g)),, establishing a lower bound on the amount of co-activations for effective learning. This design also allows for flexible trade-off between effectiveness and efficiency downstream by adjusting the activation amount.
(b) Bias from co-activated dimensions. Lexical retrievers have the capability to expand their se- mantic understanding by activating dimensions beyond the tokens presented in the input. However, this expansion can occasionally lead to unintended co-activations occurring in irrelevant dimensions, potentially introducing bias into the learning process. We refer to this situation while the i-th token V fails to reflect the relevance between g and This issue becomes more pronounced when combined with the inherent randomness introduced by Vp(p). Once this phe-
ag(g. p) =
5

---

Published as a conference paper at ICLR 2024
nomenon takes place, rectification becomes challenging, and the bias often amplifies weighting and gating distributions are often mutually dependent.
Nonparametric entry. To mitigate challenge (b), we introduce a nonparametric entry in loss com- putation. This entry uses bag-of-word vectors for queries, denoted as simi(g.) p) bow(g) (v) = Eier(,) [n) where T() is the set of tokens within g. This loss term compels (p))) con- centrate on the tokens in q without expansion, thereby preventing the irrelevant dimensions from possessing excessively large values.
(c) Bias from inactive dimensions. Another challenge arises from dimensions that remain inac- tive throughout the training process. This situation commonly arises with infrequent tokens and cross-lingual tokens. On the image side, these dimensions might exhibit substantial values during random initialization, and persist due to the lack of direct supervision. Let T(B) denote the di- mensions activated by Gg(g) for all q in batch B, the supervision only cover dimensions within T(B) S V while the remaining dimensions in (VT(B) lack direct supervision as they do not ex- plicitly contribute to the contrastive objective. Although some level of control is exerted through the normalization on (z), empirical evidence suggests that this is not sufficiently effective.
Contrastive (CTS) mask. We propose the design of contrastive mask to alleviate issue (c). Specif- ically, the contrastive mask distributes those neglected dimensions to the in-batch instances, For each 4j from the batch B, we enforce Gy(gj) to activate [V)](B) dimensions from the set V (T(),, while [79]
concurrently deactivating them in (py)). This will involve these dimensions into the computation of similarity of negative pairs sim( p) for all # P. while cancel their contribution to the posi- tive pairs sim(1;; P. )- Through this approach, VDR introduces dimension-wise supervision across the entire V space, thereby facilitating a stable and reliable disentanglement learning process.
4 EXPERIMENTAL SETUP We conduct two groups of experiments for text-to-text and cross-modal retrieval scenarios, referred to as 2DR and VDR respectively.
4.1 DATASETS
Text-to-text. we train 2DR,2: on MS MARCO passage ranking dataset (Bajaj et al., 2016) which comprises approximately 8.8 million passages and around 500 thousand queries. We conduct zero- shot evaluations on 12 datasets from the BEIR benchmark (Thakur et al. 2021). which are widely used across previous papers.
Cross-modal. we utilize the mid-scale YFCC15M dataset introduced by DeCLIP (Cui et al. containing 15 million image-caption pairs for training. Our evaluation spans ImageNet, COCO Captions (Chen et al. 2015), and Flickr30k (Plummer et al. 2015) datasets.
2022
4.2 IMPLEMENTATION DETAILS
Our experimental settings and training configuration follow DPR under text-to-text scenarios and CLIP under cross-modal scenarios. We use AdamW optimizer (Loshchilov & Hutter 2018) with learning rate that linearly increases in the first epoch and then gradually decays. All of our models are trained on NVIDIA V100 GPUs with 32GB memory.
Text-to-text. We train VDR,2: for 20 epochs with a batch size of 256 and learning rate of 2e-5. Each query paired with one negative passage provided by the MS MARCO dataset during the training. We use tied embedding and do not incorporate contrastive masking in textual retrieval.
Cross-modal We train VDRcm for 20 epochs with a batch size 4096 and a learning rate of 2c-4. The input resolution of the image encoder is 224 x 224, and the max sequence length of the text encoder is 77. We initialize the learnable temperature parameter to 0.07, adopt the same prompt engineering and ensembling techniques CLIP for ImageNet.
4.3 EVALUATION
Our model offers two inference modes based on how the query q is represented. Parametric infer- ence, denoted as VDR, follows the conventional lexical retrieval pipeline as Eg(4) = Vg(g) GG) The optimal level of sparsity is determined through grid search, involving the variation of activation amount across diverse tasks. Nonparametric inference, referred to as V employs a normal- ized woords vector for query representation, i.e., (q)(4) bow(g). For both inference modes,

---

Published as a conference paper at ICLR 2024
adhere to the conventional pipeline embed where Ep(p) = Vo(p)) Gp(p). For fair compar- ison, we set & = in all of our gating function, where d = 768 for text-to-text and d = 512 for cross-modal scenarios, which is the dimensions of the latent representation from the dense retriever baselines. We evaluate the disentanglement quality from two aspect:
External aspect: whether the disentangled representation of the natural language counterpart can identify and retrieve the corresponding data among the vast pool of candidates. We adhere conven- tional retrieval benchmark ($5) assess this aspect.
Internal aspect: whether the dimensional values can effectively explain the input data. Given the absence of definitive ground truth for this evaluation, we devise an indirect assessment method. In the retrieval benchmark, we gauge the retrieval accuracy of VDR*. For cross-modal retrieval, we additionally supplement our evaluation with case studies ($6.2) and human assessments ($6.2)
4.4 BASELINES
Text-to-text. We compare 2DR121 to several primary and advanced baselines. The primary base- lines include BM25, SPLADE-max (Formal et al. 2021a), and DPR (Karpukhin et al., 2020). No- tably, DPR and SPLADE have similar model sizes and training settings compared to our methods. The advanced retrieval baselines, include ANCE (Xiong et al. (020). UnifieR (Shen al. 2022b),
Contriever (Izacard et al. (02T)) SimLM (Wang et al. (022a)), MASTER (Zhou et al. 2022) Retro- MAE (Liu & Shao 2022), LexMAE (Shen al. 2022a). and ES (Wang et al. 2022b). These advanced baselines incorporate sophisticated techniques such as retrieval-oriented pre- treitriinngg spe-
cialized negative sampling, knowledge distillation, and access Wikipedia data during training. These additional techniques, while promising, often introduce considerable computational overhead and
manual tuning efforts. More details can found in Appendix Due to resource constraints, we will leave them for future work.
Cross-modal. We compare VDRcm primarily against CLIP (Radford et al.
2021 CLIP-BERT, and
with advanced baselines SLIP (Mu al. 2022). FILIP (Yao et al.) (021). DeCLiP L al. 2021),
and ProtoCLIP (Chen et al. 2022). Specifically, SLIP, DeCLIP, and ProtoCL.P incorporate within- modal supervision, which known to be beneficial but computationall) expensive (Andonian al.,
2022 Geng et al., 2023). FILIP leverages the finer-grained alignment between image patches and textual tokens. CLIP-BERT is a variant of CLIP that uses BERT as the text encoder, with a similar model size and training setting to us.
EXPERIMENTS
5.1
TEXT-TO-TEXT RETRIEVAL
Misic!
BME
SPLADE
DOP
IVDE.
ANCE
UnIEk
SinLAI
MANTER
RaCeMAR LEMMA ES
Resteval Pre mrining
Special Negrives
Divilbtion
Wikipolia Ansres
ArgsAm
$1.3
43.0
4013
48.
43.5
41.3
1011
44.6
42.1
43.3
300
5144
FEVEE
21.3
19.9
16.2
18.1
13.6
19.8
17.5
16.3
21.5
23.2
21.9
15.4
DEPeda
31.3
36.6
30.4
37.6
39,0
28.1
40.6
41.3
34.5
39.9
38.0
42.4
41.0
FEVER
75.3
73.0
63.8
74.8
74.0
66.9
6336
75.8
65.7
69.2
77.4
8000
58.2
PIQA
23.6
28.7
23.7
29.3
28.8
29.3
31.1
32.9
29.2
52.8
31.6
35.2
36,4
HaporQA
60.3
53.6
45.2
68.4
63.5
45.6
68.1
03.8
58.1
58.9
63.5
71.6
60.2
NECorpus
12.5
31.3
26.1
32.7
23.7
32.9
32.8
32.3
33.0
30.
34.7
36.5
NQ
32.9
46.9
45.
45.3
47.2
48.6
$1.4
49.8
47.3
SL.
518
56.2
80.0
SCIDOCS
188
15A
15.3
12.3
150
16.5
14.1
15,0
15.9
19.0
SeiFact
67.6
686
81.7
71.7
73.1
TREC-COVID
60.1
71.5
83.0
71.2
76.3
79.6
Teuch6-2000
27.7
30.2
32.0
29.0
28.5
Ave
41.1
42.4
353
44A
84.
380
45.
45.
43.1
35.1
48.
46.8
Avg (wdo NO
48.5
43.
40.4
42.4
64.5
45.6
Table I: Text-to-text retrieval results on BEIR benchmark (NDCG@ 10), t: our implementation. Bold: the best among primary baselines.
Overall performance. [uble[[peeeen effectiveness comparison of different approaches on the BEIR benchmark in zero-shot scenario. Notably, VDR,2: outperforms both DPR and SPLADE by significant margin, while maintaining nearly identical model sizes and training configurations. When compared to advanced baseline methods, VDR consistently outperforms the majority of them, without relying sophisticated and computationally expensive techniques. It's worth noting that these techniques are orthogonal to the retrieval design and can be seamlessly integrated with model to further enhance its effectiveness. This underscores the effectiveness of proposed architectural design in text-to-text retrieval scenarios, even though it was initially designed to ad- dress challenges pross-modal situations. The robust zero-shot performance exhibited by VDR

---

Published as a conference paper at ICLR 2024
highlights its resilience and versatility, making it strong foundation with the potential to benefit wide range of applications.
Remarkable effectiveness and efficiency of nonparametric inference. It is worth noting that VDR which use binary bag-of-words query representations, surpasses the majority of baselines. This observation provides two essential insights. Firstly, our model excels in disentangling textual data, allowing the binary query representation to match the representation of its target from a vast pool of candidates, thereby delivering promising retriever accuracy. Secondly, its success under- scores the nature of the current retrieval tasks, where term-based expansion and overlap play pivotal roles, is worth highlighting that VDR,2, does not need any neural network forwarding inference time, thus significantly improving retrieval speed. In $66.3 we show that V2R can achieve over 10x efficiency compared to dense retrieval, making an optimal choice under low-resource settings.
5.2
CROSS- MODAL RETRIEVAL
ImageNet
MSCOCO
PlikkrOk
Model
image toxt
tESt to image
imagg- to GEXT
SOXU-DD image
Topl
Top5
RGp3
R0
RGF5
RB:1
R-mien
R381
REE
R091
RES
KGTT0
R-meain
CLI
32.81
37.41
43.9
35.7
13.0
31.7
12.7
32.6
TCLIP BERT
32.4
56.1
23.9
47.8
00.3
13.6
33.8
45.1
31.4
44.1
71.2
80.7
27.8
54.7
63.9
57.4
IVDR...
38.7
63.6
30,9
54.5
65.4
17.4
38.1
49.7
42.7
51,0
79.3
86.7
32.4
68.1
70.7
63.4
IVDRI
11.8
28.6
38.6
SLJP
58.6
27.7
52.6
63.9
18.2
39.2
$1.0
42.1
47.8
76.5
85.0
32.3
58.7
61.7
TFILIP
39.1
64.4
21.6
46.7
59.0
13.7
31.7
41.6
35.7
46.3
74.4
83.2
30.7
58.2
68.6
60.2
PLOP
32.0
30.2
55.1
66.5
16.9
37.9
49.4
42.7
DDCTIIP
166
Table 2: Cross-modal results on ImageNet, MS COCO, and Flickr30k. f: our implementations.
Overall performance. Table 2 presents the performance of cross-modal retrievers on the Ima- geNer, MS COCO, and Flickr30k datasets. The results demonstrate the effectiveness of our approach VDRcm across various benchmarks. Notably, VDRcm consistently surpasses the primary bascline models, CLIP and CLIP-BERT, following the same training conditions. Moreover, VDR exhibits superior performance compared to most advanced baselines. It is important to highlight that many of these advanced baselines rely on multi-vector representations or intra-modal objectives, which come at the expense of considerable computational demands during both training and inference stages.
In cross-modas scenarios, the performance of nonparametric inference VDR cm is somewhat limited. We attribute this to the inherent nature of images, which often contain wealth of information not explicitly presented in their captions. As a result, relying solely on the tokens present in queries for matching proves to be challenging, and the incorporation of expansion becomes essential.
ANALYSIS
6.1 ABLATION STUDIES
We have conducted extensive ablation studies within cross-modal scenarios to gain deeper understanding of the individual components of model. In order to mitigate computational expenses, we opted to train models for 5 epochs using learning rate of 5e-4. We then eval- uated the effects of removing specific compo- nents from the model. The results in Table 3 show that the removal of any of these components lead to a decline in performance across all three datasets. This highlights the positive impact of contrastive mask, nonparametric (NP) entry, and max pooling to VDR
ImageNet
MS COCO
Flickr30k
TR
IR
TR
IR
VDR...
29.3
25.4
13.7
43,2
26.9
CTS mask
23.2
21.1
11.9
37.0
22.4
NP entry
26.4
24.4
14.1
42.6
26.2
max pooling
27.8
22.2
12.2
37.4
23.7
Figure 3: Ablation studies of different compo-
nents of VDR
6.2 INTERNAL INSPECTION OF DISENTANGLED REPRESENTATIONS
Image disentanglement. In Figure 4 (a.b), we conduct analysis of internal dimensional values within disentangled representations. The font size within the wordcloud indicates the dimensional values of this token in the weighting distribution V(x).. From the left, we observe that the vocabulary distribution remarkably aligns with the visual features of the image itself. On the right, red bounding boxes highlight specific patch groups for disentanglement. In this process, we apply max pooling only to the representations of these selected patches within the DST head. VDR effectively achieves disentanglement among different patch groups within the same image. This is particularly evident
8

---

Published as a conference paper at ICLR 2024
nasa desert ma rs equipment
trains train
Love bay norway curo
cycle
honda ded
Speed glue ahtt
Fa 1lway raflFoad
bikes aees be
(a) Image Disentanglement
(b) Image Disentanglement (Patch-level)
book
reading
N
snow
eodong
000
sleeping nn
N
Skling PONSS trails
Winter SHERPE 8
JAIN E Tulig Lelng
books
SKI
A cal loying nead to boek..
cat
cards
Accuple
ski
blue
papl rile lhnugh
snow SKIer
book
bible
snows
(c) Retrieval Reasoning
Figure 4: Different approaches for internal inspection on disentangled representations.
in its ability to precisely associate separate visual concepts, such "train" and "bay" with the respective patches in the image. Overall, these results vividly demonstrate that the disentangled representation generated by VDR exhibits rational dimensional values that efficiently explain the input data.
Retrieval reasoning. Figure 4 (c) shows the disentangled representation of the image (p),), cap- tion (g), and the clement-wise product of both (q) @ (p), which provide insight into the retrieval process. the example on the left, the dimensional values within the image representation prominently capture objects present in the image, such as "book" and "cat". On the other hand, the text representation emphasizes contextually relevant concepts like "reading" and "sleeping". By an- alyzing on (q) @ (p). VDR is able to quantify the individual contributions of each token to the retrieval process. This highlights a notable achievement of VDR in enhancing the interpretability of the retriever, enabling clearer understanding of why certain objects retrieved.
Human evaluation. We conducted human evaluations to compare VDRcm to the SOTA captioning model BLIP Li et al. (2022a) in terms of explainability. Details can be found in Appendix H The results indicate that VDR achieves a satisfactory interpretation rate of 92%, outperforming BLIP, which achieves 85%. Notably, participants express preference for approach over BLIP in 48% of the cases, underscoring that our method effectively elucidates input data and matches the explainability of the leading captioning model.
6.3 RETRIEVAL EFFICIENCY
We perform retrieval using Ik queries with cor- pus consisting of 100k data points. We employed inverted indexes for sparse retrieval. The detailed experimental setup is provided in Appendix G We show retrieval effictiienness-fffiiinne of VDR with different inference modes in Figure G The x-
C000 SW
BEIR
axis is retrieval latency per query, and the y-axis is the performance. Among the methods evalu- ated, the nonparametrie inference * proved to be the most efficient, significantly outperforming parametric inference VDR. For VDR, fewer acti- vations k results in more sparse representations, which can enhance retrieval efficiency. When less than 128, the efficiency of VDR is comparable to that of dense model.
fot age
100
Lantercy (ms/g)(-)
Figure 5: Effectivenes--effiiinnny compar- isons of different retrievers.
CONCLUSIONS
In this work, we propose VDR, a simple but effective retrieval-based disentanglement framework that leverages natural language as a form of supervision. Our approach demonstrates that naturally vocabu-
occurring linguistic counterparts of data lary space. Extensive experiments and analysis show that VDR not only yields rational disentangled representations but also enhances the effectiveness, efficiency, and robustness of the retrieval system.
effectively encourage the disentanglement

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We express our sincere gratitude to the anonymous reviewers for their invaluable comments and suggestions. Additionally, we extend our thanks to Lu Hou, Lewei Yao, and Haokun Lin from Noah's ' Ark Lab for their insightful discussions on multi-modal retrieval.
Lei Chen's ' work is partially supported by National Science Foundation of China (NSFC) under Grant No. U22B2060, National Key Research and Development Program of China Grant No. 2023YFF0725100 1000072500, the Hong Kong RGC GRF Project 16213620, RIF Project R6020-19, AQE Project AoE/E-603/18, Theme-based project TRS T41-603/20R, CRF Project C2004-21G, China NoFC 61729201, Guangdong Basic and Applied Basic Research Foundation 2019 Hong Kong ITC ITF grants MHX/078821 and PRP/004/22XX, Microsoft Research Asia Collabora- tive Research Grant and HKUSST-Weban joint research lab grants.
REFERENCES
Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16430-16441, 2022.
Junwen Bai, Weiran Wang, and Carla Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 34:10105-10118 2021.
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang. and Qun Liu. Sparterm: Learning cerm-based sparse representation for fast text retrieval. arXiv preprint ar arXiv:2010.00768, 2020.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint /6/1::6611.9900 2016.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 3598);1798-1828, 2013.
Yoshua Bengio et Learning deep architectures for ai. Foundations and trends@ Machine Learning, 2(1):1-127) 2009.
Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding- based large-scale retrieval. arXiv preprint arXiv:2002.03332. 2020.
Delong Chen, Zhao Wu, Fan Liu, Zaiquan Yang, Yixiang Huang, Yiping Bao, and Erjin Zhou. Prototypical contrastive language image pretraining. arXiv preprint arXiv:2206. 10996, 2022.
Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise simi- larities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3495- 3502, 2020.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K. Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1XXx:1154400022. 2015.
Yufeng Cui. Lichen Zhao. Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive Linguage-image pre-training: clip benchmark of data, model, and supervision. arXiv preprint arXiv:2203.05796, 2022.
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. (17114186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:20 1929, 2020.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv: 10086, 2021a.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and ex- pansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 1288-22994 2021b.
Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An image is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems, 34: 9216-9223 2021.
Shijie Geng, Jianbo Yuan, Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language-image pretraining with hierarchy-awand attention. arXiv preprint urXiv:2303.02995, 2023.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representationss 2017.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint vrXiv:2112.09111 2021.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2649-2655 PMLR, 2018.
Diederik P Kingma and Max Welling. 1rXiv:1312.611 2013.
Auto-encoding variational bayes.
arXiv preprint
Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language image pre-training, arXiv preprint arXiv:2209. 13430, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imant pre- training for unified vision- languaeee understanding and generation. In International Conference on Machine Learning, pp. 12888-12998 PMLR, 2022a.
Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative time series forecasting with dif- fusion, denoise, and disentanglemennt In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b.

---

Published as a conference paper at ICLR 2024
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language- -mnuag pre- training paradigm. arXiv preprint arXiv:2110.05500000 2021.
Sheng-Chich Lin, Hong Yang, and Jimmy Lin. In-batch negatives for knowledge distilla- tion with tightly-coupedd teachers for dense retrieval. Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pp. 163-173, 2021.
Zheng Liu and Yingxia Shao. Retromae: Pre-training retrieval-oriented transformers via masked auto-encoder. arXiv preprint urXiv22205.12033 2022.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4124 PMLR. 2019a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprini arXiv:/905.012 2019b.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly- supervie disentanglement without compromises. In International Confer- ence on Machine Learning, pp. 3348-6359. PMLR, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International conference on machine learning, pp. 4212-4221. PMLR, 2019.
Christopher Manning. An introduction to information retrieval. Cambridge university press, 2009.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerehner. dsprites: Disentanglement testing sprites dataset. https//gghubbccco//eeeennnn/tppp
2017.
Giangiacomo Mercatali, André Freitas, and Vikas Garg. Symmetry-inducee disentanglement on graphs. Advances in Neural Information Processing Systems, 35731497-1, 2022.
Bhaskar Mitra, Nick Craswell, al. An introduction to neural information retrieval. Foundations and Trends in Information Retrieval, 13(1):1-126, 2018.
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European Conference on Computer Vision, pp. 529-544. Springer, 2022.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-pheee correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. Advances in neural information processing systems, 28, 2015.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, and Ji- Ji-Rong Wen. thorough examination on zero-shot dense retrieval. arXiv preprint arXiv:2204. 12755, 2022.
12

---

Published as a conference paper at ICLR 2024
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Mehran Shakerinava, Arnb Kumar Mondal, and Siamak Ravanbakhsh. Structuring representations using group invariants. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. Lexmae: exxicon-bottlenectee pretraining for large-scale retrieval. arXiv preprint arXiv:2208. 14754, 2022a.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Kai Zhang, and Daxin Jiang. Unifier: A unified retriever for large-scale retrieval. arXiv preprint arXiv:2205. //94 2022b.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation information retrieval models. arXiv preprint arXiv:2104.08663 2021.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:2207.025755 2022a.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:22/2.03553 2022b.
Shiyu Wang. Xiaojie Guo, and Liang Zhao. Deep generative model for periodic graphs Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022c.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00800 2020.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2021.
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: survey, arXiv preprint arXiv:2211. 14876, 2022.
Kun Zhou, Xiao Liu, Yeyun Gong, Wayne Xin Zhao, Daxin Jiang, Nan Duan, and Ji-Rong Wen. Master: Multi-task pre-trained bottlenecked masked autoencoders are better dense retrievers. arXiv preprint arXiv:22 2022.
13

---

Published as a conference paper at ICLR 2024
A HYPERPARAMETERS
Our hyperparameter selection follows DPR for the text-to-text retrieval while CLIP for the cross- modal retrieval. Details can be found in Table below.
text-to-text
cross-modal
Batch Size
256
4096
Epoch
20
20
Learning Rate
2e-5
2e-4
Warmup Epoch
LR Decay
Linear
Aoncalln
Normalization
None
L2-norm
Temperature
None
0.07
Hard Negative
Max Activation Num
Max Seq Length (Q/P)
256/256 768/768
77/49
Transformer Width (Q/P)
768/768
Table 3: Hyperparameters for training VDR.
B REPRODUCTION COMPARISONS
Figure 6 and Table we present our repro- ductions of DPR and CLIP, accompanied by re- sults from other pertinent research papers. This facilitates valid reproduction and fair compari- son. Notably, our study consistently showcases the highest levels of performance in relation these foundational baselines.
Model
DPR
DPR
MS MARCO
17.7
31.7
ArguAna
17.5
40.8
Climate-FEVER
14.8
DBPedia
26.3
30.4
FEVER
56.2
63.8
FiQA
11.2
23.7
Figure Shoowcssss our replicated DPR model, which outperforms the versions reported in other studies. Therefore, we present our repli- cated baselines in main paper.
HotpoiQA
39.1
NFCorpus
18.9
26.1
NQ
47.4
43.2
SCIDOCS
7.7
10.9
SciFact
31.8
47.4
In Table 4 it's noteworthy that UniCLIP (Lee et al.) 2022), having undergone pre-training on YFCCI5M with a similar configuration, demonstrates superior outcomes. As a result, we have chosen to adopt their outcomes the cross- cross-modal retrieval aspect, with the exception of ImageNet where we have opted to utilize our own replicated scores.
TREC-COVID
33.2
60.1
Touché-2020
13.1
22.1
Avg. Best on
26.4
35.8
Figure 6: Reproduction of DPR from different sources. f: ours.
MSCOCOO
ImageNet
image-to-tent
text-to-image
image-to-tent
text-to-image
Topi
Top5
Our re-implementttion based on DECLIP's checkpoints
CLIP
32,80
57.35
16.94
39.50
50.94
10.75
26.19
35.24
34.70
65.00
74,10
23.60
46.90
58.76
SLIP!
33.57
58.60
17.94
40.42
51.82
11.22
26.48
35.36
35.60
65.60
77.30
23.40
47.32
57.96
FILIP
39.16
64.35
21.64
46.66
59.00
13.72
31.72
41.60
46.30
74.40
83.20
30.66
58.18
68.56
DeCLIP
43.24
25.34
51.20
63.44
16.59
35.24
45.41
51.30
80.70
88.50
35.50
63.04
73.02
Results reported Un:CLIP
CLIP
31.3
20.8
43.9
55.7
1300
31.7
42.7
34.9
63.9
75.9
23.4
47.2
58.9
DeCLIP
41.2
28.3
53.2
64.5
18.4
39.6
51.4
51.4
80.2
88.9
34.3
60.3
70.7
Table 4: Reproduction of cross-modal retrieval on ImageNet, MS COCO, and Flickr30k from dif- ferent sources.
14

---

Published as a conference paper at ICLR 2024
C RELIANCE ON MASKED LANGUAGE MODEL
In this section, we empirically validate the reliance of lexical retriever on the pre-trained masked language models (MLM).
We adhere to the same training pipeline of our approach, while only initializing the linear projection within the DST head of the p encoder and training it from scratch. This configuration denoted as VDRproj- Additionally, we incorporate BERT-base models as lexical retrieval baseline, without We present the training result below.
undergoing further fine-tuning, denoted BERT lex-
Model
Epoch
NDCGIO@BENT
MRRIO@MARCS
BERT
0
20.1
28.9
VDR
38.9
28.4
VDR
42.3
30.8
VDR
42.9
31.7
VDR
43.4
32.4
VDR
5
43.7
32.8
VDR
proj
0.2
Table 5: Different setup of lexical retrievers trained in the text-to-text retrieval scenarios.
Our experimental findings show that when we employ the pre-trained MLM projection, which inher- ently offers rational weighting distribution from the outset, VDR reliably improve the effectiveness and achieve best results within 5 training epochs. Conversely, when starting from scratch with the projection layer on p side, even with substantial training efforts, the VDRproj setup encounters chal- lenges in attaining effective convergence. This obstacle compromises the final outcomes and makes even fall behind the performance of the untrained baseline, BERT lex. These findings support and validate the insights presented in Section 3.3
Moreover, our observations and experiments cross-modal retrieval suggest that achieving an fective transition from crratchinnitializeds distribution rational one necessitates substantial amount of training data, a large batch size, and the inclusion of the contrasting mask.
D IMPACT OF NONPARAMETRIC ENTRY
We emphasize the essential role of incorporat- ing the nonparametric entry during training to achieve disentanglement in our model. With- out it, our model tended to assign excessive values to overly common rare tokens. We conjecture this issue arises from the interdepen- dence between the gating and weighting func- tions, which amplifies biases rather than miti- gating them.
0.25
0.54
To validate this hypothesis, we examine the em- beddings produced by model with and with- out the nomparametric entry. In Figure we la- bel model in cross-modal setting with non- parametric entry as VDR (w/ BoW), without it as VDR (w/o BoW), and BERT-based model as BERT-text. We take these encoders to em- bed text and images from the MS COCO test set into lexical representations, calculating average values for each token within these representa- tions. We then visualize the top 100 tokens us- ing word clouds and the distributions of their values using box plots. Our observations re- veal that image representations from VDR (w/o BoW) have sharper distributions, characterized
BERT-oot
vDRt BoW)
VDR text twls RoW)
VDR image (wl Rol))
VOR Image (wdo RKW
the
today
digital cleanus
Mymy into
of
eitting
em
pop
O f
in
Bray
aV
on
ing
Joe
m
M
and
##a
Ioe SHT
the
tr ave
Figure 7: Box plot (top) and word cloud (bottom) of the vocabulary distributions on MS COCO.
15

---

Published as a conference paper at ICLR 2024
by higher upper bounds and mean values in L2 norm space. However, the top 100 tokens in word cloud of VDR (w/o BoW) lacks meaningfulness and fails to convey distinct information. This implies that the omission of the nonparametrie input in VDR amplifies biases, causing specific meaningless tokens to be assigned excessive values, thereby consistently dominating the matching outcome.
We term the above phenomenon as "disentanglement laziness". Simply put, the learning process avoids the "hard work" of properly disentangling and instead takes the easy route for optimization, degrading to a entangled fashion. In bi-encoder architecture, we have noticed that relying solely on parametric components causes the model consistently assign high values tokens that are either frequently or never encountered, resulting in entangled learning within a subset of the vocabulary space. In doing so, the model seemingly "escapes" from the rigorous work of disentanglement, reducing the problem into an optimization within an entangled representation space. Interestingly, this phenomenon not exclusive to the research of disentangled representation learning. is also observed in other research, like the Mixture of Experts (MoE), where it is referred to as "load im- balance". This term alludes to the model's tendency to consistently favor certain experts, thereby causing unequal distribution of learning and optimization channels. In addressing the observed issue in our experiments, we enhance the disentanglement process by integrating the nonparamet- ric entry, which provides stable and straightforward supervision of the data, independent of any influences from the entangled parametric model.
E SPARSITY V.S. EFFECTIVENESS
E. AMOUNTS ACTIVATION
We present the effectiveness of VDR with different amounts of activation A in Table and Table
Word Length Query
VDR
Model
Doc
VDR
32
256
MARCO
34.4
34.5
34.4
AnguAna
193
167
48.8
48.6
27.3
41.7
47.0
47.2
Climate-FEVER
20
85
18.1
17.2
17.1
17.6
17.2
17.2
16.9
DBPedia
50
37.6
35.1
38.0
38.6
39.0
38.8
38.9
FEVER
85
74,8
73.7
74.0
73.9
73.9
73.9
73.9
FIQA
132
29.3
28.1
28.2
28.8
28.8
28.6
28.4
HotparQA
18
46
68.4
64.4
65.0
65.5
65.5
65.4
65.0
NPCorpus
232
32.7
32,5
33,0
32.9
32.9
32.8
NQ
79
45.8
44.6
45.8
46.4
46.9
47.0
47.2
SCIDOCS
14.8
15.2
SciFFatt
12
214
67.6
67.3
66.8
67.2
67.1
67.3
56.6
TREC COVID
67.3
67.8
67.3
Touché-2020
292
27.7
29.1
29.0
29.4
29.5
29.8
29.4
average
422
Table 6: Effectiveness of VDR with varying activation amounts k. Bold denotes the overall best result and underline denotes the best query sparsity for VDR.
VDR
ImageNer
MSCOCO
Flickr30k
K
Topl
Top5
R@l
image-to-tent
R@5
R@
R@1
text-to-image
R@5
R@:0
R@D
imago-to-tent
R@B
R@ 10
R@
text-to-image
R@5
R@
32
34.8
56.7
18.9
38.9
49.3
15.8
35.1
46.8
34.9
39.2
70.3
31.7
58.0
68.5
64
36.6
59.7
23.3
43.5
56.1
17.1
37.3
48.6
40.9
67.9
78.2
33.3
59.8
71.3
128
38.1
62.0
27.0
49.7
17.4
38.1
49.4
44.9
73.9
83.4
33.3
60.0
71.4
256
38.5
63.3
29.5
53.9
64.4
17.4
38.
49.4
49.9
77.2
85.6
32.9
60.0
71.2
512
63.6
30.9
54.5
65.4
17.4
38.1
49.7
51.0
79.3
86.7
32.4
60.1
70.7
Table 7: Effectiveness of VDR with varying activation amounts k. Bold denotes the best result.
In the text-to-text scenario, the results demonstrate that the effectiveness of VDR increases as increases, reaching peak and then decreasing. This suggests that by properly selecting the number of activation units, VDR is able to achieve considerable improvement.
In the cross-modal scenario, the results demonstrate that the effectiveness of VDR increases consis- tently as l increases in the majority cases. This suggests that a higher number of activation units can lead to better performance in cross-modal scenarios.
16

---

Published as a conference paper at ICLR 2024
E.2 AMOUNT OF VOCABULARY SIZE
We explore the effects of expanding the vocabulary size by switching from standard BERT encoders to multilingual BERT (mBERT) encoders with a vocabulary size of 1 110k, which is 4 times larger than that of BERT encoders. Our aim is evaluate if larger vocabulary and the consequent increase in sparsity would adversely affect our representation learning approach.
DPR(BBERT)
VDRMBBERT)
DPR(BERT)
VDR(BERT)
ArguAna
37.2
46.7
40.8
48.6
Climate-FEVER
16.2
DBPedia
31.3
36.4
30.4
39.0
63.8
FIQA
21.6
27.5
23.7
28.8
HotporQA
45.2
65.5
NFCorpus
25.2
33.3
26.
33.0
SCIDOCs
10.7
14,3
10.9
15.3
TREC-COVID
57.3
66.7
60.1
67.8
Touch6-2020
27.7
22.1
29.8
Avg
34.9
42.6
35.8
44.5
Table 8: Effectiveness of DPR and VDR using BERT-based and multilingual BERT-based encoders.
Table illustrates the results with increased vocabulary size. With the switch to mBERT-based encoders, the vocabulary size grows from 30k to 110k, consequently increasing the sparsity due to the activation of the same number of dimensions. Interestingly, there is a minor performance drop in VDR on the BEIR benchmark, from 44.5 to 42.6, when switching to mBERT encoders. We also observe that DPR exhibits similar trend. Notably, VDR outperforms DPR by approximately 23% in both scenarios, suggesting that the performance drop may not be directly linked to the larger vocabulary size. Instead, it could arise from mismatch between the multilingual encoder and the predominantly English downstream retrieval tasks. This finding aligns with our earlier analysis in Section 3.3 where we posited that pretrained masked language models provide solid foundation for establishing effective gating distributions. This experiment indicates that increasing the vocabulary size or sparsity does not significantly hinder the learning process.
F
CASE STUDY
bolivia
fat-og
Courtety SerteyyshA D sailing
toy Bock
Ca
Venicle
deer
Da battle
Sate patl
dus Erian
Schooner
E
Cadroron obot
Aay M dep
follow
Canion 1
Plecte te
Case
Case
Case
Mit coffee
gee
fruit
Ni bi
pair
focus
parrot crot
orienta Iee
cafe
UM0
caster vel
caannemenn Cmeent aurchase
Temon
duck
Case 5
Case
Hown leads
aniinn SIAN SMA
gus hawk dus
prairie
Whether
rine
Lawn Lawnerd
safari
fog fog
Ho Eckedi
Z0o oggical
pigeon tocks
squi Frel
ton
de rect
sown ZOO
peach egg
Case
Case
down loa
vederarzo Carzan
dump Soups
Food iici
down 1oad
vege
shel food
Kalinn siw..ret.ooo.. resident
cube
sessica
domb tee duckk ag Cupby
2 Mineu TncuExale
Sadirc abasket
asia
satco Froes
Case 13
Case 14
Case
Figure 8: More study VDR disentanglement of image.
17

---

Published as a conference paper at ICLR 2024
We provide additional case studies in Figure Cases through 8 represent successful cases as de- termined by our experts, while cases through 16 illustrate instances where the image encoder did not perform expected. For those good cases, can observe that the main concepts present in the images are aptly represented in the word cloud. This indicates that the image encoder of VDR effec- tively captures the semantic meaning of these images, producing a reasonable and understandable representation within the disentangled vocabulary space. For the unsuccessful cases, we observed some cases of misidentification or misconception. After analysis, we identified two main reasons for these errors. First, there are cases where the encoder fails to correctly identify the object within the image because resembles another object, thus skewing the results. For example, in cases 9 and 13, the encoder incorrectly identifies ducks as squirrels and dogs as bears, likely due to their similar appearances within the images. Secondly, certain images entail concepts associated with n-gram phrases, which is challenging for internal inspection or word cloud visualization. For instance, in case 10, the term "giraffe" is tokenized into three tokens: "gi". "##raf", and "##fe". While the first token, "gi", appears in the word cloud, the latter two are missing. Such n-gram concepts can be challenging to capture or infer through an internal inspection of the representation. This limitation can be traced back to the choice of tokenizer used prior to training.
G DETAILS IN EFFICIENCY MEASUREMENT
We perform retrieval using IK text queries with a pre-embedded corpus consisting of 100k data points. We employed inverted indexes for sparse retrieval. The retrieval experiments are conducted on single-threaded Linux machine with two 2.20 GHz Intel Xeon Gold 5220R CPUs. The batch size used in the experiments is one and the maximum sequence length for queries is 77. The MS MARCO and MS COCO datasets were utilized for text-to-text and text-to-image retrieval, respec- tively. The average query length for text-to-text retrieval was 6.8 and for cext-to-image retrieval was 1 1..6. The effectiveness of the retrieval methods was evaluated using the average NDCG@ 10 scores on the BEIR metric for text-to-tex retrieval and the Recall @ I metric for text-to-image retrieval on the MS COCO dataset.
H DETAILS OF HUMAN EVALUATION
This section outlines our evaluation approach to comparing the VDR with the SOTA captioning model BLIP. For VDR, we encoded images into disentangled representations. Human evaluators then selected the most understandable tokens from the top-5 dimensions, without access to the orig- inal image. In the case of BLIP, evaluators chose up to 5 tokens from -ile-ggeeerted captions that they felt best captured the essence of the caption, again without seeing the corresponding images. We also included evaluations of the full captions generated by BLIP for comparison.
To assess the effectiveness of these methods, we recruited an additional group of 10 participants from universities with diverse background, each evaluating 20 images. They were tasked with determining (1) the effectiveness of the token sets in capturing key concepts of the images and (2) which token set (VDR BLIP) provided a more accurate description of the image. To mitigate potential position bias, we randomized the order of the top-5 manually selected tokens from VDR and varied the presentation order of the two methods for each participant. All participants were presented with the same set of image samples in both phases of the evaluation. This ensured they had adequate familiarity and information for comparison in the subsequent phase.
Our findings indicated a high level of satisfaction among participants with the tokens derived from VDR, with 92% expressing satisfaction. This satisfaction rate was comparatively higher than the 85% satisfaction for the top-5 tokens selected from BLIP captions. However, it's noteworthy that satisfaction decreased to 76% when evaluating full BLIP captions. We interpret this decline as a result of two factors: firstly, BLIP captions are generally concise (ranging from 2 to 10 words), often making the top-5 tokens adequate for summarizing key concepts. Secondly, while full captions offer comprehensive description, they sometimes include specific details (like location and quantity) that may not always be accurate, leading to a reduction in overall satisfaction. When comparing the top-5 tokens from both VDR and BLIP. participants showed preference for VDR approach in 48% of the evaluations. This demonstrates that our framework effectively captures and elucidates the essential elements of the input data, rivaling the explainability of the leading captioning model.
18

---

Published as a conference paper at ICLR 2024
I ABLATION STUDY ON TEXT-TO-TEXI RETRIEVAL
We present an ablation study for text-to-text retrieval, aiming to understand the impact of different components on the performance of our text-to-text retrieval model. We evaluate various ablations to gain insights into the model's behavior. Specifically, we consider three ablations: the original implementation, "w/o elulp" setting where we replace the elulp activation with the relu activation, and "w/o bow" setting where we remove non-parametric entry loss during training.
VD
VDR
elulp
bow
X
NDCG 10
ArguAna
48.6
45.5
43.5
48.8
47.9
41.4
Climate-FEVER
17.6
16.5
18.4
18.1
17.6
15.5
DBPedia
39.0
37.6
38.3
37.6
37.4
34.4
FEVER
74.0
72.2
73.7
74.8
72.6
70.7
FIQA
28.8
29.0
28.1
29.3
28.3
25.3
HotpotQA
65.5
63.8
64.5
68.4
68.4
63.2
NFCorpus
33.0
32.4
33.2
32.7
32.4
32.3
NQ
47.2
45.5
44.9
45.8
45.0
38.8
SCIDOC'
15.3
14.5
15.4
15.2
14.0
SciFact
67.3
65.6
65.7
67.6
67.1
66.0
TREC-COVID
67.8
64.8
68.0
69.0
66.7
60.7
Touché-2020
29.8
28.8
29.4 4
27.7
27.2
21.4
Avg
44.5
43.0
43.5
44.6
43.8
40.3
Table 9: Ablation study for text-to-text retrieval on BEIR benchmark.
J FAIRNESS IN BASELINE DISTINCTION
The advanced baselines refer to baselines that employ sophisticated techniques known for signif- icantly improving the performance number but orthogonal to the retrieval design. These methods often come with significant increase in computational requirements and the need for meticulous tuning. Here, we delve into these techniques as applied in a text-to-text scenario.
(1) Retrieval-
oriented pre-training involves using harge-scale pre-training tasks (Chang et al. 2020 tailored to improve the retriever's efficiency. (2) Specialized negative sampling is well known as crucial in contrastive learning and training retriever (Lin al., 202I). (3) Knowledge distillation (Gou al., 2021) is process where knowledge from a larger model (usually cross-encoder) is transferred to simpler one (bi-encoder). While this improves performance, also increases complexity and com- putational demands.
(4) Access to Wikipedia data during training (Ren et al. 2002) potentially benefit the performance on datasets constructed from Wikipedia, such as the NQ dataset.
We primarily compare our method, VDR,2 with DPR, and VDR with CLIP-BERT. These com- parisons are apt because these methods have identical parameter counts and follow similar training pipelines. Both VDR and DPR have 217 million parameters and were trained on 500k question- passage pairs using V100 GPUs over one day. Similarly, both VDR and CLIP-BERT, having equal number of 197 million of parameters, trained on 15 million image-caption pairs for six days.
K SIGNIFICANCE TEST ON IMPROVEMENT
The results of significance tests comparing the performance of the VDR against baseline models with identical parameter counts and training pipelines are detailed below. These tests were designed to evaluate whether VDR offers statistically significant improvement over the baseline models, with the null hypothesis positing equal performance between the compared models. For text-to-text retrieval, the results are summarized in Table 9 Out of 12 datasets, VDR,2: showed statistically significant improvements in datasets, as indicated by the symbol A. Similarly, for cross cross-modee modal re- trieval, detailed in Table 10, VDRem exhibited significant improvements across all tested settings when compared to the CLIP-BERT baseline. These results collectively indicate that VDR, in both text-to-text and cross-modal scenarios, offers substantial improvements over the baseline models.
19

---

Published as a conference paper at ICLR 2024
MSCOCO
Fliskr30k
Model
image to-
image
image
image
R01
R383
RØ1
R81
RG TO
R81
R0
BERT
23.9
71.2
V
30.9A
54.5A
65.4A
174A
38.1
49.7A
51.04
79.3/
86.7A
32.4%
60.1A
70.7A
Table 10:
Statistical significance for differences with VDRmm and CLIP-BERT via two-sided student-t test, A indicates methods with significantly higher Recall with < 0.01.
DPR
VDR
ArguAna
40.8
48.8A
Himate-FEVER
16.2
18.14
DBPedia
30.4
37.6A
FEVER
63.8
74.8/
FIQA
23.7
29.34
HotporQA
45.2
68.4
NFCorpus
26.1
32.7A
NQ
45.82
SCIDOCs
10.9
15.4A
SciFact
47.4
67.6A
TREC-COVID
60.1
69.0
Touché-2020
22.1
27.7
Figure 9: Statistical significance for differences with VDR,2, and DPR via two-sided student-t test, A indicates methods with significantly higher NDCG@ 10 with p < 0.01.
20


Reviews:
[{"review_id": "aOkpRFU6z1", "reviewer": "Reviewer_fuH7", "summary": "1. The paper presents a novel approach for learning disentangled representations for text-to-text and text-to-image retrieval by leveraging an LM vocabulary as the distinct units of latent representation.\n\n2. Concretely, the paper proposes casting both the query representation and the passage representations to a |V| dimensional space, with each value in the vector indicating the token's relevance to the input. This is then passed through a gating function to obtain the final (sparse) representation for each input. The models are trained end to end in a contrastive learning framework.\n\n3. In order to extend the approach for multimodal retrieval scenarions, the authors introduce the following modifications:\n\n3.1 Leveraging elu1p + top-k instead of ReLU to control for the number of sparse tokens in the representation and ensure the number of activated units are deterministic.\n\n3.2 A non-parametric entry loss between the BoW representation of the query and the encoded representation of the data, thereby encouraging the model to avoid learning irrelevant co-activations.\n\n3.3 In order to encourage activation of tokens that are in-frequent, the authors propose a contrastive mask, that activates a fixed fraction of activations within a batch, but discards their contribution while computing similarity between the positive samples; thereby encouraging the units to participate in the loss.\n\n4. The proposed method demonstrates strong performance on both text-to-text retrieval (when trained on MS MARCO and tested on BEIR), as well as multimodal retrieval (when trained on YFCC15M and tested on ImageNet, COCO Captions, and Flickr30k).\n\n4.1 In addition to that, for text only retrieval, leveraging a BOW representation of the query (thereby removing the need for a query encoder) achieves good retrieval performance while being up to 10x faster.\n\n4.2 For the scenario of image disentanglement, based on human evaluation, for 92% cases the annotators found the disentangled representations to be a satisfactory representation of the image, compared to 85% when using a state-of-the-art image captioning model", "questions": "1. How stable is the proposed approach to increasing the vocabulary size ? Concretely, would the performance be impacted if the underlying text encoder was XLM-Roberta (with a vocabulary of 250k)? \n2. On a similar line of thought, what do the authors think about the feasibility of being able to decouple the vocabulary of the latent vector and the vocabulary used by the query encoder ? Concretely, would it be possible to have an XLM-Roberta encoder with the latent vectors being represented by a BERT vocabulary ?\n3. For the text-to-text retrieval scenario, does the non-parametric inference still work without using the non-parametric entry loss ? \n4. Would it be possible to elaborate more on how the analysis for Figure 4 (b) was done ? Is it considering only the patch embeddings for the max pooling operation in the DST head ? \n\nMinor Typographic issues\nAbstract: Our approach employ a bi-encoder -> our method employs a bi-encoder\nAbstract: Moreover, The result -> Moreover, the result\nPage 2: real-world data are -> real world data is\nPage 4: to the tokens exist -> to the tokens that exist", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well presented, I quite enjoyed reading it. The authors do a good job of presenting the approach in a clear manner. The idea of using the vocabulary from the tokenizer of a pre-trained model as the discrete latent units is quite neat. \n2. The proposed modifications (w.r.t the elu1p + top-k activation, the non parametric entry loss and the contrastive mask) are well motivated.\n3. The paper presents strong results for both text-to-text and text-to-image retrieval. For text-to-text, the results using non-parametric inference are particularly surprising and impressive, and potentially is generally applicable for tasks requiring fast inference. For the text-to-image retrieval scenario, both human evaluation as well as the qualitative analysis presented does establish the core premise of the paper of being able to leverage a text vocabulary for learning discrete latent representations.", "weaknesses": "1. Since the latent representations presented are tied to an underlying vocabulary, that inherently limits the kind of representations that the model can potentially learn. Because most of the modern day tokenizers are learned based on statistical properties, especially for models with larger vocabulary sizes (where each unit itself may not represent a meaningful entity), the representations learned may not be interpretable. The authors do showcase this limitation in one of the failure cases in the Appendix (w.r.t the tokenization of the word Giraffe).\n\n2. The human evaluation setup for BLIP captioning is artificially detrimental to the image-captioning model. Specifically, the evaluators extracted 5 tokens from the caption to best reflect the image. However, the criterion of using top-k discrete tokens is a limitation of the proposed method and thus should not be imposed on the image captioning model. At the very least, using the complete caption should also be used (potentially as an upper bound for the task of which set of tokens better describe the image).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7yHVHy55g0", "reviewer": "Reviewer_LJaD", "summary": "This work introduces Vocabulary Disentangled Retrieval (VDR), a method that aims to take a retrieval-based approach to achieve disentangled representation learning. VDR supports parametric and non-parametric inference. The latter option does not require a neural network during inference, decreasing the computation speed, making it suitable for low-resource scenarios. The method is tested on text-to-text retrieval and cross-model retrieval tasks. The baselines are split into primary and advanced baselines, and VDR outperforms many of the primary baselines in terms of NDCG and Recall. A human evaluation on 20 images, and a case study with 6 images is presented to evaluate the disentanglement of VDR.", "questions": "- The current work focusses on disentanglement, and I am wondering what the authors consider to be the difference between disentanglement and grounding in this case?\n\n- The authors claim that VDR outperforms the baselines with a significant margin, can the authors give the results of a significance test that backs up this claim?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method, VDR, supports nonparametric inference, which increases the inference efficiency. This makes this method an interesting option in lower resource settings (given budget for training).\n- The authors address an important problem, and their method could be used to help users interpret the results of retrieval system.", "weaknesses": "- I am mostly concerned about the evaluation methods used, specifically: \n\n(1) The retrieval baselines are split into primary baselines and advanced baselines. The authors position themselves in the primary group, arguing that their method does not need advanced techniques as in the advanced baseline group. I am not sure how 'advanced' is defined here, and thus whether this is a a fair distinction. This is important, as VDR outperforms the primary baselines, but does not necessarily outperform the advanced baselines. \n\n(2) One of the most important aspects of VDR is the disentangled representation. This is evaluated with a small case study of only 6 images (and some more in the appendix) and a small scale human evaluation. Except for the small size of the data samples, it is also not clear how the images were selected. Are these cherry picked, or random? There is also little information about the setup of the human evaluation, making it difficult to judge these results. Given that this is such an important part of the paper, I would have expected a larger scale evaluation.\n\n- The paper makes a couple of unsubstantiated claims, most notably:\n\n(1) Page 2: \"In practice, individuals employ natural language to interpret and distinguish objects.\" There is no source to back up this claim, and it is also not entirely clear what the authors mean by this, for example, babies can distinguish objects, before they can speak?\n\n(2) The definition of Information Retrieval on page 3 is very narrow. It does not follow the broader definition given in Manning, 2009, which the authors cite. The other source, Zhao et al., 2022, is a survey paper on dense text retrieval methods, which is also only a tiny fraction of the entire field of Information Retrieval.\n\n- Minor detail: the authors cite Kingma & Ba (Adam) for AdamW, instead of Loshchilov & Hutter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTETvOfQqq", "reviewer": "Reviewer_LxNZ", "summary": "The paper introduces an innovative approach called Vocabulary Disentangled Retrieval (VDR) for learning disentangled representations using natural language as a supervisory signal. VDR employs a bi-encoder architecture that is trained on data-text pairs, where a disentanglement head maps dense representations into a disentangled space. This space is structured such that each dimension correlates with a token from the BERT tokenizer's vocabulary, creating an interpretable, token-level representation of data.\n\nThe authors present two novel sparsification techniques to refine these disentangled representations: top-k sparsification and nonparametric sparsification using normalized binary bag-of-words. These methods are designed to maintain the interpretability of the representations while enhancing retrieval performance.\n\nExtensive experiments across 15 benchmark datasets, encompassing text-to-text and text-to-image retrieval tasks, demonstrate that VDR significantly outperforms comparable bi-encoder retrievers and existing baselines.\n\nBy utilizing sparse embeddings and a vocabulary-defined space without complex or computationally intensive methods, the paper's proposed approach provides a more interpretable and efficient solution for embedding-based retrieval across various domains.", "questions": "The authors use elu1p activation as a replacement for ReLU. Do you have experimental results that motivate this decision?\n\nDo the authors have any intuition as to why their approach works better than DPR? It is not clear to me why dense representations would perform much worse than the disentangled representations based on the vocabulary.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The approach provides interpretable representations that are robust for retrieval tasks, though not necessarily state-of-the-art. The quality of the work is reflected in its thorough experimental setup, with evaluations that cover both document and image retrieval tasks.\n\nThe clarity of the paper is another strength, as the paper is well-written and easy to understand. This clarity extends to the simplicity and ease of implementing the proposed retrievers, which also generalize to different applications such as text-to-text and text-to-image retrieval tasks.\n\nIn terms of significance, the model demonstrates a strong performance in zero-shot retrieval tasks. The VDR-bag of words, in particular, is competitive with the more complex and parametric VDR-k, offering better latency and making it appealing for low-resource applications. The encouraging results on BEIR benchmark and cross-modal benchmarks underscore the impact of the work, suggesting that the method could be further improved by integrating it with other orthogonal approaches.", "weaknesses": "In my opinion, the biggest concern with this paper is that its main innovation being the use of vocabulary as a basis for disentanglement, which is not convincingly demonstrated to enhance explainability or offers significant improvements.\n\nThe focus of the methodology and experiments appears to be on training effective retrievers using disentangled representation in the vocabulary space. However, this seems to diverge from the initial motivation, which emphasizes using natural language to guide disentanglement learning. This inconsistency complicates the narrative flow and understanding of the paper’s objectives.\n\nIn terms of empirical evaluation, while the model demonstrates improved performance in text-to-text and cross-modal retrieval tasks, there is a lack of analysis on non-retrieval tasks. This omission limits the understanding of how the disentanglement contributes to broader applications beyond retrieval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)