STAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Authors: Chen Chen
Bowen Zhang
Liangliang Cao
Jiguang Shen
Tom Gunter
Albin Madappally Jose
Alexander T Toshev
Yantao Zheng
Jonathon Shlens
Ruoming Pang
Yinfei Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Image text retrieval
sparse embedding
interpretability

Abstract: Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications. State-of-the-art contrastive approaches, e.g. CLIP, ALIGN, represent images and texts as dense embeddings and calculate the similarity in the dense embedding space as the matching score. On the other hand, sparse semantic features like bag-of-words models are more interpretable, but believed to suffer from inferior accuracy than dense representations. In this work, we show that it is possible to build a sparse semantic representation that is as powerful as, or even better than, dense presentations. We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space. Each token in the space is a (sub-)word in the vocabulary, which is not only interpretable but also easy to integrate with existing information retrieval systems. STAIR model significantly outperforms a CLIP model with +$4.9\%$ and +$4.3\%$ absolute Recall@1 improvement on COCO-5k text$\rightarrow$image and image$\rightarrow$text retrieval respectively. It also achieved better performance on both of ImageNet zero-shot and linear probing compared to CLIP.

STAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Yantao Zheng, Jonathon Shlens?, Ruoming Pang. Yinfei Yang Apple AI/ML
[chen_chen999, bowen_zhang4, Icao27, yinfetyyeeaple.com
Abstract
CLI / ALIGN
STAIR
Image and retrieval is one the founda- tional tasks in the vision and language domain with multiple real-world applications. State-of- the-art contrastive approaches, e.g- CLIP (Rad- ford et al., 2021). ALIGN (Jia et al., 2021), represent images and texts as dense embed- dings and calculate the similarity in the dense embedding space as the matching score. On the other hand, sparse semantic features like bag-of-words models are inherently more inter- pretable, but believed to suffer from inferior ac- curacy than dense representations. In this work, we show that it is possible to build sparse semantic representation that as powerful as, even better than, dense presentations. We tend the CLIP model and build sparse text and image representation (STAIR), where the im- sparse token space.
fores
D-dimensional umbedding
Visual Encoder
Text Encoder
"the bird is perched the tree"
Figure I: Learning sparse and interpretable em- bedding for a vision-language model. CLIP (Rad- ford et al., 2021) and ALIGN (Jia al., 2021) learn compact dense embedding, while STAIR aims to learn sparse and interpretable embedding a high- dimensional space. Each dimension in the sparse bedding represents (sub-)word from a large dictionary, with non-negative importance weight.
age and text mapped Each token in the space is (sub-)word the vocabulary, which is not only interpretable but also easy to integrate with existing informa- tion retrieval systems. STAIR model signifi- cantly outperforms CLIP model with +4.9% and +4.3% absolute Recall 1 improvement on COCO-5k tmxge- and image-tex re- trieval respectively. It also achieved better per- formance on both of TmageNet zero-shot and linear probing compared to CLIP.
dedicated encoders for images and text, enabling joint embedding in shared space. By enhancing the cosine similarity for aligned pairs and dissimi- larity for unmatched pairs, these contrastive models achieve remarkable performance in fine-tuning and zero-shot generalization for image-text retrieval, VQA, and image classification tasks.
1 Introduction
Learning high-quality and performant representa- tions from large-scale image-text data has been intensively studied in recent years. Improved vision-language representation benefits many ap- plications, including image-text retrieval (Chen et al., 2015; Plummer et al., 2015), VQA (Antol et al., 2015; Johnson et al., 2017). (), and image cap- tioning (Vinyals al., 2015). The resurgence of interest in contrastive learning has fueled recent ad- vances. State-of-the-a models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) employ
Despite the impressive benchmark performance, the dense embedding space usually considered as a black box and challenging to interpret. The meaning of an dense embedding is determined by its vector relationships with other embeddings, rather than corresponding directly to any human- understandable concept. This lack of direct cor- respondence with interpretable concepts hinders the transparency and interpretability. Moreover, de- ploying a retrieval model trained with a contrastive objective to real-world large-scale image-text re- trieval is a non-trivial task. Despite approximated
1f Work done while at Apple

---

nearest neighbor search (Guo et al., 2020; John- son et al., 2019a) can be used retrieve from dense embedding space, the cost can be high when scaling up to billions of images. Additionally, tradi- tional approaches like inverted indexes, commonly employed by search engines, cannot be straight- forwardly applied to the dense embeddings. The lack of interoperability further complicates the in- tegration of dense embeddings with other retrieval features without additional training. In comparison, interpretable sparse embeddings possess distinct advantages across numerous applications. More detailed discussion can be found in Appendix D.
the STAIR model and CLIP model, sharing the same architecture and datasct. Experiment results a significant improvement of STAIR model over CLIP on image-text retrieval tasks, with +4.9% and +4.3% recall on COCO-5K (Chen et al., 2015) text text=image and image->txt retrieval respectively. STAIR models also demonstrate similar or better performance on zero-shot classification and linear probing tasks including ImageNet. To quantify the interpertability of the embeddings, we define an terpretable space using BERT vocab. Experiments suggest STAIR substantially more interpretable. achieving a Top-1 accuracy 32.9% on ImageNet, outperforming CLIP's accuracy of 13.7%.
The community has explored sparse and inter- pretable image representations extensively. Early approaches, e.g. bag-of-words and topic models (Csurka et al., 2004: Fei-Fei and Perona, 2005; Lazebnik et al., 2006) were widely coupled with SIFT descriptor (Lowe, 2004) but found to exhibit inferior performance compared dense vectors (Lin et al., 2011; Sánchez et al., 2013). Other endeavors, like deep visual-semantic embedding (Frome et al., 2013) using ImageNet topies yielded more interpretability but still failed to outperform dense representations (Faghri et al., 2017).
Approach
We begin with an overview of the dual-encoder architecture to establish the basis of our research.
2.1 Dual-Encoder Contrastive Learning
Given a dataset of image-text pairs D {(x,, m:)). where x; and 1/1 represent the image and text respec- tively, a dual encoder learns a similarity Mx, y), such that the aligned pair (zz, 31) is assigned higher similarity score compared to the unmatched pairs sampled from negative sample set D/-
We hypothesize that the gap between sparse se- mantic and dense embedding stems from two pri- mary factors: (1) Previous works semantic em- bedding inadequately explored large-scale training to capture the rich semantics the image-text do- main. (2) Most existing semantic embedding meth- ods are built on a fixed vocabulary (e.g. thousand of concepts), which cannot handle voocbllvocayulary concepts. In this paper, we present new model, named STAIR. and a multi-stage training recipe to learn Sparse Text And Image Representation to tackle the aforementioned challenges. Our sparse representation not only matches but also outper- forms the dense image-text representations.
A dual-encoder architecture comprises an image encoder EMMAGR and text encoder ETEXT- Each encoder consists two components: 1) a stan- dard input-specific neural network f(-) and 2) a projection head g(.) that maps the features into embeddings in joint dense embedding space:
E(-) = g(f())
where f(-) is typically transformer model and g(-) pooling layer.
The similarity between an image æ and text y is measured by cosine similarity:
Inspired by the recent success of sparse represen- tation information retrieval field (Bai et al., 2020; Formal et al., 2021b), STAIR encodes the image and text into a sparse and grounded token space, illustrated in Figure 1. Specifically, images and text are mapped to sparse and interpretable in high-dimensional space, where each dimension is associated with an actual (sub-)word from a large dictionary, with non-zero weights. Notably, the proposed stage training recipe plays a critical role in the grounding of the sparse embedding.
E(x) E(y)
M M(x, v)
(2)
| II| E
The dual encoder trained by minimizing the contrastive loss Lcoxi
Mizao)/T
LCON = [D] Σ log Eeethe (
where D, = D/ U ((x;, vi) denotes the positive pair and its negative set for each pair, and is the softmax temperature.
We conduct a comparative analysis between

---

postiive negatlive
N
K
M
Projecdon Laye!
Thinn Phoal
Taknn Pistidion
Share
byndhomm
Hard
Call
Octio Liogt
Text Transformer
Trensformer
11
14
11
14
Sham
Eritedd Lodhap
Contrasive LoGS
Sparn
Tokan
Enbobciig
Image
Goos A1
Figure 2: Diagram of STAIR architecture. It employs dual-encoder architecture. Different than dense models like CLIP or ALIGN, STAIR maps the dense embeddings to sparse latent embedding space via a token prediction head. In addition regular contrastive loss minimize an image-text matching loss, FLOPs (Paria et al., 2020) loss is added to encourage sparsity of image and text embeddings.
2.2 STAIR
1:
Stage Image
Following CLIP (Radford et al., 2021), STAIR also adopts a dual-encoder architecture. As depicted in Figure 2, it consists of an image encoder and a text encoder, both comprising feature extrac- tion network f(-) and projection head g(-). In particular, the dense project head g(-) is replaced with Token Projection Head, which maps the representation to a sparse embedding space. vo- cabulary V used as the basis of embedding space for interpretability purpose.
Image Ensoder
Imwage Engoder
Enceder
Freader
"the bind peicted
sarchadd
the
Stage 3: Full Model training
Image Enzodor
Text Encader
Unilaash both text and Image encaders.
oe the
Training strategy for STAIR model. (1) The text output masked to only predict the weights of tokens that occur in the input. (2) The image encoder is frozen
Figure 3:
'he token projection head g(-) comprises two components: (1) mapping function that maps the input sequence h to a sequence of weights for cach token j in the vocabulary space V, and (2) pooling layer that summarizes the sequence into sparse embedding in the vocabulary space V.
mask removed from the text encoder. (3) All constraints are removed.
and adding the log function empirically improves performance by suppressing overwhelmingly large weights (Zhao et al., 2021). Consequently, the image embedding !MA. and text embedding EXxT are sparse vectors residing dimensional space defined by the vocabulary. In practice, relying solely on the ReLU opera- tor in Eq. (5) often leads to insufficient sparsity. To obtain better control over sparsity, we incorpo- rate FLOPs regularization loss (Paria al., 2020) to encourage a small number of non-zeros token embeddings in V:
For the mapping function, we leverage the BERT (Devlin et al., 2019) masked language model (MLM) prediction head p(.):
a |V|-
p(hg) e TRANNFORM((h)) +
(4)
where hy = f(-) corresponds to the jth token in the sequence of the feature extraction network out- put. The TRANSFORM function comprises FC layer with GELU activation and LAYER NORM layer, and e and b are the linear mapping and bias in MLM prediction head that maps the output to vocabulary space. The weights of e is tied with token embedding lookup table in the text encoder. For the pooling layer, we adopt the approach from Formal et al., 2021b,a aggregate the weight of token sparse embedding ENC:
FNC O
LFLOPS
(6)
By combining the contrastive loss and the FLOPS
loss, the STAIR model is optimized by:
ENC log(1 (ELU(max(p(hh))) (5)
L = LcoN + X CMAGEE + CTXTT
(7)
RELU activation non non-negativity of token weights
where A and A are FLOPs regularization weights

---

Table 1: Model Size (Unit: Millions), The number of parameters STAIR comparable CLIP.
to meaningful tokens in the interpretable space de- fined by V.
Text Encoder
Image Encoder
Stage 2: Training with frozen image encoder In this stage, out focus shifts to grounding the text embedding to the same interpretable space. The key idea is to leverage the image encoder teach the text encoder as a teacher model guide the training of the text encoder. Specifically, we freeze the im- age encoder while training the text encoder to align its image embedding using contrastive loss. After stage 2 training, both image and text embeddings are positioned within the same human human-interprees interpretablle embedding space constructed by V.
CLIP
53.7
88.6 86.8
STAIR
53.8
for image and text embeddings, respectively. No- tably, by adjusting these weights, STAIR embed- dings can achieve high level of sparsity, resulting in an effective embedding size smaller than CLIP. Details are discussed in Section 6.3.
3 Training Details
The STAIR model aims to achieve two goals: 1) aligning text and images in the sparse embedding space; 2) grounding the sparse embedding dimen- sion with human-understanddale (sub-)word in the vocabulary. In other words, the image and text embeddings should reside in space defined by basis vectors that correspond interpretable to- kens. However, in practice, we found that simply replacing the dense projection head with a token prediction head alone does not guarantee the 2nd goal outright. This is because the images and text are distinct modalities with inherent semantic gaps, and contrastive loss alone only encourages text/im- age alignment. We shown in Section 5.2 that the model learns shortcut by relying on less common tokens to bridge the modality gap.
Stage 3: Joint fine-tuning of both encoders The initial two-stage training establishes solid foundation for both encoders to produce human- interpretable embeddings. In stage 3, we enhance the performance of image-text matching by jointly finetuning both encoders.
Experiments show that this multi-stage training recipe critical for the embedding interpretability. A qualitative and quantitative comparison between training with multi-stage recipe and without can be found in Section 6.1 and Appendix C.
Experiments
4.1 Datasets
Our dataset is a combination of internal and pub- lic datasets, totaling 1.1B image-text pairs. The public datasets consists of Conceptual Caption 3M (Sharma et al., 2018) and Conceptual Captions 12M (Changpinyo et al., 2021). The internal dataset consists of 1B image-text pairs, including 134M clean licensed dataset (see Appendix A for details) and a 971M M noisy web-crawled dataset. The web- crawled dataset is mined using a similar approach as described in ALIGN (Jia et al., 2021). We fur- ther filter the data using a public CLIP model removing pairs with a CLIPScore (Hessel et al.. 2021) below 0.2.
To address this challenge, we propose a multi- stage training approach that sequentially bridges the discrepancy between the sparse embedding space and the interpretable space defined by the vocabulary as illustrated in Figure 3.
Stage Training image embedding with masked tokens In the first stage, we co-train both encoders while applying a binary mask to the text embedding. Formally, given the text input y, the masked text embedding is formulated as:
ENC
TEXT MASK
g(f(w)) MASK(y)
(8)
4.2 Configurations
where MASK, = 1 if the jth token exists in y, and g( (v)) predicts the weights of the non- mon-md to- kens. In other words, the text embedding is forced to activate the only the tokens appearing in the original text input, while disregarding others. By matching with the masked text embedding, the im- age encoder learns to ground its image embedding on the tokens from the paired text. Consequently, after the stage 1, it anchors the image embeddings
In the experiment, train CLIP model and STAIR model for comparison. For both models. we utilize transformer (Vaswani et al., 2017) as the backbone with modified CLIP-B/16 (Radford et al., 2021) configurations. The text encoder a 12-layer Transformer with 512 hidden dimensions
hhttps:/hugggiggface http://uugginggaaac.ccoaaaaaaal clip-vit- pase--

---

Table 2: Zero-shot text/image retrieval. Reporting recall @ on Flickr30K and COCO.
COCO 5K
Flickr30K
text - image
image R@5
text
image
image text
R@1
R@5
R@ 10
R@1
R@ I0
R@0
R@5
R@ 10
R@)
R@5
R@ 10
CLIP STAIR
36.2
62.2
72.2
53.4
78.3
85.6
63.0
86.7
92.5
79.6
95.5
98.1
41.1
65.4
75.0
57.7
80.5
87.3
66.6
88.7
93.5
81.2
96.1
98.4
Table 3: Zero-shot classification accuracy. Reporting the top-1 accuracy (%) across datasets.
ImageNet
Caltech-101
CIFAR- 100
SVHN
DTD
OxPet
OxFlowers
Eurosat
RESISC45
CLIP
65.1
82.3
63.2
42.0
53.6
85.8
67.7
52.4
64.3
STAIR
65.6
82.5
63.4
53.0
56.3
85,9
68.2
51.0
62.8
and attention heads, while the image encoder is a 12-layer ViT with 768 hidden dimensions and 12 attention heads. The text input is tokenized using
hancement in recall@1 for COCO-5K text->image and image-text retrieval, respectively. Similar improvements are observed on the Flickr30K.
BERT WordPiece tokenizer (Devlin et al., 2019) with vocabulary of 30,522 tokens. The maximum text input sequence length is set to 76.
4.4 Zero-Shot Visual Classification
In Table 3, we report zero-shot image classifica- tion top-1 accuracy on 9 datasets using the same prompts set from Radford et al. (2021). The re- sults indicate that STAIR either outperforms per- forms competitively with CLIP on most datasets. Particularly, STAIR shows significantly better per- formance on SVHN that requires an exact match, thanks to its token grounding capability. However, we observe that STAIR also struggles with spe- cialized
The CLIP model trained for 600K steps using the LAMB optimizer (You et al., 2020) with a learn- ing rate of le and a weight decay of le 2. The learning rate is warmed up for the first 10k steps and then linear decay to 0. The STAIR model goes through 3 stages, with each stage trained for 300K, 300K, and 600K steps, using the same configura- tion CLIP. To mitigate catastrophic forgetting, a smaller max learning rate of 1e is adopted in stage 3. FLOPs regularization weights are set to d1 X2 -3 by default, following the quadratic growth in Formal et al., 2021a. Unless specified otherwise, all models are trained using global batch size of 66,884 All our experiments were conducted using 64 A100 GPUs.
tasks, such as Eurosat (Helber et al., 2019) and RESISC45 (Cheng et al., 2017), which is consistent with the observations from Radford et al. (2021).
4.5 Linear Probe of Visual Classification
We also compare the linear probe performance us- ing the same datasets Table 4. We observe that the STAIR model consistently achieves superior results compared to CLIP, even on Eurosat and RE- SISC45, where it exhibits weaker performance in zero-shot. This improvement can be attributed to the larger embedding dimensionality of 30,522 in the STAIR model, compared to 512 in CLIP. More- over, with sparsity enforced in STAIR, there no extra cost on storage and computation compared the dense embeddings (further discussed in Section 6.3).
is noted that the proposed STAIR model does not introduce additional parameters compared to CLIP, because the linear projection e for both the image and text modalities parameters with token embedding lookup table as described in Section 2.2. The detailed number of learnable parameters for each model are summarized in Table 1.
4.3 Zero-Shot Text Image Retrieval
Table 2 shows the @ (K=1, 5, 10) perfor- mance of image/text retrieval tasks on Flickr30K (Plummer et al., 2015) and COCO-SK (Chen et al., 2015). The metrics are reported with prompt of "a photo of added before the original caption, following Radford al. (2021). We observe sig- nificant improvement of STAIR models compared to the CLIP baseline, with 4.9% and 4.3% en-
Interpretability
5.1 Quantatively analysis
Interpreting high-dimensional embeddings poses a challenge because their representations in the em- bedding space Rd not naturally correspond to

---

Table 4: Linear probe classification accuracy. Reporting the top-1 accuracy (%) across datasets.
ImageNet
Caltech-101
CIFAR- 100
SVHN
DTD
OxPet
OxFlowers
Furosat
RESISC45
CLIP STAIR
77.0
94.9
76.7
65.8
80.6
88.8
97.6
95.1
92.6
78.1
96.1
77.7
72.5
82.7
91.7
98.2
95.9
93.9
Improved interpretability of the STAIR model. We report the top-K accuracy (%) of the la- bel among all of the vocabulary in the tokenizer on the ImageNet, CIFAR-100, and CalTech- 101 datasets.
Table 5:
Input
ImageNet
Top-1
Top-10
Top-50
Top- Top-100
Mitopper: 0.32 etboarrding: 0.32 #ttee: 0.31, aeghan: 0.31, #tak: 0.31, Mtaava: 0.31, 0.31,
STAIR
32.9
69.0
83.8
87.7
CLIP
0.31, thjjr: 0.35, httjang: 0.31, oetong: admel: 0.3, Athuz: #dbel: 0.3, ARchar:
CLIP
13.7
34.3
47.0
51.9
CIFAR-100
Top-1
Top-10
Top-50
Top- 100
panda: 2.38, bear: 1.63, zoe: 1 1.19, widdfla:
cat 1, suitoase: 1.14, lugagge: 1.08,
STAIR
amused: 0.83. mustering 0.81, irawwiing: pelie
STAIR
10.3
56.8
75.4
80.7
dhiw:: 1.1, annoldd: 1.01,
giant: 1.01, mammals: 0.58
0.78, packingg: 0.77, kitten:
CLIP
8.0
28.9
44.7
50.5
CalTech-101
Top-1
Top-10
Top-50
Top-100
STAIR
29.3
45.4
56.0
64.8
Figure Examples of relevant (sub-)word) in the lexicon space and the similarities in CLIP and STAIR. STAIR provides more interpretable visual concepts compared to CLIP. ## indicates subword from the vocabulary.
CLIP
8.1
24.2
38.9
43.8
vocabulary of the BERT WordPiece Tokenizer (De- vlin et al., 2019), approximating a lexicon space that covers any human-interpretable concepts. Note that, under this definition, the embedding space becomes the same as the interpretable space for STAIR and F reduces to an identity function. Sim- consider ilar to zero-shot image classification, an image embedding easier interpret if closer to its ground-truth -ruth class in the lexicon space. This task is generally more challenging than the tradi- tional zero-shot classification because the candi- dates now become the entire vocabulary, which is much more than the predefined classes.
easily understandable human concepts. To over- come this challenge, Kim et al. (2018) proposed leveraging an interpretable space R" spanned by vectors e representing human-intenn interpeaatteeeelalee con- cepts. From this standpoint, interpreting the em- bedding can be seen as a mapping F Rd + RH For image-text dual encoder models, suitable interpretable space that connects multimodalities is lexicon space spanned by basis vectors rep- resenting humaa-understandable tokens, words, phrases, and/or sentences. The interpretation can then be determined by the similarity between the embedding and each basis vector, Sim(( c). The lexicon space is crucial for comparing interpreta- tions between different models. For example, if we have an embedding from image of a dog but the lexicon space does not include concept for "dog." it becomes challenging to understand the image embedding itself.
We compare the interpretability of CLIP and STAIR on three datasets, ImageNet, CIFAR-100. and Caltech- 101 and use the Top-K accuracy as quantitative metrics. Specifically, a ground-truth class is tokenized into separated sub-words e the vocabulary, we consider the maximum sim- ilarity over the sub-words, max; (Sim(, as the final similarity. As shown in Table 5, our sults demonstrate that STAIR exhibits significantly superior interpretabilityy compared to CLIP in the interpretable space.
Zero-shot image classification is restricted form of functionally-groundee interpretability eval- uation (Doshi- Velez and Kim, 2017) as its inter- pretable space is predefined by its classes and dataset specific. However, in practice, the inter- pretable space can be both lack of human labels and unlimited (Ghorbani et al., 2019). To lift the constraint, we expand our interpretable space as the
5.2 Qualitatively analysis
We also conduct a qualitative examination com- pare the interpretability of embeddings from the STAIR and CLIP models. Figure 4, we present the top Sim(-, ct)) (sub-)words in the interpretable

---

wavelick blrd
woddings lle gnutt
OUOTO TRE
gul
subsuectes Rataiist
berress Iriit flick recepttions
sydney
ashere ceesto
werves refetroduuce
photograph
siicing cakes tradiitinally
cake yu marrianc cheesa
presperde
Vabe compotnorss
Ming WAV
australaa
1R00T narbour
nlumage schleswiu
grapes wedding
BUStral
(a) Caption: A seagull standing on the sand of beach.
(b) Bride and grooms arms cutting the wedding cake...
(c) couple of people on kayak boats the middle...
Visu Tooustee mimy concourse window materri Muhts
troicmmbria hending lamb scataa
Inch livestock
beeing ana nippón
iivrvalriime spotting
creditorr
gutction
airports airline trand airlines
airport delle alrorft
unmnay wottic
driving sheep goats flick
bustest
iapana Irrorrt nights departing rep republlssed
airilnes
aarcraft
rendsite
(d) An airport filled with planes sitting on tarmacs.
(e) Sheared sheep on roadway taken from vehicle.
(l) A plane on the runway is being led by tow cart.
suechneicediaae mating snbspcies auhile yeese brant wilditle
republished mmm relevised leasi plechr istinss
minista bey an roads
ABEEY ROAD KNS
Westminster
thanksgiving dinner
minster nwnorthwest abbey
photograph goose
chicken stutiing
trrkey roast
by read ahly
vession caurtshio SWaES
tastel GaUS holidays
meal bread
(g) There a sign in front of brick house.
(h) Two ducks floating together on wa- ter.
(i) A table set for a traditional Thanks- giving dinner.
Figure 5: Predictions from STAIR model. Top 20 tokens predicted by STAIR for image. Font size indicates prediction weight. The original caption is shown below the image. Detailed weights are provided in Appendix F.
Input
STAIR Dragg
space defined by the BERT WordPiece vocabulary for each image. We see that STAIR is better at capturing visual concepts that humans can easily understand than CLIP, which is consistent with our quantitative analysis. Additionally, we observe that the top tokens from CLIP tend to have similar matching scores while STAIR avoids the problem by adopting Eq. (5) in the Token Projection Head. Figure 5 shows more examples of (sub-)words with the highest weights from STAIR embeddings given each image. The results suggest that STAIR is capa- ble of grounding images to tokens that are semanti- cally related. For example, it can infer "wedding" from the picture depicting a bride and groom cut- ting a cake. This grounding and interpretability ability of STAIR is highly valuable for debugging and understanding the model's behavior. For ex- ample, observe bias towards activating the "https" token in many predicted embeddings. We find that mainly stems from substantial portion of web-mined content in our training data, where "https" frequently appears in associated texts.
120
tanded
-13
seeting (Feepd
pen
on dkit on sinvey hill. with linge recntonn in the buckground.
perton riding fownhil
Searchly
purch baod
pupul
1:01
1400
1.56
Ahono
Jago
1.50
1.99
(ow/r)
2.11
1.55
0.36
1.20
0.74
1.40
0.29
tadee
1.15 Dm
1.52
1.00
Nera
Seam
skimich
recouiity
steodion
2.50
1460
Nuttlee
grop of ditlewer calored beddy heom sitting on top of Mue table.
Seart
1.20
t5ye
0.55
Sorbec
1.50
milgknn
1.5
r8p4
1.9
0.70
1:40
1.18
beam
Figure 6: Examples of top predictions and weights from STAIR models. STAIRs predic- tions are disconnected from the actual input meaning, while STAIR predictions from multi-stage training are grounded to input semantics. ## indicates subword from the vocabulary.
for comparison. Figure 6 illustrates the predic- tions from STAIR and the multi-stage STAIR models, respectively. We observed that STAIR tends to redefine the semantic meaning of tokens and repurpose them to match images and text. For instance, tokens such as "em- inent" and "nottingham" are redefined as topics related to teddy bears, as they consistently appear for teddy bear related images. Similarly, "clue" is reinterpreted as a concept related to skiing. While we can deduce the new semantic meaning through reverse engineering, this task is non-trivial. In con-
6 Analysis and Discussion
6.1 Necessity of Multi-stage training
Multi-stage training strategy plays crucial role in ensuring STAIR embedding grounded meaning- ful tokens. To demonstrate its necessity, we train a single-stage model, denoted as

---

Stage 1 Stage
Acchhaatodd Toiens
2000
FLO'S regularization weight
Accurecye1 recallss:
1e
60%
Stage
1500
10
40%
1000
20%
300
0%
Imagelitt
COCO
COCO
Flickr2kk
Flickra0k
imagefiet, FICkK30K; COCO 5R; ImogeWet, COCO Sk: Fickrroh
(a) Number of Activated Tokens. Suhscripts f and denote results for text and image, respectively. The horizontal line represents an embedding size of 512 of CL.IP baseline. The effective text embedding size in STAIR is significantly smaller than CLIP when À 2 1e" enabling faster retrieval.
Figure 7: Ablation multi-stage training. Stage I establishes solid foundation for zero-shot performance. Stage enhances performance primarily on ImageNet. Stage 3 provides additional improvements on both Ima- geNet and COCO/Flickr30K.
FLOPs regularization weight
1e-2
1
80%
Accuracya1 Reccliia
trast, the inclusion of multi-stage training ensures that tokens retain their original meaning, resulting in a heman-readable embedding 3.
60%
40%
20%
6.2 Ablation Multi-stage training
0%
Imageliet
text
this section, we qualitatively study the effect of the multi-stage training strategy on zero-shot transfer. Figure 7 shows the top-1 accuracy of Im- ageNet classification and recall 1 on COCO-5K and Flickr30K image-text retrieval for each stage separately. We observe that even in stage reason- able performance can already be achieved, despite the restricted activation of text tokens. Stage 2 pro- vides additional improvements, particularly in the classification task. Stage 3 significantly enhances the text-image matching capability, as evidenced by the improvements in all metrics.
Performance zero-shot transfer. More sparsity
embedding lowers the zero-shot transfer performance.
Figure & Ablation on FLOOSs regularization weights.
complexity of sparse embeddings dot product is lin- ear to the smaller number of non-zero units in two embeddings, STAIR models are more efficient in conducting similarity computation during retrieval compared to CLIP. Moreover, we observe that more tokens are activated in the image embeddings than in the text embeddings. One explanation is that the image semantics is usually broader and more gen- eral while the text meaning is more specific. We also notice that when À is large, text embedding tends to be more sparse especially when the inputs are shorter. On the other hand, the regularization weights show negative impact on zero-shot per- formance, particularly in the case of retrieval tasks.
6.3 Ablation on Sparsity
Embedding sparsity is crucial for ensuring the ef- ficiency of similarity computation and retrieval speed. In STAIR, the strength is controlled by the FLOPs regularization weights. To study its impact, we train three STAIR models with regularization weights A = X2 E {1e-2, -3 1e te. We examines their text and image embedding sparsity, iee. the number of tokens with non-zero weights in the predictions, well as their zero-shot trans- fer performance on is ImageNet, COCO-5k, and Flickr30k summarized in Figure 8.
Related Work
Image and text retrieval Image-text retrieval ap- proaches can be categorized into dual-encoder and cross-encoder approaches. In the dual-encoder ap- proach, images and text are encoded separately as dense embeddings. DeVISE (Frome et al., 2013) was of the pioncering dual-encoder models. With the advent of transformers, Radford al. (2021) proposed CLIP that leverages large-scale pretraining datasets and established new of the-art across multiple benchmarks. Finetuning the visual model (Dosovitskiy et al., 2021) and lan-
The results suggest that, when À 2 1e-3, the ef- fective number of tokens in STAIR is significantly lower than the dense embedding dimension of 512 used in CLIP for text embeddings. Since the time
"Interestingly, the STAIR model is still able to achieve comparable performance to multi-stage training on various retrieval and classification tasks. Further details on STAIR training and quantitative results can he found in Appendix C.

---

guage model (Devlin et al., 2019) further improves performance. On the other hand, the cross-encoder approach like UNITER (Chen et al., 2020) employs a single encoder for both image and text inputs. While cross-encoder models capture fine-grained alignment between image and text but are slower at inference, dual-encoder models offer faster re- trieval speed with precomputed embeddings.
important to acknowledge its limitations. We dis- cuss the following aspects areas for future im- provement and exploration.
Interpretability Loss of Word Order The vocabulary-based representation employed by STAIR, utiliziig unigram vocabulary for sparse embeddings, demonstrates strong performance in various scenarios. However, it may encounter chal- lenges in interpretability for humans when dealing with phrases or sentences that rely on the specific order words, such as distinguishing between "dog chasing cat" and "cat chasing dog". While the sparse embeddings still maintain competitive per- formance in terms of text-image semantic match- ing, humans may find it difficult to infer the in- tended semanties solely from individual activated tokens. To address this limitation, future research could explore incorporating bigram representations or combining sparse and dense embeddings to cap- ture a wider range of linguistic nuances and en- hance intcrpretability.
STAIR follows the dual-encoder approach but introduces a novel aspect by using sparse embed- dings instead of dense embeddings. This choice of sparse embeddings in STAIR leads to improved in- terpretability and better retrieval performance com- pared to dense embeddings.
Document retrieval via sparse embedding Sparse embedding has been widely used in infor- mation retrieval (Dai and Callan, 2019; Bai et al., 2020; Jang al., 2021). By hashing the sparse embeddings, retrieval can be efficiently performed using an inverted index system. Our work draws inspiration from SPLADE (Formal et al., 202 employing sparse embeddings. However, unlike SPLADE, our approach tackles retrieval across modalities. The semantic gap between images and text makes designing joint sparse embedding space challenging. Additionally, grounding images and text to meaningful tokens non-trivial task. In STAIR, we propose a streamlined approach that enables fast retrieval speed, interpretability, and high retrieval accuracy in the multimodal setting.
Prediction Bias Another limitation pertains potential prediction biases that can arise from the training data, particularly due to web-mined con- tent (Chang et al., 2019; Dev al., 2021). The prevalence certain concepts or biases web data can influence the model's predictions, leading to unintended biases in image-text retrieval or classi- fication tasks, as well as impacting interpretability. Addressing this issue requires careful considera- tion of the training data sources and strategies mitigate biases during model training.
8 Conclusion
In this paper, we introduced STAIR (Sparse Text And Image Representation), an approach that en- codes image and text inputs into sparse embeddings within sparse token space. We also employed multi-stage training strategy ensure that the em- beddings are grounded in meaningful tokens. By comparing STAIR with the CLIIP model, we ob- served that STAIR achieved superior performance on image-text retrieval tasks and demonstrated bet- ter results on various zero-shot and linear probing classification tasks. Furthermore, through quantita- tive and qualitative analysis, we illustrated the in- terpretability advantage of our sparse embeddings over dense embeddings, making them more easily understandable for humans.
Computation Overhead of Multi-Stage Train- ing We acknowledge that the multi-stage training process in STAIR requires more iterations than the baseline CLIP model, resulting in increased computation overhead. It is worth noting that the additional computational cost through the multi- stage training necessary achieve interpretabil- ity. We provide fair comparison by reporting the performance of a single-stage STAIR model trained under the same computation budget in Appendix C. The results demonstrate that even the single- stage STAIR model performs competitively with or outperforms CLIP, showcasing the efficacy of the approach within given computational constraints.
Limitations
A Acknowledgements
While STAIR demonstrates several strengths in image-text representation and interpretation, it is
We thank our teammates from Visual Intelligence, foundation model, and other Apple groups for their

---

feedback and suggestion. Sepcial thanks goes to Navdep Jaitly, Brandon McKinzie, Xianzhi Du for the early review of the paper, Floris Weers, Vaishaal Shankar for linear probing experiments setup and Cybertron team for infrastructure supports.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Ar- jun Subramonian, Jeff Phillips, and Kai- Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1968-1994.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of NAACI-IILT, pages 4171- 4186.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Finale Doshi- Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.000 082088608.
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang. Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. Sparterm: Learning base- sparse representation for fast text retrieval. arXiv preprint arXiv:2010.00768
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations.
Kai-Wes Wei Chang, Vinodkumar Prabhakaran, and Vicente Ordonez. 2019. Bias and fairness in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNPP): Tutorial Abstracts.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612.
Soravit Changpinyo. Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 3558-3568.
Li Fei-Fei and Pietro Perona. 2005. A bayesian hierar- chical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition (CVPR'05) volume 2, pages 524-531. IEFE.
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2021a. Splade v2: Sparse lexical and expansion model for informa- tion retrieval. arXiv preprint arXiv:2109.1000
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:7504.0033
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021b. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288-2292
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
Andrea Frome, Greg S Corrado, Jon Shiens, Samy Ben- gio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic em- bedding model. Advances in neural information pro- cessing systems, 26.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. 2017. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, (05(10):1865-188) 1883.
Amirata Ghorbani, James Wexler, James 7ou, and Been Kim. 2019. Towards automatic concept- based explanations. Advances in Neural Information Pro- cessing Systems, 32.
Gabriella Csurka, Christopher Dance Lixin Fan, Jutta Willamowski, and Cédric Bray. 2004. Visual cate- gorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV, vol- ume pages 1-2. Prague.
Ruiqi Guo. Philip Sun, Erik Lindgren, Quan Geng. David Simcha, Felix Chem, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning, pages 3887-3896. PMLR.
Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv: 1910. 10687.

---

Shayan Hassantabar, Zeyu Wang, and Niraj K Jha. 2021. Scann: Synthesis of compact and accurate neural net- works. IEFE Transactions Computer-Aided De- sign of Integrated Circuits and Systems, 4009):0122- 3025.
Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06) volume 2. pages 2169-2178.
Patrick Helber. Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226.
Liunian Harold Li. Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded sanuuge-image pre-training. In Proceedings of the IEFF/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10965- 10975.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528.
Jimmy Lin, Xueguang Ma. Sheng-Chiell Lin, Jheng- Hong Yang, Ronak Pradcep. and Rodrigo Nogueira. 2021a. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356- 2362.
Kyoung-Rok Jang, Junmo Kang, Giwon Hong, Sung- Hyon Myaeng. Joohee Park, Taewon Yoon, and Heecheol Seo. 2021. Ultra-high dimensional sparse representations with binarization for efficient text re- trieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 1016-1029.
Sheng-Chieh Chieh Lin, Theng-Hong Yang. and Jimmy Lin. 2021b. Contextualized query embeddings for con- versational search. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 1004-1015.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR.
Yuanqing Lin, Fengjun Lv, Shenghuo Zhu, Ming Yang, Cour, Kai Yu, Liangliang Cao, and Huang. 2011. Large-scale image classification: Fast feature extrac- tion and svm training. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, pages 1689-1696.
Jeff Johnson, Mathhjs Douze, and Hervé Jégou. 2019a. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-544 547.
David G Lowe. 2004, Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019b. Billion-scale similarity search with GPUs IEEE Transactions on Big Data 733):335-447.
Biswajit Paria, Chih-Kuan Yeh, Ian EH Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020. Minimizing flops to learn efficient sparse represen- tations. In International Conference on Learning Representations.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei- Fei.Fei. C Lawrence Zimick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-pha to-hrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pages 2641-2649,
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. In- terpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In In- ternational conference machine learning, pages 2668-2677. PMLR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark. et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning. pages
PMLR.
Carlos Lassance and Stéphane Clinchant. 2022. An efficiency study for splade models. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2220-2226.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. 2022. Perceptual grouping vision-language mod- els. arXiv preprint 120.992210.0999.

---

Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. 2013. Image classification with the fisher vector: Theory and practice. International journal of computer vision, 105:222-245.
B Impact of training data and global batch size
In Section 3, both the CLIP and STAIR models were trained on the I.IB dataset with a batch size of 16,384, which was determined by our computa- tion budget. To investigate the impact of training data and global batch size, we conducted additional experiments. Specifically, we trained a CLIP model with a batch size of 32,768, denoted as CLIP 32K- and compared its performance with the OpenAT CLIP-B/16 benchmark (Radford et al., 2021). The benchmark model was trained on a 400M dataset with batch size of 32,768, as shown in Table 6, Our implemented CLIP32x performs competitively on ImageNet and achieves significantly better sults on retrieval tasks when trained on our larger I.IB dataset.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 2556-2565, Melbourne, Australia. Association for Computational Linguistics.
Tao Shen, Xiubo Geng. Chongyang Tao, Can Xu, Xiao- long Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2022. Lexmae: Lexicon- botteeee-bbtteeeeeed pre- training for large-scale retrieval. arXiv preprint arXiv:2208. 14754.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Furthermore, we observed that the choice of global batch size has a noticeable impact on both zero-shot classification and retrieval benchmarks. Despite using a smaller batch size, the STAIR model outperforms other models on the COCO image/text retrieval task. This highlights the ef- fectiveness of the STAIR approach even with a reduced batch size.
Oriol Vinyals. Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. 2020. Large batch optimization for deep learning: Training bert in 76 minutes. In International Confer- ence on Learning Representations.
It is important to emphasize that our primary focus is on introducing novel approach to image- text representation learning. Despite smaller batch size, fair comparisons can be made when models are trained under similar condition. The STAIR approach is not limited to specific model and can be applied to other dual-encoder architectures such as ALIGN (Jia et al., 2021) and GLIP (Li et al., 2022).
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee, 2021. Sparta: Efficient open-domain question answering via sparse transformer matching retrieval. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 565-575.
C Single-stage training
In this section, we investigate the performance of the STAIR model when trained without the multi- stage training strategy described in Section 3. We denote this variant as STAIR model shares the same archi- tecture as the multi-stage STAIR model depicted in Figure 1, and it is trained using the same configura- tions for 600K steps as the CLIP model described in Section 4.2. The FLOPs regularization weights A and A are set to the same values the STAIR model, i.e., \ = X2 13
A Clean Licensed Dataset
The High Quality Image Text Pairs (a.k.a. HQITP- 134m) dataset consists of approximately 134m di- verse and high quality images paired with descrip- tive captions and titles. Images range in spatial resolution from 320 to 2048 pixels on the short side. All images are JPEG format and most are RGB. Each example image is associated with ti- tle, and a list of several captions. On average, these captions consist of 20.1 tokens, with the shortest caption consisting of just one token and the longest spanning over 1000 tokens. This dataset was li- censed to our industrial rescarch lab by a third party for commercial use.
The
Table presents the zero-shot image-text re- trieval and classification performance of the two versions of the STAIR model and the baseline CLIP model. Remarkably, the STAIR F-

---

Table Zero-shot transfer of open-source CLIP vs our CLIP using 32K batch size. We report recall K on Flickr30K and COCO, and top-1 accuracy (%) ImageNet. We attach 16K batch-size STAIR metrics for reference.
ImageNet Acc@!
COCO SK
3lickr30K
Data Size
Batch Size
text - image
image + text R@1
text + image
image - text R@1
R@1
R@1
CLIP ORENAI
400M
32,768
68.6
33.3
63.5
82.3
CLIPaa 32x
32,768
39.0
57.4
67.0
85.0
CLIP
1.1B
16,384
36.2
53.4
79.6
STAIR
1.1B
16,384
41.1
57.7
81.2
Table Zero-shot text/image retrieval and classification of STALRSINGLE-SSSRE We report recall and top-1 accuracy (%) Bold indicates the best overall performance.
COCO 5K
Flickr30K
text + image
image + text
text + image
image + text
R@I I
R@5
RØ 10
R@1
R@5
R@ 10
R@
R@5
R@
R@
R@5
R@ 10
CLIP
36.2
62.2
72.2
53.4
78.3
85.6
63.0
86.7
92.5
79.6
95.5
98.1
STAIR SINIRRINGGE--5T SINGLR-STAGE
40.5
64.9
74.9
57.7
80.3
87.4
66.7
88.6
93.7
81.0
96.7
98.5
STAIR
41.1
65.4
75.0
57.7
80.5
87.3
66.6
88.7
93.5
81.2
96.1
98.4
Caltech- 101
CIFAR- 100
ImageNet
SVHI
DTD
OxPet
OxFlowers
Eurosat
RESISC45
CLIP
65.1
82.3
63.2
42.0
53.6
85.8
67.7
52.4
64.3
STAIR SOOLES
64.0
81.5
63.7
39.4
56.7
85.9
67.0
49.5
62.2
STAIR
65.6
82.5
63.4
53.0
56.3
85.9
68.2
51.0
62.8
model achieves similar performance to the multi- stage trained model in terms of zero-shot tex- t/image retrieval and classification tasks. Both STAIRSIGGLE-STAEE and STAIR models outperform the CLIP model across most of the metries.
often rely on approximated neighbor search tech- niques, such k-means clustering and product quantization, handle large-scale scenarios. These operations be computationally costly and may require additional approximations or quantization to reduce memory usage, leading to loss of pre- cision. Additionally, updating the index with new data often necessitates extra computation or even re- re-computation, operations like k-means cluster- ing and product quantization are data-dependent.
However, it is important to note that the in- terpretability of the STAIR model is considerably worse, as indicated in Table 8. As discussed in Section 5.2, the STAIRSINOLS-STAGE model functions as modtl clustering algo- rithm that repurposes words weighted token cen- troids. The contrastive objective trains the model to prioritize matching the aligned image and text using these token centroids. However, it lacks the capability to effectively constrain the predicted to- kens to their original human-readade meanings compared to multi-stage training.
In contrast, token-based retrieval systems in STAIR do not face these challenges, as the indexing is based directly on the tokens themselves. Further- more, the tokens in STAIR optimized with FLOPs regularizer to encourage a uniform distri- bution among sparse tokens, which benefits the retrieval process. Moreover, the interpretable na- ture of the token-based system provides additional advantages, such as the ability to build customized query trees using logical operators, leverage token- based blacklists/hhitelistss on both the query and index side, and combine with other token-based features in the inverted index system. This ben- efits applications like web image search, where supplementary signals such as image text and surrounding textual content play important role be- sides the image content itself.
D Token Based vs Embedding Based Search
The development of mmbeediing-based retrieval systems has gained significant attention in recent years (Hassantabar et al., 2021: Johnson et al., 2019a; Lin al., 2021b). While these systems show promising progress, deploying large-scale embedding- based search systems still poses sev- eral challenges. Firstly, embedding- mmbedddig-bbsed systems

---

Table 8: Interpretability of STAIR SINGL We report the top-K accuracy (%) of the label among all of the vocabulary. Bold indicates the best overall performance.
ImageNet
CIFAR-100
CalTech
Top-1
Top-100
Top-1
Top-100
Top-1
Top-100
CLIP STAIR
13.7
51.9
8.0
50.5
8.1
43.8
0.0
0.6
0.0
0.3
0.0
0.2
STAIR
32,9
87.7
10.3
80.7
29.3
64.8
Table 9: Retrieval Speed Comparison. STAIR achieves better retrieval speed compared to CLIP with comparable retrieval accuracy (%).
E.1
Text Encoder Free Localization
The STAIR model is capable of localizing image regions related to an arbitrary query (c.g., an object) from the vocabulary without relying on the text side for inference
QPS
R@ 10
R@25
CLIP
6.02
42.1
55.2
Recall that the image encoder uses a vision trans- former that divides the original image into grids. Equation introduces the mapping function p(.), which projects each token/grid representation to the vocabulary space, where each dimension represents the activation of the corresponding token from the vocabulary. Building this, we can identify the regions correlated with query by examining the activation scores of the input tokens at each grid. Figure 9 illustrates examples of images given arbi- trary text queries and their corresponding activation heatmaps. In the first example, we visualize the activation maps for the queries "German Shepherd Dog", "Dog", and "Cat". Both "German Shepherd Dog" and "Dog" align well with the actual dog in the image. In contrast, the activations for the "Cat" query spread across the entire image. Additionally, we observe that the activation heatmap for "Dog" aligns better compared to the heatmap for "German Shepherd Dog". This is because the multi-token query is decomposed into multiple tokens from the pre-defined vocabulary, and tokens "German" and "Shepherd" often represent other concepts rather than dog. As mentioned in Section 8, better- tuned vocabulary, which includes "German Shep- herd Dog" as single token, would help address this issue.
STAIR
14.1
42.5
55.7
D.1 Retrieval Speed
Retrieval speed between dense and sparse embed- ding has been extensively studied in the informa- tion retrieval community (Shen et al., 2022; Las- sance and Clinchant, 2022). For instance, the Shen et al. (2022) demonstrates that a sparse model can achieve similar or better efficiency and accuracy trade-offs compared to dense model. One can tune the retriever, such as using the top N tokens, which still results in strong performance. However, latency and accuracy depend heavily on the imple- mentation and configuration of retrieval systems.
We conducted a preliminary comparison of CLIP and STAIR models using the popular retrieval li- braries FAISS (Johnson et al., 2019b) and Py- scrini (Lin et al., 2021a) on a 1M-scale dataset sampled from COCO (Chen et al., 2015). The pre- liminary results are summarized in Table 9 showing that STAIR is capable of achieving 2x speed over CLIP while maintaining comparable retrieval ac- curacy. Comprchensive comparison would require more extensive experimentation, which we will leave as future work.
We hypothesize that the localization capability of STAIR models stems from the max pooling op- eration in Eq. 5. According to Ranasinghe et al., 2022, simply changing the image and text encoder poolers to max pooling can significantly enhance the localization and segmentation capability of a CLIP model. While the CLIP model still requires text encoding to align with the patch representa-
E Text Encoder Free Applications
The token grounding capability of the STAIR model opens up possibilities for more efficient ap- proaches existing tasks. In this section, we ex- plore two potential applications: 1) Text Encoder- Free Localization, and 2) Text Encoder-Fred Image- Text Retrieval.
"In cases where query is not in the vocabulary or consists of multiple tokens, the query are tokenized into in-vocabulary tokens, and their average is used for localization.

---

raage Drugated
Gunan Shealerd
DPg
Ce:
Retrivall
Retrieval
Image Encoder
Text Encoder
Image Enceder
"the bid porched
"Uh bire perched
Oui Diood
E
Figure Dual encoder text encoder free archi- tecture. Text-eneoder- free architecture contains only image encoder and uses MASK as text embedding.
Ime Modda
Zebra
Laga
as S compared to the original STAIR. While it may not match the performance of the dual encoder baseline, the results are encouraging given its potential. We observe that the text-encoder- free STAIR demonstrates relatively stronger per- formance ImageNet classification compared to text/image retrieval tasks. This discrepancy is due to the fact that ImageNet classes are generally more concise. In contrast, captions from COCO-5k and Flickr30k often contain more stop words, which are treated with equivalent importance semantically meaningful tokens by MASK. This indicates signif- icant room for improvement in text-encoder- performance, which we leave as future work.
Figure 9: Qualitative examples of text inference free localization in STAIR Left side shows the original images from ImageNet or COCO 5K. Right side shows the queries and their activation heatmaps. (L. indicates the query is the original ImageNet label of the image. (The heatmap colors are normalized for better visualiza- tion, not representing real values.)
tions from the image, STAIR does not rely on any inference from the text side. These findings high- light the potential of STAIR models for localization, segmentation, and open vocabulary detection tasks, which we leave as future work.
F Image Prediction Weights
We provide the detailed weights of predicted tokens for each image from Figure 5 in the Table 11 for reference.
E.2 Text Encoder Free Image-Texx Retrieval
Another notable advantage of the STAIR model is its ability enable rext-encoder-fres retrieval systems. Figure 10 compares the dual-encoder ar- chitecture with the text-encoder-free architecture. Specifically, we utilize the image encoder of STAIR to generate sparse image embeddings, while the texts are directly converted into MASK in the vo- cabulary space after tokenization, serving the sparse text embeddings. Unlike the dual-encoder approach that requires inference from both the im- age and text encoder, this architecture only relies on the former, making it suitable for applications with restricted latency requirements. Note that also differs from retrieval systems built with fixed taxonomies using inverted indexes since it can han- dle any free text inputs.
Table 10 summarizes the zero-shot performance of the text-encoder-free STAIR model, denoted

---

Table 10: Zero-shot of STAIR with and without text encoder inference. We report recall @ K on Flickr30K and COCO, and top-1 accuracy (%) ImageNet.
ImageNet Acc@1
COCO 5K
Flickr30K
text + image
image + text
text -) image
image + text R@1
R@1
R@1
R@1
STAIR
65.6
41.1
57.7
81.2 66.6
STAIR
50.0
21.0
27.2
40.4
47.0
Table 11: Detailed weights of STAIR predicted tokens. Top 20 tokens predicted by STAIR for an image. ## indicates subword from the vocabulary.
Hitgul: 2.09, ##ll: 1.56, photograph:
cake: 1.75, wedding: 1.47, cakes: 1.31.
sydney: 1.98, australia: 1.92, australian:
1.35, plumage: 1.19. waves: 1.18,
flick: 1.26, tier: 1.25, grapes: 1.23, fruit:
1.42. opera: 1.33, kay: 1.32, ##dling:
ashore: 1.13, beaches: 1.12, beach: 1.1,
1.18, weddings: 1.15, receptions: 1.09,
1.27, paddle: 1.17, canoe: 1.16, harbour:
bird: 1.09, reintroduced: 1.08, version: 1.07, subspecies: 1.04, schleswig: 0.97,
marriage: 1.0, ##ir: 0.99, #Milly: 0.98,
1.1. rowing: 1.04, competitors: 1.01, swimming: 0.97, regatta: 0.96, prosper- ous: 0.95, aquatics: 0.88, race: 0.85, in- habit: 0.85, waterfront: 0.79, pink: 0.79,
slicing: 0.98, fruits: 0.88, herries: 0.87,
flick: 0.89, vague: 0.84, wave: 0.81, stillness: 0.79, turbulent: 0.79, repub-
marriages: 0.86, traditionally: 0.86,
cheese: 0.86, lighted: 0.84, parties: 0.82
lished: 0.78, coasts: 0.78
wider: 0.78
airport: 1.6, airline: 1.35, airlines: 1.22,
sheep: 1.74, herd: 1.54, goats: 1.36,
nippon: 2,26, ana: 2.03, airline: 1.7,
window: 1.22, airports: 1.21, delta:
cumbria: 1.32, flick: 1.29, livestock:
aircraft: 1.4, airlines: 1.37, 77: 1.36, air-
1.21, nexus: 1.06, lax: 0.99, concourse:
1.29, driving: 1.27, goat: 1.24, scot-
port: 1.3, el: 1.16, flights: 1.09, depart-
0.97, #lisk: 0.92, gate: 0.87, flights:
land: 1.09, trafii: 1.04, lamb: 1.02,
ing: 1.06, ##7: 1.01, 1.01, republished: 0,92, busiest: 0.91, boeing: 0.9, livery: 0.88, aviation: 0.87, creditors: 0.85, spotting: 0.8, japan: 0.75, commons: 0.72.
0.86, traveled: 0.84, view: 0.82, airways:
loch: 1.01, tyrol: 0.94, skye: 0.92, un-
0.79, aircraft: 0.76. mildly: 0.76, Hilr:
usual: 0.92, flock: 0.91, animals: 0.9, heading: 0.89, roadside: 0.87, inaccessi-
0.75, rainy: 0.75, traveling: 0.75
ble: 0.85
ABBEY ROAD NW8
MUSTIM MOTER
abbey: 2.7, westminster: 2.43, mw: 2.03,
geese: 1,86, goose: 1.75, canada: 1.4,
thanksgiving: 1.72, turkey: 1.46, roast:
##minster: 1.87, northwest: 1.82, west:
photograph: 1.31, mating: 1.18, ducks:
1.26, very: 1.25, pie: 1.23, christmas:
1.78, ab: 1.67, ##lbey: 1.62, roads: 1.6.
1.18. #Hoides: 1.14, brant: 1.13, sub-
1.04, ##kcans: 1.02, mmm: 1.01, chicken:
##3: 1.59, #itey: 1.56, road: 1.51, abby:
species: 1.12, wildlife: 1.11, awhile: 1.1. courtship: 1.08, version: 1.04. swans: 0.97, lake: 0.97, pond: 0.95, sired: 0.95. synchronized: 0.94, duck: 0.94, flick:
0.99, holidays: 0.88, feast 0.82, repub-
1.38, sign: 1.23, minister: 1.22, peter-
lished: 0.81, stuffing: 0.78, dinner: 0.76,
borough: 1.21, ##mist: 1.13, 1.0,
bread: 0.76, tasted: 0.75, convinces: 0.75, televised: 0.74, meal: 0.73
agreed: 0.97, n: 0.94
0.93


Reviews:
[{"review_id": "VBix0ymHeJ", "reviewer": "Reviewer_gpBi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful comments. In the following sections, we provide clarification and respond to the comments.\n\n> *One major part of the training data of this model is 1B internal image-text pairs, which may not be released and cause difficulty to reproduce the experimental results.*\n\nThanks for highlighting the issue of training dataset accessibility. We would like to emphasize that the paper's main findings remain robust across varying training data. It is worth noting that our findings were consistent when utilizing publicly accessible datasets like LAION. However, our organization later forbade us from using such data due to concerns around copyright and safety [1,2].  As a result, we employed an internal version of a large-scale dataset for our experiments. \nTo further address reviewer's concern, we quickly conducted preliminary experiments on single-stage STAIR using the COYO dataset [3], up to 300K step. By examining the training curve against our current dataset, the [training curves](https://ibb.co/MN68G9h) are remarkably similar. Based on this, we believe that the COYO model would attain similar level of performance as reported in the paper. We hope this addresses the concern of reproducibility.\n\n> *As mentioned in Section 6.1, multi-stage training plays a crucial role in ensuring STAIR embedding grounded to meaningful tokens. But there is no clear explanation about this or detailed discussion about the contribution of each training stage. As claimed in lines 228-230. all parameters including BERT and MLM head are trained at the stage 1, the only difference between stage 1 and STAIR_{SINGLE-STAGE} is the text masking strategy. But this masking strategy doesn't seem to resolve the redefining problem ( As declared in lines 218-220, this masking strategy is designed to avoid model learning a shortcut by relying on less common tokens to bridge the modality gap).*\n\nTo provide clarity, we conducted a quantitative ablation study on the contribution of each training stage for both zero-shot image classification and text-image retrieval tasks, as discussed in Section 6.2 (lines 440-452).\n\nRegarding the role of text masking in grounding embeddings to the meaningful tokens, it's crucial to understand that the redefining problem is that arises from the contrastive loss primarily emphasizing text/image alignment rather than token interpretability (as noted in lines 212-218). This is also why the STAIR_{SINGLE-STAGE} learns a shortcut through less common tokens as long as it minimizes the contrastive loss. In contrastive, the multi-stage training incrementally bridges the modality, \n\n* In the 1st stage, text masking forces the image encoders to learn to produce the meaningful tokens existing in the original input text. This anchors image embeddings to meaningful tokens,  text embeddings still lack grounding during inference when text masking isn't available.\n* In the 2nd stage, the frozen image encoder guides the text encoder to generate such meaningful tokens without masking assistance.\n* The 3rd stage involves joint fine-tuning, further enhancing model performance.\n\nThe qualitative and quantitative comparison between the STAIR_{SINGLE-STAGE} and STAIR on embedding interpretability are given in Figure 6 and Table 8. We will ensure this aspect gets more clarity in the final version of our paper.\n\n[1] [Getty Images is suing the creators of AI art tool Stable Diffusion for scraping its content](https://www.theverge.com/2023/1/17/23558516/ai-art-copyright-stable-diffusion-getty-images-lawsuit)\n\n[2] [Birhane, Abeba, Vinay Uday Prabhu, and Emmanuel Kahembwe. \"Multimodal datasets: misogyny, pornography, and malignant stereotypes.\" arXiv preprint arXiv:2110.01963 (2021)](https://arxiv.org/abs/2110.01963)\n\n[3] [COYO-700M: Image-Text Pair Dataset](https://github.com/kakaobrain/coyo-dataset)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel representation method for vision and language balancing performance, interpretability and efficiency.\n\n2. Sufficient experiments verify the effectiveness of this method.", "reasons_to_reject": "1. One major part of the training data of this model is 1B internal image-text pairs, which may not be released and cause difficulty to reproduce the experimental results. \n\n2. As mentioned in Section 6.1, multi-stage training plays a crucial role in ensuring STAIR embedding grounded to meaningful tokens. But there is no clear explanation about this or detailed discussion about the contribution of each training stage.  As claimed in lines 228-230. all parameters including BERT and MLM head are trained at the stage 1, the only difference between stage 1 and STAIR_{SINGLE-STAGE} is the text masking strategy. But this masking strategy doesn't seem to resolve the redefining problem ( As declared in lines 218-220, this masking strategy is designed to avoid model learning a shortcut by relying on less common tokens to bridge the modality gap).", "questions_to_authors": "", "paper_topic_and_main_contributions": "Taking into account the interpretability and efficiency of image-text retrieval, this paper first proposes sparse text and image representation in human-readable tokens. Their method achieve better performance than CLIP on image-text retrieval and zero-shot image classification.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vb5Qn02eHP", "reviewer": "Reviewer_GqY9", "summary": "", "questions": "Please refer to Reasons to Reject", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful comments. In the following sections, we provide clarification and respond to the comments.\n\n> *Although I agree that the sparse embedding mentioned is more interpretable, this interpretability seems unnecessary in image-text retrieval. Dense embeddings that machines can understand are enough. On the other hand, the dense embeddings maybe achieve better approximations to natural semantic (manifold) spaces.*\n\nThank you for your agreement that sparse embedding is more interpretable. In terms of the comparison between sparse and dense models, we acknowledge the impressive performance of dense embeddings. Nevertheless, we would like to emphasize that interpretable sparse embedding possess distinctive advantages (as discussed in line 56-69).\n\nFor instance, retrieval systems used in large-scale web image search engine heavily relies on traditional methods like inverted index. Moreover, in web image search scenarios, supplementary signals such as image alt text and surrounding textual content play important role besides the image content itself. Integrating the STAIR sparse embedding with an inverted index system is straightforward, thanks to its interpretable token space, while it would be more complicated for dense embeddings.\n\nAdditionally, as discussed in Section 4.2, we note the sparse embedding can achieve significantly better performance on retrieval tasks that demand precise match, such as the SVHN task aimed at an exact number match. \n\nLastly, it’s worth noting that STAIR’s image encoder alone is capable of mapping the image into meaningful tokens, thereby introducing novel possibilities for existing tasks like localization and text-image retrieval, without requiring the use of the text encoder. Further elaboration can be found in Appendix F.\n\nIn conclusion, we believe both dense and sparse models offer their own distinctive advantages in different image-text retrieval tasks. Our work reveals the possibility to build the strong sparse embedding for image-text retrieval, and we leave the further study including combining dense and sparse representation as a future work. We will add more discussion in the final version.\n\n> *The paper claims the proposed STAIR can tackle “…are built on a fixed vocabulary, which cannot handle out-of-vocabulary concepts” in line 87-90. However, the sparse token space adopts the vocabulary as the basis of embedding space. How does STAIR deal with words out of the vocabulary? It seems that there is no essential difference from the dense situation.*\n\nIn terms of handling out-of-vocabulary tokens in the input, one big difference between our model and traditional BoW models is that our approach uses a WordPiece model like Bert. In contrast, most traditional BoW models’ vocabulary is based on the entire words, not word or sentence pieces. We will update the paper to make it clear.\n\n> *It is not clear whether the performance improvement comes from the sparse and grounded embedding or the designed training recipe. How will performance change when porting the training recipe to models with dense embedding? There seems to be a lack of ablation experiments in this regard.*\n\nWe would like to emphasize that STAIR_{single stage}, introduced in Section 6.1, is an immediate model variant between dense model and STAIR. It employs the sparse embedding but lacks the  proposed multi-stage training recipe  proposed in our work. In Appendix F, we present an ablation study comparing STAIR, STAIR_{single stage}, and dense models. On one hand, both sparse models (STAIR and STAIR_{single stage}) outperform the dense model across most metrics in zero-shot image classification and text-image retrieval tasks as demonstrated in Table 7. On the other hand, multi-stage training recipe is the key to achieve embedding interpretability (i.e. grounded to meaningful tokens) as illustrated in Table 8.\n\n> *In addition, STAIR needs to be applied to other CLIP or ALIGN based models to verify its effectiveness.*\n\nThanks for the valuable suggestion. Could you kindly recommend specific alternative CLIP or ALIGN based models for comparison? We would greatly appreciate this input.\n\nTo provide further clarity, our primary focus is to introduce a novel approach to image-text representation learning that is sparse and interpretable. The current implementation is based on the ViT-B16, and we have ensured fair comparisons between CLIP and STAIR under similar conditions. We believe our approach is adaptable to other CLIP or ALIGN variants and acknowledge it as a promising avenue for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explores the possibility of employing sparse and grounded token space in image-text retrieval to make embeddings interpretable, with performance improvement on CLIP.", "reasons_to_reject": "1.Although I agree that the sparse embedding mentioned is more interpretable, this interpretability seems unnecessary in image-text retrieval. Dense embeddings that machines can understand are enough. On the other hand, the dense embeddings maybe achieve better approximations to natural semantic (manifold) spaces.\n\n2.The paper claims the proposed STAIR can tackle “…are built on a fixed vocabulary, which cannot handle out-of-vocabulary concepts” in line 87-90. However, the sparse token space adopts the vocabulary as the basis of embedding space. How does STAIR deal with words out of the vocabulary? It seems that there is no essential difference from the dense situation.\n\n3.It is not clear whether the performance improvement comes from the sparse and grounded embedding or the designed training recipe. How will performance change when porting the training recipe to models with dense embedding? There seems to be a lack of ablation experiments in this regard. \n\n4.In addition, STAIR needs to be applied to other CLIP or ALIGN based models to verify its effectiveness.", "questions_to_authors": "Please refer to Reasons to Reject", "paper_topic_and_main_contributions": "Despite remarkable performance, the dense embedding space used in CLIP and ALIGN models is challenging to interpret, i.e., it has no direct relation with human understandable concepts. The paper proposes a sparse image and text model to map image and text in sparse token space, which is interpretable and better than dense embedding on cross-modal aligning. Besides, the paper proposes a multi-stage training recipe for the grounding of sparse embedding with understandable concepts. A series of experiments are performed to demonstrate the effectiveness of the proposed model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9xEdZdNOJL", "reviewer": "Reviewer_9dcF", "summary": "", "questions": "* How does STAIR compare with CLIP in terms of actual retrieval speed?\n* What is the impact of regularization weights $\\lambda$ on interpretability? Is there a trade-off between interpretability and retrieval performance when increasing the regularization weights?\n* Could you provide more information about the training cost, e.g., the total training time and the infrastructure being used?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful comments. In the following sections, we provide clarification and respond to the comments.\n\n> *The claim of STAIR's greater efficiency compared to CLIP lacks quantitative evaluation. How does STAIR truly compare to CLIP in terms of actual retrieval speed?*\n\nThank you for raising the concern of text image retrieval efficiency. Retrieval speed between dense and sparse embedding is a topic well-studied in the information retrieval community. ([1, 2]). However, latency and accuracy depend heavily on the implementation and configuration of retrieval systems. We conducted a preliminary comparison of CLIP and STAIR models using the popular retrieval libraries FAISS and Pyserini on a 1M-scale dataset.\n\n|Model   |\tQPS  |  R@10\t|  R@25 |\n| -------- | ------ | --------- | ------- |\n|CLIP     |\t6.02\t |  42.1%\t|  55.2%   |\n|STAIR   |\t14.1\t |  42.5%\t|  55.7%   |\n\nWe will include these preliminary results in the final version, but a comprehensive comparison would require more extensive experimentation, which we will leave as future work.\n\n> *How do the regularization weights $\\lambda$ impact interpretability? Is there a trade-off between interpretability and retrieval performance when increasing these weights?*\n\nWe would like to clarify that the regularization weights $\\lambda$ have minimal impact on the interpretability metrics. The defined interpretability is based on the accuracy of top K important tokens aligning with predefined classes. While changing the weights controls whether less important tokens get activated in the output embedding, it is unlikely to alter the order of tokens importance. \n\nThat said, there is almost no trade off between interpretability and retrieval performance. As the regularization weights increase, less tokens are expanded. For interpretability, if the predefined class token happens to get dropped, the metrics gets lower correspondingly, otherwise remain unaffected. For the retrieval performance, increased regularization weights also lower metrics as demonstrated in Figure 8(a). Therefore, interpretability and retrieval performance would move in the same direction when changing regularization weights.\n\n> *Can you provide more details about training costs, including total training time and the infrastructure employed?*\n\nThank you for bringing up the question of the training cost. All our experiments were conducted using 64 A100 GPUs. The total training time for each model is as follows\n\n|Model   |\tTotal steps  |\tWall time (days)\t|  \n| -------- | ------ | --------- | \n|CLIP     |\t600k  |\t12.4\t| \n|STAIR_{SINGLE-STAGE} |\t600k  |\t12.1\t|  \n|STAIR   |\t1200k  |\t20.9|\n\nWe acknowledge that the multi-stage training process in STAIR does require more iterations than the baseline CLIP model. As a fair comparison, we report the performance of a single-stage STAIR model trained under the same computation budget in Appendix D. This model still outperforms competitively compared to CLIP. We will include these details in the final version of the paper.\n\n> *It is unclear whether STAIR can enhance interpretability and downstream performance when combined with more advanced VLMs like BLIP*\n\nThanks for highlighting the potential with more advanced VLM. We would like emphasize that our primary focus is to introduce a novel approach to image-text representation learning that is sparse and interpretable. Our proposed approach can also be applied to other dual encoder variants. For more complicated architectures beyond the basic dual encoder, additional research works are required. We are happy to collaborate with the community and enhance our approach in the future work.\n\n\n[1] [Shen, Tao, et al. \"Lexmae: Lexicon-bottlenecked pretraining for large-scale retrieval.\" arXiv preprint arXiv:2208.14754 (2022).](https://arxiv.org/abs/2208.14754)\n\n[2] [Lassance, Carlos, and Stéphane Clinchant. \"An efficiency study for splade models.\" Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022.](https://arxiv.org/abs/2207.03834)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The problem addressed by this paper, i.e., learning interpretable image and text representations, is important, which can facilitate the transparency of large VLMs.\n* The proposed methods, i.e., the sparse embedding space and the multi-stage training strategy, are reasonable and effective.\n* The experiments are well-designed and the results are convincing, which validate the advantage of STAIR in terms of both interpretability and effectiveness across various downstream vision-language tasks.\n* The limitations of this work are properly recognized in the paper, which is accompanied by discussions on potential solutions and future directions.\n* The paper is well-written and easy to follow.", "reasons_to_reject": "* No quantitative evaluation supports the claim that STAIR is more efficient than CLIP in text-image retrieval.\n* It is unclear whether STAIR can still enhance the interpretability and downstream performance, when combined with more advanced VLMs, e.g., BLIP[1]. \n\n[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation", "questions_to_authors": "* How does STAIR compare with CLIP in terms of actual retrieval speed?\n* What is the impact of regularization weights $\\lambda$ on interpretability? Is there a trade-off between interpretability and retrieval performance when increasing the regularization weights?\n* Could you provide more information about the training cost, e.g., the total training time and the infrastructure being used?", "paper_topic_and_main_contributions": "This paper addresses the problem of learning interpretable text and image representations. Unlike the prevailing vision-language models (VLMs) that rely on abstract dense embeddings, this work introduces an interpretable sparse embedding space, namely STAIR, where each dimension associates with a (sub-)word from the BERT vocabulary. This work also proposes a multi-stage training strategy to (1) align learned image and text representations effectively and (2) ground them in meaningful (sub-)words. The empirical results demonstrate that, compared with the CLIP model that utilizes dense embeddings, the embeddings of STAIR are not only more interpretable to humans but also achieve superior performance in image-text retrieval, zero-shot and linear probing visual classification tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies text-image retrieval and proposes a method whereby text and images are mapped into a sparse, interpretable space where individual dimensions correspond to wordpiece tokens. This is in contrast to models such as CLIP that utilize dense and less interpretable embeddings. Nevertheless, the paper demonstrates performance improvements over CLIP using its interpretable embeddings.

Pros (from reviewers):
The proposed method is novel and appealing from both an interpretability standpoint as well as a retrieval accuracy standpoint
The experiments are extensive and convincing
The limitations and future work are given adequate consideration in the text
The paper is clear and well-written

Cons (from reviewers)
The paper uses a closed-source 1B example training set
The multi-stage training procedure of the model is not described in as much detail as it could be
It is unclear from the experiments where the performance gains over CLIP are coming from. Is it due to the well-tuned training recipe, or from the sparse nature of the proposed embeddings themselves?
Decision: Accept-Main