NEXIN: Negative Exclusion Inference
Authors: Aastik, Pragya Paramita Sahu, Aniruddha Bala, Pinaki Bhaskar


Reviewer paper summaries: This paper introduces NEXIN: Negative Exclusion Inference, a training/tuning free way of adapting vision-language models to perform better on negative queries (i.e. "A man not wearing a hat"). NEXIN works by splitting queries into two components, a positive component and a negative component, and building a new query vector which is a linear combination of the original query, the positive component, and the negative component. This paper also introduces a new measure for retrieval, barrier potential, inspired by charge potentials in electrons, which can better "map the capabilities to the fine-grained understanding of negation". Performance on a custom set of negated data, NEX-Bench (300 samples from Flickr-1K and Visual genome) show that precision and recall are notably improved when using the linear combination vector as a retrieval target compared to without.

The paper introduces a methodology for improving the alignment of text and image in the case of a text that negates an object. For example, if one wishes to search for an image of a "man without hair", one would like to encounter precisely that image. However, according to the authors, vision-and-language models tend to miss the negation, thereby returning images of men *with* hair. So, their methodology tackles this challenge. Furthermore, they create a benchmark dataset to evaluate this capability. Finally, they introduce a new metric for measuring the quality of the results in this task.
Summary of strengths: This paper tackles an interesting and useful question: how can we improve the performance of vision-language retrieval in the presence of negative expressions. It is well known that negative expressions are hard for existing vision and language models, and it's a fairly clever (and to my knowledge, novel) idea to exploit vector arithmetic in language embeddings to improve recall performance in retrieval systems. The introduced benchmark, NEX-Bench, while small, could potentially be useful to other members of the community. The introduced metric "barrier potential" is an interesting place to gain inspiration from, and could potentially be shown to be an interesting way of measuring tasks that have explicit hard tradeoffs.

- The methodology promises to increase performance without requiring retraining or annotations
- The benchmark they provide seems useful, although by making it public it will probably be read by future pre-trained large language models, thereby artificially increasing their performance on this benchmark.
Summary of weaknesses: While there are some (potential) strengths, there are also many notable weaknesses:
- **Data Release**: It is not clear that any data/artifacts will be released, particularly (1) code for computing the barrier potential and (2) the NEX-Benchmark. While the papers claims that the NEX-Benchmark will be released (L392), the checkpoint seems to indicate that company policy restricts the disclosure of many essential details to clear and open science including infrastructure, data collection tools, annotation policies, etc. 
- **Motivation of the modified text encoding**: It's not immediately clear to me from the paper why a linear combination of text embeddings should lead to the negative expression behavior present in the results. While this may be exploiting the anagram effect (Vylomova et al. 2015), this isn't discussed at any point in the paper, and it's not validated in the results. Some additional discussion here would be quite nice: what happens at the extremes of the hyperparameters (esp. given that the parameters are unbounded), and what are the kinds of phenomena that are expected? Further, it's not clear to me why the original query is included here, and if the original query is necessary. 
- **Motivation of the Barrier potential metric**: In my opinion, this paper does not clearly motivate the need for the barrier potential metric vs traditional retrieval measures. The justification in section 3.3 is that "These metrics fail in measuring a lot of semantic and compositional understanding of the system" (L305), and the justification in the introduction is "[Recall @ N] fails to map the capabilities to the fine-grained understanding of negation" (L112) but this needs to be significantly expanded on: what particularly are the challenges that traditional measures face, and why does barrier potential solve these challenges? It's clear to me that there is a good reason for introducing a new measure based on a "hardness" of retrieval, but it's not well described here. It's also not clear to me that the negative retrieved samples represent true "hard negatives" - isn't it better for a retrieval system to return a "woman with a hat" instead of a "cat" given the query is "woman without a hat"? Finally, I'm not entirely certain why barrier potential was chosen is the underlying motivation - why should we be drawing inspiration from holes of electrons in search measures? I could come up with my own potential answers to this questions, but this should be described in the paper itself. 
- **Exploration of hyperparameters**: While Figure 6 explores how the parameter $\eta_{ex}$ is related to the barrier potential, this paper is missing exploration of the other parameters including $\eta_{r}$ and the missing parameter in Eq (1) which is a weight on the original query. These explorations for both *positive* and *negative* values of the $\eta$ values would be interesting, since it would help show how 
- **Data diversity/Size**: Because the paper does not include the dataset, it's very challenging to evaluate the diversity of the proposed benchmark, or the difficulty of the benchmark. Further, the dataset is quite small, only 300 queries, limiting it's potential applications. From the few examples in Figure 7, it seems like the queries are fairly simple -- do such simple queries often appear in practice? Or do real world search situations demand more complex language processing?
- **Flexibility**: As discussed in Section 3.1, the NEXIN approach requires splitting queries into "positive" and "negative" components, however this paper does not disclose the method for doing this, beyond saying that it is a "simple rule-based word matching system"). This is a key component of the method, and without knowing more about the system, it's hard to evaluate the flexibility of this approach. it's also unclear if a simple rule-based system is sufficient for the approach - the paper claims on L261 that it is unlikely that such queries appear in practice, but does not provide any supporting evidence to if that's the case.  
- **Efficiency**: In addition to computing three times as many text embeddings, this method also requires a parsing step (of variable complexity). While the paper claims (L272) that this is a low-cost process, it would be good to explicitly test and present this performance (either in the paper, or in an appendix). In search applications containing many thousands or millions of queries, such an overhead may be nontrivial. 
- **Train vs. Test Splits**: It's not discussed in the paper how the $\eta$ values are chosen, and if they are correctly using train/validation/test splits.
- **Variance Analysis**: There is no analysis of the variance of the results, which makes it hard to understand the significance of the results, especially for the barrier potential measure, where the magnitudes of the differences are not intuitive.

### Bigger issues:
- 026: The fact that the improvement is higher on your benchmark than on previous benchmarks is a red flag about your benchmark, because it sounds like it was constructed to highlight the performance of your system.
- 219-225: I don't see enough evidence that existing models cannot process negation. There is no discussion of existing literature on this subject, and the only evidence shown is based on three searches performed by the authors of the paper on a specific model and dataset. Furthermore, the dataset they chose is one in which, per their own words (114-116), does not contain many negated captions.
	- 228: how should I interpret the cosine similarity of 0.38+/-0.02? Is it low or high? It is, after all, significantly below 1.
- Please provide a formula for S_t. Your explanation indicates that S_t is the "average of the scores of nth and n + 1th ranked image". This leads to a confusion in terminology, because you say that n is the number of positive images, so there is no such thing as an n+1th ranked image.
- Equation 5 introduces K_i and C without explanation, so I am unable to reproduce the results.
- 384: Although the standard benchmarks do not contain sufficient exclusion captions to validate your methodology, you should still show results on existing benchmarks to demonstrate that your methodology does not worsen performance on existing benchmarks.
- Table 1: all cells should probably have the same number of decimals. Also, I think three decimals is too much here.
- Section 4.3: they don't say what dataset they used to choose the value of the hyperparameters
- Table 2 should show the effect of NEXIN on inclusion captions as well. Also, Table 1 shows the standard metrics on the combined dataset, while Table 2 shows the new metric on the separate datasets. Why is that? Ideally you should have both standard and new metrics on both the combined and the separate datasets, with and without NEXIN.
- 256-267: they do not provide the list of words and/or the regular-expression formula they used to select the exclusion caption, which is a crucial part of this pipeline. Furthermore, they say that a small or large language model could be used for this, but they don't give any results of using this option. Thus, I think they should stick to the rule-based method, but be more specific about what method that is.

### Smaller methodological issues:
- 025: 20% improvement in what metric/task?
- 029-036: Citation needed
- 041: define Semantic Image Search and cite if appropriate
- 110-112: Citation needed
- 114-116: Citation needed
- 208-210: To reflect the fact that both T_\epsilon and I_\epsilon are embeddings in the same space, I would suggest naming them \epsilon_T and \epsilon_I instead. Actually, later on you use the terminology $E$ for embeddings. Maybe you should just use this here already.
- 277: why do you need the term \eta_r E_t(Q_r)? Conceptually what you want to do is to subtract the exclusion embedding, but why add the root embedding too?
- 302-303: Citation needed.
- 310-312: Citation needed, or further explanation.
- 317-326: Citation needed for the explanation about semiconductors.
- 346-352: can you specify if this applies to the "raw" embedding or the "corrected" embedding from Eq 3?
- 411-413: Citation needed.
- No comment is made about how to extend this work to languages other than English.
Comments, suggestions and typos: There are significant typos in the paper leading to reduced understanding - this paper would benefit from additional proof reading.

**Comments after author discussion**: I have read the author response, and adjusted my score accordingly. See response comments for deatils.

- 015: define acronym VLM
- 020: Proposed -> The proposed (missing article) => applies to multiple locations
- 021: on-top-of -> on top of (incorrect hyphenation) => applies to multiple locations
- 049: its -> their
- 066: no parentheses for in-text citations (\citet instead of \cite or \citep) => applies to multiple locations
- 070: define acronym VL
- Figure 1 caption: image -> images
- 090: prompt based -> prompt-based (missing hyphenation) => applies to multiple locations
- Figure 7 caption: The second sentence should be “Compared to **the** results without NEXIN in **Figure** 1 **,** addition of NEXIN shows significant improvem**e**nt in search results.”
- 101: missing comma before "which"
- 101: define acronym E2E
- 210: "it's" -> "its"
- 212: remove "a"
- Figure 3: "cosine product" -> "cosine similarity"
- 219-225: these two sentences convey essentially the same information, so you should remove one of them.
- 271: the "x" in Q_ex should be in the subscript: Q_{ex}
- 287: double parentheses on the citation => applies to multiple locations
- 323: "and powers our world" sounds colloquial and is unnecessary
- Why do you need to specify "hard" in "hard positive" and "hard negative"?
- 346: E should be E_i, correct?
- Figure 6: the caption is wrong
- 261: I think "thespian" is not the right word
Overall assessment scores: 1.5

2