SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhihao Fan $^{1}$ , Zhongyu Wei $^{1}$ , Zejun Li $^{1}$ , Siyuan Wang $^{1}$ , Xuanjing Huang $^{1}$ , Jianqing Fan $^{2}$
$^{1}$ Fudan University,  $^{2}$ Princeton University
{fanzh18,zywei,zejunli20,wangsy18,xjhuang}  $@$  fudan.edu.cn, jqfan@princeton.edu
Matching model is essential for Image-Text Retrieval framework. Existing research usually train the model with a triplet loss and explore various strategies to retrieve hard negative sentences in the dataset. We argue that current retrieval-based negative sample construction approach is limited in the scale of the dataset thus fail to identify negative sample of high difficulty for every image. We propose our TAiloring neGative Sentences with Discrimination and Correction (TAGS-DC) to generate synthetic sentences automatically as negative samples. TAGS-DC is composed of masking and refilling to generate synthetic negative sentences with higher difficulty. To keep the difficulty during training, we mutually improve the retrieval and generation through parameter sharing. To further utilize fine-grained semantic of mismatch in the negative sentence, we propose two auxiliary tasks, namely word discrimination and word correction to improve the training. In experiments, we verify the effectiveness of our model on MS-COCO and Flickr30K compared with current state-of-the-art models and demonstrates its robustness and faithfulness in the further analysis.
The task of image-text retrieval takes a query image (sentence) as input and finds out matched sentences (images) from a candidate pool. The key component of the retrieval framework is the similarity computation of an image-sentence pair and it aims to assign higher scores to positive pairs than negative ones. Triplet loss is widely applied for training. Take image-to-text as example<sup>1</sup>, it constructs two image-sentence pairs using an image and two sentences (one is relevant and the other is not), and the optimization process increases the similarity of the
Figure 1: Diagram of matching scores (a) and two examples (b) in Flickr30K (Plummer et al., 2015).
(a) The diagram plots a triplet (image, positive sentence, negative sentence) as a dot is defined by matching score of the positive pair on the X-axis and that of the negative pair on the Y-axis. The matching scores are also computed by CLIP(ViT-B/32) (Radford et al., 2021).
Image	Sentence	Score
P1:A man with a gray beard rides his bike on the beach of the ocean.
N1: Man on bike, with bike clothing and helmet on, having trouble maneuvering through sand from beach.
G1:A woman with a gray beard rides his bike on the beach of the ocean.	0.40
0.34
0.41
P2:A little girl is posing on some pumpkins within an area surrounded by flowers.
N2:A girl wearing a red and black striped shirt is sitting on a brick wall near a flower garden.
G2:A little girl is posing on some pumpkins within a beach surrounded by flowers.	0.47
0.36
0.45
(b) Two images with the positive sentence (P), the most difficult negative one (N) retrieved from dataset by CLIP and the generated negative one (G). The score is the cosine similarity computed by CLIP and larger is better. The underlined red words are non-correspondence ones to the image.
positive pair while decreasing that of the negative one. Previous research (Xuan et al., 2020) reveals that models trained with harder negative samples, i.e., sentences that are more difficult to be distinguished, can generally achieve better performance. In this line of work, researchers explore various strategies to search mismatched sentences for a query image, from randomly choosing mismatched sentences to using the most similar one. The search
To keep the presentation simple and clear, we use imaged-to-text as example to represent tasks in both ways throughout the paper.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2667 - 2678
Figure 2: Framework of TAiloring neGative Sentences with word Discrimination and Correction (TAGS-DC).
scope moves from a single training batch (Karpathy and Fei-Fei, 2015; Faghri et al., 2018; Kiros et al., 2014; Socher et al., 2014; Lee et al., 2018; Li et al., 2019) to the whole dataset (Chen et al., 2020a; Zhang et al., 2020). Although promising results have been reported by searching for harder negative samples in a larger scope, the effectiveness is limited by the scale of the dataset.
To compare the effectiveness of these strategies, we randomly sample 3,000 images in Flickr30K (Plummer et al., 2015) and plot training triples constructed in Figure 1. Each dot stands for a triple (image, positive sentence, negative sentence), and X-axis is the matching score of the positive image-sentence pair while Y-axis is that of the negative one. In general, triples located on the left of the dotted line are more difficult to be distinguished because matching score of the negative pair is higher than the positive one or comparable. As we can see, triples obtained by searching the most difficult mismatched sample in the batch are largely located on the right of the dotted line, and the matching scores of negative pairs are much smaller with a gap larger than 0.05 on average (in the right of the solid line). When enlarging the searching scope to the whole dataset, triples move up in positions, and around  $40\%$  of negative pairs obtain higher matching scores than positive ones. However, there are still  $18\%$  of images that can only recruit negative samples with a matching score 0.05 lower than its positive counterpart. This confirms the limitation of retrieve-based negative sample construction strategy.
To have a better understanding, we present two triples in Figure 1 i.e.,  $(P_{1},N_{1})$  and  $(P_{2},N_{2})$  (denoted as black cross). It shows that negative sentences  $N_{1}$  and  $N_{2}$  describe scenes with significant
differences compared with the query images, therefore, they are easy to be distinguished. Given that a high percentage of images obtain these low-quality negative sentences in the dataset, we believe it is necessary to collect negative samples beyond retrieval. Instead of searching for original sentences in the dataset, we explore constructing artificial negative samples by editing positive sentences. We demonstrate two generated sentences in Figure 1,  $G_{1}$  replaces "man" with "woman" on  $P_{1}$  and  $G_{2}$  replaces "area" with "beach" on  $P_{2}$ . The generated sentences obtain comparable or even higher matching scores than positive ones. We further generate artificial sentences for all images to form a new set of triples. These triples are plotted in Figure 1 as pink dots. We can see all of them located on the left side of the dotted line, which means they are more difficult to be distinguished.
In this paper, we propose TAiloring neGative Sentences (TAGS) by rewriting keywords in positive sentences of a query image to construct negative samples automatically. In specific, we employ the strategy of masking and refilling. In masking, we construct scene graph for the positive sentence and mask elements in the graph (objects, attributes, and relations). Refilling replaces the masked original words with mismatched ones to construct the negative sample. In the training process, we further propose two word-level tasks, word discrimination and word correction, to incorporate fine-grained supervision into consideration. Word discrimination requires the model to distinguish which words lead to the mismatch, and word correction demands the regeneration of the original words. Both tasks evaluate the capability of the model to identify minor differences between synthetic sentences and positive ones. During inference, the output of two tasks
can provide fine-grained information through highlighting and revising mismatched words, and these can be regarded as the explanation for the decision made by the model to improve the interpretability. We evaluate our model on MS-COCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015). Experiment results show the effectiveness of our model.
Our contributions are three-fold: (1) We propose a generation-based method to construct negative samples to improve the training efficiency of image-text retrieval model. (2) To fully exploit the synthetic negative sentences, we propose two training tasks, word discrimination and word correction, to incorporate the fine-grained supervision to enhance the multi-modal local correspondence modeling. (3) Our model generates state-of-the-art performance on two public datasets MS-COCO and Flickr30K.
The overall framework of TAiloring neGative Sentences with word Discrimination and Correction (TAGS-DC) is shown in Figure 2. For each positive image-text pair  $(I_i, T_i)$ , we first generate negative sentences  $\mathbb{T}_i^-$  through scene-graph based masking and refilling  $T_i$  on the basis of masked language model (MLM) in §2.1. Second, we utilizes both retrieved and synthetic negative sentences for the training of image-text matching (IRTM and ISTM) in §2.2, where synthetic negative sentences are exploited in sentence-level. Third, we propose to train the synthetic sentence generator in a dynamic way to keep pace with the upgrading of matching model. Fourth, in §2.4, we apply word-level tasks of word discrimination (WoD) and word correction (WoC) on  $\mathbb{T}_i^-$  to discover their differences with  $T_i$  for further training. MLM, IRTM, LSTM, WoC and WoD share the same backbone  $M_\theta$  and have their own heads, namely,  $H_{MLM}$ ,  $H_{ITM}$ ,  $H_{WoC}$  and  $H_{WoD}$ . The detailed training step is illustrated in Algorithm 1 in appendix.
2.1 Scene-graph based Sentence Generation and Filtering
In general, negative sentences with more overlapped words with positive sentences tend to obtain higher matching scores with the query image, thus are more difficult to be distinguished. Therefore, we propose to edit relevant sentences to construct negative samples for a query image. After the sen
tence generation, we control the quality by filtering the false negative sentences. To ensure the editing operates on key semantic units of the sentence, we use a strategy based on scene-graph.
2.1.1 Scene-graph based Sentence Editing
The module of sentence editing takes a relevant sentence of the query image as input and outputs a synthetic sentence. It first identifies some key semantic units in the sentence and replaces them with other words. We employ a masked language model for this process following two steps namely, masking and refilling.
To identify the key semantic of a sentence, we construct the scene graph for a relevant sentence through scene graph parser of SPICE (Anderson et al., 2016) following  $\mathrm{SGAE}^2$  (Yang et al., 2019). We then collect objects, relations, and attributes as candidates for masking. To control the semantic offset of the synthetic sentence  $T_{i}^{(k)}$ , we randomly mask  $15\%$  tokens of sentence.
In the step of refilling, we use the output head  $H_{MLM}$ , which is a two-layer feed-forward network (FFN), on top of the backbone  $M_{\theta}$  for masked language modeling. Thus, image  $I_{i}$  also gets involved in MLM to guide the refilling later. The detailed computation of  $\mathcal{L}_{MLM}$  is shown in Eq. (1), where  $\circ$  is the function composition and  $NLL$  is the loss of negative log-likelihood.
$$
M L M: H _ {M L M} \circ M _ {\theta} \left(I _ {i}, T _ {i} ^ {(k)}\right)\rightarrow T _ {i} / T _ {i} ^ {(k)} \tag {1}
$$
$$
\mathcal {L} _ {M L M} = N L L \left(M L M \left(I _ {i}, T _ {i} ^ {(k)}\right), T _ {i} / T _ {i} ^ {(k)}\right)
$$
Then during refilling process, we put  $T_{i}^{(k)}$  into MLM to produce the logit scores, then sample the synthetic sentence  $T_{i}^{(k,l)}$  following the distribution which originates from the logit with temperature  $\tau$  as Eq. (2).
$$
T _ {i} ^ {(k, l)} \sim \operatorname {S o f t m a x} \left(M L M \left(I _ {i}, T _ {i} ^ {(k)}\right) / \tau\right) \tag {2}
$$
We conduct the masking and refilling steps for  $K$  and  $L$  times to generate candidate synthetic sentences.
2.1.2 False Negative Sample Filtering
It hurts the training of using sentences that are relevant to the query image as negative samples (Chuang et al., 2020; Huynh et al., 2020). Therefore we propose a filtering process to remove
$^{2}$ https://github.com/yangxuntai/SGAE
false negative ones of synthetic sentences. In vision and language datasets, each image is annotated with multiple descriptive sentences. For example, there are five in MSCOCO and Flickr30K. For a synthetic sentence, if its replaced tokens are completely included in these annotated sentences, we will treat it as relevant. Based on this, we filter synthetic sentences which are relevant.
2.2 Image Text Matching
Given an image  $I_{i}$  and a sentence  $T_{j}$ , the retrieval model assigns a matching score  $s \in [0,1]$  of  $(I_{i}, T_{j})$  with an output head  $H_{\mathrm{ITM}}$ , which is a one-layer FFN, as Eq. (3).
$$
I T M: H _ {I T M} \circ M _ {\theta} \left(I _ {i}, T _ {j}\right)\rightarrow s \tag {3}
$$
Triplet loss (TripL) is widely applied in image text matching. With a hyper-parameter  $\alpha$ , it takes a query image (text)  $U$  as an anchor for the matched (positive) image-text pair  $(U,V)$  against the mismatched (negative) pair  $(U,W)$  as the following equation.
$$
\begin{array}{l} T r i p L _ {\alpha} (U, V, W) \\ = \max  (\alpha - I T M (U, V) + I T M (U, W), 0) \tag {4} \\ \end{array}
$$
Matching on Retrieved Cases During training, for each positive image-text pair  $(I_i,T_i)$ , we retrieve a negative image  $I_{i}^{-}$  and a sentence  $T_{i}^{-}$ , then employ the loss of ITM in Eq. (5) for training,
$$
\mathcal {L} _ {I R T M} = \operatorname {T r i p} L _ {\alpha} \left(I _ {i}, T _ {i}, T _ {i} ^ {-}\right) + \operatorname {T r i p} L _ {\alpha} \left(T _ {i}, I _ {i}, I _ {i} ^ {-}\right) \tag {5}
$$
Matching on Synthetic Sentences First, we pick up these relatively better generated negative sentences. In practice, we compute the matching score between each synthetic negative sentence and  $I_{i}$  as Eq. (6), and keep a synthetic negative sentence pool  $\mathbb{T}_i^-$  to make each of them as difficult as possible.
$$
\mathbb {T} _ {i} ^ {-} = \underset {T _ {t} ^ {-} \in \left\{T _ {i} ^ {(k, l)} \mid T _ {i} ^ {(k, l)} \neq T _ {i} \right\}} {\operatorname {a r g m a x} - m} I T M \left(I _ {i}, T _ {t} ^ {-}\right) \tag {6}
$$
where  $\text{argmax-m}$  is to pick out  $m$  sentences that earn the top- $m$  matching scores.
Second, with synthetic sentences  $\mathbb{T}_i^-$  in Eq. (6), we utilize them and the positive one  $T_{i}$  to compute the triplet loss, and get  $\mathcal{L}_{ISTM}$  in Eq. (7).
$$
\mathcal {L} _ {I S T M} = \frac {1}{| \mathbb {T} _ {i} ^ {-} |} \sum_ {T _ {t} ^ {-} \in \mathbb {T} _ {i} ^ {-}} \operatorname {T r i p} L _ {\alpha} \left(I _ {i}, T _ {i}, T _ {t} ^ {-}\right) \tag {7}
$$
2.3 Dynamic Training Strategy of Negative Sample Generation for Image-Text Matching
The naive choice of MLM is to keep a pre-trained static one: pre-training a MLM in advance and fixing its parameters during the training of ITM. Recall that  $\mathcal{L}_{ISTM}$  encourages the ITM model to learn the pattern of synthetic sentences and keep them away from the image, we consider that negative sentences generated by the static MLM would be no longer difficult for the ITM model as the training goes on. We propose to use the dynamic MLM that shares the  $M_{\theta}$  with ITM for mutual improvement. Through the sharing, MLM continuously learns what is more relevant to the positive sentences and produces challenging negative ones for the improvement of ITM. The stronger ITM helps MLM to better identify the semantic alignment of image and keywords. MLM achieves the improvement synchronously with ITM through interaction.
2.4 Auxiliary Tasks to Incorporate Fine-grained Supervision
$\mathcal{L}_{ISTM}$  only provides sentence-level supervision and we argue it does not fully exploit the synthetic negative sentence. We introduce two auxiliary tasks to utilize the word-level difference and further enhance the model capability in multi-modal local correspondence modeling.
Word Discrimination The task is to determine whether each word of the synthetic sentence  $T_{t}^{-} \in \mathbb{T}_{i}^{-}$  is matched with  $I_{i}$ , and we regard the replaced words of  $T_{t}^{-}$  as mismatched ones and others as matched ones. The target label  $G_{t}$  of  $T_{t}^{-} \in \mathbb{T}_{i}^{-}$  is determined following  $G_{t,j} = 1$  if  $s_{i,j} = s_{t,j}$  else 0, where  $s_{i,j}$  and  $s_{t,j}$  are the  $j$ -th token of  $T_{i}$  and  $T_{t}^{-}$ . We set up a new output head  $H_{WoD}$ , and the objective of word discrimination is in Eq. (8).
$$
\begin{array}{l}W o D: H _ {W o D} \circ M _ {\theta} \left(I _ {i}, T _ {t} ^ {-}\right)\rightarrow G _ {t}\\\vdots\end{array}\tag {8}
$$
$$
\mathcal {L} _ {W o D} = N L L \left(W o D \left(I _ {i}, T _ {t} ^ {-}\right), G _ {t}\right)
$$
Word Correction This task is to correct these mismatched words in  $T_{t}^{-}$  as Eq. (9). The task not only requires the model to comprehensively understand the gap between the synthetic negative sentences and the original positive ones, but also word-dependency knowledge and local cross-modal alignment to fill the gap.  $H_{WoC}$  is the output head for word correction, and the objective is
MS-COCO	Flickr30K
Model	Image-to-Text	Text-to-Image	RSum	Image-to-Text	Text-to-Image	RSum
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
SCAN	50.4	82.2	90.0	38.6	69.3	80.4	410.9	67.4	90.3	95.8	48.6	77.7	85.2	465.0
MMCA	54.0	82.5	90.7	38.7	69.7	80.8	416.4	74.2	92.8	96.4	54.8	81.4	87.8	487.4
AOQ	55.1	83.3	90.8	41.1	71.5	82.0	423.8	72.8	91.8	95.8	55.3	82.2	88.4	486.3
UNITER+DG	51.4	78.7	87.0	39.1	68.0	78.3	402.5	78.2	93.0	95.9	66.4	88.2	92.2	513.9
Unicoder-VL	62.3	87.1	92.8	46.7	76.0	85.3	450.2	86.2	96.3	99.0	71.5	90.9	94.9	538.8
LightningDOT(B)	64.6	87.6	93.5	50.3	78.7	87.5	462.2	86.5	97.5	98.9	72.6	93.1	96.1	544.7
ERNIE-ViL(B)	-	-	-	-	-	-	-	86.7	97.8	99.1	75.1	93.4	96.3	548.4
UNITER(B)	64.4	87.4	93.1	50.3	78.5	87.2	460.9	85.9	97.1	98.8	72.5	92.3	96.1	542.7
TAGS-DC(B)	66.6	88.6	94.0	51.6	79.1	87.5	467.4	87.9	98.1	99.3	74.5	93.3	96.3	549.4
CLIP	58.4	81.5	88.1	37.8	62.4	72.2	400.4	88.0	98.7	99.4	68.7	90.6	95.2	540.6
LightningDOT(L)	65.7	89.0	93.7	53.0	80.1	88.0	469.5	87.2	98.3	99.0	75.6	94.0	96.5	550.6
ERNIE-ViL(L)	-	-	-	-	-	-	-	89.2	98.5	99.2	76.7	94.1	96.7	554.4
UNITER(L)	65.7	88.6	93.8	52.9	79.9	88.0	468.9	87.3	98.0	99.2	75.6	94.1	96.8	551.0
TAGS-DC(L)	67.8	89.6	94.2	53.3	80.0	88.0	472.9	90.6	98.8	99.1	77.3	94.3	97.3	557.4
Table 1: Overall performance of the image-text retrieval.  $B$  and  $L$  are the base and large settings.
MS-COCO	Flickr30K
Model	Image-to-Text	Text-to-Image	RSum	Image-to-Text	Text-to-Image	RSum
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
TAGS w/ WM	64.9	87.8	93.3	51.1	78.9	87.4	463.4	85.9	97.6	99.1	74.2	93.0	96.1	545.9
TAGS w/ SG	64.1	87.6	93.4	50.9	78.8	87.3	462.1	85.5	97.4	98.9	73.3	92.6	96.0	543.7
TAGS	65.4	88.4	93.6	51.3	79.0	87.5	465.2	87.2	97.8	99.2	74.4	93.1	96.1	547.8
Table 2: Effectiveness of Different Modules. TAGS  $w/ WM$  means replace the scene-graph based masking with word masking in TAGS. TAGS  $w/ SG$  means replace dynamic generator with static generator in TAGS.
shown in Eq. (9).
$$
\operatorname {W o C}: H _ {\operatorname {W o C}} \circ M _ {\theta} \left(I _ {i}, T _ {t} ^ {-}\right)\rightarrow T _ {i}
$$
$$
\mathcal {L} _ {W o C} = N L L \left(W o C \left(I _ {i}, T _ {t} ^ {-}\right), T _ {i}\right)
$$
2.5 Overall Training
Details of our training step are shown in Algorithm 1 in appendix. The overall training loss of our model has five components as Eq. (10) with hyperparameters  $\lambda_{IRTM}$ ,  $\lambda_{MLM}$ ,  $\lambda_{ISTM}$ ,  $\lambda_{WoD}$  and  $\lambda_{WoC}$ .
$$
\begin{array}{l} \mathcal {L} = \lambda_ {I R T M} \mathcal {L} _ {I R T M} + \lambda_ {M L M} \mathcal {L} _ {M L M} \\ + \lambda_ {I S T M} \mathcal {L} _ {I S T M} + \lambda_ {W o D} \mathcal {L} _ {W o D} + \lambda_ {W o C} \mathcal {L} _ {W o C} \tag {10} \\ \end{array}
$$
During inference, we employ the ITM to determine the matching score of the query image (text) and the candidate text (image) as Eq. (3).
Dataset We evaluate our model on MS-COCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015). In MS-COCO, each image is accompanied with 5 human annotated captions. We split
the dataset following (Karpathy and Fei-Fei, 2015) with 113,287 images in the training set and 5,000 images in the validation and test sets, respectively. Flickr30K (Plummer et al., 2015) consists of 31000 images collected from the Flickr website, and every image contains 5 text descriptions. We take the same splits as in (Karpathy and Fei-Fei, 2015), with 1000 images for validation and 1000 images for testing, and the rest for training. 500
Models for Comparison We compare our model with some competitive approaches, including MMCA (Wei et al., 2020), and AOQ (Chen et al., 2020a). We also compare with methods based on vision language pre-trained models: UNITER+DG (Zhang et al., 2020), UnicoderVL (Li et al., 2020), LightningDOT (Sun et al., 2021), UNITER (Chen et al., 2020b), CLIP (Radford et al., 2021) and ERNIE-ViL (Yu et al., 2020).
Implementation We employ the pre-trained UNITER (Chen et al., 2020b) with base (B) and large (L) settings as our backbone.
Evaluation Metrics We report recall at K (R@K) and Rsum. R@K is the fraction of queries for which the correct item is retrieved among the closest K points to the query. RSum is the sum of R@1+R@5+R@10 in both image-to-text and text-to-image.
3.1 Overall Performance
The overall result is shown in Table 1. TAGS is the model trained with generated negative samples, using the dynamic training strategy. TAGS-DC is our model built on top of TAGS, further trained using two auxiliary tasks. In the base setting, our model achieves the best performance in terms of all metrics except R@1 and R@5 of in text-to-image on Flickr30K. In the large setting, our model also outperforms other models across all metrics except R@5 MS-COCO text-to-image and Flickr30K image-to-image R@10. Compared with UNITER(L), our model achieves an improvement of 4.0 and 6.4 RSum points in MS-COCO and Flickr30K.
We further demonstrate the effectiveness of different modules, namely, scene-graph based masking (denoted as PM), dynamic sentence generation (denoted as DG), and fine-grained training tasks (denoted as WoD and WoC) in Flickr30K. Original TAGS is trained with PM and DG. TAGS-DC is further trained with WoD and WoC.
Scene-graph VS Word based Masking We replace the scene-graph based masking with word-based masking (denoted as WM) to form TAGS w/ WM. Detailed results are shown in Table 2. WM follows the original sampling method of UNITER (Chen et al., 2020b) that randomly sample  $15\%$  tokens to mask, and PM is introduced in §2.1. TAGS outperforms TAGS w/ WM in terms of all metrics, and this verifies the effectiveness of PM.
Dynamic VS Static Generator We replace DG with a static sentence generator (denoted as SG) to form TAGS w/ SG. The difference between TAGS and TAGS w/ SG lies in that the former shares the parameters of ITM and MLM while the latter does not. Both of them are initialized with the pre-trained UNITER-base and share the same hyperparameters. In detail, we set  $\lambda_{MLM} = 0.1$  and  $\lambda_{ISTM} = 0.001$ . The static generator is fixed as
a fine-tuned UNITER+MLM model. The performance of TAGS w/ SG is not so good as TAGS. This demonstrates the effectiveness of DG.
WoD and WoC In Table 2, TAGS-DC outperforms TAGS in both MS-COCO and Flickr30K. This reveals that word discrimination and correction contribute to the performance of ITM.
4.1 Difficulty Distribution of Samples from Dynamic and Static Generator