SlowGuess/NAACL_2022_OCR huggingface dataset:
Yujie Lu<sup>1</sup>, Wanrong Zhu<sup>1</sup>, Xin Eric Wang<sup>2</sup>, Miguel Eckstein<sup>1</sup>, William Yang Wang<sup>1</sup>
<sup>1</sup>University of California, Santa Barbara, CA, USA
{yujielu, wanrongzhu, miqueleckstein, wangwilliamyang}@ucsb.edu
$^{2}$ University of California, Santa Cruz, CA, USA
xwang366@ucsc.edu
Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective—imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pretrained vision-and-language models. Extensive experiments on GLUE (Wang et al., 2018) and SWAG (Zellers et al., 2018) show that iACE achieves consistent improvement over visually-supervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances. $^{1}$
Cognitive neuroscience studies reveal neural activation in vision-related brain areas when reading text (Just et al., 2004) and show a tight relationship between brain areas processing linguistic and visual semantic information (Popham et al., 2021). In addition, visual imagery improves comprehension during human language processing (Sadoski and Paivio, 1994). Such imagination empowers hu
Entailment?
Neutral?
Premise: A senior is waiting at the window of a restaurant that serves sandwiches.
Hypothesis: A man is waiting in line for the bus.
Figure 1: Rendering visual imagination is an intuitive way to activate perception for linguistic understanding, e.g. natural language inference.
man brains with generalization capability to solve problems with limited supervision or data samples.
However, the field of Natural language Understanding has mainly been focused on building machines based solely on language, ignoring the inherently grounded imagination from the external visual world. These studies either learn text-only representations from language corpora (Devlin et al., 2019; Zhuang et al., 2021; Lan et al., 2020) or implicitly involve retrieved visual supervision in pretrained language models (Tan and Bansal, 2020). Thus, their approaches appear limited in transferring the connection between language understanding and visual imagination to downstream tasks, which are essential to solving low-resource circumstances. In addition, these methods are limited to text-only augmentations, whereas visual imaginations leverage cross-modal augmentations to deal with low-resource situations.
Human brains are multi-modal, integrating linguistic and perceptual information simultaneously. Intuitively, the machines could achieve a higher-level understanding of natural language and better learning transference by imitating the procedure of human imagination behavior.
Inspired by this, we propose to understand lan
<sup>1</sup>Source code and pre-trained models are publicly available at https://github.com/YujieLu10/IACE-NLU
Human Language Technologies, pages 4392 - 4402
guage with the integration of linguistic and perceptual information via introducing imagination supervision into text-only NLU tasks. To imitate the imagination-augmented understanding process as shown in Figure 1 with text-only data, we devise a procedure with two steps: 1) pre-train a visually-supervised Transformer over paired text and images retrieved from large-scale language corpus and image set, and 2) construct the imagination with a generative model and fine-tune on downstream NLU datasets by learning the paired imagination and natural language in a cross-modal embedding. We show a detailed description of the cross-modal imagination process for a specific Natural Language Inference task in Figure 2. In this way, we utilize machine imagination to improve the performance of natural language understanding.
We adopt the few-shot learning setting to study the potential of using less human effort of annotation for our proposed iACE to learn the natural language with the help of imagination. Large margin performance gain in both extreme and normal few-shot settings demonstrate the effectiveness of iACE in solving problems with limited data samples. In the full data setting of GLUE (Wang et al., 2018) and SWAG (Zellers et al., 2018), we observe the consistent performance gain of our proposed iACE over the visually-supervised approach (e.g., VOKEN (Tan and Bansal, 2020)) upon four language base models (e.g., BERT, RoBERTa).
In summary, the main contributions of our work are as follows:
- We propose to solve the text-only learning problem in natural language understanding tasks from a novel learning perspective: imagination-augmented cross-modal language understanding.
- To address the problem mentioned above, we devise iACE to generate imaginations in a cross-modal representation space to guide the fine-tuning of the visually supervised language models.
- Experimental results in the few-shot setting validate the consistent superiority of iACE over baselines in tackling the low-resource situation. In full settings, iACE maintains the improvement in GLUE and SWAG.
Figure 2: A detailed view of our iACE framework finetunes on natural language inference task.
Visually-aided Language Learning Previous research attempt to introduce visual information to improve language learning on various Natural Language Processing (NLP) scenarios, including but not limited to machine translation (Grubinger et al., 2006; Elliott et al., 2016), information retrieval (Funaki and Nakayama, 2015; Gu et al., 2018), semantic parsing (Christie et al., 2016; Shi et al., 2019), natural language inference (Xie et al., 2019), bilingual lexicon learning (Kiela et al., 2015; Vulic et al., 2016), natural language generation evaluation (Zhu et al., 2021), spatial commonsense reasoning (Liu et al., 2022) and language representation learning (Lazaridou et al., 2015; Collell et al., 2017; Kiela et al., 2018; Bordes et al., 2019; Lu et al., 2019; Li et al., 2019; Sun et al., 2019; Luo et al., 2020; Chen et al., 2020; Li et al., 2020; Tan and Bansal, 2020; Radford et al., 2021). While most of these studies acquire visual information through retrieval from the web or large-scale image sets, a recent line of studies attempt to generate visual supervision from scratch. The visual information can either be provided in the form of representation (Collell et al., 2017; Long et al., 2021) or concrete images (Gu et al., 2018; Zhu et al., 2021). Though previous studies generate machine imagination, they only tackle specific tasks, such as machine translation (Long et al., 2021) or information retrieval (Gu et al., 2018). To the best of our knowledge, we are the first to utilize machine abstract imagination from large pretrained vision and language models to improve general NLU tasks. Recently, VOKEN (Tan and Bansal, 2020) incorporate retrieved token-level visual information into existing transformer models and achieve consistent improvement. iACE is different from this work for two aspects: 1) we explicitly encode visual imag
ination during fine-tuning. 2) we propose a novel model to borrow knowledge from imagination in both training and inference.
Few-shot Natural Language Understanding Natural Language Understanding (NLU) is a subfield in NLP that involves a broad range of tasks such as question answering, sentiment analysis, and textual entailment. Researchers have collected specific language corpus (Wang et al., 2018; Zellers et al., 2018; Xu et al., 2020) to train the machines on NLU learning. However, the general language understanding problem remains a challenge. Few-shot learning is a learning paradigm that aims to predict the correct class of instances with a relatively small amount of labeled training examples (Fink, 2004; Fei-Fei et al., 2006). It has been receiving increasing attention for its potential in reducing data collection effort and computational costs and extending to rare cases. To deal with data-scarcity in NLU problems, previous research introduces external knowledge (Sui et al., 2021), utilizes meta-learning (Geng et al., 2019; Bansal et al., 2020; Han et al., 2021) and adopts data augmentation to generate labeled utterances for few-shot classes (Murty et al., 2021; Wei et al., 2021). Recent studies (Radford et al., 2019; Brown et al., 2020) have shown that large-scale pre-trained language models are able to perform NLU tasks in a few-shot learning manner. The pre-trained multimodal models also display similar few-shot learning ability (Tsimpoukelli et al., 2021). Unlike previous studies on pre-trained multimodal Transformers that target solving multimodal tasks, our study introduces imagination from the visual world into language models and aims to improve NLU.
We illustrate how we solve the existing text-only learning problem in natural language understanding tasks as the Imagination-augmented Cross-modal Language Understanding (ICLU) problems in Section 3.1. Then we give a detailed illustration of our proposed iACE's architecture in Section 3.2. Finally, we describe the procedure and training protocol of the perceptual-enhanced linguistic understanding paradigm in Section 3.3.
NLU is concerned with understanding the semantic meaning of the given utterances. Data pieces for NLU can be structured as  $(x_{\text{context}}, \mathcal{X}, y)$ ,
where  $x_{\text{context}}$  represents the text context,  $\mathcal{X} = \{x_1, x_2, \dots, x_m, m \in \mathbb{N}\}$  denote a set of text snippets, and  $m$  denotes the number of text samples for a specific task. The model learns to predict the ground truth label  $y$ , which is either regression or a classification label. While NLU is usually regarded as a language-only task, we attempt to solve it from a cross-modal perspective by introducing the novel ICLU problem.
In our ICU problem, data pieces are structured as  $(x_{\text{context}}, i_{\text{context}}, \mathcal{X}, \mathcal{I}, y)$ , in which  $i_{\text{context}}$  represents the visual context related to the text context, and  $\mathcal{I} = \{i_1, i_2, \dots, i_n, n \in \mathbb{N}\}$  denotes the imagination set. The "imagination" refers to the images that are visualized from the text. Here,  $n$  is the number of visualized sentences for a specific task, which is the same as  $m$  by default.
To solve this problem, we devise a novel iACE to construct imagination from textual data and learn the bi-directional alignment between the imagination and text. Specifically, for each piece of text  $x_{j}$  in the sentence set  $\mathcal{X}$ , we first follow (Esser et al., 2021; Radford et al., 2021) and use a generative model to render a descriptive illustration  $i_{j}$ . The visualized imagination will later serve as the visual input in the ICU problem.
Overview Figure 3 provides an overview of the iACE framework. iACE consists of two modules: 1) the imagination generator  $G$ , 2) the imagination-augmented cross-modal encoder  $E_{c}$ . Given the textual sentence  $x = \{w_{1}, w_{2}, \dots, w_{k}, k \in \mathbb{N}\}$  ( $w_{j}$  denotes the  $j$ -th token in the sentence),  $G$  generates corresponding visual imagination  $i$ . The cross-modal encoder then encodes  $x$  and  $i$  as  $t$  and  $v$ , respectively. iACE explicitly provides imagination supervision to the visually-supervised Transformer during fine-tuning on downstream NLU tasks.
Imagination Generator Previous studies introduce visual supervision through retrieval from the web or image sets. However, it is hard to find visuals that perfectly match the topics discussed in each text snippet, especially for the relatively complicated text input for the NLU tasks. Such misalignment between the input text and the retrieved visuals might hinder the model from general language understanding learning. Out of consideration for cross-modal feature alignment, we choose to render specific visualization corresponding to each piece of input text from scratch. Specifically,
Figure 3: Overview of iACE. The generator  $G$  visualize imaginations close to the encoded texts by minimizing  $\mathcal{L}_{GAN}$ . The cross-modal encoder  $E_{c}$  learns imagination-augmented language representation. Two-step learning procedure consists of: 1) pre-train a Transformer with visual supervision from large-scale language corpus and image set, 2) fine-tune the visually supervised pre-trained Transformer and the imagination-augmented cross-modal encoder on downstream tasks.
we construct imagination of the textual input with a large-scale vision and language model guided generative framework - VQGAN+CLIP (Crowson et al., 2022). For each piece of input text  $x$ , we treat it as the prompt and use the VQGAN (Esser et al., 2021) model to render the imagination  $i$  with  $128 \times 128$  resolution and 200-step optimization. At each optimization step, we use the CLIP (Radford et al., 2021) model to assess how well the generated image corresponds to the text.
$$
\mathcal {L} _ {G A N} = 2 \left[ \arcsin \left(\frac {1}{2} \| \boldsymbol {t} - \boldsymbol {v} \|\right) \right] ^ {2} \tag {1}
$$
To be specific, CLIP encodes the input text  $x$  and the corresponding imagination  $i$  as  $t$  and  $v$ , and the training objective is to minimize the distance between  $t$  and  $v$  in the cross-modal embedding space.
Cross-modal Encoder We adopt CLIP as the cross-modal encoder to encode the input text and the generated imaginations. CLIP (Radford et al., 2021) is trained on large-scale image-text pairs and is able to align visual and textual input in the embedding space. Specifically, we use the ViT-B/32 version of Vision Transformer as the image encoder, and Transformer (Vaswani et al., 2017) with the architecture modifications described in (Radford et al., 2019) as the text encoder. For each modality, the self-attention (SA) module is applied to model the regions of imagination or the words
of the text as follows:
$$
S A (F) = \operatorname {c o n c a t} \left(\operatorname {s o f t m a x} \frac {F W _ {j} ^ {Q} F W _ {j} ^ {K ^ {\mathrm {T}}}}{\sqrt {d _ {k}}} F W _ {j} ^ {V}, \dots\right) W \tag {2}
$$
where  $F$  denotes the set of regions of the imagination or the words of the textual sentence.  $W_{j}^{Q}, W_{j}^{K}$ , and  $W_{j}^{V}$  represents the weight in the  $j$ -th head for query, key and value respectively.  $d_{k}$  is the dimension of the embedding.  $W$  is the weight matrix for multiple heads.
To solve the ICLU problem, we learn the bidirectional relationship between the text input and the visualized imagination. We apply late fusion on the text feature  $t$  and visual feature  $v$  to construct the cross-modal feature. Given the set of visual features  $S_{v}$  and textual features  $S_{t}$ , the fused embedding  $X_{S}$  can be given with:
$$
X _ {S} = \left[ R e L U \left(W _ {t} S _ {t} + b _ {t}\right), R e L U \left(W _ {j} S _ {v} + b _ {j}\right) \right] \tag {3}
$$
where  $W$  and  $b$  are of two separate fully connected layers to the visual and text embeddings. The fused embeddings  $X_{S}$  will go through two fully connected layers before we receive the final imagination-augmented language representation.
Visually-supervised Transformer We implement the visually-supervised Transformer language model proposed in Tan and Bansal (2020). The model architecture is a BERT-like pure-language-based masked language model.
Step 1:Pre-training on Large-scale Language and Vision Datasets
3.3 Learning Procedure
We introduce a novel paradigm to better understand natural language by incorporating existing language models with visual imagination. As shown in Figure 3, the procedure consists of two steps: (1) pre-train the visually-supervised Transformer, and (2) fine-tune the framework with imagination on downstream tasks.
Step 1: Visually-supervised Pre-training We pre-train a visually-supervised Transformer following the scheme proposed in VOKEN (Tan and Bansal, 2020), which extrapolates cross-modal alignments to language-only data by contextually mapping language tokens to the related images. In addition to masked language modeling, VOKEN proposed a voken classification task: given a set of tokens with masks, the model is asked to predict the best-matching image (the voken) for each tokens. The pre-training loss can be given as:
$$
\mathcal {L} = - \lambda_ {1} \sum_ {w _ {j} \in \hat {s}} \log q _ {j} (w _ {j} | \check {s}) - \lambda_ {2} \sum_ {w _ {j} \in \hat {s}} \log p _ {j} (v (w _ {j}; s) | \check {s}) \tag {4}
$$
Here  $s$  is the token set,  $\hat{s}$  is the masked tokens, and  $\check{s}$  is the unmasked tokens. The  $q_{j}$  and  $p_{j}$  represent the conditional probability distribution of the  $j$ -th token given the token  $w_{j}$  and voken  $\nu(w_{j}; s)$  respectively, and  $\lambda_{1}$  and  $\lambda_{2}$  are the balance factor of the masked language modeling task and the voken-classification task. The cross-modal classification task enables the model to learn the matching between the tokens from the language corpus (e.g., wiki) and its most-related images from the image set (e.g., MSCOCO).
Step 2: Imagination-augmented Fine-tuning We use GLUE (Wang et al., 2018) and SWAG (Zellers et al., 2018) as the downstream datasets in the following sections. Our proposed iACE learns to minimize the cross-entropy loss below:
$$
\mathcal {L} _ {\text {I m a g i n e}} = - \sum_ {j = 1} ^ {| D |} \sum_ {k = 1} ^ {K} y _ {k} \log p _ {k} \left(d _ {j} (\boldsymbol {t}; \boldsymbol {v}) | D\right) \tag {5}
$$
where  $j$  denotes the  $j$ -th data sample in dataset  $D$ , and  $K$  is the class number. The  $p_k$  represents the conditional probability distribution of  $d_j$ . During fine-tuning, the visually-supervised Transformer language model only relied on the textual input to
make predictions. The loss are computed as:
$$
\mathcal {L} _ {\text {L a n g}} = - \sum_ {j = 1} ^ {| D |} \sum_ {k = 1} ^ {K} y _ {k} \log p _ {k} (d _ {j} (\boldsymbol {t}) | D) \tag {6}
$$
Notice that we use MSE loss for the regression task. The imagination-augmented loss and pure-language based loss are summed up with a balance factor  $\lambda$  in a jointly training schema as:
$$
\mathcal {L} = \lambda \mathcal {L} _ {\text {I m a g i n e}} + (1 - \lambda) \mathcal {L} _ {\text {L a n g}} \tag {7}
$$
We use Adam Optimizer with a learning rate  $1e - 4$  for the GLUE benchmark and  $2e - 5$  for the SWAG dataset. We discuss more details in Section 4.
Datasets & Metric We conduct experiments to evaluate the performance of our proposed method over SST-2 (Socher et al., 2013), QNLI (Rajpurkar et al., 2016), QQP (Iyer et al., 2017), MultiNLI (Williams et al., 2018), MRPC (Dolan and Brockett, 2005), STS-B (Agirre et al., 2007) from GLUE (Wang et al., 2018) Benchmark, and SWAG (Zellers et al., 2018) dataset. We construct few-shot setting subsets by taking  $0.1\%$ ,  $0.3\%$ , and  $0.5\%$  of training instances as the Extreme Few-shot Setting, and  $1\%$ ,  $3\%$ , and  $5\%$  as the Normal Few-shot Setting. We train the model with the subsets and evaluate its performance on the complete development set. We use accuracy as the default evaluation metric and compare such results in the following sections.
Baselines We choose BERT (Devlin et al., 2019) and RoBERTa (Zhuang et al., 2021) as the base language models, and apply our iACE framework on top of their small and base architectures for comparison. A recent study proposes a visually-supervised language model VOKEN (Tan and Bansal, 2020) that introduces visual supervision into language model pre-training by borrowing external knowledge from retrieved images of the tokens. In natural language understanding tasks, VOKEN achieved improvements over language-based baselines BERT and RoBERTa. Thus we also use VOKEN built upon these language-based models as a set of powerful baselines. In the following experiments, each model is first pre-trained with visual supervision introduced in (Tan and Bansal, 2020) upon the four base models (BERTsmall, BERTbase,
Extreme Few-shot	SST-2	QNLI	QQP	MNLI
0.1%	0.3%	0.5%	0.1%	0.3%	0.5%	0.1%	0.3%	0.5%	0.1%	0.3%	0.5%
VOKEN(Bertbase)	54.70	77.98	80.73	50.54	51.60	61.96	44.10	60.65	65.46	37.31	54.62	58.79
iACE(Bertbase)	77.98	80.96	81.42	51.64	58.33	64.03	49.36	63.67	71.17	40.07	56.49	59.57
VOKEN(Roberta base)	70.99	71.10	77.86	54.37	62.23	65.78	62.32	67.25	70.18	48.59	49.76	58.23
iACE(Roberta base)	75.34	78.66	83.60	54.79	65.03	65.83	65.43	68.11	70.77	48.94	52.74	59.39
Normal Few-shot	1%	3%	5%	1%	3%	5%	1%	3%	5%	1%	3%	5%
VOKEN(Bertbase)	81.40	86.01	84.75	64.17	77.36	80.19	72.55	78.37	80.50	60.45	62.73	72.35
iACE(Bertbase)	82.45	87.04	86.47	65.09	79.54	80.52	74.31	78.69	80.52	62.15	70.43	73.73
VOKEN(Roberta base)	83.78	84.08	87.61	75.00	81.16	81.23	73.14	79.09	79.63	63.51	70.68	74.02
iACE(Roberta base)	83.83	84.63	89.11	79.35	81.41	81.65	73.72	79.38	79.81	65.66	70.76	74.10
Table 1: Model-agnostic Improvement in Few-shot Setting. iACE and VOKEN upon BERT and RoBERTa base size architecture are fine-tuned in Extreme Few-shot (0.1%, 0.3%, 0.5%) and Normal Few-shot setting (1%, 3%, 5%). For the few-shot setting, we use large and stable datasets from GLUE Benchmark. We compare accuracy on SST-2, QNLI, QQP, and MNLI and the average of accuracy and F1 score on QQP. BEST results are highlighted.
RoBERTa<sub>small</sub> and RoBERTa<sub>base</sub>). Then the models will be fine-tuned on downstream tasks.
Notice that base models and VOKEN use pure-language training objectives during fine-tuning. Neither of them utilizes the visual signals inherent in the downstream language corpora. In contrast, our iACE explicitly introduces visual imagination supervisions into fine-tuning and inference stages.
Implementation Details We train RoBERTa with the same configurations as a robustly optimized pre-training approach based on BERT of the same size.  $\mathrm{BERT}_{small}$  has 6 repeating layers, 512 hidden dimension.  $\mathrm{BERT}_{base}$  has 12 repeating layers, 768 hidden dimension.
The imagination of the texts is generated interactively by using VQGAN+CLIP, with  $128 \times 128$  size, 500 iterations. We use pre-trained VQGAN (imagenet $_{f16}$ ) and CLIP (ViT-B/32). We leverage CLIP (ViT-B/32) as our language and vision model for premise and hypothesis, and imagination of them. The text and image dimension is 512. The dropout rate is set to 0.1. We use Cross-Entropy loss for our cross-modal classification. Each model was first pre-trained on 4 Titan RX GPUs for 30 epochs with early stopping and a batch size of 32 and a sequence length of 126. The optimizer used is Adam with a learning rate of  $2e - 4$  and a weight decay of 0.01. The models are then fine-tuned on GLUE benchmark and SWAG dataset for 3 epochs with 32 batch size. We adopt the joint training strategy for our proposed iACE and visually supervised transformer during fine-tuning. The learning rate of the Adam optimizer is set as  $1e - 4$  and  $2e - 5$  for GLUE and SWAG, respectively.
4.2 Few-shot Learning Results