K-LITE: Learning Transferable Visual Models with External Knowledge
Authors: Sheng Shen
Chunyuan Li
Xiaowei Hu
Yujia Xie
Jianwei Yang
Pengchuan Zhang
Zhe Gan
Lijuan Wang
Lu Yuan
Ce Liu
Kurt Keutzer
Trevor Darrell
Anna Rohrbach
Jianfeng Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: external knowledge
task-level transfer
language-image pre-training

Abstract: The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, based on the broad concept coverage achieved through large-scale data collection process. Alternatively, we argue that learning with external knowledge about images is a promising way which leverages a much more structured source of supervision and offers sample efficiency. In this paper, we propose K-LITE (Knowledge-augmented Language-Image Training and Evaluation), a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in natural language with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts; In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is released at https://github.com/microsoft/klite. 

K-LITE: Learning Transferable Visual Models with External Knowledge
Sheng Shen" Chunyuan Li Xiaowei Hu"t, Jianwei Yang', Yujia Xie Pengchuan Zhangi, Zhe Gan Lijuan Wang', Lu Yuan Ce Liut, Kurt Keutzer Trevor Darrell', Anna Rohrbach', Jianfeng Gao Microsoft
University of California, Berkeley
Abstract
The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to de- scriptive captions. This form of supervision ensures high generality and usability the learned visual models, due to the broad concept coverage achieved via large- scale data collection process. Alternatively, knowledge is a promising way which leverages a much more structured source supervision and offers sample efficiency. We propose K-LITE simple strategy to leverage external knowledge for building transferable visual systems: In train- ing, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge knowledge-augmenter augmented models show significant improvement in transfer learning performance over existing methods.
argue that learning with external
1 Introduction
One of the core aspirations in computer vision (CV) is to develop systems that endow computers with the ability to effectively learn general visual representations, which can be transferred to a variety of downstream recognition datasets with arbitrary visual concepts in the wild. Though excellent performance has been achieved on standard benchmarks, the traditional supervised approaches are limited to learning fixed set of concepts, e.g., 22K concepts on ImageNet [1S] or 18K concepts on JFT-300M 32] This leads to a few issues: (i) Annotating each individual vision dataset is not only labor intensive, but also results in narrow of visual concepts; (it) Visual models trained such datasets are good at one task (with the given concept set) and this task only, and show poor transfer learning performance customized datasets that usually come with different of concepts (27
To tackle this problem, recent large-scale Inguage-augmenter visual models, such as CLIP wide variety of images with natural language
ALIGN 336] and Florence [0II, trained supervision that is abundantly available on the Internet. These models demonstrate strong zero-shot transfer capabilities, since they acquire open-set recognition abilities through problem reformulation from classification to retrieval. Moreover, model generalization is improved as natural language
"qqual Technical Contribution Project Lead
Language Image Training and Evaluation 2 OOur code is available at hattts://githhi com/esccrosoft/klit.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

supervision typically contains rich semantics. While these models usually perform well on recogniz- ing common objects, they still struggle on visual concepts that are absent or rare in the pre-training stage. To ensure good transfer performance, it is required to train such models on huge datasets with sufficient concept coverage(e.g, >400M image-text pairs), which is both labor and compute expensive.
Instead of scaling the number of image-text pairs increase concept coverage, we propose to leverage structured external knowledge to augment language supervision. The inspiration comes from how humans generalize to novel concepts: instead of trying to memorize all concepts, humans leverage the structured knowledge such as definitions and concept hierarchy. For example, when we visit a Japanese restaurant for the first time, we may struggle to understand the menu by only looking at the dish names (e.g., Takoyaki, Sashimi), as it is hard to imagine what they are. However, it becomes much clearer once waiter introduces these concepts (Figure D. leading to success in ordering food (i.e., matching content to name). Similar intuitions have been exploited in computer vision for class-level transfer
94 but not yet for task-level transfer settings (similar to that of CLIP).
To this end, we explore systematic approach to ac- quire and learn with external knowledge sources from databases such as WordNet [ and Wik- tionary
Takoyaki
ball-shaped Japanese dumpling made of bottes, with diced actopus, tempura scraps, pickied ginger, and green onion.
62) to train more transferable and sample- efficient visual models. The concept descriptions and concept hierarchies are purely textual, and the process of collecting external knowledge is fully automatic without extra human annotation. The ac- quired knowledge typically provides information that is shared between seen and unseen concepts to facilitate effective transfer. Specifically, rare con- cepts, e.g., Takoyaki, Sashimi in Figure D. are explained with more common concepts. Such knowl- edge sources are generally available for a variety of domains and datasets, making it possible to build generic approach for task-level transfer.
Sashimi
A consisting thin slices or pleces raw fish or meat.
Figure I: Motivating examples: knowledge
explains the content of the rare dish concepts.
Our main findings and contributions can be summarized as follows:
We present the first strong evidence that external knowledge can benefit large-scale task-level transfer for two core CV problems, image classification (IC) and object detection (OD), by exploring external knowledge sources, including WordNet and Wiktionary.
A simple and effective strategy K-LITE Evaluation) is proposed: The acquired knowledge is appended to the original textual concepts as model input during pre-training and evaluation. can be viewed as an automatic knowledge-awang language prompting, which makes it casier for the model to access relevant information shared between the training and evaluation data. modularized approach is also developed to enable efficient adaptation from vanilla visual models to their knowledge knowledge-augmentee augmented versions.
Language Image Training and
To demonstrate the generality of the K-LITE, we instantiate it with two recent visual models and develop our knowledge-augmente
counterparts: UniCL (95] for IC and GLIP (50) for OD, Extensive experiments in zero-shot and few-shot learning settings demonstrate that knowledge- augmented models can significantly improve over prior work. Notably, our model can achieve similar zero-shot performance to previous methods using only half of pre-training image-text pairs in some scenarios, demonstrating sample efficiency of the proposed approach.
2 Related Work
Zero-shot Visual Recognition: Zero-shot learning, i.e., classifying images where there is lack of labeled training data, has been studied for decades 21I and its popularity has recently increased further [94]. Based on the technique evolution, can be broadly categorized into two generations: the traditional class-level zero-shot and recently popular task-level zero-shot setting.
Class-level Trunsfer: Class-level zero-shot learning aims to recognize object classes whose instances have not been observed during training. The goal for the zero-shot learning methods is to associate
observed and non-ovserved classes through some form of auxiliary information, which can be either
implicit such pre-trained semantic embeddings 9 i8 [[]
explicit such as attributes 21 [3]

---

knowledge graphs 11 74], or rules and ontologies 23 Please, refer to the recent survey on knowledge-aware zero-shot learning for a more detailed review 12] Recently, it has been suggested to move away from the restricted nature of standard zero-shot evaluation and make the task more practical by including training classes at test time, i.e. generalized zero-shot learning setting [94] Despite the progress in this area, the traditional setting is typically limited to studying zero-shot transfer across classes in a single domain with manually defined splits, such as Animal with Attributes (AwA) AEI. Birds-200 [8)] SUN attributes [67, and ZS- ZS-ImageNet [73 26 Concurrently, [5 explore leveraging external knowledge to improve long-tailed visual recognition within individual domains, which falls into the category of class-level transfer.
351 text 18 19
70
level Transfer: Another line of work focuses on task-level zero-shot transfer
B6 177. They pre-train visual models on hundreds of millions of web-crawled image-caption or image-tags pairs, and evaluate their transfer ability by directly performing inference in a wide range of downstream datasets, without tuning the model weights. We argue that the task-level transfer is more practical and attractive than class-level transfer, as it is more relevant to real-world scenarios, where we may want to develop models that can serve many visual recognition applications.
Our work bridges the gap between the two lines of works above: it borrows the spirit of exploring knowledge in class-level transfer, and generalizes it for task-level transfer, leveraging the best of both worlds. To summarize, our work is different in two major aspects: (i) Settings. We focus on the task-level transfer learning across domains, i.e., from large publicly available datasets to diverse set of downstream datasets in different domains, and demonstrate that external knowledge benefits task-level transfer. (it) Modeling. Existing class-level transfer works are built upon pre-trained visual features/hackhones and shallow word embeddings or tf-idf scores, and only train the classifiers. One representative example is DeViSE [25], where a skip-gran -raa-grmmmm word embedding model and an image classifier are fine-tuned jointly. In contrast, we are training large Transformer-basee models in an end-to-end manner from scratch as in CLIP/ALIGN, providing the first empirical evidence that external knowledge can help train a general visual backbone.
Knowledge-Ine Intensive Models: In natural language processing (NLP), with the increase of model capacity via pre-trained language models [[7]. there emerges the need for more knowledgeable models [55] with advanced functionalities such as making use of encyclopedic [88] 4 o
and com- monsense knowledge 77 1022. To address this, a large number of language models augmented with
external knowledge sources have been proposed 68
45
56
100
7 achieving strong perfor-
mance on a variety of NLP tasks 66 Please refer to a recent survey
98 for a comprehensive
review.
In vision-and-language (V+L) domain, researchers have also started exploring knowledge-intensive tasks, e.g., OK-VQA 61 and WebQA [9] They often require additional information sources (e.g., factual and commonsense knowledge) beyond the QA pairs, compared to the established tasks such as VQA
R 34 and image captioning [c 2). Hence, existing pre-trained models 49
33
58
5
81
103
B8
30
would perform poorly on these knowledge-intenee intensivve V+L tasks
To address
the problem, acquiring external knowledge becomes an essential component for success
X 60 96
The success of knowledge in NLP and V+L tasks inspires to ask natural question: Can learn a transferable visual backbone model with external knowledge? Thus, we dissect and borrow the vital elements such as knowledge sources [3 52 studies for core computer vision tasks.
and modeling techniques 90 100). and carry out
3 Knowledgg-Augmenter Visual Models
Problem setup. Computer vision systems have achieved strong transfer performance, when learning with large-scale image-label data 39 and image-caption data 77.. Recently, it has been demonstrated in 195 100 superior performance over either of them. We follow the setting in 195]. and define unified triplet-wise data format D = {(x,, bn yv M- where æ E X' is an image, t E T is its language description, and y E )' is label indicating the index of the unique language description in the dataset. In general form, the language description is a text sequence = [t,." ... tzl. It ranges from simple category names representing visual concepts when L is small, more free-form and semantic-rich sentences such as captions when L is relatively large.
that the unification of image-label and image-text formats into image-text-label achieves
In this paper, we assume there exists an external knowledge source S, where one may use the language description as a query to seek additional knowledge deseription s E S for f. Given these triplet data instances D and an external knowledge source S, our goal is to learn generic visual-
3

---

Original
Query Construction
Knowledge Acquisition
WORDNET:
Image Classification
Dataset
Pretrolning
Tash Transfer
Imageliet22 GCC/YFCC
Imagelett datasets
Image
Language
Query
Knowledge
Wiktionary
Object Detection
Language-Image Learning
Pretraining Object365
Task-level Transfer
datasets
Knoweedge-Augmented Language-Image Learning
Figure 2: Left: Illustration of data construction process of the proposed knowledge-augmentt language-image learning, in contrast to the baseline language-image learning. The query g constructed from Eq.[ The same process is performed for both pre-training and downstream tasks. Right: The proposed strategy is applied to IC and OD for task-level transfer.
semantic representations, which are readily transferable to a wide range of downstream datasets, whose category names not necessarily observed during training. In Figure 2 we visually illustrate the proposed nhowledge-augmentte process and two considered application scenarios.
3.1 External Knowledge
Query Construction. For a text sequence associated with an image, different tokens may play different roles in contributing describing the main semanties of the image. Humans leverage this prior inherently in parsing the sentences to understand the image. Further, it is infeasible to employ the entire sequence as query, as this may lead the lack of coverage in the knowledge bases. Instead, we propose to construct a query E Q as compact form of original language description t, represented with the words that convey main concepts of the image.
Specifically, we consider divide-and-conquee approach. For short text sequences t such as category names in image-tag/label datasets (e.g., ImageNet (15).. we directly the category query. For long text sequences t such as captions (e.g.. YFCC (4I), we first parse the sentence to extract the noun-phrases, among which the most rare noun-phrase over the corpus is used as query for this sentence. The intuition is to convert the rare concepts into "explanations" represented in common words using external knowledge. Noun phrases are useful for summarizing the sentence and thus inferring what is being talked about in the image. For example, in "professional boxer is introduced to the crowd", the noun-phrases are "boxer". "professional boxer". and "the croud". We summarize the query construction process Jquery below for clarity:
the
when is category, class tag name, when t is a caption.
q Aquerg (t)
The most rare noun-phrase(
Knowledge Acquisition from External Sources. We consider three knowledge sources S to enrich the language descriptions t. They constructed based the two knowledge bases: WordNet (63 and Wiktionary [62]. To measure the breadth of a knowledge base, we define the concept coverage as the percentage of non-empty knowledge items retrieved for given set of issued queries.(i) WordNet [53 is a lexical database which links words into semantic relations including synonyms, hyponyms, and meronyms. Both nouns and verbs are organized into hierarchies, defined by hypernym relationships. The synonyms in WordNet are grouped into synsets, expressing the same distinct concept. The synsets serve as natural link between language and vision domains. For example, ImageNet is an image database organized according to the WordNet hierarchy, where each node is depicted by hundreds/thousaads images. (ii) Wiktionary (62) is web-based content dictionary of terms (including words, phrases, proverbs, linguistic reconstructions) These entries may contain definitions, illustrations, usage examples etc.. Next, we provide our knowledge retrieval process Gretrieve (g) for each source S, followed by example result for the query "boxer":

---

WordNet Hierarchy Swn pah- A WordNet node of the query is located, then repeatedly search its parent node. The words along the traversal path are recorded as the knowledge.
[boxer, combatant, person, causal_agent, physical_entity. entity]
WordNet Definition Swa_def- The definition from the synsets is used explain the query. someone who fighta with his fists for sport
Wiktionary Definition Swiki defn- The query is used for dictionary look-up in Wiktionary, and the corresponding definition used.
a fighter in boxing match
When multiple meanings (senses) exist for given query, we simply consider the first one for simplic- ity, and leave more sophisticated designs as future work. After querying each knowledge source, we represent the external knowledge for each language description é in the form of concatenation of its query and the corresponding retrieved result: [g. 8). This external knowledge introduces additional supervision signals to guide visual models to learn better aligned as we explain later. We next describe how to encode knowledge in multimodal models improve transfer in the image-level task of image classification and the region-level task of object detection.
representations,
3.2 Image Classification
Recent works that learn visual models with language supervision 7 often employ a dual-encoder architecture. For each image an image encoder model fe parameterized by 0 first represents as a visual feature vector ò E Ix é = (x)).. For each language description t T, we encode with text encoder f (t) parameterized by Φ, and get the [EOS] feature as the vector representation of the sentence à E IR Px11 û = (t). In this paper, we further leverage this text encoder to encode the external knowledge. First, the query q is represented
natural language p = 9prompt (q) using the language prompt in [71]. Depending on the language input t and our augmentation scheme, the mhowleggeaaugmente text sequence t T (k stands for knowledge) is represented as:
t
p, q, s],
when t is category, class tag name, when t is a caption, and a concat scheme is used, when t is a caption, and a combine scheme is used.
it q. s), }
(2)
For example, for category name boxer or for caption = professional boxer is introduced to the croud, we have q = boxer, and the corresponding language description are:
th photo of cool boxer: boxer fighter in a boxing match
t =professional boxer introduced the crowd ; boxer fighter in boxing match
We re-use the same text encoder fo to encode t as the original t as supervision for image æ. Training. For i-th image æ, and j-th language description t, in batch B, we normalize their feature fo(a.) vectors in a hyper-sphere using [J((= and uy = [T and their similarity is calculated as u. Vj. bidirectional supervised contrastive objective considered to train the model:
[n(t,
min (0.0)
Lic Lizt + Li2i, with
Σ log exp(Tu, vx)
Σ log exp(Tu v,)
Lizt
Σ 1EB
and Lizi
Σ 3E8
P())) (EP(1)
Σ jEB exp(pee ru, v,)
EiEs exp(Tu, v,)
kE kER
where P(t) = {Ailk E B.yk y: 2(j) = {kiK- E B, yk = yj and T is a temperature hyper- parameter controlling the strength of penalties on hard negative samples. Note ① is general form; it reduces to the training objective of CLIP IZI or ALIGN (3 when there is a one-to-one mapping between image and paired caption in a batch, i.e., P(i) {i) and Q6) (}}.
Evaluation. Given a downstream image classification task with a custom set of category names, we represent them with the aaueeegg-augmeetee prompt form in ( they are fed into the pre- pre-trained text encoder f to obtain the class embedding. The test image x is encoded with fo (x: and compared all class embeddings get its label from the best matching class.
Extensions with Modularized Modeling. In our study, we found it is key to ensure consistency between training and evaluation stages: if model is trained with knowledge, it performs well when
5

---

also evaluated with knowledge, Similarly, if model is trained without knowledge (e.g. CLIP/UniCL), adding knowledge directly in the evaluation stage results in performance drop. However, due the limited knowledge coverage in existing knowledge bases, 8 could be empty for a large number of queries q- When the low coverage happens for a downstream evaluation dataset, may result in a training-evaluatioo inconsistency for our aeageegg--ggeeeeeed models, and thus lower performance.
is desired have a modularized model that can switch between "with" and "without" knowledge settings. Inspired by [90], we propose to employ adapters 33 to build the network branch to encode language th, where serial MLP adapters are inserted after each self-attention and MLP modules for all Transformer layers of the text encoder, and t" is passed through fo and adapters. Meanwhile, the original f is reserved as the branch to encode vanilla natural language t. The proposed adapter-modulayyy modularized architecture can also be used for efficient stage-wise continual pre-training: one may start with a vanilla language-iman model pre-trained on D, and continue pre-train the adapters with www.edge-augmeett data (D,S) to build its knowledge version.
3.3 Object Detection
Object detection (OD) typically involves two tasks: LOD = Lels + Lioes where the localization task Lise aims to locate the presence of objects in an image with a bounding box, and classification task fas determines what object categories are present in that box. Similar to the IC task above, we improve the categorization task of individual boxes Las, in OD with external knowledge, and keep Lie the same. Specifically, we leverage GLIP 50 to reformulate OD as a phrase grounding task, by grounding each region proposed by Live
53 to phrases in text sequence. For language encoding, we form t = ig. s): this is different from
first augment category name t into its IC in that p is excluded, as no prompt engineering is used OD, as in (50)- In the original GL.PP, a sequential text encoding scheme is used: a concatenated long sequence (q:..... ; 4Kk over category names is considered as the text encoder input. In our case, simple concatenation [q1. ... 9xx 8K] will quickly break the max length requirement of the language encoder.
To resolve the issue, we propose a parallel text encoding scheme: each th is passed through language encoder independently, we use the top-layes feature of [CLS] token as the contextual vector repre- sentation RR Px1 of th: ū = ft Given K categories, they can be encoded in parallel in a batch; the encoded phrase feature sequence is the concatenation U E RPXK, U ... ūg]. The region encoding is the same as in GLIP. The feature pyramid is V E MxP. V = ff((x)) where M is the number of box features. The alignment scores S ground are computed:
S Sground VU, = ground? T),
(4)
where T E {0, 1 is the target indicating match or no match, and MMS; T) is the focal loss (53 The grounding model, consisting of the image encoder fe. the language encoder f and cross-modal interaction head introduced in [50], is trained end-to-end by minimizing the loss defined in Q the evaluation stage, the external knowledge is also retrieved, and encoded in the same parallel encoding manner to enrich the category names in the downstream OD tasks.
Experimental Results
In this section, we examine our showledge-augmentee approach answer two research questions. Q1: To what extent external knowledge benefits visual transfer learning, including sample-efficieng in pre-training and downstream? Q2: Why does external knowledge help zero-shot transfer (illustrated with success and failure case studies)?
4.1 Settings
Evaluation benchmark. We apply the proposed knowledge-augmentg models to two task-level transfer settings defined in ELEVATER benchmark A7, which evaluates the transferability of the learned visual representations in the wild. We study our models based on the datasets described in Table I The license, PII, and consent details of each dataset are in the respective papers. Due to the limited computational resources, the pre-training datasets are constrained to the large publicly available datasets used in 95 50) This setting is defined as the "Academic Track" in [47. which friendly to the academic community to allow reproducibility of the results. The number of visual concepts is identical to the number of categories for datasets with category names (e.g., ImageNet

---

Pre-training #Concepts
Downstream Concept Overlap (%)
Task
#Instances
Vocab, Size
#Ins/#C.
Dataset
ImageNet-Ik IK 20-datasets
ImageNet-21K
13M
19.2K / 18.4K
13.5K / 12.9K
591 537
11.82
13.26
IC
GCC-3M
70
3.3M
681K
64.5K
29.6K 13.0K
9.5 1:30
35.97
19.73
GCC-12M
1
12M
10,2M/
7228K
1.24M/ / 264K
5.6 i 353
61.02
31.34
YFCC-14M
M4
14M
14.2M / 1.25M
2.41M/ / 473K
8.3 + 1384
65.23
34.65
Datasct
LVIS
13-datascts
OD
Object-365
85
9.6M
365 / 365
452/
26.3K 12.4K
13.46
21.26
Table 1:
Statistics of training and test datasets used in our experiments. #lnstances indicates #Image for IC and #Regions for OD, respectively. For #Concept and Vocabulary size, we report numbers for the full set and for items with frequency larger than 5. #Ins/C. reports the mean and standard derivation for the numbers of instances per concept.
Training Method
Knowledge S
ImageNet-1KK
ICinW (20 datasets)
28.16
4.93
27.15
17.10
SwaJier
27.43
29.03
28.15
28.66
1-branch, from scratch
Swa_der
22.87
29.31
26.97
29.14
Switi def
22.05
30.23
29.03
33.44
branch, continue pre-training
Swiii aet Switi det 28.16
28.16
28.40/28.00
27.15
30.73/30.91
2- branch, from scratch
32.52/32.44
27.15
32.46/33.49
Table 2: Zero-shot task transfer performance after pre- pre- training ImageNet-221 dataset. The top block studies the effectiveness of knowledge sources S, and the bottom block studies the modularized approach. For each downstream task, the 1st and 2nd column reports the results without and with knowledge, namely green cells indicate "a match", orange cells indicate "a mismatch" w.r.t. adding knowledge in training and evaluation. In the bottom block, we report two numbers when evaluated with knowledge: using the knowledge branch only, and using two branches selectively.
2 to extract and Object-365). For image-text data (bottom 3 rows of the IC block), we use Spacy the noun phrases. We also use a merged version GCC-3M and GCC-12M denoted as GCC-15M, Given the pool of concepts, we calculate the number of unique words and report it the vocabulary size. For Concepts and Vocab Size, we report numbers: first for the full set, second for items with frequency larger than 5. The latter provides sense of "long-tailness, #Instance/ / #Concept) illustrates the varied trade-off over different datasets: image diversity, semantic richness and long-tailness. For example, YFCC is the most long-tail dataset in IC, as it has low mean value and the largest standard derivation value in #Instance #Concept. The concept overlap is computed as the percentage of concepts in a downstream dataset that are covered by the pre-training dataset. For 20-datasets and 1 13-datasets, the averaged overlap across individual datasets is reported. measures the gap (or difficulty) in concept transfer between the pre-training and the downstream data. The dataset statistics are detailed in Section B.I in Appendix.
The statisties (e.g., ratio of
Zero/Few-shot image classification. Following UniCL 195 this task evaluates to what extent a model understands novel concepts. We pre-train on ImageNet-2IK [15] and GCC 776 /TCCFCC [4
[5] and suite of 20 datasets (ICinW) proposed in 477 We use the same text prompts as in [71 and report scores averaged over 20 datasets.
datasets, and report results ImageNet-1K 1K
UniCL/Florence [TOI] show superior performance to CLIIP ALIGN counterparts; UniCL is Florence in a controlled academic setting, trained on the large publicly available datasets (55]
Zero-shot object detection. Following GLIP 50
we pre-train on Object365 771 and transfer the learned visual representations for object detection on LVIS [2] and a suite of 13 small OD datasets (ODinW) proposed in 50 A7
check the generalization ability. The box mAP is reported.
4.2 Image Classification
ImageNet-21K pre-training. We start by pre- pre-training on ImageNet-211K where all ImageNet- IK images are excluded. We still see small amount of concept overlap in Table and hypothesize that some category names are given based on different level of WordNet hierarchy. The benefits this setting are two-fold: it ensures distinctively less concept overlap, and all concepts can find their full WordNet knowledge. We report the results in Table 2 For each checkpoint, we report the results without and with knowledge in the evaluation stage. We confirm two major findings below.
7

---

ImageNet-IK
ICinW (20 datasets)
Training Data
Method
Dataset
Samples
Zero-shot
Zero-shot
Linear Probing
Fine-tuning
ImageNet-2IK
13M (full)
UniCL
28. 16
27.15
53.07
4.15
55.96 = 2.50
13M (full)
K-LITE
30.23
33.44
53.92
1.6
57.81
1.48
14M (half)
UniCL
34,43
34.30
53.50
2.22
56,45
= 2.48
YFCC-14M
14M (half)
K-L,TR
36.67
36.50
49.48
1.21
55.88
1.64
ImageNet-21K
14M (half)
K-Tara
42.36
36.50
54,28
166
52.11
27M (full)
UniCL
43.06
35,99
55.96
1.39
58.25 1. 2.9
27M (full)
K-LITE
45.67
38.89
57.06
58.24
15M (half)
UniCL
41.64
36.31
53,86
+:73
59.04 =: 3.13
GCC-15M
15M (half)
K-LITB
44.26
39.53
55.91
=: 2.53
58.20
=: 3.19
ImageNet-21K
15M (half)
K-LITE
47.30
40.32
57.38
2:70
60.72
2.29
28M (full)
UniCL
46.83
38,90
57.92
3.31
60.99
28M (full)
K-LITB
48.76
41.34
58.56
63.39
Table Overall comparisons of our Knogeeege-aggmented models. Each model is pre-trained with 32 epochs following CLIP [ZT]/UniCL. 195]. It indicates that the Combine scheme is used for the image-caption data, otherwise the default the Concat scheme decribed in Section 3.2 The linear probing and fine-tuning results are reported for 5-shot settings over 3 random seeds.
F1: All three knowledge sources are beneficial. In Section 3.1 we have introduced WordNet hierarchy Swn_path. WordNet definition Swn, def. and Wiktionary defimiioon Swiki der as external knowledge sources. It is shown that all three of them are effective, improving the zero-shot accuracy by absolute gain 1-2% on ImageNet-1K (from 28.16% 16% to 30.23%) and 2-6% on the dataset suite in average (from 27.15% to 33.44%), respectively. Among them, Wiktionary definition Swiki turns out to be the most effective, therefore, we use it as the default knowledge source throughout the remaining experiments.
F2: The modularized approach is effective. Our results in Table reveal that training/test inconsistency in terms of involving knowledge can dramatically degrade the model performance. For example in the 1st row, the baseline UniCL is pre- pre-trained without knowledge, its performance decreases from 28. 16% 4.93% when knowledge is added in the test stage. In contrast, in the 4th row, our K-LITE is pre-trained with knowledge, its performance decrease from 33.44% to 29.03% if knowledge is excluded in the evaluation stage. Therefore, we consider modularized approach with 2-branch in the model. First, we continue pre-train our modularized model from a 32-epoch knowledge checkpoint by only updating the Adapters on www.lleggggggggneeeee image-text pairs for 10 epochs. It already shows a performance gain from 27.61% to 28.40% This suggests more affordable solution to obtain Knowledge-augmentt
models from existing models. We can further boost the performance 28.90% if we evaluate with two branches, each of which only passes its corresponding language version. Finally, we also train the modularized model from scratch, and demonstrates a significant gain (absolute 4%) on ImageNet- IK, and over 5% improvement the 20 datasets. For fair comparisons with knowledge-free models, we train our models with one branch in the rest of experiments.
To demonstrate the performance of K-LITE in the extreme large-scale settingss we leverage the largest checkpoint of Florence [101 trained on 800M image-text pairs, and continue pre-training the model on ImageNet-21KK with external knowledge. It improves the zero-shot ImageNet- 1K accuracy of Florence from 83.74% to 85.80% As an ablation baseline, continuing pre-training without knowledge yields 85.35%. The absolute 0.45% performance gain shows that external knowledge can still benefit transfer learning, though huge amount of pre-training data is employed.
Pre-training on image-text-label data. The unification of image-label and image-caption as image- text-label has been demonstrated superior over either one of them (95). Therefore, we report overall results on the combined data in Table The few-shot learning results are reported with training examples, using two model adaptation method: linear probing and full model fine-tuning. The average numbers over random seeds are reported. K-LITE improves its knowledge- UniCL in almost all the cases. Importantly, K-LITE can outperform UniCL using only half of the pre-training image-text pairs in several cases. It demonstrate the high sample-efficiency of K-LITE, and that external knowledge is an effective source consider, when collecting large-scale image-text pairs to develop language-augmenter visual models at scale. We also compare K-LITE and UniCI. with Swin-Bas Base on the joint data including ImageNet-21K,, GCC15M and YFCC15M. K-LITE
counterpart
8

---

Knowledge covenge
Concapt overtap with knowledge
Origno concest overap
Performance gain with knowledge
100
Concapt overlap (%)
A0-
-50
Score (%)
-2-
+20.2
26.4
23-
+17.8
140
633.3
60
5.05
81
PateiCamesyy
DrD
Commrral
Staddontaans
Figure 3: Performance improvement analysis with external knowledge. External knowledge can
largely improve concept overlap between pre-training and evaluation stages, hence usually yields higher recognition scores. Knowledge coverage indicates the percentage of concepts that exist the knowledge base for each downstream dataset.
& English martgold Any of the Old World plants, of genus Calendula, flowers.
V Lobster bioquer thick creamy soup made from fish, shellfish vegetables
orange, yellow
Wallfower: Any of short-lived herbs shrubs the Erysimum genus
Hot and sour soup Any one of several soups, served various Asian cuisines, both spicy
with bright yellow to red flowers
(a) Success examples. The two datasets with largest improvement in Fig. Flowers 102 and F0od101. The description of the parent concept, material, shape, color etc. clarifics the concepts, boosting performance for the graingr classification tasks.
bus motor vehidle transporting numbers people joads
builddin''s usually covered with corpeting. sitting
crop land arabie land
car: wheeled vehicle that moves
cheir An iamm of furniture used to sit comprising seat. legs or whenls
independently with at least three wheels
permanent crop land arabie land
(b) Failure examples. The two datasets with the largest performance loss Fig. Left (EuroSat): both class names have the same knowledge. Middle & Right (VOC2007): The knowledge contains spurious words that confuse the models.
Figure 4: Success and failure cases on image classification. For each image, the top row is the knowledge-based prediction, and the bottom row is the baseline prediction (no knowledge).
improve the zero-shot performance of UniCL from 52.18% to 57.78% ImageNet-IK, and from from 43.20% 45.47% on ICinW.
Breakdown Analysis. Next, we ask why does external knowledge improve the zero-shot task transfer performance on a broad range of datasets? To answer this question, we compare the breakdown performance on all 20 dataset in Figure 3 for the ImageNet-21K checkpoints trained with and without Wiki knowledge. Out of 20 datasets, external knowledge shows superfor/comparableffffeee performance to the baseline on 16/1/3 datasets, respectively. One prominent observation is that Wiki knowledge improves concept overlap for train-evaluation from 13.26% 51.24% by average. is easy to understand, concepts are explained in more commonly used words in Wikitionary, providing bridge for train and evaluation. This reflected by the increased height of blue bar for most datasets in Figure 5 Interestingly. for all datasets with increased accuracy scores, there shows an increase of the concept overlap. In summary, knowledge is an effective approach to improve concept overlap, a prerequisite for good task transfer performance. In Figure 4 (a), we provide success examples after adding knowledge; more examples are shown in Appendix.
augmented approach does not help mainly belong to two scenarios: (i) No externl knowledge was extracted from the given knowledge base (i.e.,
Limitations. The failure cases where
Wiktionary in this case), e.g., StanfordCars and FGVC Aircraft. They often require Nomain-sspecife knowledge explanations to define car brand (e.g., Volvo C30 Hatchback 2012) or an aircraft model type (e.g., 737-200), while Wiktionary can hardly provide such professional definitions. (ii) While knowledge is available, the quality is too low to provide useful information. Figure g (b), provide failure examples from two datasets with the biggest performance loss after adding

---

Method
LVIS
W (13 datasets)
APr
APc
APF
SINIS
Son
Sun
Swai det
Sun pum
Sun def
Swit det
GLIP-A 50
14.2
13.9
23.4
18.5
28.8
Baseline GI.IP
8.6
14.0
23.1
17.9
17.6
17.1
17.2
15.0
27.5
26.8
21.0
18.5
K-LITE
14.8
18.6
24.8
16.9
21.3
18.7
21.4
20.5
25.0
30.3
28.4
31.7
Table 4: Zero-shot task transfer performance on OD. APr/APc/APt indicates the AP values for rare, common, frequent groups of categories on LVIS. Cell coloring follows the same protocol as in Table 2 without external knowledge.
GLIP is implemented with parallel text encoding Section 3.3
knowledge. In Figure 6 in the Appendix, we show more examples where knowledge only yields slight improvement. To summarize, promising future research direction is to improve the knowledge quality to be more related to the given classification tasks.
4.3 Object detection
We evaluate the model's ability recognize diverse objects on LVIS 28 and 13 downstream datasets used in [50] in zero-shot setting. We report on Mini Val containing 5,000 images on LVIS. Our K-LITE GL.IP is trained with Wiktionary definitions. The results are presented in Table 3] The 1st. row are the original numbers reported in [50], using the sequential text encoding. The 2nd row is our implementation of GLIP using parallel text encoding, whose effectiveness is validated by the comparable numbers with 1st row. The 3rd row is
-uggeedee GLIP, i.e., K-LITE. The benefit using external knowledge is evident. On LVIS, the categories are divided into rare, common, frequent groups, based on the number of training images per category. K-LITE improves the detection performance for all three groups with an average of 2.8 points on LVIS, and particularly brings a 4.7 points improvement the enriched semantics of external knowledge significantly helps the model recognize concepts with decent number of instances. Since LVIS has its knowledge source SLVIS, mostly built upon WordNet definitions (88) we evaluate our model with SLVIS- We alter the external knowledge source to Swm_path Swn_ded def- and Swiki_dr def in evaluation, which yields mAP 18.7, 21.4, 20.5, respectively. It verifies that our knowledge source extraction process is reliable. Similarly, K-LITE improve the 13 downstream OD datasets from 28.8 (or 27.5 with its own knowledge-free counterpart) to 31.7. The success and failure examples of OD are shown in Section B.5 in Appendix.
MiniVal APc over the GLIP-A reported in 500. We conclude that
5 Conclusions
In this paper, we have presented a phowledge-augmentt approach K-LITE to learn a generic visual model for task-level transfer. General external knowledge sources including WordNet and Wiktionary are explored to enrich the natural language supervision, which is then used in both the language-imag pre-training stage and the prompt-based evaluation stage. We have demonstrated the generality and effectiveness of K-LITE in two core computer vision problems: image classification and object detection. Extensive experimental results show that our method can achieve superior performance over existing methods on 20 IC datasets and 13 OD datasets, respectively. K-LITE also outperforms its knowledge-free counterpart UniCL using half of the pre-training data in the large-scale academic data setting, demonstrating that leveraging external knowledge is promising direction in improving pre-training sample-efficieng for learning transferable visual models.
Acknowledgments
The authors gratefully acknowledge Chenguang Zhu, Wenhao Yu, Yuwei Fang for the early insightful discussions the use of dictionary for rare concepts in NLP task, Hao Cheng for the inspirations of external knowledge in open-domain QA. The project is partly done in the MER-BBerkeeey collaboration program
| The work depends publicly available knowledge databases; original authors and contributors who made their "knowledge" public. Sheng Shen and Kurt Keutzer supported by Samsung SAIT, Intel corporation, Intel VLAB team, Intel One-API center of excellence, as well as funding through BDD and BAIR. The work of Sheng Shen, Anna Rohrbach and Trevor Darrell was supported in part by DoD including DARPA's LwLL., PTG and/or SemaFor programs, as well as BAIR's 's industrial alliance programs.
acknowledge all the
10

---

References
[1] FER 2013: Kaggle challenges in representation learning facial expression recognition. attpps//www kaggle. com/
[2] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
[4] Sören Auer, Christian Bizer. Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for web of open data. In The semantic web, pages 722-735. Springer, 2007.
[5] Sven Bambach, Stefan Lee, David Crandall, and Chen Yu. Lending hand: Detecting hands and recognizing activities in complex egocentric interactions. ICCV, 2015.
[6] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In ACM SIGMOD, 2008.
[7] Schastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426, 2021.
[8] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Foodd-01-mmiiig discriminative components with random forests. ECCV, 2014.
[9] Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk, Webqu: Multihop and multimodal qa. arXiv preprint arXiv:2109.00590 2021.
[10] Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In ICCV, 2017.
[11] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
[12] Jiaoyan Chen, Yuxia Geng, Zhuo Chen, Ian Horrocks, Jeff Z Pan, and Huajun Chen. Knowledge-aware zero-shot learning: Survey and perspective. IJCAI, 2021.
[13] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017.
[14] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[15] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: hierarchical image database. CVPR, 2009.
large-scale
[16] Li Deng. The MNIST datahase of handwritten digit images for machine learning research. IEEE signal processing magazine, 2012.
[17) Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[18] Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write a classifier: Zero-shot learning using purely textual descriptions. In ICCV, 2013.
[19] Mohamed Elhosciny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the "beak"; Zero shot learning from noisy text description at part precision. In CVPR, 2017.
[20] Mark Everingham. Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 2010.
[21] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, 2009.
[22] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested 101 object categories. In CVPR workshop, 2004.
11

---

[23] Rob Fergus, Hector Bernal, Yair Weiss, and Antonio Torralba. Semantic label sharing for learning with many categories. In ECCV, 2010.
[24] Jannik Fritsch, Tobias Kuehnl, and Andreas Geiger. A new performance measure and evaluation benchmark for road detection algorithms. In ITSC. IEFE, 2013.
[25] Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: deep visual-semantic embedding model. NeurIPS, 2013.
[26]
Yanwei Fu and Leonid Sigal. Sumi-supereised vocabulary-informe learning. In CVPR, 2016.
[27] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthis Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trainee trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 1rXiv:1811.1221,. 2018.
[28] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. CVPR, 2019.
[29] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002008909, 2020.
[30] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and- aad-language navigation via pre-training. In CVPR, 2020.
[31] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[32] Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
[33] Neil Houlsby. Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter--fflcient transfer learning for nlp. In ICML, 2019.
[34] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[35] Dinesh Jayaraman and Kristen Grauman. Zero shot recognition with unreliable attributes. arXiv preprint arXiv:1409.4322 4307.327, 2014.
[36] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918. 2021.
[37] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. NeurlPS, 2020.
[38] Wonjac Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- laiio-and-language transformer without convolution or region supervision. arXiv preprint arXiv:2102.0333 2021.
[39] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver. Jessica Yung. Sylvain Gelly, and Neil Houlshy. Big transfer (bit): General visual representation learning. In ECCV, 2020.
[40] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[42] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. TACI, 2019.
[43] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Learning to detect unseon object classes by between-class attribute transfer. In CVPR, 2009.
[44] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Attribute bared classification for zero-shot visual object categorization. PAMI, 2013.
12

---

[45] Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinnich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledgeeinllnnivee NLP tasks. NeurIPS, 2020.
[46] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. In ICCV, 2017.
[47] Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang. Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. ELEVATER: benchmark and toolkit for evaluating language-angmented visual models. In NewrIPS Track on Datasets and Benchmarks, 2022.
[48] Junnan Li, Ramprasaath Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation lcarning with momentum distillation. arXiv preprint wwrXiv2217.0077
2021.
[49] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 190.::9908.03355 2019.
[50] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Neng Hwang, al. Grounded language-image pre-training, CVPR, 2022.
[51] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020.
[52] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-iman image pre- pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021.
[53] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[54] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[55] Angli Liu, Jingfei Du, and Veselin Stoyanov. Knowledge-augmentt language model and its application to unsupervised named-entity recognition. arXiv preprint D.XXv:9904.044999 2019.
[56] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-BERT: Enabling language representation with knowledge graph. In AAAI, 2020.
[57] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo, Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, 2021.
[58] Jiasen Lu, Dhruy Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-laagee language tasks. arXiv preprint arXiv: 1908. 02265, 2019.
[59] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1xXiv:106.515// 2013.
[60] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based VQA. In CVPR, 2021.
[61] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. CVPR, 2019.
[62] Christian M Meyer and Iryna Gurevych. Wiktionary: A new rival for expert-built lexicons? Exploring the possibilities of collaborative lexicography. na, 2012.
[63] George A Miller. WordNet: An electronic lexical databaxe. MIT press, 1998.
[64] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. arXiv preprint arXiv:21 12750 2021.
[65] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing. IEEE, 2008
[66] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
13

---

[67] Genevieve Patterson and James Hays. SUN attribute database: Discovering, annotating. and recognizing scene attributes. In CVPR, 2012.
[68] Matthew E Peters, Mark Neumann, Robert I. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. arXiv preprint arXiv: /009.09990..0.. 04164, 2019.
[69] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.0222 2020.
[70] Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den Hengel. Less is more: zero-shot learning from online textual documents with noise suppression. In CVPR, 2016.
[71] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0000 00020, 2021.
[72] Scott Reed, Zeynep Akata, Honglak Lec, and Bernt Schiele. Learning deep representations of fine-grained visual deseriptions. In CVPR, 2016.
[73] Marcus Rohrbach, Michael Stark, and Bernt Schiele. Evaluating knowledge transfer and zero-shot learning in large-scale setting. In CVPR, 2011.
[74] Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum. Learning to share visual appearance for multiclass object detection. In CVPR, 2011.
[75] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang. Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
[76] Piyush Sharma, Nan Ding, Sehastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In ACI., 2018.
[77] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and lnn-lane tasks? ICLR, 2022.
[78] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NeurlPS, 2013.
[79] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, 2017.
[80] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN, 2011.
[81] Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.088: 2019.
[82] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effective- ness of data in deep learning cra. In /CCV, 2017.
[83] Hao Tan and Mohit Bansal. Lxmert: Learning cross- coss-modality encoder representations from transformers. In EMNLP, 2019.
[84] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 2016.
[85] Changyao Tian, Wenhai Wang. Xizhou Zhu, Xiaogang Wang, Jifeng Dai, and Yu Qiao. VI-Itr: Learning class-wise arXiv:2111.13579, 2021.
representation for long-tailed visual recognition. arXiv preprint
[86] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[87] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equivariant cnns for digital pathology. In MICCAI, 2018.
[88] Denny Vrandečié. Wikidata: A new platform for collaborative data collection. In WWW. 2012.
14

---

[89] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-200 dataset, 2011.
[90] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, al. K- K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint rrXiv:2002.01808 2020.
[91] Xiaolong Wang. Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. CVPR, 2018.
[92] Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: leaming rank with joint word-image embeddings. Machine learning, 2010.
[93] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozheh Mottaghi. Multi-modal answer validation for knowledge-basee VQA. arXiv preprint arXiv:2103.22444 2021.
[94] Yonggin Xian, Christoph HI Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a compre- hensive evaluation of the good, the bad and the ugly. PAMI, 2018.
[95] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, La Yuan, Ce Liu, and Jianfeng Gao. Unified contrastive learning in image-text-Lude space. CVPR, 2022.
[96] Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang- An empirical study of GPT-3 for few-shot knowledge-based VQA, arXiv preprint arXiv:2109.05014, 2021.
[97] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language- -mnag pre-training, arXiv preprint arXiv:2111.07783 2021.
[98] Da Yin, Li Dong, Hao Cheng. Xiaodong Liu, Wei Chang, Furu Wei, and Jianfeng Gao. A survey of ineteeegeeittensivee nlp with pre-trained language models. arXiv preprint arXiv:2202.0777 2022.
[99] Fei Yu, Jiji Tang, Weichong Yin, Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint MoKiv:2006.1633:2 2020.
[100] Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, and Meng Jiang. Dict-hert: Enhancing language model pre-training with dictionary. arXiv preprint 2rXiv:2110.06490 2021.
[101] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11422. 2021.
[102] Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. Transomes: From linguistic graphs to commonsense knowledge. arXiv preprint arXiv:2005.00206 2020.
[103] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision- langugge models. In CVPR, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] in
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Table
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] in Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] in Appendix
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "pdkjOUgS6cx", "reviewer": "Reviewer_nX1x", "summary": "The paper tackles the poor transfer learning performance of current visual models. It presents a systematic approach to leveraging external knowledge databases to train more transferable and sample-efficient visual models. The paper presents sufficient evidence that external knowledge can benefit large-scale task-level transfer for two core vision problems, image classification, and object detection. The model can achieve similar zero-shot performance to previous methods when using only half of the pre-training image-text pairs in some scenarios.", "questions": "Please see the weaknesses.", "limitations": "Limitations and potential negative societal impacts are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear. Leveraging external knowledge is promising to increase the concept coverage of vision-language models.\n2. Novel solution. Two external knowledge sources including WordNet and Wiktionary are explored to enrich the natural language supervision, which is then used in both the language-image pre-training stage and the prompt-based evaluation stage.\n3. Sufficient experiments. The experiments on two downstream tasks with several datasets support the main claims of the paper.\n\nWeaknesses:\n1. For knowledge acquisition, when multiple meanings exist for a given query, the method simply considers the first definition. Does the authors analyze the proportion of polysemy and the correct rate of the first-word sense?\n2. In order to train more transferable and sample-efficient visual models, the paper instantiates K-LITE with two tasks to demonstrate the generalization ability. Have the authors tested on more tasks, like image-text retrieval?\n3. In Table 2, why is the performance of 2-branch lower than 1-branch from scratch?\n4. Why not consider other types of external knowledge sources like ConceptNet and knowledge graphs?  What are the advantages of the currently used knowledge sources over others?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "laGyL86QAfl", "reviewer": "Reviewer_Q5MT", "summary": "This paper proposes to use external knowledge for learning multi-modal representation from text and images. The authors extract a query given a text description attached to an image. They use the query to extract more informative descriptions from the freely available external knowledge. The externally obtained textual description is then paired with the image to enhance the jointly learned visual representation. ", "questions": "As I mentioned above, the authors could perform some interesting experiments to elaborate more on performance. For instance, which classes benefit more from exploring external knowledge? Can it be somehow quantified based on a rareness score? According to the main idea proposed in the introduction, the rare classes should benefit more from external knowledge. You already have a measure to determine the rareness of noun phrases used in the query extracting process. Can you use the same measure to see if there is a correlation between the rareness of a given class and the obtained boost after using external knowledge? ", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper's idea is simple, effective, and properly presented. The paper is technically sound, and the experiments match NeurIPS standards.\n\nThe only weakness I see in this paper is that the authors are content with reporting the final performance to validate the proposed ideas. They show more detailed analyses in the appendix. However, I believe conducting a set of ablation studies will strengthen the paper. See below for more details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "egqfJJbnlhN", "reviewer": "Reviewer_vBeu", "summary": "This paper proposes a knowledge base augmented image-language model. The main architecture is an image-text contrastive training like CLIP. The key idea is for the text input, a \"keyword\" (a noun) is extracted and used as a query to search in multiple knowledge bases to get the word hierarchy or definition. This additional textual information is also input to the image-language model as a language prompt. The model is evaluated on multiple downstream datasets on image classification and object detection tasks. Results show incorporating an external knowledge base is effective. \n", "questions": "1. In Table 3, the bold number 21.3 is confusing. There is a 21.4 in the same row under LVIS tab. What's the meaning of bold font here?\n2. In the query construction section (L144) the paper chooses a single noun as a query to search in the knowledge base. Are there more discussions on this design choice? For example, if it's helpful to query `adj+noun` for a more accurate definition, or sometimes `verb+noun` if the text input describes an action.\n", "limitations": "There is no major negative societal impact of the work. The paper also briefly discussed it in the appendix.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper acknowledges related works (L111) that incorporate knowledge base in the NLP or vision-language models. The paper focuses on the transfer ability (transfer to specialized datasets and rare objects) of such a knowledge base augmented model, which is novel in the field. \n\nThe quality of the paper is very good. The ablation studies are well conducted. The experimental results are strong. \n\nIn general, the paper is clearly written and well organized. \n\nThe experimental results show the proposed method works well on a wide range of specialized datasets for both image classification and object detection tasks. This method of incorporating a knowledge base can be applied to other research.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QGdeMBRQJe", "reviewer": "Reviewer_AT54", "summary": "The paper proposes K-LITE which leverages external knowledge as a more structured source of supervision signals for building transferable visual models with knowledge-augmented language-image pre-training. External knowledge can enrich the natural language supervision by explaining rare concepts with more common concepts thus increasing the concept coverage. K-LITE proposes systematic steps to construct queries to get knowledge from external sources such as WordNet or Wiktionary to facilitate the pre-training and evaluation for image classification and object detection tasks. To bridge the gap between pre-training and evaluation, the paper utilizes adapters to further modularize the original language encoder with external knowledge data. With extensive experiments conducted, the paper demonstrates K-LITE’s great generality for downstream datasets and data effectiveness on both image classification and object detection tasks.", "questions": "1. In Section 3.1, how to compute the rare level of a noun phrase? Note that in Line 153, the paper said that the most *rare* noun-phrase over the corpus is used as a query for the sentence.\n2. For the example sentence in Line 156, will all the three noun phrases “boxer”, “professional boxer”, and “the crowd” be used as queries? If the answer is yes, this is not aligned with “the most *rare* noun-phrase” in Line 153.\n3. Typo in Line 279: “therefor” should be “therefore”, “mudularized” should be “modularized”.", "limitations": "The paper has reported limitations and offered satisfactory explanations. No obvious negative societal impact was found.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow. The motivation of leveraging external knowledge to help language-image pre-training is natural and clear.\n2. With knowledge-augmented training and evaluation, K-LITE achieves superior task-level transfer performances on 20 image classification and 13 object downstream tasks. In addition, K-LITE can get comparable performance as UniCL with only half of the pre-training data, showing great data efficiency.\n3. The paper provides detailed and persuasive analyses and discussions to confirm the design choices of K-LITE. The analysis of train-evaluation concept overlap (increased from 13.26% to 51.24% with knowledge) well explained why external knowledge can facilitate language-image pre-training and task-level transfer to downstream IC and OD tasks.\n\nWeaknesses:\n1. While the motivation is natural, it is not surprising. Since many prior works have leveraged external knowledge for language-only and vision-language tasks before. Even for know-augmented visual recognition, there is at least a similar idea proposed [1], but the K-LITE paper didn’t cite [1]. Therefore, the technical novelty seems relatively limited.  *[1] Tian C, Wang W, Zhu X, et al. VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition[J]. arXiv preprint arXiv:2111.13579, 2021.*\n\n2. Table 1 reveals that training/test inconsistency in terms of involving knowledge can dramatically degrade the model performance. Thus, to ensure higher performance, during evaluation, detailed explanations for noun phrases always need to be concatenated as knowledge prompts. This seems to be not aligned with the motivation of the paper and the spirit of utilizing external knowledge. For example, if we already what is “sashimi”, we don’t need to find the definition every time when we see the word later. I think this paper didn’t solve this motivation-reality gap very well. Even though applying K-Adapter can alleviate the problem, it didn’t bring any improvement in terms of accuracy compared with knowledge-free pre-training according to Table 1.\n3. Since detailed explanations are appended to noun phrases, the length of the input sequence will be much longer than the previous knowledge-free category name or caption. This will definitively result in longer inference time and larger training costs. Although K-LITE enjoys better data efficiency, the author should also provide the training and inference budgets to show whether K-LITE possesses another aspect of efficiency.\n4. From Figure 4 in the main paper and Figure 5 in the appendix, we can find that the performance and improvement of K-LITE largely depend on the quality of the external knowledge base. The method itself cannot guarantee an improved performance at the cost of external organized/structured data under some circumstances (just as the Limitations subsubsection stated).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept