SlowGuess/ACL_2020_OCR huggingface dataset:
Tian Jin*
IBM Thomas J. Watson Research Center
Yorktown Heights, New York 10598
tian.jin2@ibm.com
Zhun Liu\*
Bellevue, WA 98004
zhunliu@microsoft.com
Shengjia Yan
Tandon School of Engineering
Brooklyn, NY 11201
sjyan@nyu.edu
Alexandre Eichenberger
alex@us.ibm.com
Louis-Philippe Morency
morency@cs.cmu.edu
Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose  $\mathbf{N}^3$  (Neural Networks from Natural Language) - a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model.  $\mathbf{N}^3$  leverages language descriptions to generate parameter adaptations as well as a new task-specific classification layer for a pre-trained neural network, effectively "fine-tuning" the network for a new task using only language descriptions as input. To the best of our knowledge,  $\mathbf{N}^3$  is the first method to synthesize entire neural networks from natural language. Experimental results show that  $\mathbf{N}^3$  can out-perform previous natural-language based zero-shot learning methods across 4 different zero-shot image classification benchmarks. We also demonstrate a simple method to help identify keywords in language descriptions leveraged by  $\mathbf{N}^3$  when synthesizing model parameters.
A person with generic world knowledge can learn to perform a new task based on verbal instructions. On the other hand, despite recent successes in deep
Figure 1: An illustration of  $\mathbf{N}^3$ . A list of object class descriptions is fed to the  $\mathbf{N}^3$  model to produce a CNN classifier that can classify images belonging to the object classes described.
learning, it remains challenging to re-purpose pretrained visual classification models to recognize a new set of objects without labeling a new training dataset. A natural question emerges from this observation: can a computer also learn to recognize new objects, simply by reading the descriptions of them in natural language? Concretely, can we create visual classifiers using language descriptions of the objects of interest?
In this paper, we introduce a new paradigm for synthesizing task-specific neural networks for image classification simply from language descriptions of the relevant objects. We propose  $\mathbf{N}^3$  - Neural Networks from Natural Language, a metamodel that takes a list of object descriptions as input to produce a classification model for these objects, as illustrated in Figure 1. The capability of producing task-specific neural network from lan
This work was performed when Zhun Liu was affiliated with Carnegie Mellon University.
$^{1}$ Code is released at https://github.com/tjingrant/n3cr.
6994
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6994-7007
guage descriptions makes  $\mathbf{N}^3$  ideal to a wide range of zero-shot tasks (Wah et al., 2011; Zhu et al., 2017; Elhoseiny et al., 2017; Zhu et al., 2017; Nilsback and Zisserman, 2006) where we do not have visual training data but can still easily obtain language descriptions of the objects of interest.
Prior zero-shot learning methods aim to achieve a similar goal of applying pre-trained networks on unseen classes. In zero-shot image classification, a typical method of generalizing to unseen classes is to construct class embeddings to augment the classification layer of the pre-trained network or take retrieval-based approaches while utilizing generic visual features from pre-trained networks (Akata et al., 2015; Kodirov et al., 2017; Elhoseiny et al., 2013; Lei Ba et al., 2015; Reed et al., 2016).
Extending on the idea of generating classification layers for the pre-trained network,  $\mathbf{N}^3$  modifies the parameters of all layers in the pre-trained network. While the underlying pre-trained network in previous approaches only extracts generic visual features,  $\mathbf{N}^3$  makes it possible to extract task-specific ones. This approach effectively increases the capacity at which semantic information in the descriptions can affect the pre-trained network.
In our experiments, we evaluated our proposed  $\mathbf{N}^3$  method with 4 popular zero-shot image classification benchmark datasets. We performed ablation studies to understand the importance of synthesizing task-specific feature extractors, the necessity of a pre-trained visual classification model and the effects of language representation choices on the efficacy of  $\mathbf{N}^3$ . In addition, we provide a simple approach to help interpret what aspects in the language descriptions are  $\mathbf{N}^3$ -generated models examining when making predictions.
To summarize, our contributions are 3-fold:
1. We propose a novel meta-model  $\mathbf{N}^3$  to synthesize task-specific neural network models using natural language descriptions.
2. We demonstrate  $\mathrm{N}^{3}$  's superior efficacy in solving natural language guided zero-shot learning problems. Our analysis shows that  $\mathrm{N}^{3}$  's ability to tailor neural network models to extract task-specific features plays an important role in achieving such superior accuracy.
3. We show that  $\mathbf{N}^3$  can aid the interpretation of predictions of synthesized models as they can be traced back to both supportive and refutative evidence within language descriptions.
In this section, we are situating our work in the context of zero-shot learning and dynamic parameter generation for neural networks.
Zero-shot Learning Zero-shot learning studies how we can generalize our models to perform well for tasks without any labeled training data at all. Achieving classification accuracy above chance-level in such scenarios requires modeling the relationships between the seen classes during training and the unseen classes during testing. A typical and effective method is to manually engineer class attribute vectors to obtain representations of seen and unseen classes in a shared attribute space (Duan et al., 2012; Kankuekul et al., 2012; Parikh and Grauman, 2011; Zhang and Saligram, 2016; Akata et al., 2016). Yet such a method requires laborious engineering of class attributes, which is not feasible for large-scale and/or fine-grained classification tasks (Russakovsky et al., 2015; Khosla et al., 2011; Welinder et al., 2010). Hence, there is also work in zero-shot learning that attempts to leverage textual data as object class representations (Lei Ba et al., 2015; Elhoseiny et al., 2013). The majority of these models are committed to embedding-based retrieval approaches, where classification is re-formulated as retrieving the class embedding with maximal similarity (Akata et al., 2015; Kodirov et al., 2017). While they can handle well the case where there is an indefinite number of classes during test time, such approaches suffer from extra computation cost at inference time since they need to traverse all the seen data points. Moreover, these models often rely on a pre-trained feature extractor for input, which is usually fixed and cannot be further adapted for the unseen classes (Akata et al., 2015; Kodirov et al., 2017; Elhoseiny et al., 2013). While there is a handful of work that tries to modify the parameters in-place during test time, they either rely on shallow language representation with limited expressiveness (Lei Ba et al., 2015) or require a significant amount of textual descriptions per class to train their model (Reed et al., 2016), both of which are not ideal. In our work, we aim to learn a model that can synthesize parameters for entire neural networks to adapt to the new tasks using short descriptions of the object classes. This leads to better metadata efficiency of our proposed method.
Dynamic Parameter Generation As mentioned before,  $\mathbf{N}^3$  dynamically generates classification models for designated classes. Dynamic parameter generation has been explored in the context of generating recurrent cells at different time-steps of RNNs (Ha et al., 2016), constructing intermediate linear models for interpretability inside neural networks (Al-Shedivat et al., 2017), and contextual parameter generation for different language pairs in multilingual machine translation (Platanios et al., 2018). As mentioned in Section 2, some zero-shot learning methods can also be viewed as generating classifier parameters (Lei Ba et al., 2015; Elhoseiny et al., 2013). However, many of the previous work directly or indirectly mentions the challenge of memory and computation complexity - after all, the output of the parameter generation model are large matrices that are excessively high dimensional. To tackle this issue, previous work either only generate very simple linear layers (Lei Ba et al., 2015; Elhoseiny et al., 2013; Al-Shedivat et al., 2017), or impose low-rank constraints on the weights to mitigate the memory issues (Ha et al., 2016). In our work, we utilize the architecture of sequence-to-sequence models and treat the weight matrices to be generated as a sequence of vectors. This allows parameter generation for entire neural networks with little memory bottleneck.
3  $\mathbf{N}^3$  Methodology
In this section, we describe our approach for synthesizing task-specific neural networks to recognize new objects with their natural language descriptions and a generic pre-trained model. We denote a list of natural language descriptions for  $K$  objects, each containing  $L$  tokens as  $D = \{d_{k,l}\}_{k=1,l=1}^{k = K, l = L}$ . We denote a pre-trained model with parameters  $\Theta$  as  $\mathcal{F}(\cdot; \Theta)$ , so that for a set of images  $X$ ,  $\mathcal{F}(X; \Theta)$  produces the classification prediction  $Y$ .
We can now precisely formulate our problem as follows: given a pre-trained classification model,  $\mathcal{F}(\cdot ;\Theta)$ , and the natural language description of  $K$  object classes,  $D$ , adapt the original parameters  $\Theta$  to the specialized parameters  $\Theta^{\prime}$  so that the finetuned  $\mathcal{F}(\cdot ;\Theta^{\prime})$  model accurately classifies the  $K$  objects described in  $D$ .
3.1 Synthesizing Task-specific Models via Parameter Adaptation
Our method draws inspiration from transfer learning, which is often employed when the training
dataset is small. Transfer learning entails training a neural network from a generic pre-trained model to one that solves a new, often task-specific problem. Thus, we can similarly formulate  $\mathbf{N}^3$  to synthesize task-specific model parameters by adapting existing ones in the generic pre-trained model with the guidance of natural language task descriptions. While transfer learning updates the pre-trained parameters using signals derived from task-specific training data,  $\mathbf{N}^3$  relies only on language descriptions to achieve the same objective. Concretely, the adapted parameters  $\Theta'$  are computed as follows:
$$
\Theta^ {\prime} = \Theta + \mu \cdot \Phi (D; \Gamma) \tag {1}
$$
where  $\Phi (\cdot ;\Gamma)$  is a function with parameter  $\Gamma$  , mapping natural language descriptions to parameter adaptations for all parameters in the pre-trained model. Since the transfer learning process often proceeds with a tiny learning rate to restrict the effect of fine-tuning, we introduced a trainable scaling factor  $\mu$  to similarly regulate the effect of parameter adaptation. The initial value of  $\mu$  is a hyperparameter. In our experiments, we used an initial  $\mu$  value of  $1e^{-3}$  to mimic the effect of using a small learning rate for transfer learning. The specific value of  $1e^{-3}$  is derived from the default learning rate used in the PyTorch transfer learning tutorial (Chilamkurthy). In a later section, we evaluate the necessity of this scaling factor as well as the effect of a range of initial values of  $\mu$
3.2 Making Adaptation Computationally Feasible via Hierarchical Attention and Layer Sharing
The mapping  $\Phi$  from natural language descriptions to parameter adaptations is particularly high-dimensional. Constructing  $\Phi$  with the transformer block (Vaswani et al., 2017) is not straight-forward for our scenario because it requires  $O(N^2)$  size of memory where  $N$  is the length of the input/output sequence and our  $N = K\times L$  can be prohibitively large. Thus, to reduce the memory consumption of  $\Phi$ , its attention span must be restricted to a small but semantically relevant subset of all input elements. To this end, we designed  $\Phi$  to be a two-level hierarchy of transformer blocks, as illustrated in 2. The first level of transformers, named the Tokens2Label Encoder, encodes the natural language descriptions of each object class to a label embedding vector. Intuitively, this level summarizes the described visual features of an ob
Figure 2: Hierarchical encoder for descriptions and decoder for parameter adaptations.
ject class to a single, fixed-sized embedding vector. Thus, attention in this level has a maximum span of  $L$  spanning all tokens in each description. The second level, named the Labels2Adaptation Encoder-Decoder, encodes the sequence of label embedding vectors and decodes them into parameter adaptations of multiple layers. This level of transformer blocks examines the characteristics of all encoded object classes and determines how to adapt the pre-trained model parameters to classify images corresponding to these object labels. In this level, the model attention has a maximum span of  $K$  spanning all the object classes. Moreover, due to the sheer number of layers in state-of-the-art CNN models, we initialize layer-specific adaptation decoders for each layer and decode from shared hidden states encoded by the adaptation encoder. Only through these measures can we materialize the high-dimensional mapping  $\Phi$  under reasonable memory constraints.
Putting everything together, for a pre-trained network with parameter  $\Theta$ ,  $\mathbf{N}^3$  applies the mapping  $\Phi$  to input descriptions to generate a parameter adaptation  $\Delta \Theta$  with the same shape as  $\Theta$ . The adaptation  $\Delta \Theta$  is multiplied with a trainable scaling factor  $\mu$  to control its impact on the pre-trained model. The scaled parameter adaptation  $\mu \cdot \Delta \Theta$  is then combined with its pre-trained counterpart  $\Theta$  via point-wise addition.
The mapping  $\Phi$  from object descriptions  $D$  to  $\Delta \Theta$  contains two parts. The Tokens2Label Encoder transforms the set of tokens contained in  $K$  entries
of object class descriptions, each with length up to  $L$ , denoted as  $\{d_{k,l}\}$ , to a set of  $K$  object label embedding vectors  $\{c_k\}_{k = 1}^K$ :
$$
c _ {k} = \operatorname {E n c o d e r} _ {\mathrm {T 2 L}} \left(d _ {k, 1: L}\right), k = 1, \dots , K \tag {2}
$$
Subsequently, Labels2Adaptation Encoder-Decoders translate the object label embedding vectors into a parameter adaptation matrix  $\Delta \Theta$ . Parameters in a typical neural network often have more than two-dimensions, but for simplicity in our setup, they are always viewed as a two-dimensional matrix consisting of a sequence of parameter columns. 1 Viewing the object-label embeddings  $\{c_i\}_{i=1}^k$  as an input sequence and the parameter adaptation  $\Delta \Theta$  as an output sequence of  $M$  columns  $\{\Delta \theta_m\}_{m=1}^M$ , Labels2Adaptation Encoder-Decoders can be expressed as:
$$
\mathbf {h} _ {1: K} = \operatorname {E n c o d e r} _ {\mathrm {L 2 A}} \left(c _ {1: K}\right) \tag {3}
$$
$$
\Delta \theta_ {m} = \operatorname {D e c o d e r} _ {\mathrm {L} 2 \mathrm {A}} ^ {\mathrm {m}} \left(\mathbf {h} _ {1: K}\right), m = 1, \dots , M \tag {4}
$$
$$
\Delta \Theta = \operatorname {C o n c a t} \left(\left[ \Delta \theta_ {1}; \Delta \theta_ {2}; \dots ; \Delta \theta_ {M} \right]\right) \tag {5}
$$
Finally, pre-trained parameter  $\Theta$  is adapted to  $\Theta^{\prime}$  using the following equation, with  $\mu$  being a trainable scaling factor:
$$
\Theta^ {\prime} = \Theta + \mu \cdot \Delta \Theta \tag {6}
$$
3.3 Training Methodology
We formulate the training of  $\mathbf{N}^3$  as the following optimization problem. Optimal parameters  $\Gamma$  for  $\mathbf{N}^3$  model  $\Phi (\cdot ;\Gamma)$  should map a list of language descriptions for  $K$  class objects  $\mathcal{D} = \{\mathcal{D}_1,\dots ,\mathcal{D}_K\}$  to parameter adaptations  $\Delta \Theta = \Phi (\mathcal{D};\Gamma)$ , such that the cross entropy loss between ground truth label  $Y$  and model prediction  $\mathcal{F}(X;\Theta ,\Phi (\mathcal{D};\Gamma))$  is minimized for all image-label pairs  $(X,Y)$  in the training set.
Thus, to train  $\mathbf{N}^3$  on a image dataset  $\mathcal{I}$  with labels  $\mathcal{L}$ , class descriptions  $\mathcal{D}$  and a pre-trained model  $\mathcal{F}$  to produce a  $K$ -way classification model, we first draw meta-batches of  $K$  class labels  $\mathcal{L}_K \in \mathcal{L}$ . Then, a subset of the image dataset  $\mathcal{I}_{\mathcal{L}_K}$  and description dataset  $\mathcal{D}_{\mathcal{L}_K}$  corresponding to the drawn labels  $\mathcal{L}_K$  are constructed. We then draw minibatches of images  $\mathcal{B}$  and ground-truth labels  $Y$
For instance, the parameter of a convolutional layer  $W$  of shape  $[K,C,kH,kW]$  where  $K$  is the number of output channels,  $C$  the number of input channels,  $kH$ ,  $kW$  the kernel height and width, is viewed as a sequence of  $K$  parameter columns, with a column size of  $C\times kH\times kW$ .
6997
from  $\mathcal{I}_{\mathcal{L}_K}$ . For each mini-batch  $\mathcal{B}$ , distinct parameter adaptations  $\Delta \Theta$  are generated by evaluating  $\Phi (\cdot ;\Gamma)$  at  $\mathcal{D}_{\mathcal{L}_k}$ . Batch loss is then calculated as  $\frac{1}{|\mathcal{B}|}\sum_{i}\ell (Y_{i},\mathcal{F}(X_{i};\Theta ,\Delta \Theta))$  where  $\ell (\cdot ,\cdot)$  refers to cross-entropy loss. Since the meta-model  $\Phi (\cdot ;\Gamma)$  and the pre-trained model  $\mathcal{F}$  are fully differentiable, gradients can be propagated back to meta-models to optimize meta-model parameters  $\Gamma$ .
We evaluate  $\mathbf{N}^3$  by comparing its efficacy in solving natural language guided zero-shot learning problems with prior state-of-the-art methods.
In this section, we introduce our training method, datasets and evaluation protocols.
To evaluate the  $\mathbf{N}^3$ , we select 4 standard zero-shot image classification datasets and collected natural language descriptions for their object classes.
Caltech-UCSD-Birds 200-2011 (CUB) (Wah et al., 2011) contains images of 200 species of birds. Each species of bird forms its own class label. In total, there are 11,788 images in this dataset.
Animal with Attributes (AWA) (Lampert et al., 2014; Xian et al., 2017) is another dataset to evaluate zero-shot classification methods. It consists of 50 classes of animals with a total of 37322 images.
North America's Birds (NAB) is a dataset used by prior state-of-the-art methods related to our task. Following the established practices (Zhu et al., 2017; Elhoseiny et al., 2017), we consolidated the class labels into 404 distinct bird species. The consolidation process combines closely related labels (e.g., 'American Kestrel (Female, immature)' and 'American Kestrel (Adult male)') into a single label (e.g., 'American Kestrel'). We end up with 48,000 images of 404 classes of bird-species.
Flowers-Species (FS) is a dataset we built based on Oxford Flowers (Nilsback and Zisserman, 2006), another commonly used zero-shot dataset. The original contains label categories that are a mixture of species and genera. Some genus includes thousands of species, yet the dataset examples only cover a fraction of them. Such mismatch creates biases in the dataset that fundamentally cannot be addressed through learning from external descriptions. This hence undermines its utility as a test of our proposed method: for instance, when  $\mathbf{N}^3$  is
asked to generate classifier to decide whether an object is of label "anthurium", which is a genus of around 1000 species of varying visual appearance, the efficacy of our generated model can only be evaluated based on a representative samples that cover most of the species within the genus "anthurium". However, the dataset only contains a tiny number of (i.e., 105) correlated (species-wise) samples, making such evaluation neither comprehensive nor conclusive in the context of our task objective and may introduce unexpected noise in evaluation results. Therefore, we decided to filter out the genera from the original Oxford Flowers dataset, leaving only the species as class labels, as an effort towards homogenizing the sample spaces implied by the class labels and the image dataset. This leaves us with 55 classes and 3545 images.
For each dataset, we collect language descriptions for object classes from websites like Wikipedia. To collect language descriptions from Wikipedia, we use the python package Wikipedia (Goldsmith) to access structured representation of Wikipedia pages, and extract section content under "Description" to be used as object class descriptions. If no Wikipedia entry exists for a specific object class, we resort to manually searching for the object class description on Google. Furthermore, we truncate these textual excerpts to a maximum length of 512 tokens to avoid excessively long descriptions and the accompanying computational issues.
Recent study showed that previous zero-shot learning evaluation protocols are inadequate and proposed a set of rigorous evaluation protocols for attribute-based zero-shot learning methods (Xian et al., 2017). Although both our tasks and datasets differ, we nevertheless followed Xian et al.'s guiding principles of their Rigorous Protocol and developed our evaluation protocol:
- Similar to Rigorous Protocol, we used two meta-splits Standard Splits (SS) and Proposed Splits (PS) to evaluate all methods; the Standard Splits are established meta-splits and Proposed Splits are meta-splits that guarantees the exclusion of ImageNet-1K classes from the test set.
- Due to class imbalance, Rigorous Protocol proposes to use per-class averaged accuracy for more meaningful evaluation. Thus, to evaluate meta-model on a meta-split containing
6998
Datasets	Total	Standard Split/Proposed Split
Training	Validation	Testing
CUB (Wah et al., 2011)	200	100	50	50
AWA2 (Zhu et al., 2017; Elhoseiny et al., 2017)	50	30	10	10
NAB (Zhu et al., 2017)	404	324	40	40
FS (Nilsback and Zisserman, 2006)	55	35	10	10
Table 1: Number of Class Labels in Our Meta-Split (both Standard Split and Proposed Split). Within each meta-split, training, validation and testing class labels are disjoint.
the set of classes  $C$ , we calculate the per-class averaged accuracy as shown in Equation 7.
$$
A c c _ {C} = \frac {1}{| C |} \sum_ {c \in C} \frac {\text {s a m p l e s i n c}}{\text {s a m p l e s i n c}} \tag {7}
$$
- Unlike Rigorous Protocol which uses ResNet-101 model, we use ResNet-18 as our pretrained model; such choice helps reduce the output dimensionality of  $\mathbf{N}^3$  by reducing the number of parameters  $\mathbf{N}^3$  adapts.
- Our method is unique in that permutations of the classes belonging to the same metasplit count as distinct tasks and therefore, to account for variations, we test our models on 10 different permutations of test set classes and report the medium value of the relevant evaluation metric.
We have tabulated the number of class labels used for training, validation and testing in Table. 1.