SlowGuess/NAACL_2021_OCR huggingface dataset:
Songyang Zhang $^{1*}$ , Linfeng Song $^{2}$ , Lifeng Jin $^{2}$ , Kun Xu $^{2}$ , Dong Yu $^{2}$  and Jiebo Luo $^{1}$
szhang83@ur.ROCHESTER.edu, jluo@cs.rochester.edu
$^{2}$ Tencent AI Lab, Bellevue, WA, USA
{lfsong, lifengjin, kxxkunxu, dyu}@tencent.com
We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model (Kim et al., 2019) as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction.
Constituency parsing is an important task in natural language processing, which aims to capture syntactic information in sentences in the form of constituency parsing trees. Many conventional approaches learn constituency parser from human-annotated datasets such as Penn Treebank (Marcus et al., 1993). However, annotating syntactic trees by human language experts is expensive and time-consuming, while the supervised approaches are limited to several major languages. In addition, the treebanks for training these supervised parsers are small in size and restricted to the newswire domain, thus their performances tend to be worse
Sentence: A squirrel jumps on stump.
Sentence: Man starts to play the guitar fast.
(b)
Figure 1: Examples of video aided unsupervised grammar induction. We aim to improve the constituency parser by leveraging aligned video-sentence pairs.
when applying to other domains (Fried et al., 2019). To address these issues, recent approaches (Shen et al., 2018b; Jin et al., 2018; Drozdov et al., 2019; Kim et al., 2019) design unsupervised constituency parsers and grammar inducers, since they can be trained on large-scale unlabeled data. In particular, there has been growing interests in exploiting visual information for unsupervised grammar induction because visual information can capture important knowledge required for language learning that is ignored by text (Gleitman, 1990; Pinker and MacWhinney, 1987; Tomasello, 2003). This task aims to learn a constituency parser from raw unlabeled text aided by its visual context.
Previous methods (Shi et al., 2019; Kojima et al., 2020; Zhao and Titov, 2020; Jin and Schuler, 2020) learn to parse sentences by exploiting object information from images. However, images are static and cannot present the dynamic interactions among visual objects, which usually correspond to verb phrases that carry important information. There-
* This work was done when Songyang Zhang was an intern at Tencent AI Lab.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1513-1524 June 6-11, 2021. ©2021 Association for Computational Linguistics
fore, images and their descriptions may not be fully-representative of all linguistic phenomena encountered in learning, especially when action verbs are involved. For example, as shown in Figure 1(a), when parsing a sentence "A squirrel jumps on stump", a single image cannot present the verb phrase "jumps on stump" accurately. Moreover, as shown in Figure 1(b), the guitar sound and the moving fingers clearly indicate the speed of music playing, while it is impossible to present only with a static image as well. Therefore, it is difficult for previous methods to learn these constituents, as static images they consider lack dynamic visual and audio information.
In this paper, we address this problem by leveraging video content to improve an unsupervised grammar induction model. In particular, we exploit the current state-of-the-art techniques in both video and audio understanding, domains of which include object, motion, scene, face, optical character, sound, and speech recognition. We extract features from their corresponding state-of-the-art models and analyze their usefulness with the VC-PCFG model (Zhao and Titov, 2020). Since different modalities may correlate with each other, independently modeling each of them may be sub-optimal. We also propose a novel model, Multi-Modal Compound Probabilistic Context-Free Grammars (MMC-PCFG), to better model the correlation among these modalities.
Experiments on three benchmarks show substantial improvements when using each modality of the video content. Moreover, our MMC-PCFG model that integrates information from different modalities further improves the overall performance. Our code is available at https://github.com/Sy-Zhang/MMC-PCFG.
- We are the first to address video aided unsupervised grammar induction and demonstrate that verb related features extracted from videos are beneficial to parsing.
- We perform a thorough analysis on different modalities of video content and propose a model to effectively integrate these important modalities to train better constituency parsers.
- Experiments results demonstrate the effectiveness of our model over the previous state-of-the-art methods.
Our model is motivated by C-PCFG (Kim et al., 2019) and its variant of the image-aided unsupervised grammar induction model, VC-PCFG (Zhao and Titov, 2020). We will first review the evolution of these two frameworks in Sections 2.1–2.2, and then discuss their limitations in Section 2.3.
2.1 Compound PCFGs
A probabilistic context-free grammar (PCFG) in Chomsky normal form can be defined as a 6-tuple  $(S, \mathcal{N}, \mathcal{P}, \Sigma, \mathcal{R}, \Pi)$ , where  $S$  is the start symbol,  $\mathcal{N}$ ,  $\mathcal{P}$  and  $\Sigma$  are the set of nonterminals, preterminals and terminals, respectively.  $\mathcal{R}$  is a set of production rules with their probabilities stored in  $\Pi$ , where the rules include binary nonterminal expansions and unary terminal expansions. Given a certain number of nonterminal and preterminal categories, a PCFG induction model tries to estimate rule probabilities. By imposing a sentence-specific prior on the distribution of possible PCFGs, the compound PCFG model (Kim et al., 2019) uses a mixture of PCFGs to model individual sentences in contrast to previous models (Jin et al., 2018) where a corpus-level prior is used. Specifically in the generative story, the rule probability  $\pi_r$  is estimated by the model  $g$  with a latent representation  $\mathbf{z}$  for each sentence  $\sigma$ , which is in turn drawn from a prior  $p(\mathbf{z})$ :
$$
\pi_ {r} = g _ {r} (\mathbf {z}; \theta), \quad \mathbf {z} \sim p (\mathbf {z}). \tag {1}
$$
The probabilities for the CFG initial expansion rules  $S \to A$ , nonterminal expansion rules  $A \to BC$  and preterminal expansion rules  $T \to w$  can be estimated by calculating scores of each combination of a parent category in the left hand side of a rule and all possible child categories in the right hand side of a rule:
$$
\begin{array}{l} \pi_ {S \to A} = \frac {\exp (\mathbf {u} _ {A} ^ {\top} f _ {s} ([ \mathbf {w} _ {S} ; \mathbf {z} ]))}{\sum_ {A ^ {\prime} \in \mathcal {N}} \exp (\mathbf {u} _ {A ^ {\prime}} f _ {s} ([ \mathbf {w} _ {S} ; \mathbf {z} ]))}, \\ \pi_ {A \rightarrow B C} = \frac {\exp (\mathbf {u} _ {B C} ^ {\top} [ \mathbf {w} _ {A} ; \mathbf {z} ])}{\sum_ {B ^ {\prime} , C ^ {\prime} \in \mathcal {N} \cup \mathcal {P}} \exp (\mathbf {u} _ {B ^ {\prime} C ^ {\prime}} ^ {\top} [ \mathbf {w} _ {A} ; \mathbf {z} ])}, \\ \pi_ {T \rightarrow w} = \frac {\exp \left(\mathbf {u} _ {w} ^ {\top} f _ {t} ([ \mathbf {w} _ {T} ; \mathbf {z} ])\right)}{\sum_ {w ^ {\prime} \in \Sigma} \exp \left(\mathbf {u} _ {w ^ {\prime}} ^ {T} f _ {t} ([ \mathbf {w} _ {T} ; \mathbf {z} ])\right)}, \tag {2} \\ \end{array}
$$
where  $A, B, C \in \mathcal{N}, T \in \mathcal{P}, w \in \Sigma$ ,  $\mathbf{w}$  and  $\mathbf{u}$  vectorial representations of words and categories, and  $f_{t}$  and  $f_{s}$  are encoding functions such as neural networks.
Optimization of the PCFG induction model usually involves maximizing the marginal likelihood of a training sentence  $p(\sigma)$  for all sentences in a corpus. In the case of compound PCFGs:
$$
\log p _ {\theta} (\sigma) = \log \int_ {\mathbf {z}} \sum_ {t \in \mathcal {T} _ {\mathcal {G}} (\sigma)} p _ {\theta} (t | \mathbf {z}) p (\mathbf {z}) d \mathbf {z}, \tag {3}
$$
where  $t$  is a possible binary branching parse tree of  $\sigma$  among all possible trees  $\mathcal{T}$  under a grammar  $\mathcal{G}$ . Since computing the integral over  $\mathbf{z}$  is intractable,  $\log p_{\theta}(\sigma)$  can be optimized by maximizing its evidence lower bound  $\mathrm{ELBO}(\sigma ;\phi ,\theta)$ :
$$
\begin{array}{l} \operatorname {E L B O} (\sigma ; \phi , \theta) = \mathbb {E} _ {q _ {\phi} (\mathbf {z} | \sigma)} [ \log p _ {\theta} (\sigma | \mathbf {z}) ] \tag {4} \\ - \operatorname {K L} [ q _ {\phi} (\mathbf {z} | \sigma) | | p (\mathbf {z}) ], \\ \end{array}
$$
where  $q_{\phi}(\mathbf{z}|\sigma)$  is a variational posterior, a neural network parameterized with  $\phi$ . The sample log likelihood can be computed with the inside algorithm, while the KL term can be computed analytically when both prior  $p(\mathbf{z})$  and the posterior approximation  $q_{\phi}(\mathbf{z}|\sigma)$  are Gaussian (Kingma and Welling, 2014).
2.2 Visually Grounded Compound PCFGs
The visually grounded compound PCFGs (VC-PCFG) extends the compound PCFG model (C-PCFG) by including a matching model between images and text. The goal of the vision model is to match the representation of an image  $\mathbf{v}$  to the representation of a span  $\mathbf{c}$  in a parse tree  $t$  of a sentence  $\sigma$ . The word representation  $\mathbf{h}_i$  for the  $i$ th word is calculated by a BiLSTM network. Given a particular span  $c = w_i, \ldots, w_j (0 < i < j \leq n)$ , we then compute its representation  $\mathbf{c}$ . We first compute the probabilities of its phrasal labels  $\{p(k|c, \sigma) | 1 \leq k \leq K, K = |\mathcal{N}|\}$ , as described in Section 2.1. The representation  $\mathbf{c}$  is the sum of all label-specific span representations weighted by the probabilities we predicted:
$$
\mathbf {c} = \sum_ {k = 1} ^ {K} p (k | c, \sigma) f _ {k} \left(\frac {1}{j - i + 1} \sum_ {l = i} ^ {j} \mathbf {h} _ {l}\right), \tag {5}
$$
Finally, the matching loss between a sentence  $\sigma$  and an image representation  $\mathbf{v}$  can be calculated as a sum over all matching losses between a span and the image representation, weighted by the marginal of a span from the parser:
$$
s _ {i m g} (\mathbf {v}, \sigma) = \sum_ {c \in \sigma} p (c | \sigma) h _ {i m g} (\mathbf {c}, \mathbf {v}), \tag {6}
$$
where  $h_{img}(\mathbf{c},\mathbf{v})$  is a hinge loss between the distances from the image representation  $\mathbf{v}$  to the matching and unmatched (i.e. sampled from a different sentence) spans  $\mathbf{c}$  and  $\mathbf{c}'$ , and the distances from the span  $\mathbf{c}$  to the matching and unmatched (i.e. sampled from a different image) image representations  $\mathbf{v}$  and  $\mathbf{v}'$ :
$$
\begin{array}{l} h _ {i m g} (\mathbf {c}, \mathbf {v}) = \mathbb {E} _ {\mathbf {c} ^ {\prime}} [ \cos (\mathbf {c} ^ {\prime}, \mathbf {v}) - \cos (\mathbf {c}, \mathbf {v})) + \epsilon ] _ {+} \\ + \mathbb {E} _ {\mathbf {v} ^ {\prime}} \left[ \cos (\mathbf {c}, \mathbf {v} ^ {\prime}) - \cos (\mathbf {c}, \mathbf {v}) + \epsilon \right] _ {+}, \tag {7} \\ \end{array}
$$
where  $\epsilon$  is a positive margin, and the expectations are approximated with one sample drawn from the training data. During training, ELBO and the image-text matching loss are jointly optimized.
2.3 Limitation
VC-PCFG improves C-PCFG by leveraging the visual information from paired images. In their experiments (Zhao and Titov, 2020), comparing to C-PCFG, the largest improvement comes from NPs (+11.9% recall), while recall values of other frequent phrase types (VP, PP, SBAR, ADJP and ADVP) are fairly similar. The performance gain on NPs is also observed with another multi-modal induction model, VG-NSL (Shi et al., 2019; Kojima et al., 2020). Intuitively, image representations from image encoders trained on classification tasks very likely contain accurate information about objects in images, which is most relevant to identifying  $\mathrm{NPs}^1$ . However, they provide limited information for phrase types that mainly involve action and change, such as verb phrases. Representations of dynamic scenes may help the induction model to identify verbs, and also contain information about the argument structure of the verbs and nouns based on features of actions and participants extracted from videos. Therefore, we propose a model that induces PCFGs from raw text aided by the multi-modal information extracted from videos, and expect to see accuracy gains on such places in comparison to the baseline systems.
3 Multi-Modal Compound PCFGs
In this section, we introduce the proposed multimodal compound PCFGs (MMC-PCFG). Instead
<sup>1</sup>Jin and Schuler (2020) reports no improvement on English when incorporating visual information into a similar neural network-based PCFG induction model, which may be because Zhao and Titov (2020) removes punctuation from the training data, which removes a reliable source of phrasal boundary information. This loss is compensated by the induction model with image representations. We leave the study of evaluation configuration on induction results for future work.
of purely relying on object information from images, we generalize VC-PCFG into the video domain, where multi-modal video information is considered. We first introduce the video representation in Section 3.1. We then describe the procedure for matching the multi-modal video representation with each span in Section 3.2. After that we introduce the training and inference details in Section 3.3.
3.1 Video Representation
A video contains a sequence of frames, denoted as  $V = \{v_{i}\}_{i=1}^{L^{0}}$ , where  $v_{i}$  represents a frame in a video and  $L^{0}$  indicates the total number of frames. We extract video representation from  $M$  models trained on different tasks, which are called experts. Each expert focuses on extracting a sequence of features of one type. In order to project different expert features into the same dimension, their feature sequences are feed into linear layers (one per expert) with same output dimension. We denote the outputs of the  $m$ th expert after projection as  $\mathbf{F}^{m} = \{\mathbf{f}_{i}^{m}\}_{i=1}^{L^{m}}$ , where  $\mathbf{f}_{i}^{m}$  and  $L^{m}$  represent the  $i$ th feature and the total number of features of the  $m$ th expert, respectively.
A simple method would average each feature along the temporal dimension and then concatenate them together. However, this would ignore the relations among different modalities and the temporal ordering within each modality. In this paper, we use a multi-modal transformer to collect video representations (Gabeur et al., 2020; Lei et al., 2020).
The multi-modal transformer expects a sequence as input, hence we concatenate all feature sequences together and take the form:
$$
\mathbf {X} = [ \mathbf {f} _ {a v g} ^ {1}, \mathbf {f} _ {1} ^ {1}, \dots , \mathbf {f} _ {L _ {1}} ^ {1}, \dots \mathbf {f} _ {a v g} ^ {M}, \mathbf {f} _ {1} ^ {M}, \dots , \mathbf {f} _ {L _ {M}} ^ {M} ], (8)
$$
where  $\mathbf{f}_{avg}^{m}$  is the averaged feature of  $\{\mathbf{f}_i^m\}_{i = 1}^{L_m}$ . Each transformer layer has a standard architecture and consists of multi-head self-attention module and a feed forward network (FFN). Since this architecture is permutation-invariant, we supplement it with expert type embeddings  $\mathbf{E}$  and positional encoding  $\mathbf{P}$  that are added to the input of each attention layer. The expert type embeddings indicate the expert type for input features and take the form:
$$
\mathbf {E} = \left[ \mathbf {e} ^ {1}, \mathbf {e} ^ {1}, \dots , \mathbf {e} ^ {1}, \dots , \mathbf {e} ^ {M}, \mathbf {e} ^ {M}, \dots , \mathbf {e} ^ {M} \right], \tag {9}
$$
where  $\mathbf{e}^m$  is a learned embedding for the  $m$ th expert. The positional encodings indicate the location
of each feature within the video and take the form:
$$
\mathbf {P} = \left[ \mathbf {p} _ {0}, \mathbf {p} _ {1}, \dots , \mathbf {p} _ {L _ {1}}, \dots , \mathbf {p} _ {0}, \mathbf {p} _ {1}, \dots , \mathbf {p} _ {L _ {M}} \right], \tag {10}
$$
where fixed encodings are used (Vaswani et al., 2017). After that, we collect the output of transformer that corresponds to the averaged features as the final video representation, i.e.,  $\Psi = \{\psi_{avg}^{i}\}_{i=1}^{M}$ . In this way, we can learn more effective video representation by modeling the correlations of features from different modalities and different timestamps.
3.2 Video-Text Matching
To compute the similarity between a video  $V$  and a particular span  $c$ , a span representation  $\mathbf{c}$  is obtained following Section 2.2 and projected to  $M$  separate expert embeddings via gated embedding modules (one per expert) (Miech et al., 2018):
$$
\boldsymbol {\xi} _ {1} ^ {i} = \boldsymbol {W} _ {1} ^ {i} \boldsymbol {c} + \boldsymbol {b} _ {1} ^ {i},
$$
$$
\boldsymbol {\xi} _ {2} ^ {i} = \boldsymbol {\xi} _ {1} ^ {i} \circ s i g m o i d \left(\boldsymbol {W} _ {2} ^ {i} \boldsymbol {\xi} _ {1} ^ {i} + \boldsymbol {b} _ {2} ^ {i}\right), \tag {11}
$$
$$
\pmb {\xi} ^ {i} = \frac {\pmb {\xi} _ {2} ^ {t}}{\| \pmb {\xi} _ {2} ^ {i} \| _ {2}},
$$
where  $i$  is the index of expert,  $\pmb{W}_1^i$ ,  $\pmb{W}_2^i$ ,  $\pmb{b}_1^i$ ,  $\pmb{b}_2^i$  are learnable parameters, sigmoid is an element-wise sigmoid activation and  $\circ$  is the element-wise multiplication. We denote the set of expert embeddings as  $\Xi = \{\pmb{\xi}^i\}_{i=1}^M$ . The video-span similarity is computed as following,
$$
\omega_ {i} (\mathbf {c}) = \frac {\exp \left(\mathbf {u} _ {i} ^ {\top} \mathbf {c}\right)}{\sum_ {j = 1} ^ {M} \exp \left(\mathbf {u} _ {j} ^ {\top} \mathbf {c}\right)},
$$
$$
o (\boldsymbol {\Xi}, \boldsymbol {\Psi}) = \sum_ {i = 1} ^ {M} \omega_ {i} (\mathbf {c}) \cos \left(\boldsymbol {\xi} ^ {i}, \boldsymbol {\psi} ^ {i}\right), \tag {12}
$$
where  $\{\mathbf{u}_i\}_{i = 1}^M$  are learned weights. Given  $\Xi^{\prime}$ , an unmatched span expert embeddings of  $\Psi$ , and  $\Psi^\prime$ , an unmatched video representation of  $\Xi$ , the hinge loss for video is given by:
$$
\begin{array}{l} h _ {v i d} (\Xi , \Psi) = \mathbb {E} _ {\mathbf {c} ^ {\prime}} [ o (\Xi^ {\prime}, \Psi) - o (\Xi , \Psi)) + \epsilon ] _ {+} \\ + \mathbb {E} _ {\Psi^ {\prime}} [ o (\Xi , \Psi^ {\prime}) - o (\Xi , \Psi) + \epsilon ] _ {+}, \tag {13} \\ \end{array}
$$
where  $\epsilon$  is a positive margin. Finally the video-text matching loss is defined as:
$$
s _ {v i d} (V, \sigma) = \sum_ {c \in \sigma} p (c | \sigma) h _ {v i d} (\Xi , \Psi). \tag {14}
$$
Noted that  $s_{\text{vid}}$  can be regarded as a generalized form of  $s_{\text{img}}$  in Equation 6, where features from different timestamps and modalities are considered.
During training, our model is optimized by the ELBO and the video-text matching loss:
$$
\mathcal {L} (\phi , \theta) = \sum_ {(V, \sigma) \in \Omega} - \operatorname {E L B O} (\sigma ; \phi , \theta) + \alpha s _ {v i d} (V, \sigma), \tag {15}
$$
where  $\alpha$  is a hyper-parameter balancing these two loss terms and  $\Omega$  is a video-sentence pair.
During inference, we predict the most likely tree  $t^*$  given a sentence  $\sigma$  without accessing videos. Since computing the integral over  $\mathbf{z}$  is intractable,  $t^*$  is estimated with the following approximation,
$$
\begin{array}{l} t ^ {*} = \arg \max  _ {t} \int_ {\mathbf {z}} p _ {\theta} (t | \mathbf {z}) p _ {\theta} (\mathbf {z} | \sigma) d \mathbf {z} \tag {16} \\ \approx \underset {t} {\arg \max} p _ {\theta} (t | \sigma , \pmb {\mu} _ {\phi} (\sigma)), \\ \end{array}
$$
where  $\pmb{\mu}_{\phi}(\sigma)$  is the mean vector of the variational posterior  $q_{\phi}(\mathbf{z}|\sigma)$  and  $t^*$  can be obtained using the CYK algorithm (Cocke, 1969; Younger, 1967; Kasami, 1966).
DiDeMo (Hendricks et al., 2017) collects 10K unedited, personal videos from Flickr with roughly  $3 - 5$  pairs of descriptions and distinct moments per video. There are 32994, 4180 and 4021 videosentence pairs, validation and testing split.
YouCook2 (Zhou et al., 2018) includes 2K long untrimmed videos from 89 cooking recipes. On average, each video has 6 procedure steps described by imperative sentences. There are 8713, 969 and 3310 video-sentence pairs in the training, validation and testing sets.
MSRVTT (Xu et al., 2016) contains 10K videos sourced from YouTube which are accompanied by 200K descriptive captions. There are 130 260, 9 940 and 59 794 video-sentence pairs in the training, validation and testing sets.
Following the evaluation practice in Zhao and Titov (2020), we discard punctuation and ignore trivial single-word and sentence-level spans at test time. The gold parse trees are obtained by applying a state-of-the-art constituency parser, Benepar (Kitaev and Klein, 2018), on the testing set. All models are run 4 times for 10 epochs with different random seeds. We evaluate both averaged corpus-level
F1 (C-F1) and averaged sentence-level F1 (S-F1) numbers as well as their standard deviations.
4.3 Expert Features
In order to capture the rich content from videos, we extract features from the state-of-the-art models of different tasks, including object, action, scene, sound, face, speech, and optical character recognition (OCR). For object and action recognition, we explore multiple models with different architectures and pre-trained dataset. Details are as follows:
Object features are extracted by two models: ResNeXt-101 (Xie et al., 2017), pre-trained on Instagram hashtags (Mahajan et al., 2018) and finetuned on ImageNet (Krizhevsky et al., 2012), and SENet-154 (Hu et al., 2018), trained on ImageNet. These datasets include images of common objects, such as, "cock", "kite", and "goose", etc. We use the predicted logits as object features for both models, where the dimension is 1000.
Action features are extracted by three models: I3D trained on Kinetics-400 (Carreira and Zisserman, 2017), R2P1D (Tran et al., 2018) trained on IG65M (Ghadiyaram et al., 2019) and S3DG (Miech et al., 2020) trained on HowTo100M (Miech et al., 2019). These datasets include videos of human actions, such as "playing guitar", "ski jumping", and "jogging", etc. Following the same processing steps in their original work, we extract the predicted logits as action features, where the dimension is 400 (I3D), 359 (R2P1D) and 512 (S3DG), respectively.
Scene features are extracted by DenseNet-161 (Huang et al., 2017) trained on Places365 (Zhou et al., 2017). Places365 contains images of different scenes, such as "library", "valley", and "rainforest", etc. The predicted logits are used as scene features, where the feature dimension is 365.
Audio features are extracted by VGGish trained on YouTube-8M (Hershey et al., 2017), where the feature dimension is 128. YouTube-8M is a video dataset where different types of sound are involved, such as "piano", "drum", and "violin".
OCR features are extracted by two steps: characters are first recognized by combining text detector Pixel Link (Deng et al., 2018) and text recognizer SSFL (Liu et al., 2018). The characters are then converted to word embeddings through word2vec (Mikolov et al., 2013) as the final OCR features, where the feature dimension is 300.
Face features are extracted by combining face de
tector SSD (Liu et al., 2016) and face recognizer ResNet50 (He et al., 2016). The feature dimension is 512.