SlowGuess/NAACL_2021_OCR huggingface dataset:
Worldly Wise (WoW) - Cross-Linguual Knowledge Fusion for Fact-based Visual Spoken-Question Answering
Kiran Ramnath<sup>1</sup>, Leda Sari<sup>1</sup>, Mark Hasegawa-Johnson<sup>1</sup>, and Chang Yoo<sup>2</sup>
<sup>1</sup>University of Illinois, Urbana-Champaign
$^{2}$ KAIST
$^{1}\{$  kiranr2, lsari2, jhasegaw\} @illinois.edu
2{cd_yoo}@kaist.ac.kr
Although Question-Answering has long been of research interest, its accessibility to users through a speech interface and its support to multiple languages have not been addressed in prior studies. Towards these ends, we present a new task and a synthetically-generated dataset to do Fact-based Visual Spoken-Question Answering (FVSQA). FVSQA is based on the FVQA dataset, which requires a system to retrieve an entity from Knowledge Graphs (KGs) to answer a question about an image. In FVSQA, the question is spoken rather than typed. Three sub-tasks are proposed: (1) speech-to-text based, (2) end-to-end, without speech-to-text as an intermediate component, and (3) cross-lingual, in which the question is spoken in a language different from that in which the KG is recorded. The end-to-end and cross-lingual tasks are the first to require world knowledge from a multi-relational KG as a differentiable layer in an end-to-end spoken language understanding task, hence the proposed reference implementation is called Worldwide (WoW). WoW is shown to perform end-to-end cross-lingual FVSQA at same levels of accuracy across 3 languages - English, Hindi, and Turkish.
Imagine being able to ask your voice assistant a question in any language, to learn some trivia about your favorite movie star. This task falls in the realm of Knowledge-based Question Answering (QA). One such challenging QA task is that of Fact-based Visual Question Answering (FVQA) (Wang et al., 2018) which seeks to imitate how humans leverage background common-sense knowledge when answering visual questions. This task ensures that answering each question about an image requires external knowledge not directly available within the image or the text of the question. (see Fig. 1). The external information is provided in the form of knowledge graphs, which are multi-relational
Figure 1: Example of a fact-based visual question Question - Which object in this image can be found in a Jazz Club?
Supporting fact - You are likely to find [[a trumpet]] in [[a jazz club]]
Subject, Predicate, Object - (Trumpet, AtLocation, Jazz Club)
Answer - Trumpet
graphs, storing relational representations between entities. The entities could be single words or phrases of words that denote objects or concepts. Such tasks, though widely studied, exist mostly for well-resourced languages (Goyal et al., 2017; Wang et al., 2018). These languages generally also have mature Automatic Speech Recognition (ASR) systems and language models. The accompanying Knowledge Graphs (KGs) also tend to be limited to languages that are well-resourced (Auer et al., 2007; Tandon et al., 2014; Liu and Singh, 2004). Against this background, it is worthwhile to think of building end-to-end systems which directly use speech signals as input, that can readily harness huge knowledge repositories stored in another language, instead of requiring Tabula Rasa learning.
With these motivations, the main contributions of this paper are two-fold: 1) A new task referred to as Fact-based Visual Spoken-Question Answer
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1908-1919 June 6-11, 2021. ©2021 Association for Computational Linguistics
ing (FVSQA) along with the release of 5 hours of synthetic-speech data in each of the three languages - English, Hindi, and Turkish. 2) An end-to-end architecture Worldwide-Wise (WoW) capable of answering questions trained directly on speech features in all three languages. To the best of our knowledge, this is the first work to perform KG knowledge acquisition using only a speech signal as input, without the requirement for a pre-trained automatic speech recognizer as a system component.
Worldly-Wise (WoW) is readily generalizable to other languages, even those without an ASR-system. This is possible because of two reasons - a) it obtains speech features as Mel-Frequency Cepstral Coefficients and does not require ASR-based text-conversion or speech feature extraction from a language-specific pretrained network, and b) for knowledge acquisition, it does not require the entity label to be in the same language as the question, instead leveraging neuro-symbolic entity representations in the form of KG embeddings. These KG embedding methods, trained to remedy KG sparsity by performing missing-edge prediction, learn transferable entity-features that encode the local and global structures in KGs. This also permits the architecture to use an image representation technique called 'Image-as-Knowledge' (IaK). This uses a co-attention mechanism that attends to important entities in the image and time-steps in a question, thus allowing for improved answer retrieval. The IaK technique was first presented by (Ramnath and Hasegawa-Johnson, 2020) for the goal of performing FVQA over incomplete KGs, but is applied to a speech signal as opposed to a textual question. We revisit its important details below in the relevant sections.
We report experimental results on synthetic speech data in the aforementioned diverse languages to demonstrate its effectiveness. Hindi and Turkish are simulated as under-resourced languages by denying the system access to any text, ASR, or machine translation to or from those languages, thereby requiring the system to learn the mapping from Hindi and Turkish speech signals to the KG knowledge stored in English. Through this work, we hope to motivate research in expanding spoken language understanding (SLU) in under-resourced languages through models which circumvent the need for parallel text labelled resources.
2 Related Work: Multimodal SLU
Spoken language understanding (SLU) has a long history. It is well established in speech literature that using speech audio features in an end-to-end fashion for Language Understanding tasks is nontrivial compared to text. There are several difficulties in using speech directly as input such as long length of inputs making it difficult to densely capture context, presence of spoken accents, gender, environmental noise, and acoustic information, etc. which all pose challenges for use in end-to-end semantic reasoning on it.
For most of its history, SLU was developed in a pipelined fashion, with ASR feeding text to a natural language understanding system, e.g., to the best of our knowledge, the only published uses of SLU with knowledge graphs that fit this description is (Woods, 1975). Recent research in end-to-end multimodal SLU bypasses the need for ASR by leveraging a parallel modality such as image (Harwath et al., 2016; Kamper et al., 2019) or video (Sanabria et al., 2018), or a non-parallel corpus of text (Sari et al., 2020), to guide learning speech embeddings such that the speech input can be used in a downstream task.
In speech-based VQA applications, the most common approach is a two-step approach which consists of an ASR followed by text-based VQA (Zhang et al., 2017). However, these systems are not generalizable to under-resourced or unwritten languages for which we cannot train an ASR system. Therefore, in this study, we will explore using neural speech embeddings, which are guided by the information in the KG, for achieving FVSQA.
3 Related Work: Knowledge Graphs
Knowledge graphs (Suchanek et al., 2007; Auer et al., 2007; Bollacker et al., 2008) are effective ways of representing objects or concepts and their inter-relationships. Such relational representations are formally defined in the Resource Description Framework (RDF) as triples  $f =$  (subject, predicate, object), where (subject, object) are entities, predicate is the relation connecting the two entities. (Halford et al., 2010) showed that such linked representations correlate highly with human cognition. Furthermore, KGs can be classified as Closed-World or Open-World. The former assumes that non-existent fact triples must necessarily be false, while the latter assumes that the KG could be incomplete, and there
fore missing edges could be either true or false. While closed-world assumptions hold for domain-specific KGs, common-sense KGs extracted from web-scale datasets do not respect this assumption (Galárraga et al., 2013; Dong et al., 2014).
3.1 KG embeddings
Common-sense KGs extracted from web-scale datasets are usually incomplete. KG embedding techniques (Bordes et al., 2013; Sun et al., 2019; Socher et al., 2013; Nickel et al., 2011; Dong et al., 2014; Dettmers et al., 2018) have been studied as a means to remedy incompleteness of large-scale KGs. These embeddings have been shown to transfer well to other tasks that require knowledge acquisition over the KGs.
KG Embedding methods usually assign scores or truth-probabilities to each fact triple by learning latent features for entities and relationships. These methods learn a score mapping  $\phi(h, r, t): \mathcal{E} \times \mathcal{R} \times \mathcal{E} \to \mathbf{R}$  where  $\mathcal{E}$  is the set of all entities,  $\mathcal{R}$  is the set of all relation-types.  $h, t \in \mathcal{E}$  are the head (subject) and tail (object),  $r \in \mathcal{R}$  is the directed relationship that connects the two. The observed KG can be expressed as  $\mathcal{G} \subset \mathcal{E} \times \mathcal{R} \times \mathcal{E}$ , which in turn is a subset of  $\mathcal{G}_o$ , the unknown set of all true edges in the world that the KG seeks to represent. The embeddings  $(h, r, t)$  are learned so that the score  $\phi(.)$  is high for edges not just in  $\mathcal{G}$  but also for those in  $\mathcal{G}_o$ , and low for edges outside of it.
Distance-based models (Bordes et al., 2013; Sun et al., 2019; Trouillon et al., 2016; Bordes et al., 2011) learn embeddings  $h$ ,  $r$  and  $t$  in order to minimize the distance between  $t$  and  $f(h, r)$ , for some projection function  $f(\cdot)$ . Common-sense KGs are often based on free text, therefore most entities occur rarely; an example is the entity "lying on" in Fig. 2. Since it is very challenging for distance-based methods to perform completion of commonsense KGs, very few previous benchmarks have approached this task (Li et al., 2016; Malaviya et al., 2020). In (Ramnath and Hasegawa-Johnson, 2020), it was shown that Entity-Relation Multi-Layer Perception (ERMLP) (Dong et al., 2014), which uses an MLP to produce the score  $\phi(h, r, t)$  for each fact triple, works better for FVQA in comparison to TransE and RotatE.
3.2 KGQA
Knowledge-graph question answering (KGQA) is the task of answering questions regarding facts
that can be inferred/retrieved from a KG given the question, image and the graph. Language-only benchmarks include (Bordes et al., 2015; Berant et al., 2013), vision-and-language benchmarks include (Sanket Shah and Talukdar, 2019; Marino et al., 2019; Wang et al., 2018). In (Wang et al., 2018), FVQA is approached as a parsing and fact-retrieval problem, while (Narasimhan and Schwing, 2018) directly retrieves facts using lexical-semantic word embeddings. In Out-of-the-box (OOB) reasoning (Narasimhan et al., 2018), a Graph Convolutional Network (Kipf and Welling, 2017) is used to reason about the correct entity, while (Zhu et al., 2020) (the current State-of-the-Art in the complete-KG FVQA task) added a visual scene-graph (Krishna et al., 2016) and a semantic graph based on the question alongside the (OOB) KG reasoning module. In (Ramnath and Hasegawa-Johnson, 2020), FVQA is tackled on incomplete KGs using KG embeddings to represent entities instead of word-embeddings, as the latter are shown to be inadequate for this task.
Among other KGQA works closely related to our approach, (Huang et al., 2019) answer a text question using minimum-distance retrieval of translational KG entity and relation embeddings, thereby achieving SOTA results on SimpleQuestions with supporting knowledge bases Freebase2M and Freebase5M (Bollacker et al., 2008). In (Lukovnikov et al., 2017), authors use character-level embeddings for SimpleQuestions. In (Saxena et al., 2020), KG Embedding-based reasoning over missing edges is performed on the text-only benchmarks Webquestions (Berant et al., 2013) and MetaQA (Zhang et al., 2018), where they also perform multi-hop reasoning. Amongst KGQA baselines involving the visual modality, the OKVQA benchmark (Marino et al., 2019) provides outside common-sense knowledge in the form of supporting text. The accompanying external knowledge is acquired using a neural network parse of the fact text. KVQA (Sanket Shah and Talukdar, 2019) provided KGs as outside knowledge, and they tackled the task using face-recognition and entity-linking to answer several different types of questions.
4 Task Formulation
This section introduces a new task called FVSQA and presents a new dataset collected for this task.
Overall architecture
Figure 2: Our architecture for FVSQA. (1) Object and scene detectors find constituent entities in images. (2) The image is represented as a collection of KG embedding features for these detected entities. (3) MFCC features for the spoken question are passed via an LSTM. (4) The co-attention mechanism described in Fig. 3 fuses the image and question encoding, then (5) passed through successive fully-connected layers, whose (6) last layer is used as a query. (7) The closest entity to this query is retrieved as the answer.
4.1 FV(S)QA
FVSQA is similar to FVQA in all aspects but for the modality of the question  $q$ ; in FVSQA it is a speech input instead of a text input.
The following condition holds for questions in the FVQA (Wang et al., 2018) benchmark: for each (question, image, answer) triplet in the dataset  $(q_{i}, I_{i}, y_{i}) \in \mathcal{D}$ , exactly one supporting fact in the knowledge graph  $(f_{j} = (h, r, t) \in \mathcal{G})$  exists such that the correct answer  $y_{i}$  is either the head or the tail of  $f_{j}$ , and such that at least one of the two entities is visible in the image.
The companion knowledge-graph is constructed from three diverse sources: ConceptNet (Liu and Singh, 2004), Webchild (Tandon et al., 2014), and DBPedia (Auer et al., 2007). ConceptNet provides common-sense knowledge about entities, DBPedia mainly conveys hypernym (i.e. parent-child) relationships, while Webchild covers many different kinds of comparative relationships between entities (these are considered as a single relationship-type for FVQA).
Answering questions in FVQA is to perform the following operation
$$
\hat {y} = \underset {e \in \mathcal {E}} {\operatorname {a r g m a x}} p (y = e \mid q, I, \mathcal {G}), \tag {1}
$$
i.e., retrieving that entity which is most likely to be
Knowledge Base	Total facts	Questions
DBPedia	35152	817
ConceptNet	119721	4652
Webchild	38576	357
Table 1: Distribution of facts and questions across the KBs (Wang et al., 2018)
the correct answer given a question  $q$  and image  $I$ , and given the graph  $\mathcal{G}$ .
The FVSQA task formulation is identical, except that the question is not textual but spoken. We study the task when the question is spoken in one of three languages – English, Hindi, Turkish.
4.2 Data Description
The dataset contains 2190 images sampled from the ILSVRC (Russakovsky et al., 2015) and the MSCOCO (Lin et al., 2014) datasets. 5826 questions were obtained via crowdsourcing on Amazon Mechanical Turk which concern 4216 unique supporting facts (Table 1). FVSQA provides the same five train-test splits as FVQA, where each split contains images and questions roughly in the ratio 1:1. The accompanying KG consists of roughly 194500 facts, about 88606 entities. In total, the dataset contains 13 relations: \( R \in \{Category, HasProperty, RelatedTo, AtLocation, IsA, HasA, CapableOf,
Figure 3: A co-attention mechanism fuses the image and question representations. First, self-attention provides a single question-embedding of the speech signal (bottom orange circle). Next, the question-embedding functions as a context vector to guide the visual attention weights (top orange circle). The final image embedding is a vector present in the span described by its constituent entities' KG Embedding vectors.
UsedFor, Desires, PartOf, ReceivesAction, CreatedBy, Comparative}.
The next section describes how the multi-lingual speech data is generated.
4.2.1 Data Generation - Text Translation
The text questions in FVSQA dataset are in English. To generate spoken questions in Hindi and Turkish, we first translate the questions using Amazon Translate  $\mathrm{API}^1$  from English. We manually review the questions to ensure intelligibility of questions. These translated texts are only used for speech data generation; these are not available to the network during either training or inference.
4.2.2 Data Generation - Text-to-Speech
We use Amazon's Polly  $\mathrm{API}^2$  to generate spoken questions for each language. The generated speech is in mp3 format, sampled at  $22\mathrm{kHz}$ . For a given language, all questions were generated using the same voice. The voices used were Joanna for English, Aditi for Hindi, and Filiz for Turkish. We again manually review and ensure intelligibility of speech data so generated.
5 Our Approach
Fig. 2 depicts the architecture we use for FVSQA. As shown in the figure, co-attention fuses an image  $I$  and question  $q$  to form a query vector  $\nu$ . This query vector is then used to retrieve the answer from the KG as
$$
\hat {y} (q | I) = \underset {e \in \mathcal {E}} {\operatorname {a r g m a x}} \nu (q, I) ^ {T} e. \tag {2}
$$
The following sections address representations of the question, KG, and image, the information fusion function  $\nu (q,I)$ , and the loss function. The image and KG representations are identical to those considered in (Ramnath and Hasegawa-Johnson, 2020), however, their goal is different from ours, as they perform monolingual text-FVQA over incomplete KGs.
5.1 Question representation
We represent the speech waveforms using Mel-Frequency Cepstral Coefficient features. We set the window-length to 25 ms and stride-size of 10 ms. For each time-step, we follow standard convention of using 39-dimensional vectors - the first 12 cepstral coefficients and the energy term, along with delta and double-delta features to gather contextual information as well.
5.2 KG Representation
To discriminate between a true and false fact, a binary classification-based KG Embedding model is used. Training a meaningful classifier would require presenting it with both positive and negative examples, but the observed KG  $\mathcal{G}$  has only positive samples. This leads us to a 'chicken and egg' problem - KG Embeddings are supposed to mitigate the very problem of incompleteness, yet they need some negative edges to actually learn a good score function. Some heuristics have been empirically found to work well in overcoming this problem. Under the Locally Closed World Assumption (LCWA) (Dong et al., 2014), negative samples can be generated by randomly corrupting the tail entity of existing facts. The KG embedding loss function penalizes the network when a true edge has a low truth-probability, and a false edge has a high truth-probability. But some false facts may be more difficult for the model to classify as false than the others. (Sun et al., 2019) introduced a self-adversarial negative sampling strategy so that the loss function reflects this, and each false fact's contribution to the loss is scaled by the truth-probability assigned by the network during training. Thus, false edges with a higher truth-probability are penalized more heavily than false edges with lower truth-probabilities.
Based on each true fact  $f_{i}$ , a total of  $n$  adversarial facts are generated and used to train discriminative embeddings using noise contrastive estimation (Gutmann and Hyvarinen, 2010). Thus the knowledge graph embedding loss  $\mathcal{L}_{KGE}$  in
<sup>1</sup>https://aws.amazon.com/translate/
$^{2}$  https://aws.amazon.com/polly/
Module	No. of parameters
ERMLP	27, 306, 901
LSTM	12,480
Image representation	0
Visual Attention	340
Textual Attention	40
FVSQA MLP	193860
Table 2: Number of parameters in WoW
cludes the arithmetic inverse of the sum of the log probability that each observed edge is true  $(\ln \sigma (\phi (f_i)))$  , plus the expected log probability that the adversarial edges are false  $\left(\ln \sigma (-\phi (f_j')) = \ln \left(1 - \sigma (\phi (f_j'))\right)\right)$
$$
\begin{array}{l} \mathcal {L} _ {K G E} = - \sum_ {i = 1} ^ {| \mathcal {G} |} \left(\ln \sigma (\phi (f _ {i})) + \right. \\ \left. \sum_ {j = 1} ^ {n} p _ {i} \left(f _ {j} ^ {\prime}\right) \ln \sigma \left(- \phi \left(f _ {j} ^ {\prime}\right)\right)\right) \tag {3} \\ \end{array}
$$
where expectation is with respect to the probability  $p_i(f_j')$ . This probability is tuned using a temperature hyperparameter  $\alpha$  as
$$
p _ {i} \left(f _ {j} ^ {\prime}\right) = \frac {\exp \left(\alpha \phi \left(f _ {j} ^ {\prime}\right)\right)}{\sum_ {k = 1} ^ {n} \exp \left(\alpha \phi \left(f _ {k} ^ {\prime}\right)\right)}. \tag {4}
$$
Eq. (3) is used to train embeddings of the head  $(h)$  and tail  $(t)$ , which are applied to the FVSQA task as described in the next several subsections. Eq. (3) also trains relation embeddings  $(r)$  and MLP weights for the ERMLP scoring function  $(w_{MLP})$ ; these quantities are not used for the downstream FVSQA task.
5.3 Image as Knowledge (IaK) Representation
We revisit the IaK representation first described by (Ramnath and Hasegawa-Johnson, 2020). For the FVQA task, (Narasimhan and Schwing, 2018) established the importance of representing images as a bag-of-visual concepts instead of using features from pretrained networks. This is a simple one-hot encoding of all object and scene detections found in the image. IaK instead represents each image as a contextually-weighted sum of KG entity vectors of detected visual concepts. (Ramnath
and Hasegawa-Johnson, 2020) showed its superior performance for text-FVQA.
Detecting Objects: We use Torchvision's COCO object-detector to detect the 80 COCO (Lin et al., 2014) object classes. The detector used was a Faster RCNN network (Ren et al., 2015) with a ResNet50 backbone (He et al., 2016), and feature pyramid network (Lin et al., 2017). Another detector (ZFTurbo, 2018) trained on OpenImages 600 classes detections was used; we then retain only those classes which are present in ImageNet 200 object detection classes as well as in (Wu et al., 2016). The overlap obtained is almost exact; fewer than 10 classes were not found.
Detecting Scenes: A WideResNet (Zagoruyko and Komodakis, 2016) detector trained on the MIT365 places dataset (Zhou et al., 2017) detects the scenes depicted in each image. Only those classes which were used for constructing the FVQA KG (i.e. the 205 classes from MIT205 places dataset) are retained.
Upon detecting objects and scenes in each image, their corresponding entity KG embeddings are retrieved from KG. IaK then represents each image as a concatenation of entity embedding vectors.
More specifically,  $I_{i} = [e_{i}^{1},\dots ,e_{i}^{m}]\in \mathbf{R}^{N_{e}\times m}$  where  $N_{e}$  is the embedding dimension, and  $m$  is the number of visual concepts detected in the image.
5.4 Fusion Function  $\nu$
As shown in Fig. 3, a co-attention mechanism fuses the image and question representations. To compute a contextual-query for the image-attention, we first obtain a self-attention weighted question representation  $A(q_{i})$  as:
$$
A \left(q _ {i}\right) = \sum_ {t = 1} ^ {\left| q _ {i} \right|} \alpha_ {q} ^ {t} q _ {i} ^ {t}, \quad \alpha_ {q} ^ {t} = \frac {\exp \left(w _ {\alpha_ {q}} ^ {T} q _ {i} ^ {t}\right)}{\sum_ {t = 1} ^ {\left| q _ {i} \right|} \exp \left(w _ {\alpha_ {q}} ^ {T} q _ {i} ^ {t}\right)}, \tag {5}
$$
where  $\alpha_{q}^{t}, w_{\alpha_{q}}$  are respectively the attention paid to time-step  $w_{t}$ , and the weight parameters of the attention network used to compute the attention-scores.
Then, using  $A(q_{i})$  as a query, a contextual attention-weighted summary of the image  $A(I_{i})$