SlowGuess/NAACL_2021_OCR huggingface dataset:
Method	MR	MRR	Hits@1	Hits@3	Hits@10
ERMLP	11194	0.156	0.132	0.152	0.197
Table 3: KG Embedding accuracy
Language	Hits @1	Hits @3
English	49 ± 0.62	61.85 ± 1.13
Turkish	48.96 ± 1.14	61.56 ± 0.79
Hindi	49.29 ± 0.73	61.26 ± 0.93
English - ASR + text-FVQA	54.07 ± 1.15	65.52 ± 0.75
Table 4: FVSQA Performance of WoW architecture across different languages
is obtained as:
$$
A (I _ {i}) = \sum_ {j = 1} ^ {m} \alpha_ {I} ^ {j} e _ {i} ^ {j},
$$
$$
\alpha_ {I} ^ {j} = \frac {\exp \left(w _ {\alpha_ {I}} ^ {T} \left[ \begin{array}{c} A (q _ {i}) \\ e _ {i} ^ {j} \end{array} \right]\right)}{\sum_ {k = 1} ^ {m} \exp \left(w _ {\alpha_ {I}} ^ {T} \left[ \begin{array}{c} A (q _ {i}), \\ e _ {i} ^ {k} \end{array} \right]\right)} \tag {6}
$$
where  $\alpha_{I}^{j},w_{\alpha_{I}},e_{i}^{j}$  are respectively the attention paid to concept  $j$  in the image, the weight parameters of the image-attention network, and the  $j^{\mathrm{th}}$  constituent concept of the image.
$A(I_{i})$  represents a mapping:  $R^{N_e\times m}\to R^{N_e}$  which is the attention-weighted convex combination of its inputs, thus  $A(I_{i})$  is a vector drawn from the span of the entities present in the image.  $A(q_{i})$  represents a mapping:  $R^{39\times T}\rightarrow R^{39},T$  being the length of the spoken question signal.
Finally, a query vector is obtained by fusing the attention-weighted image and question vectors in the following manner:
$$
\nu \left(q _ {i}, I _ {i}\right) = h \left(A \left(I _ {i}\right), A \left(q _ {i}\right); w _ {\nu}\right) \tag {7}
$$
where  $h(\cdot)$  is a two-layer fully-connected network with ReLU activation functions. As prescribed in STTF (Narasimhan and Schwing, 2018), late fusion is used wherein both the question and image vectors are separately passed through one fully-connected layer before being concatenated.
5.5 Loss function
The loss function in Eq. 8 mirrors the answer prediction mechanism, in that the network is penalized whenever the cosine-similarity between the produced query and ground-truth answer deviates from 1.
$$
\mathcal {L} _ {F V Q A} = \sum_ {i} \left(1 - y _ {i} ^ {T} \hat {y} \left(q _ {i} \mid I _ {i}\right)\right) \tag {8}
$$
where  $\hat{y} (q_i|I_i)$  is as given in Eq. (2).
Apart from the MFCC feature generation, the rest of the experimental setup is similar to that described in Seeing-is-Knowing (Ramnath and Hasegawa-Johnson, 2020). It is briefly recapped in the sections below.
6.1 Training the KG Embeddings
For training KG Embeddings, the entire KG is split as  $80\%$  training set and  $20\%$  test set. The embedding dimensions for both entity and relation embeddings are  $N_{e} = N_{r} = 300$ . The batch size used is 1000. ERMLP is trained for 25,000 epochs. Adam optimizer is used for which the learning rate was initialized as 0.01 and then it is scaled down by a factor of 0.1 after every 10,000 epochs. The hyper-parameter search for the learning rate was performed by choosing among values in the set  $\{0.0001, 0.001, 0.01, 0.1\}$ . The temperature hyperparameter  $\alpha$  for the self-adversarial probability parameterization is set to 1 for all experiments. The number of adversarial samples  $n$  generated for each positive sample is 16.
ERMLP is parameterized as a three-layer neural network. The size of the first layer is  $3N_{e}$  since it takes the concatenated head, relation, and tail embeddings as input. Subsequent layers are  $2N_{e}$  and  $N_{e}$  in size respectively, which are finally capped by a single sigmoid unit to output the truth probability  $\phi (h,r,t)$ . The activation functions used by the hidden layers are the Rectified Linear Unit (ReLU), which outputs max{0, x} for an input  $x$ . All layers are fully connected and none of them use dropout.
The KG Embeddings accuracy is measured using the standard metrics: Hits @1, Hits @3, Hits @10. These determine how often each correct tail/head gets ranked in the top 1, 3, or 10 ranked facts for
Question 1: Which object is used for banging out rhythms in this image?
Figure 4: Visualizing co-attention maps produced by Worldly Wise
Answer Source: Image
Answer: Drum
Answer predicted: Drum
Question 2: Where can object in the center of image be found?
SPO triple: {Airplane, AtLocation, Airport}
Answer Source: Image
Answer: Airport
Answer predicted: Runway
SPO triple: {Drum, UsedFor, banging out rhythms}
Speech attention summed over all time-steps of a word
Speech attention summed over all time-steps of a word
Visual Attention for all three voices
each ground-truth  $(h,r) / (r,t)$  pair. Mean Rank is a metric often used to gauge the performance of KG Embeddings. It measures the mean rank of each true fact  $f_{i}\coloneqq (h,r,t)$  in the dataset when ranked by its truth-probability for a given  $(h,r)$  pair. An allied metric is the Mean Reciprocal Rank  $= \frac{1}{|\mathcal{D}|}\Sigma_i\frac{1}{R_i}$
6.2 Training WoW
A maximum of  $m = 14$  visual concepts are detected in each image. We report Hits @1 and Hits @3 for each model. All the results are based on performing K-fold cross validation across the five train-test splits; the numbers reported are mean and standard deviation. To train the fusion function  $\nu$ , the optimizer used is Stochastic Gradient Descent with a batch size of 64. The training runs for 100 epochs with a learning rate of 0.01 and a weight decay of 1e-3. Fully-connected layers use a dropout probability of 0.3.
All models were trained using GPU servers
provided by Google Colab. The training for the ERMLP takes approximately 3 hours, while training  $\nu (q,I)$  on one train split takes roughly 2 hours.
7.1 Cross-lingual FVSQA
Aided by ERMLP, WoW is able to perform FVSQA at the same levels of accuracy across English, Hindi, and Turkish. FVSQA is trained using the best performing KG embedding model demonstrated in (Ramnath and Hasegawa-Johnson, 2020) and its performance is highlighted in Table 3. To verify the superiority of ERMLP over word-embeddings, we compare a model trained with KG entities represented as averaged word embeddings instead. This representation fails to train an end-to-end system even for English, the final accuracy being close to  $0\%$ .
For English, we additionally investigate an ASR + Text-based system, where the FVQA model is trained on gold-standard textual questions, and dur
ing inference-time, an ASR-converted speech transcript of the question is provided. The ASR system is based on the pre-trained Kaldi ASpIRE model<sup>3</sup> which was originally trained on augmented Fisher English dataset. The resulting FVQA system performs better than an end-to-end system for English. This indicates some joint-training strategies for speech and text-based systems could help increase accuracy for the end-to-end speech system. However, our experiments on sharing the lower layers of the network between speech and text-systems did not improve accuracy of the end-to-end speech system for English.
7.2 Attention mechanism visualizations
We can see in Q.1, Fig. 4 that for each language, the speech signal can perform as a good query vector to calculate contextual visual attention as per Eq.(5). The resulting IaK attention maps are interpretable, and in cases where the network predicts the wrong answer, provide an insight into the reason for the network's failure as in Q.2.
Furthermore, the speech self-attention maps are also coherent and informative. The alignment of time-steps in the speech signal with boundaries is generated alongside the question generation. This information, however, is not used while training the network, and is only used to investigate the attention mechanism. Fig. 4 also shows attention accumulated by each word over all time-steps of the word's utterance. We can clearly see that the relevant time-steps are attended to, depending on the image and the question itself. To the best of our knowledge, this is the first work to jointly learn attention-based speech representations guided by external KG knowledge.
A new task FVSQA is presented in this work, along with an architecture that can perform cross-lingual knowledge acquisition for question-answering. In the process, we demonstrate the first task to perform knowledge acquisition directly using a speech signal as an input. This knowledge acquisition for speech can be extended to other tasks such as audio caption-based scene identification (Harwath et al., 2016) and multi-modal word discovery (Harwath et al., 2018). Future work will include extending FVSQA to a multi-speaker setting, gathering spo
Figure 5: Example of a fact-based visual question Question - Which animal in this image is man's best friend?
Supporting fact - [[dogs]] are [[man's best friend]]  
Subject, Predicate, Object - (Dog, HasProperty, man's best friend)
Answer - Dog
ken data from real-world speakers, as well as extending it to languages without an ASR system.
9 Ethical Impact
We now turn to discuss the ethical implications of this work. Worldwide-Wise relies on leveraging cross-lingual knowledge resources for question answering. While this approach yields enormous benefits, care must be taken to evaluate appropriateness of the source of knowledge depending on the language. What may be considered as conventional wisdom in one culture or language may not be true for another. An example of how this manifests in our dataset is shown in Fig. 5. The knowledge graph conveys conventional wisdom in English that 'A dog is man's best friend', and therefore the expected answer to this question is 'Dog'. However, in regions where Hindi is spoken, the answer could equally be expected to be 'Cow' that appears in the image. This example is quite informative, and if such an instance can occur in the extreme, it could lead to fairness issues. This highlights the fundamental tradeoff involved in training such a cross-lingual system on knowledge generated in another language. Governance of such a system is therefore essential to ensure cultural appropriateness and fairness in different contexts.
3https://kaldi-asr.org/models/m1
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Zachary Ives, and et al. 2007. DBpedia: A nucleus for a web of open data. In Proc. 6th International Semantic Web Conference. Springer.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD Conference, pages 1247-1250.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. CoRR, abs/1506.02075.
Antoine Bordes, Nicolas Usunier, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In In Advances in Neural Information Processing Systems 26, pages 2787-2795. Curran Associates, Inc.
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Proc. 25th AAAI Conference on Artificial Intelligence, pages 301-306.
Tim Dettmers, Pascal Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D knowledge graph embeddings. In Proc. AAAI, pages 1811-1818.
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Patrick Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In Knowledge Distillation and Data Mining, pages 601-610.
Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. 2013. Amie: association rule mining under incomplete evidence in ontological knowledge bases. In Proc. 22nd Internat. Conf. World Wide Web (WWW).
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy.
Graeme S. Halford, William H. Wilson, and Steven Phillips. 2010. Review relational knowledge: the foundation of higher cognition.
David Harwath, Galen Chuang, and James Glass. 2018. Vision as an interlingua: Learning multilingual semantic embeddings of untranscribed speech. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4969-4973, Calgary, AB.
David Harwath, Antonio Torralba, and James Glass. 2016. Unsupervised learning of spoken language with visual context. In Advances in Neural Information Processing Systems, pages 1858-1866.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. 2019. Knowledge graph embedding based question answering. In ACM International Conference on Web Search and Data Mining.
Herman Kamper, Aristotelis Anastassiou, and Karen Livescu. 2019. Semantic query-by-example speech search using visual grounding. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7120-7124. IEEE.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR).
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. Lecture Notes in Computer Science, 8693:740-755.
Hugo Liu and Push Singh. 2004. Conceptnet: A practical commonsense reasoning toolkit. BT Technology Journal, 22:211-226.
Denis Lukovnikov, Asja Fischer, Jens Lehmann, and Soren Auer. 2017. Neural network-based question answering over knowledge graphs on word and character level. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, page 1211-1220, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020. Commonsense knowledge base completion with structural and semantic context. Proceedings of the 34th AAAI Conference on Artificial Intelligence.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR).
Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. 2018. Out of the box: Reasoning with graph convolution nets for factual visual question answering. In Advances in Neural Information Processing Systems 31, pages 2654-2665.
Medhini Narasimhan and Alexander G. Schwing. 2018. Straight to the facts: Learning knowledge base retrieval for factual visual question answering. In European Conference on Computer Vision (ECCV), pages 460-477.
M. Nickel, V. Tresp, and H.P. Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proc. International Conference on Machine Learning.
Kiran Ramnath and Mark Hasegawa-Johnson. 2020. Seeing is knowing! fact-based visual question answering using knowledge graph embeddings.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28, pages 91-99.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV).
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018. How2: a large-scale dataset for multimodal language understanding. arXiv preprint arXiv:1811.00347.
Naganand Yadati Sanket Shah, Anand Mishra and Partha Pratim Talukdar. 2019. KVQA: Knowledge-aware visual question answering. In Proc. AAAI Conference on Artificial Intelligence.
Leda Sari, Samuel Thomas, and Mark Hasegawa-Johnson. 2020. Training spoken language understanding systems with non-parallel speech and text. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8109-8113. IEEE.
Apoory Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498-4507, Online. Association for Computational Linguistics.
R. Socher, D. Chen, C.D. Manning, and A.Y. Ng. 2013. Reasoning with neural tensor networks for knowledgebase completion. In Advances in Neural Information Processing Systems, page 926-934. Curran Associates, Inc.
Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In IN PROC. OF WWW '07, pages 697-706. ACM.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In Proc. International Conference on Learning Representations, pages 1-18.
Niket Tandon, Gerard de Melo, Fabian M. Suchanek, and Gerhard Weikum. 2014. Webchild: harvesting and organizing commonsense knowledge from the web. In Seventh ACM International Conference on Web Search and Data Mining, WSDM, pages 523-532, New York, NY, USA.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of Machine Learning Research, pages 2071-2080, New York, New York, USA. PMLR.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2018. FVQA: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2413-2427.
William A. Woods. 1975. Motivation and overview of SPEECHLIS: An experimental prototype for speech understanding research. IEEE Trans. Acoustics, Speech, and Signal Processing, 23(1):2-10.
Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton Van Den Hengel. 2016. What value do explicit high level concepts have in vision to language problems? In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 203-212.
Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. In BMVC.
ZFTurbo. 2018. Keras-retinanet for open images challenge 2018. https://github.com/ZFTurbo/Keras-RetinaNet-for-Open-Images-Challenge-2018.git.
Ted Zhang, Dengxin Dai, Tinne Tuytelaars, Marie-Francine Moens, and Luc Van Gool. 2017. Speech-based visual question answering. arXiv preprint arXiv:1705.00464.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In Thirty-Second AAAI Conference on Artificial Intelligence.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452 - 1464.
Zihao Zhu, Jing Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu. 2020. Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1097-1103. International Joint Conferences on Artificial Intelligence Organization. Main track.