20Learning unknown objects from speech and vision- Crossmodal audio-visual concept discovery -Abstract
In order for AI to visually perceive the world around it and to use language to communicate, it needs a dictionarythat associates the visual objects in the world with the spoken words that refers to them. We explore a neuralnetwork models that learn semantic correspondences between the objects and the words given images andmultilingual speech audio captions describing that images. We show that training a trilingual model
simultaneously on English, Hindi, and newly recorded Japanese audio caption data offers improved retrieval
performance over the monolingual models. Further, we demonstrate the trilingual model implicitly learns
meaningful word-level translations based on images. We aim for a future in which AI discovers concepts
autonomously while finding the audio-visual co-occurrences by simply providing media data that exists in theworld such as TV broadcasting. We also consider the application to large-scale archive retrieval and automatic
annotation that involves interactions between different sensory modalities such as vision, audio, and language.
EnglishHindiJapaneseEnglishAudio-visualconcept
discovery(2) Learn semantic correspondences(1) Collect images and speech
between visual objects and spoken
waveforms describing those images.
words.

Japanese(3) Highlight relevant regions ofimage as they are being described.

Evaluation of embedding spacelearned from image and audio captions

Learning neural network embeddings
for images and spoken audio captions
Paired image and audio captions are more similarin embedding space than mismatched pairs(1) Audio-visual retrieval performance
Recall scores for the top 10 hits (1,000 image/caption pairs)

Image CNN（Pre-trained VGG16）

Monolingual model: 0.45

*CNN: Convolutional Neural Network

Meanpooling

Hindi(2) Cross-lingual audio2audio retrieval performance

Embeddingspace

Recall scores for the top 10 hits (1,000 cross-lingual captionpairs)

w/ using images: 0.50

w/o using images: 0.01
English CNN

Exploring visually grounded speech-tospeech translation

Mean
pooling

Hindi

Two small children are walking on a dirt

English road in what seems to be a cornfield

Hindi CNN…
Unpooledembeddings

MeanpoolingJapaneseJapanese CNN

Meanpooling…
Mismatchedaudio captions

Japanese

子供二人組が道を歩いている。
えー、左右には背の低い草木が生い茂っている

children cornfield

EnglishMultilingual model: 0.50

Similarity matrix
between English andJapanese captions

子供草木*Joint research results with MIT Computer Science and Artificial Intelligence Laboratory

References[1] Y. Ohishi, A. Kimura, T. Kawanishi, K. Kashino, D. Harwath, and J. Glass, “Crossmodal Search using Visually Grounded Multilingual
Speech Signal,” IEICE Technical report on Pattern Recognition and Media Understanding (to appear)[2] D. Harwath, G. Chuang, and J. Glass, “Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech,” InProc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2018), April 2018.

ContactYasunori Ohishi Email: cs-liaison-ml at hco.ntt.co.jp
Media Recognition Group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.