Corpus Construction andSemantic Analysis ofIndonesian Image Description
Khumaisa Nur’aini1,3, Johanes Effendi1, Sakriani Sakti1,2,Mirna Adriani3, Sathosi Nakamura1,2
1Nara Institute of Science and Technology, Japan

2RIKEN, Center for Advance Intelligence Project AIP, Japan
3Faculty of Computer Science, Universitas Indonesia, Indonesia

1Outline Background Related Works Corpus Construction
 Quality Assessment
 Syntactic and Semantic Analysis
 Conclusion

2

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Background3

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Background Sentence-based image description has become an active
research topic for computer vision and NLP Applications: Automatic image description (X. He et al. 2017, A. Karpahty et al. 2014)
 Image retrieval based on textual data (Y. Fend et al. 2010)
 Visual Question Answering (Z. Yang et al. 2010)
 Multimodal MT (L. Specia et al. 2016, D. Elliot et al. 2017)

 Available datasets
Contain image and English text description (Flickr8K, Flickr30K, MSCOCO)
Extended to difference languages:
 Flickr30K has been extended to German, French, and Czech
 MSCOCO has been extended to Japanese Flickr8K has been extended to ChineseIndonesian image description does not exist yet!
This paper: Construct of image description in the Indonesian language
Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 20184

Related Works

5

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Related Works Sentence-based image description in a new language
 Direct image captioning Text translation (manually or automatically by MT)

 Most existing works use the translation method A new dataset in target languages will have identical meaning withthe source language It is argued that an image can represent a universal concept.
Thus, given the same image, the text descriptions in different
languages shall have identical semantic meaning
 However:
Neuroscience studies found a difference in visual perceptions basedon different cultural backgrounds
Further study of the effect of cultural background on visual
perception may be necessary
Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

6

Related Works Multi30K: Multilingual Image Description(D. Elliot et al. 2016) The only existing work that did both direct captioning and translation
 30K English-German image description(1) Translation English-to-German without given the images(2) Direct captioning of images in German without given
the English description
 Analysis of the difference in sentence length
Result: The German translations are longer than the
independent captioning (11.1 vs. 9.6 words)

In this study, we attempted to investigate the differenceby calculating syntactic and semantic distance

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 20187

Corpus Construction

8

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Corpus Construction
 Utilize image description corpus from WMT Multimodal
machine translation challenge WMT Training set: Flickr30K (31,783 images, 5 English desc./image) WMT Dev set : 1015 images, 5 desc./image
 WMT Test set 2017 : 1000 images, 1 desc./image
 WMT Test set 2018 : 1071 image, 1 desc./image

 Construct image description in Indonesian Language(1) Translation English-to-Indonesian without giving the images(2) Direct captioning of images in Indonesian without giving
the English description

 Analysis the difference in syntactic and semantic distance

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 20189

Translation
 English-to-Indonesian Translation (Eng2Ind_Translation)
 Automatic translation with Google Translate API Data: Flickr30K training set, dev set, and test set 2017-2018
 Resulting 166,061 translation

 Manual Validation by Indonesian crowdworkers(Eng2Ind_PostEdit) Post-editing to correct any errors in translation results
without having the corresponding images
 Crowdworkers- Native Indonesian (4M, 5F)- 20-30 years old- Minimum works: 250 sentences per session
 Data: Only dev set and test set 2017-2018
 Resulting 7,146 post-edited sentences

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 201810

Direct Captioning Direct Image Captioning (Ind_Caption)
 Indonesian captioning without having English description or
English-to-Indonesian translation (suggested range: 5-25 words/sent) Crowdworkers- Native Indonesian (7M, 15F)
- 20-30 years old- Minimum works: 200 images (one caption/image) per session
 Data: 10K of Flickr30K training set, dev set and test set 2017-201811

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Quality Assessment

12

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Quality of Automatic Translation
 Investigate the quality of Eng2Ind_Translation
by treating Eng2Ind_PostEdit as the reference Sentence Length No significant difference in the number of the words per sentence
between Eng2Ind_Translation and Eng2Ind_PostEdit
 About 12 words per sentence Translation error rate (TER) (M. Snover, et al., 2006)

 Minimum number of edits (ins, del, sub, shift) in the translation sothat it exactly matches the corresponding reference Average TER was about 5%

 The quality of Eng2Ind_Translation is still acceptable
Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 201813

Syntactic and SemanticAnalysis
14

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Translation vs Direct Captioning
 Syntax Analysis
 End2Ind_Translation sentences are 7.5% longer than the sentencesin Ind_Caption
 Frequencies of POS tag

15

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Translation vs Direct Captioning Semantic Analysis

 Semantic distance between Eng2Ind_Translation and Ind_Caption
 Semantic embedding with Word2Vec/FastText- Word2vec treats each word in a corpus like an atomic entity andgenerates a vector for each word
- FastText treats each word as composed of character ngrams
 Semantic distance16

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Translation vs Direct Captioning Semantic Analysis

 Semantic dist. between Ind_Caption and Eng2Ind_Translation are always
farther away than the distance among Eng2Ind_Translation themselves
 Almost 50% of Indonesian image descriptions lies outside of thethreshold (max dist. among translations)
17

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Translation vs Direct Captioning Semantic Analysis

Shortest Distance (Image a3)
Eng_Caption

A black dog is running along the beachFurthest Distance (Image b2)
Green Bay Packer player cooling off

Eng2Ind_Translation Seekor anjing hitam berlari di sepanjang
pantai

Pemain Green Bay Packer sedang
mendinginkan diri

Ind_Caption

Pemain dengan nomor punggung 4

Seekor anjing hitam sedang berlari-lari dipantai

Ind2Eng_Translation A black dog is running around the beachPlayer whose number is 4

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 201818

Conclusion

19

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018

Conclusion
 Constructed Indonesian image description
 Eng2Ind_Translation: English-to-Indonesian automatic translations(WMT training set Flickr30K, dev set and test sets 2017-2018)
 Eng2Ind_PostEdit: Manual post-edits on Eng2Ind_Translation(WMT dev set and test sets 2017-2018)
 Ind_Caption: Direct Indonesian captioning(10K of Flickr30K, dev set and test sets 2017-2018)

 Analysis
 Syntactic: Sentence length of Eng2Ind_Translation > Ind_Caption
 Semantic: Almost 50% Indonesian image descriptions lies outside
the threshold (max dist. among translations) An image may represent a universal concept, but visual perception
greatly depends on cultural backgrounds Currently: Given the images, we construct the captions for Indonesian Further work:- Extend to other ethnic languages- Given identical captions or translated version, investigate whether
people from different cultural backgrounds can produce similar images

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 201820

Thank You

21

Sakriani Sakti @ AHC Labs, NAIST, Japan | SLTU 2018 | August 29th-31st, 2018