別紙41 (課程博士(英文))Date of submission (month day, year)

July 07山,2021

Department ofCompuler science

StudentlD Numberand Engineering
Pplicant's nameYosh北飢'u lshida

189303
Supervisors

Jun Miura

Rizalsetya perdana

Abstract (Doctor)
T洲e ofThesis

ContextuaⅡZation ofMultimodal sequence Models on hれage to Textin story Generation

Approx 80o words

A major achievement in artificialinte11igence (AD, particularly in visual
rec0牙nition, is developin牙 a machine capable of understanding a complex visualScene. Beyond understandin牙 the visual object, the other research subfield in AI,natura11an牙Ua今e 牙eneration systems, attempt to develop machines capable ofdescribing objects W北h human langua目e. understanding dynamic visual scenes
then describing them in words is easy for humans, but this task is difficult formachines. Research work in generatin今 Sentence descriptions from visualrepresentations, e.牙., ima牙e captionin牙, is continuously impl'oving as computing
techn010gy, social networkin牙 Platforms, and a1今or北hms continue to evolve.1n the
real'1ife implementation, ima牙e to text system helps a visuaHy disabled person
Understand images and possibly needed by a computer device with usability
assistance. Although the ima牙e captionin目 algor北hm has been shown to achieve
Success, it is lim北ed to describing only a sin今le ima牙e with a literal object
description

The cul'rent image captionin牙 System cannot process a sequence of images
directly with the output of multiple cohesive sentences. shifting from a single
ima目e, static moment, and generating no"context'description toward a sequence ofima牙es that depict the dynamic event with an output cohesive narrative isCha11en牙ing. visual storyte11ing is an ima牙e'to'text task that comes with the more
Complicated scenario describing an ima牙e sequence into story style sentences.1t
Utilizes the advance of computer vision capabilities in recognizing the complex

Visual object as a human・1ike inferencin牙 ability in terms of structure andSubjectivity. previous approaches in visual storyte11ing systems generate
Iess・than'accurate narrative sentences compared with the human、目enerated
Story. several drawbacks Were accused of describin号' 1iteral objects only,
monotonous sentences, and low'1exical variance so that the generated story suffersfrom being less coherent. Furthernlore,1ess'context stories led to inaccurate
information delivery due to the absence ofthe visual object vaHdation mechanism
and the minimum nun〕ber oflearning resources for the langua牙e model. Based on
the aforementioned problems and limitations, we consider in)proving the output
quality, i.e., generatin牙 a contextualized narrative story
This dissertation develops techniques and models focusin牙 on generatln牙 anarrative text based on visual sequences close to a human'generated story. TO

accomplish the achievements of the mentioned objectives, we break down theProposed approach into three sub'works. First, we desi牙n the experiment to build
the ilnage'text feature pair representation as a sin今Ular data point. This approachis named multimodal instance、based transfer learning tested in an lmage

Captioning task. As an initial part of the whole architecture, a simple setting isConsidered by representing the non、sequential feature with no coherent output

expectation objective. second, the absence of non、visual concept words in the曾enerated sentence story, an important component ln composln牙 a narratlve, wasdiscussed. A non'visualconcept is a word entity that accompanies the literal object

that the visual object detection algor北hm cannot recognize.＼入7e investi曾ate theCorrelation of ima牙e・text pairs t0 今enerate new feature representation with theUnderlying concept of canonical correlation analysis in figuring out the drawback
Third, the lack ofcontext in the previous approach's outputs leads to delivering anerroneous message and unwanted context. Thus, we attempt to improve thearchitecture with context、awareness by supplementin牙

newfeatures andincorporatin牙 externa11an今Ua牙e resources on the decodin牙 language generation
Sta牙e

The question of how to represent lmage'text palrs lnto

newdatarepresentation was an early problem statement in this dissertation. To express a
Pair of image・text data into a sin牙le data point, simple concatenation of ima牙efeature and word representation vectors was not applicable due to the difference ofdata distribution between modalities. A single data point in the form of a vector

Should represent both modalities within a pair of ima曾e、text data. we employ abinary hashing mechanism that 牙enerates a mapping between ori宮inalspaces intoa Hamming space structured as binary codes. The new mappin曾 representatlon

Was applied to transfer learning among the image captionin今 datasets to confirm
the effectiveness. This method is particularly effective for single pair

image'sentence comparison only; a new question was raised about how ifthe pairs
are sequences ofimages paired with multiple sentences
We formulate the investigation by exploitin号 the multimodal pairs'
Correlation to map the sequential pattern feature of image and text pairs for theVisual storyte11ing task. To extract the sequential pattern from the array ofima牙es, we attempt to maximize the cross、modal correlation by extendin牙 theCanonical correlation analysis, thus suitable for the sequential settin今. The

Proposed end・to・end arch北ecture particularly aims to encode the non'visualConcept from the visual representation. As its objective is only to maximize theIikelihood of input with the tar晉et sequence, it lacks the semantic correlation andresults in the low'context output

Considerin目 the aforementioned limitation, we improve the architecture byextending the encodin今 and language 今eneration decoding process to generate aCoherent, object'focused, and contextualized sentence story. To overcome theinaccurate context, we incorporate the visual object detection feature to validate
the literal object durin牙 the encodin牙 Process. From the lan曾Uage decodin今, weUtilize the pre・trained lan牙Uage 今eneration model to contextualize the sequence

今enel・ated lan牙Ua牙e.1n this study, a11 Sequential pattern learnin牙 employs theSelf、attention mechanism that excels compared with the other approach

Experimental results on the vlsT dataset demonstrate the effectiveness of
Our proposed contextualized lan目Uage generation and multimodal representatlon
Over the baseline based on automatic metrlcs