Name

YU Hongtao

学位論文の要旨
Abstract of Thesis
研究科

ヘルスシステム統合科学

SchoolInterdisciplinary Science and Engineering in Health Systems

専 攻ヘルスシステム統合科学

DivisionInterdisciplinary Science and Engineering in Health Systems

学生番号78501204

Student No.

氏 名NameYU Hongtao

学位論文題目 Title of Thesis（学位論文題目が英語の場合は和訳を付記）

Study on Benefits of Semantically Audiovisual Interaction on Unisensory Working Memory（視聴覚意味的相互作用が単感覚作業記憶に与える影響に関する研究）
学位論文の要旨 Abstract of Thesis

In daily life, human is often surrounded by stimuli from different sensory modalities (e.g., auditory,visual, tactile). To better master the external situation, human brain must integrate different sensory
signals to sufficiently perceive the external environment. Especially, it must be noted thatapproximately 80% information is derived from auditory and visual channels. The ability to integrate
the visual and auditory signal to a complete and coherent cognition is highly depending on audiovisual
integration (AVI). It is worth noting that AVI not only facilitates the instant perception performancebut also facilitates subsequent cognitive performance. Image that you must keep the phone numberof a new friend in your mind. Memory encoding process will be facilitated if this friend writes thenumber while repeat the number using friend's own voice or suppressed when the friend writes thenumber while making an irrelevant joking. Although evidence had been showed that visual working
memory retrieval can be accelerated by previous semantically congruent AVI, it remains argued thatwhether faster memory retrieval was contributed by a coherent multisensory representation ormodality-specific unisensory representation. Considering the multisensory evidence indicated that
coherent multisensory representation formation was contributed by semantically congruent AVI.
Early multisensory studies reported that benefits of coherent multisensory representation forunisensory perception performance were asymmetric. Less effective auditory perception performancecan receive more multisensory benefits compared with visual perception performance. Some studies
also indicated the perception asymmetric performance can also modulate subsequent cognitive
processing. Therefore, one possible research plan for exploring the unresolved question wasinvestigating the unisensory (i.e., visual and auditory) working memory performance under different
multisensory memory encoding environments. The main aim of this present thesis was to investigate
how semantically audiovisual interaction differentially modulate subsequent unisensory working
memory performance.
In part 1, we investigate whether semantically congruent AVI during the encoding stage of shortterm memory can differentially modulate subsequent unisensory visual and auditory short-term
memory performance by applying a delayed matching-to-sample paradigm (DMS). The reaction time

results revealed significantly faster unisensory short-term retrieval performance under thesemantically congruent audiovisual encoding condition. The present study suggested that coherent
multisensory representation formation might be optimized by semantically congruent multisensory
integration with modal-based attention in memory encoding, and can be rapidly triggered bysubsequent unisensory memory retrieval demands. For exclusively accelerated auditory short-term
retrieval, we suggested that coherent multisensory representation formation is strengthened by asemantically congruent visual stimulus that is not the attention focus during the memory encoding
stage. Importantly, during the memory retrieval stage, a less effective auditory stimulus can trigger
optimized multisensory representation, thereby facilitating rapid memory retrieval processing.
Especially, it must be noted DMS was widely used in previous short-term memory as well as working
memory studies. To further evaluate the possibility that unisensory memory retrieval was also
involved in working memory but not limit to short-term memory, we also tentatively evaluate theresult reliability under three interference condition: distractor, interruption and no interference. For
the interruption condition, the reaction time results revealed a significant difference in visual working
memory retrieval between semantically congruent bimodal memory encoding and unimodal memory
encoding. These results indicates that semantically congruent bimodal encoding accelerates theunisensory short-term memory and working memory retrieval.
In part 2, based on the part 1, we further investigate whether the interaction benefits between
semantically congruent AVI and top-down attention can further modulate the subsequent unisensory
visual and auditory working memory retrieval performance. The results reconcile and extend previous
multisensory working memory studies by demonstrating that semantically congruent bimodal
presentation with divided-modality attention can accelerate subsequent unisensory working memory
retrieval, especially less effective auditory working memory retrieval. This result indicated thatsufficient semantically congruent bimodal presentation (e.g., divided-modality attention) not only
facilitates immediate behavioral perceptual performance but can also strongly impact subsequent
unisensory working memory performance. Moreover, compared with insufficient multisensory
integration (e.g., modality-specific selective attention), sufficient multisensory integration (e.g.,divided-modality attention) requires more resources for an individual to fully encode and integratevisual and auditory information and maintain a robust multisensory representation, leading to fewer
available resources for subsequent unisensory working memory retrieval. Especially, we conduct a
control experiment to evaluate whether participants remember the visual or auditory stimulus by using
verbal naming method. In lined with our previous experiment result, the result of control experimentalso revealed a faster auditory memory retrieval under the semantically congruent AVI with dividedmodality attention condition, indicating the verbal naming effect was not an important factor for faster
auditory memory retrieval, one possibility we tentatively suggested that divided attention resource to
two modalities might be led to sufficient multisensory integration, and then lead to robust
multisensory representation formation.
In part 3, based on the part 2, we further investigate whether interaction between semanticallycongruent AVI and top-down attention can differentially modulate the unisensory visual and auditory
working memory performance by affecting the memory encoding or retrieval stage. The firstexperiment evaluates whether unisensory working memory retrieval was benefiting fromsemantically congruent AVI. The results only found a weak significant difference for auditory

NameYU Hongtao

working memory retrieval under the semantically congruent and incongruent multisensory retrieval
condition. And then, the second experiment evaluate whether unisensory working memory retrieval
was not only benefit from the multisensory retrieval benefits but also can benefit from themultisensory encoding benefits. For visual working memory retrieval, significantly faster reaction
time was found when semantically congruent audiovisual pairs were presented during the memory
encoding and retrieval stage of working memory, indicating the coherent multisensory representation
formation was facilitated by semantically congruent audiovisual encoding, and then visual probe
trigger the multisensory representation even under the task-irrelevant, auditory stimulus interference
condition. For auditory working memory retrieval, faster memory retrieval was only found insemantically congruent audiovisual encoding condition, it is reasonable to assume that a coherent,robust multisensory representation was constructed during semantically congruent multisensory
memory encoding because of task irrelevance, but semantically congruent visual stimuli provide more
redundant information. And then, during the working memory retrieval stage, a less effective auditory
stimulus can trigger optimized multisensory representation and then achieve rapid memory retrieval
processing.
In conclusion, first, we found that unisensory working memory retrieval (i.e., especially auditorymodality) can be accelerated by previous semantically congruent AVI, indicating the possibility that
coherent multisensory representation formation was contributed by semantically congruent AVI.
Further, we found semantically congruent bimodal presentation with divided-modality attention canaccelerate subsequent unisensory working memory retrieval, especially less effective auditory
working memory retrieval, indicating the possibility that multisensory representation formationstrongly depends on the sufficient attention resource. Finally, we found auditory memory retrieval
unisensory memory retrieval can gain more multisensory benefits from memory encoding stage but
not retrieval stage, indicating that memory retrieval stage might depends more on the extent to which
the probe information overlaps the previous encoded information. Especially, a less effective auditory
probe can trigger the coherent multisensory representation and then achieve rapid memory retrieval
processing whatever the semantic information provided by a task-irrelevant visual stimulus wascongruent or incongruent.