Mayumi Bono, Assistant Professor, Digital Content and Media Sciences Research
Division

Breaking through the conventional wisdom of language, from the field of communication

How can human beings communicate so well? Even though everyday conversation isoften full of grammatical errors, the listener usually correctly understands what thespeaker wants to say. I would like to gain a deeper understanding of the structures ofthe Japanese conversation and of Japanese sign language conversation through a close
analysis of interactions, including speech and gestures.

Focusing on what linguistics overlooks

Originally my area of specialization is linguistics, which has traditionally focused ongrammar. Until now, the main subject of the research has been text. However, spoken
language not simply conveys meaning through speech , for example using intonation,
but also involves simultaneous communication through other modalities such as eye
direction and body movement.

To understand these multimodal language activities of human beings, I will adopt an
approach of comparing sign language with spoken language, utilizing the methods ofcomparative linguistics. Such methods involve looking individually at matters such ashow hand and mouth movements differ when comparing the two. Incidentally, did you
know that sign language is a natural language just like spoken language, and that itwas not created artificially? Spoken language has an abundant corpus and computer
processing of such language has already seen significant advances; the fact that methods
for writing down everyday interactions in detail and accumulating data on suchinteractions still have not been established is a grievous oversight.

1

There is no method for recording in textual form sign language, for which originally nowritten language existed. For example, while sign language includes numerous
variations corresponding to dialects, today these are in the process of being lost, and will
indeed be lost if not retained in data form. In addition, using multimodal data to showthat sign language is a language just like spoken language will unravel peoples’
misunderstandings of sign language and open up a route toward reforming people’s
awareness of sign language.

Toward development of a multimodal corpus

Accordingly, I am collecting data from a variety of locations. For example, changes areapparent in conversational structures, such as the conversation of a six-person groupsplitting into two or three groups and then joining together again, and it is clear thatcertain rules apply, such as participants taking turns speaking. The “takoyaki party”
activity is well suited to observing how multiple modalities combine together. It is very
interesting how two activities overlap as participants appear to converse with theirsense of hearing while using their sense of sight to check on the takoyaki as they cook.

The rules governing how a conversation starts and ends are also an issue that remains
unclear. However, since this is an issue that has been taken into account in today’s
media design, the future advent of next-generation media such as teleconferencing andhumanoid robots will provide an excellent opportunity for groundbreaking research inthis area. I am working to proactively create a variety of opportunities in which I cangather data, taking a long-term point of view.

Interviewed and summarized: Rue Ikeya

2