JAIST Repository
https://dspace.jaist.ac.jp/Title

マルチモーダル深層学習に基づいたタイ手話における指文
字認識：新しいベンチマークとモデル構成

Author(s)WUTTICHAI, VIJITKUNSAWAT

CitationIssue Date

2024-06

Type

Thesis or Dissertation

Text version

ETDURLhttp://hdl.handle.net/10119/19331

RightsDescription

Supervisor: Nguyen Minh Le, 先端科学技術研究科, 博士

Japan Advanced Institute of Science and Technology

氏

名

WUTTICHAI VIJITKUNSAWAT

学

位の種類

博士（情報科学）

学位記番号

博情第 528 号

学 位 授 与 年 月 日論文題目論 文 審 査 委 員

令和 6 年 6 月 24 日
Deep Multimodal-based Finger Spelling Recognition for Thai Sign
Language: A New Benchmark and Model Composition
Nguyen Le MinhJAISTProfessor
Shinobu Hasegawa

JAISTProfessor

Naoya Inoue

JAISTAssoc. Professor

Kiyoaki Shirai

JAISTAssoc. Professor

Masnizah Mohd Universiti Kebangsaan Malaysia Assoc. Professor

論文の内容の要旨
Video-based sign language recognition is vital for improving communication for the deaf and hard ofhearing. However, due to a lack of resources, creating and maintaining the quality of Thai signlanguage video datasets is challenging. To address this issue, we assess multiple models with a novel
dataset of 90 signs, covering the full letters of alphabets, vowels, intonation marks, and numbers, asdemonstrated by 43 signers. We investigate seven deep learning models with three distinct
modalities for our analysis: video-only methods (including RGB-sequencing-based CNN-LSTM andVGG-LSTM), human body joint coordinate sequences (processed by LSTM, BiLSTM, GRU, andTransformer models), and skeleton analysis (using TGCN with graph-structured skeleton
representation). A thorough assessment of these models is conducted across seven circumstances,encompassing single-hand postures, single-hand motions with one, two, and three strokes, and twohand postures with static and dynamic point-on-hand interactions. The research highlights that theTGCN model is the optimal lightweight model in all scenarios. In single-hand pose cases, acombination of the Transformer and TGCN models of two modalities delivers outstanding
performance, excelling in four particular conditions: single-hand poses, single-hand poses requiring
one, two, and three strokes. In contrast, two-hand poses with static or dynamic point-on-handinteractions present substantial challenges, as the data from joint coordinates is inadequate due tohand obstructions stemming from insufficient coordinate sequence data and the lack of a detailed
skeletal graph structure. The study recommends integrating RGB-sequencing with visual modality
to enhance the accuracy of two-handed sign language gestures. Moreover, experimental results on
our dataset show that our method outperforms previous state-of-the-art methods significantly in fiveout of seven conditional hand pose experiments, especially two-hand poses.
Keywords: Thai Finger Spelling, Sign Language Recognition, Deep Learning, Multimodal Learning,Benchmark Dataset.

論文審査の結果の要旨
The major contribution of the thesis is exploring Thai sign language translation using deeplearning. The first significant contribution involves developing a comprehensive video databasefor Thai Finger Spelling (TFS). This database comprises 10,467 videos demonstrating 90 uniqueletters in various handshapes with one or both hands, featuring contributions from 43 diverse
signers with different appearances and backgrounds.
The second contribution involves extensive research into designing and developing a finger
spelling recognizer for TFS based on the collected dataset. The proposed method undergoes
analysis through extensive experiments across three modalities and different representation
learning techniques. These include CNN-LSTM and VGG-LSTM models for RGB sequences,LSTM, BiLSTM, GRU, and Transformer models for the coordinate sequence of the human body's
joint structure, and TGCN for the skeleton modality's graph structure.
Experimental results on various settings and the proposed dataset highlight the advantages anddisadvantages of deep learning methods for recognizing Thai Sign Language.
In the third contribution, the thesis conducts comprehensive statistical tests, encompassing bothin-sample and out-of-sample evaluations, to identify the most efficient model rigorously. This
ensures the recommended model is practical and effective for real-world use. The thesis ispresented a nice story for Thai Sign Language which is presented in five well-structured chapters.
The publication qualifies for granting the results of the thesis.
Overall, this is an excellent dissertation and we approve awarding a doctoral degree to Wuttichai
Vijitkunsawat.