Learning Co-Speech Gesture for Multimodal Aphasia Type Detection
Authors: Daeun Lee
Sejung Son
Hyolim Jeon
Seungbae Kim
Jinyoung Han
Venue: EMNLP 2023 Main (2023)
Keywords: Aphasia
NLP
Applications
Speech
Multimodality
TL;DR: Recognizing the importance of analyzing co-speech gestures for detecting aphasia types, we propose a multimodal graph neural network based on speech and corresponding gesture patterns.

Abstract: Aphasia, a language disorder resulting from brain damage, requires accurate identification of specific aphasia types, such as Broca's and Wernicke's aphasia, for effective treatment. However, little attention has been paid to developing methods to detect different types of aphasia. Recognizing the importance of analyzing co-speech gestures for distinguish aphasia types, we propose a multimodal graph neural network for aphasia type detection using speech and corresponding gesture patterns. By learning the correlation between the speech and gesture modalities for each aphasia type, our model can generate textual representations sensitive to gesture information, leading to accurate aphasia type detection. Extensive experiments demonstrate the superiority of our approach over existing methods, achieving state-of-the-art results (F1 84.2%). We also show that gesture features outperform acoustic features, highlighting the significance of gesture expression in detecting aphasia types. We provide the codes for reproducibility purposes.

Learning Co-Speech Gesture for Multimodal Aphasia Type Detection
Daeun Lee"; Sejung Son'r, Hyolim Jeon ,Sunggbee Kim", Jinyoung Han 1:2
Department of Applied Artificial Intelligence, Sungkyunkwan University, Seoul, South Korea DDepartment of Human-Al Interaction, Sungkyunkwan University, Seoul, South Korea 3 DDppartment of Computer Science & Engincering, University of South Florida, Tampa, FL., USA {delee12, jinyounghanJeskku edu (maze0717, syfla1512]@g skku. edu, seungbae@usf.ed edu
Abstract
Non-Fluent Comprehension
Aphasla
Trans. Motor Aphasia
Aphasia, a language disorder resulting from brain damage, requires accurate identification of specific aphasia types, such as Broca's 's and Wernicke's i aphasia, for effective treatment. However, little attention has been paid to de- veloping methods to detect different types of aphasia. Recognizing the importance of ama- lyzing co-speech gestures for distinguish apha- sia types, we propose a multimodal graph neu- ral network for aphasia type detection using speech and corresponding gesture patterns. By learning the correlation between the speech and gesture modalities for each aphasia type, model can generate textual representations sen- sitive to gesture information, leading to accu- rate aphasia type detection. Extensive exper- iments demonstrate the superiority of our ap- proach over existing methods, achieving state- of-the-art results (F) 84.2%). We also show that gesture features outperform acoustic fea- tures. highlighting the significance of gesture expression in detecting uphasia types. We pro- vide the codes for reproducibility purposes'.
Tokens
Tokens
["]
[
the
Fluent Non-Comprenension
Wernicke's Aphasia
Trans. Sensory Aphasia
Tokens
Tokens
began
M
pray
[']
and,
[|:10b]
[|]]]]]]]]
Figure 1: Variations in gestures observed across differ- ent types of aphasia. Each aligned data (text, audio, gesture) is extracted using Automatic Speech Recogni- tion (ASR) ($3).
the traditional assessments for PWA are known to be time-consuming and costly (Wilson et al., 2018), identifying the pathological language impairment from speech data has received great attention (Qin et al., 201 8a; Fraser al., 2014). Clinically, de- tailed diagnosis of aphasia type imperative for proper treatment procedures; however, little atten- tion has been paid to developing a classification model for the types of aphasia.
1 Introduction
Aphasia is language disorder caused by brain structure damage affecting speech functions, com- monly triggered by stroke and other factors such as brain injury, dementia, and mental disorder (Broca al.. 1861; Wasay et al., 2014). Depending on prominent symptoms and severity, aphasia can be classified into the eight types (Kertesz, 2007) such as Broca's 's and Wernicke's aphasia as shown in Fig- ure 5 in Appendix. People with aphasia (PWA) can face various communication challenges due to limited language processing and comprehension capabilities, which can result in difficulties with social interaction (El Hachioui et al., 2017). Since
For diagnosing various types of aphasia, analyz- ing the co-speech gestures of PWA can be essential. PWA often rely on non-verbal communication tech- niques, especially gestures, as an additional com- munication tool due to difficulties in word retrieval and language errors (Preisig et al., 2018; de Kleine et al., 2023). Therefore, the same word can be interpreted differently depending on the accom- panying gestures for different aphasia types. For example, as shown in Figure 1, individuals with Broca's 's aphasia show a higher frequency of iconic
'Equal contribution.
Conresponning author
(Code: https: ht/.//tbb.cmm///aaLLL..... wtwp:///o////////////ae Sww.////tA.S.K//aaaaiiiioooooo al A--Aphasia-Tyaae Type-yyteccetinneeeoeee Detection_EMM...PP...

---

gestures because Broca's area damage results in preserving semantic content but being impaired flu- ency (Albert et al., 1973). On the other hand, indi- viduals with Wernicke's 's aphasia tend to derive lim- ited assistance from gestures since fluent but inco- herent speech pathology Hence, since the meaning of the word 'I' from PWA with Broca's aphasia may have more latent information, utilizing both speech (i.e., linguistic and acoustic) and gesture (i.e., visual) information is crucial in classifying aphasia types.
tion by adjusting the weights of pre-trained word embeddings with the refined representations of dis- fluency tokens. Consequently, a multimodal Fusion Encoder ($4.4) is applied to incorporate aphasia- type-specific multimodal representations to predict the final aphasia type ($4.5).
2002).
The extensive experiments show that our model performs better than the prior methods in detect- ing aphasia types. We find that applying the GNN can effectively generate multimodal features by capturing relations between speech and gesture in- formation, enhancing performance. Our analysis reveals that gesture features play more critical roles in predicting aphasia types than acoustic features, implying that gesture expression is unique at- tribute of PWA with different types. We exemplify that the qualitative analysis based on the proposed model can help clinicians understand aphasia pa- tients more comprehensively, helping to provide timely interventions. To the best of our knowledge, this is the first attempt that uses gesture and speech information for automatic aphasia types detection.
In combining two different modalities, speech and gesture, applying the existing multimodal fu- sion models is challenging due to the following two limitations. First, the existing models were usually built upon emotion recognition or sentiment anal- ysis benchmarks (Zadch et al., 2016; Busso al., 2008) mainly consisting of facial information; yet, PWA struggles in using faces caused by the im- paired emotion processing (Multani et al., 2017). Second, these models tended to capture single link across the modalities, e.g., linking happiness- related words smiling faces (Yang et al., 2021; Tsai et al., 2019), but there can be multiple links across the modalities depending on aphasia types. (e.g., 'I' may be connected to multiple gestures based on aphasia types, as shown in Figure 1).
2 Related Work
Aphasia analysis and detection. To identify the pathological symptoms shown in narrative speech of PWA, researchers have focused on linguistic features (e.g., word frequency, Part-of-Speech (Le et al., 2018), word embeddings (Qin et al., 2019a)), and acoustic features (e.g., filler words, pauses, the number of phones per word (Le and Provost, 2016; Qin et al., 2019b)). With the advances in automated speech recognition (ASR) that can make the tran- scription of aphasic speech into text (Radford al., 2022; Baevski et al., 2020), there have been end-to- end approaches that do not require explicit feature extraction in assessing patients with aphasia (Chat- zoudis et al., 2022; Torre et al., 2021). While these works reveal valuable insight into detecting apha- sia, little attention has been paid to identifying the types of aphasia, which can be crucial for proper treatment procedures. Considering the importance of understanding gesture patterns for identifying the types of aphasia (Preisig et al., 2018), we pro- pose a multimodal aphasia type prediction model using both speech and gesture information.
To address these challenges, we propose to ap- ply a graph neural network to generate multimodal features for each aphasia type, enabling the acquisi- tion of rich semantics across modalities (Banarescu al., 2013). Due to the heterogeneity among mul- timodalities in clinical datasets (Pei et al., 2023), a unique graph structure can help distinguish di- verse patterns (Zhang et al., 2022) by learning the correlation between speech and the corresponding gestures of PWA with different aphasia.
Specifically, we construct heterogeneous net- work that represents the relationships between speech (particularly including disfluency-related keywords and acoustic information) and gesture modalities based on their co-occurrence by us- ing data from AphasiaBank (Forbes et al., 2012: WacWhinney et al., 2011) 1), shared database for aphasia research. Then, Speech- Sreech-Gesture Graph En- coder ($4.2) extracts three node embeddings for disfluency-related keywords, audio, and gesture tokens by aggregating information from different sources. Before fusing the modalities, the Gesture- aware Word Embedding Layer ($4.3) generates textual representations sensitive to gesture informa-
Multimodal learning for detecting language im- pairment. Multimodal language analysis helps to understand human communication by integrat- ing information from multiple modalities for tasks such as sentiment analysis (Zadeh et al., 2017) and

---

emotion recognition (Yoon et al., 2022). How- ever, previous studies simply concatenated fea- tures to identify language impairment in speech data (Rohanian et al., 2021; Balagopalan et al., 2020), which lacks comprehension of the complex connections between modalities (Cui et al., 2021; Syed et al., 2020). To capture interconnections be- tween modalities, recent studies used Transformer- based multimodal models (Lin al., 2022; Rah- man et al., 2020a) that can extract contextual infor- mation across modalities, utilizing factorized co- attention (Cheng et al., 2021). self-attention (Haz- arika et al., 2020), or cross cross-mode modal attention (Tsai et al., 2019). While the prior work tended to capture a single link between modalities, e.g., linking the 'happy' (linguistic modality) to smiling face (vi- sual modality) (Yang et al., 2021; Tsai et al., 2019), little attention has been paid to learning multiple links across the modalities, which can be crucial in aphasia type detection. As shown in Figure 1, speech and gesture modalities may have multiple links across the different aphasia types. Hence, to distinguish aphasia types, we propose to apply a graph neural network to generate multimodal fea- tures for each aphasia by learning the correlation between speech and gesture patterns of PWA.
Table I: Summary of data statistics of the aligned modal- ities with 50 tokens for each user. (V: impaired, X: not impaired),
Impaired
Aphasia
Data Statisties
Labels
Flu.
Com.
Types
Subj.
Samp.
Dur. (8)
Ctrl
X
x
Control
194
1,815
17.14
Anomic
143
927
31.80
Flu
x
Conduction
62
424
26.27
Non
Sensory
3
15.83
X
V
Com.
Wernicke
188
24.68
Non
Motor
23
43.01
x
Flu.
Broca
73
271
34.75
Total
3,651
23.78
data preprocessing.
Data Preprocessing We utilize Whisper (Radford et al., 2022), a popular ASR model, to generate au- tomated transcriptions. Unlike previous ASR sys- tems focusing on transcribing clean speech (Torre et al., 2021), Whisper can capture filler words (e.g.. um, uh) and unintelligible disfluency words (e.g.. [*]) the token level. As depicted in Table 8 in Ap- pendix, the Word Error Rate (WER) results show that the average WER for all participants is compa- rable to the previous research (Weiner et al., 2017). Furthermore, our research reveals that the text pre- processing results in a decrease in WER due to the frequent presence of fillers and disfluencies in the text of PWA, except for those with non-fluent aphasia types (e.g., Non-Fluent)..
3 Aphasia Dataset
We then align the text tokens with corresponding gesture and audio tokens using the provided times- tamps (Louradour, 2023). We extract the first frame (1 FPS) representative image for the gesture to- kens. To augment the dataset, chunk the aligned modalities with 50 tokens for each user except any segment less than seconds long, as shown in Fig- ure 1. Note that we exclude the segments of the interviewer the video recordings since their sub- jective reactions can potentially affect the patient's s performance (Parveen and Santhanam, 2021; Qin et al., 2018b).
Data Collection We sourced our dataset from the AphasiaBank (MacWhinney et al., 2011; Forbes et al., 2012), shared database used by clinicians for aphasia research; the corpus information is summarized in Table 6 in Appendix. The dataset provides video recordings of the language eval- uation test process between a pathologist and a subject, which also includes human-annotated tran- scriptions and subjects' demographic information. Among the evaluation tests, we chose the Cin- derella Story Recall task (Bird and Franklin, 1996), where pictures from the Cinderella storybook are shown participants, who are then asked to recall and retell the story spontaneously. The test has been proven as a helpful tool that can offer valu- able insights into a subject's ' speech and language skills (Illes, 1989).
Finally, the dataset consists of 507 subjects and their 3,651 aligned modalities, with an average duration of 23.78 seconds, as shown in Table 1.
Aphasia Type Detection
Each data is categorized into one of four types: Control, Fluent, Non-Compa Comprehension. Non-Fluent, based on the ability auditory comprehension and fluency skills, as shown in Table 1. Due to the limited amount of data available for training, spe- cific types of aphasia, such as Global aphasia and Transcortical mixed aphasia, are excluded during
Figure 2 illustrates the overall architecture of the proposed model for detecting aphasia types. The model comprises four main components: Speech- Gesture Graph Encoder ($4.2), Gesture-aware Word Embedding Layer ($4.3), Multimodal Fusion Encoder ($4.4), and Aphasia Type Prediction Decoder ($4.5).

---

Aphasia Type Prediction
4.2.1 Heterogeneous Graph Construction
YO
In a preliminary analysis, we observed statistical differences between words, especially disfluency- related keywords, and aligned gesture features across different aphasia types (see Table 11 in Ap- pendix). Based on the findings, we construct a set of heterogeneous graphs G {Gi, ........i for Cj that represents the relationships between disfluency-related keywords and multimodalities (i.e., gesture, audio) based on their co-occurrence. Note that we use the same disfluency tokens for each graph in G to capture the differences between users with different aphasia types. Disfluency keywords are extracted based on their frequency in ASR transcriptions, as shown in Table 9 in Appendix. Unlike previous approaches (Day et al., 2021). we do not remove "stop words' because most disfluency-related words associated with such 'stop words,' which are typically filtered out in text classification tasks. Formally, each graph is defined with the set of disfluency nodes Vy. which are connected to aligned gesture Vo and audio Va nodes in aligned multimodal data a as follows.
Fully Connected Layer
Addition
U
VA
Multimodal Fusion Encoder
(4)
h
(K)
Hetero AGG.
RoBERTa
hl
Gesture-aware Word Embedding
AGG Node AGG
GL
[*] am um Disfluency Tokens
Speech-Gesture Graph Encoder
Ag* Ag* Ad
(1) Textual Representation. In order to extract the representation of disfluency node in Vy, we use the pre-trained RoBERTa (Liu et al., 2019) that is pre-training language model shows a robust performance across various NLP tasks. After tokenizing the set of words {.rm) into ef, we encode them as follows.
G3 (V,, Ve, Va, Efa.
BI-LSTM
Bi-LSTM
47
10
of of
Video
The overall architecture of the proposed model: ① Speech-Gesture Graph Encoder ($4.2), (2) Gesture-ant aware Word Embedding Layer ($4.3), ( Mul- timodal Fusion Encoder ($4.4), and ( Aphasia Type Prediction Decoder ($4.5).
Figure 2:
h RRBERRT E IRmX4
(2)
4.1 Problem Statement
where de is the dimension of the textual feature. If any disfluency keywords are not included in the vocabulary of the pre-trained tokenizer, we add new tokens to represent these keywords.
Suppose have set of aphasia dataset C C= [c) is structured on an n length of se- quences, including text tokens (LT), gesture tokens (v:",, and audio tokens (a")) as described in Figure 1, which can be represented as C; uf a.). Then, the proposed model is defined as multi-class classification problem that classifies an aphasia data e into an Aphasia type y1 (Control, Fluent, Comprroereenni,,nn,nnnnoooo Non-Flumm)
Visual Representation. To utilize the gesture information for gesture nodes V,, extract the pose landmarks using MediaPipe" (Lugaresi al., 2019) API that generates body pose landmarks. providing image-based 3-dimensional coordinates. Since our video data captures participants seated, we exclude the lower body keypoints and leverage the 23 upper body keypoints. Visual feature e/ is fed into a bidirectional LSTM to derive context representation, reflecting the dynamic nature of an individual's gestures. Finally, the hidden state vectors concatenated follows.
4.2 Speech-Gesture Graph Encoder
We propose a Speech-Gesture Graph Encoder that can generate multimodal features specific to each aphasia type by learning the correlation between speech and gesture patterns in identifying the aphasia types.

---

In this way, three aggregated representations for disfluency, gesture, and audio tokens, respectively, are obtained as follows.
= LSTM (e), LSTM (et)) (3) Acoustic Representation. For audio feature extraction, we use OpenSmile (Byben et al., 2010), an open-source audio processing toolkit, with extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) (Eyben et al., 2015). We extract 25 low-level acoustic descriptors (LLDs) in each second, including loudness, MFCCs (Mel-frequency cepstral coefficients), and other relevant features. We then average the values all 25 features to obtain audio features ef, After that, the audio hidden state vectors are extracted, adopting the same approach for extracting visual features as follows.
H (hl K
(9)
4.3 Gesture-aware Word Embedding Layer
To obtain textual features h sensitive to gesture information, we update the pre-trained RoBERTa word embedding weights (Liu et al.. 2019) with updated disfluency representations f as follows.
f h ( (RoBERTa(e)) E R
(10)
h 3RoBERTa(e)) Ra
(11)
Subsequently, the final multimodal representation for is derived as follows.
4C I.STM hG1) LSTM (cf, M++1 (4)
H; (h, A (/00
4.2.2 Cross-relation Aggregation
(12)
To update the features of the target node V. adopt GraphSAGE (Hamilton al., 2017). widely used GNN model that supports batch training without requiring updates across the entire graph. This recursive approach involves updating the embeddings of each node by aggregating information from its immediate neighbors using aggregation function at each search depth (k)) Specifically, the representation hyt of node V is updated by combining hyy with information obtained from h "N(rY> which represents the neighboring nodes of V at step k. The initial output is bn lv. and the series of updating processes are defined follows.
4.4 Multimodal Fusion Encoder
To fuse multimodal representations, utilize Multimodal Transformer (Tsai al., 2019) that effectively integrates multimodal features by capturing contextual information. We employ two cross-attention layers and a self-attention mechanism for generating each modality feature as shown Figure 7 in Appendix. For example, when the model generates textual feature, each layer (i.e., (V-T)), (A+T)) uses the visual and acoustic representations
the key/value pairs, respectively, and textual representation as the query vector. After concatenating two features from each layer, a self-attention Transformer fuses them to generate cross-modal representation UT, Next, the model concatenates three cross-modal features (i.e., UT.U" (AA) to derive the final multimodal representation UTV A as follows (See Tsai al. (2019) for more details).
(k) (N(V) NodcAGG* ((h vu EVE
(5)
h - -ww (h @ ))(m))))
(6)
the target node has multiple relations from different types of source nodes {r,})=1 N(w)] (
UTVA Uf e UY e U.A
(13)
V has heterogeneous representations as follows.
(k) hy [hy"), "iiv",,y"" h. "3
Finally, we add the pooled embeddings of [CLS] token in h& and the UVA as follows.
(7)
where is the number of source node types among Since latent
Ú - U-U h CLS
(14)
the neighbors of target node information from distinct source node embeddings can affect the target node representation differently, we aim to aggregate the fine-grained interplay between them without losing valuable knowledge by maintaining independent distributions. Finally, we derive feature of node V at step l follows.
4.5 Aphasia Type Prediction
To predict the types of aphasia for c, the final prediction vector is generated as follows.
ÿ ((RELU(F(Ú))
(15)
where F a dully-connected layer and ReLU is an activation function. Finally, the cross-entropy
f (k) HeteroAGG (1()
(8)

---

loss is calculated using the probability distribution y and classification score y obtained follows.
Table 2: Comparison performance between the pro- posed model and baseline models, with results averaged over Group Stratified 5-fold cross-validation. Results marked with an asterisk (*) indicate statistical signifi-
L= Σ ve logig
(16)
cance compared to MAG (p 0.05) according to the
Wilcoxon's 's signed rank test.
where b is the batch size.
Label (Fl (FI-soore)
Model
Non
Non
Total
Ctrl
Flu
5 Experiments
Com.
Flu.
SVM
0.338
0.670
0.000
0.000
0.000
5.1 Experimental Settings
Aphasiu&
RF
0.742
0.871
0.719
0.000
0,400
Dementia
DI
0.688
0,824
0.652
0.090
0.285
Table 10 in Appendix shows that we split the dataset into the train and test sets with an 8:2 ra- tio. In all our experiments, we ensure that users in the test set are entirely disjoint and do not overlap with those in the training set. For reproducibility, detailed experimental settings are summarized in Appendix B.I.
Delection
LR 0.655
0.773
0.696
0.000
0.000
AdaBoost
0.719
0,882
0.662
0.000
0,303
MISA
0.761
0.899
0.741
0.000
0,349
0.761
0.885
0,750
0.000
0.410
Fusion
MAG
0.725
0.838
0.698
0,000
0.514
SP-Tram.
0.756
0.893
0.742
0,000
0.324
30 Tok.
0.732
0.905
0.676
0,303
0.446
Ours
50 Tok.
0.842
0.949
0.840*
0.125
0.530
Table 3: Ablation study to examine the effectiveness of $4.2 Speech-GGsture Graph Encoder and $4.4 Multi- modal Fusion Encoder.
5.2 Baselines
To conduct extensive performance comparisons, we consider two categories of baseline methods: (i) aphasia & dementia detection models and (ii) mul- timodal fusion baselines. A detailed explanation of the baselines is summarized in Appendix B.2.
Component
Prec
Rec
F1
LSTM
0.777
0.790
0.780
Node agg.
Mean
0.722
0.743
0.722
Pool
0.781
0,807
0.787
BiLSTM
0.837
0.852
0.842
Mean
0.785
0.763
0.771
Aphasia & Dementia Detection Baselines. Since predicting aphasia types has yet to be explored, we compare the approaches from the related tasks that only detect the presence or absence of aphasia. Ad- ditionally, compare the popular models for de- mentia detection: (i) Logistic Regression (LR) (Cui al., 2021; Syed al., 2020), (ii) Support Vec- tor Machines (SVM) (Qin al., 2018b; Rohanian et al., 2021), (iii) Random Forest (RF) (Cui al., 2021; Balagopalan et al., 2020), (iv) Decision Tree (DT) (Qin et al., 201 8b; Balagopalan and Novikova, 2021), and (v) AdaBoost (Cui et al., 2021).
Hetero
Sum
0.759
0.774
0.761
agg.
Max
0.794
0,803
0.783
Min
0.837
0.852
0.842
Concat
0.746
0.609
0.568
Multiply
0.776
0.808
0.784
Fusion
Add
0.789
0.793
0.751
SP-Trans.
0.757
0.781
0.758
MulT
0.837
0,852
0.842
predicting aphasia types. We observe that the pro- posed model can relatively accurately predict mi- norities such as Non Comprehensiin (N=191) than other baselines. We also investigate how the number of tokens decided in creating an aligned multimodal dataset (Figure 1) affects the model performance. As demonstrated in Table 2, we find that the Fl-score is higher for the model using 50 tokens than 30 tokens, which indicates the im- portance of comprehending longer sequences for accurate prediction of aphasia types. While the utilization of 30 tokens improves the performance of Non Comprehension, we attribute this im- provement to the larger size of the dataset based on 30 tokens, as indicated in Table in the Appendix.
Multimodal Fusion Baselines. We compare the proposed model with the following four exist- ing multimodal fusion methods: (i) MulT (Tsai al., 2019), (ii) MISA (Hazarika et al., 2020), (iii) MAG (Rahman et al., 2020b), and (iv) SP- Transformer (Cheng et al., 2021).
6 Results
6.1 Model Performance
Table 2 shows the weighted average F1 F1-score of the baselines and the proposed model across the aphasia types. The proposed model outperforms all the baselines (FF 84.2%) regardless of the apha- sia types. Specifically, deep learning-basee mul- timodal fusion models perform better than mod- els that simply concatenate multimodal features in
6.2 Ablation Study
We perform ablation study to examine the effec- tiveness of each component of the proposed model. Analysis on Model Components. As described

---

in Table 3, the model performance significantly improves when we use the BiLSTM node aggre- gator (Tang et al., 2020). Given that our dataset structured on a sequence level, RNN-based models exhibit higher performance with their capacity to capture long-term dependencies. Also, the Min het- erogeneous aggregate function (Wang et al., 2019) can effectively learn relationships between sources from multiple modalities. We also compare the multimodal fusion models, including Transformer- based multimodal fusion models (Tsai al., 2019; Cheng et al., 2021) as a Multimodal Fusion En- coder. As shown in Table 3, MulT (Tsai et al., 2019) model performs better at 84.2% of Fl-score. suggesting its capability to generate a more infor- mative multimodal representation.
Table 4: Comparison of the impact $4.2 Speech-
Gesture Graph Encoder and $4.3 Gesture-aware Word Embedding Layer on cach modality.
Component
Metrics
Graph Encoder
Update Embed.
Modality
Prec
Rec
F1
X
X
Text
0.692
0.727
0.700
X
X
Acoustic
0.455
0.566
0.501
X
X
Visual
0.591
0.689
0.629
X
X
T+V+A
0.749
0.786
0.765
X
T+V+A
0.774
0.814
0.792
V
T+V
0.767
0.781
0.773
V
T+A
0.771
0.740
0.747
V
T+V+A
0.837
0.852
0.842
improves when using refined word embeddings (F1 84.2%) compared just pre-trained word em- beddings (FI 79.2%). This implics that gesture- sensitive text features can help the multimodal fu- sion encoder assign different weights to each modality considering the meaning of words.
Analysis on Different Modalities. To analyze the significance of each modality in detecting aphasia types, perform analysis of unimodal models trained with each modality. For unimodal apha- sia type detection, we solely utilized the unimodal Transformer encoder. As shown in Table 4, the model trained with text features performs better (F1 70.0%). This indicates that linguistic features are more informative for identifying language im- pairment disorders (Cui et al., 2021; Chen et al., 2021). Additionally, visual features outperform acoustic features (F1 62.9%), suggesting that PWA exhibit distinct characteristics in their gesture ex- pression. When constructing a graph G. employing only the edge f between disfluency and gesture tokens results in higher performance (F1 77.3%) compared to using the edge E between disflu- ency and audio tokens (FI 74.7%), as described in Table 4. However, considering all three modalities together shows the best performance, which reveals that learning linguistic features along with visual and acoustic characteristics and their relations is more effective than relying solely on one modality.
We further conduct experiments to determine the optimal number m of disfluency token nodes Vy in the graph G, ranging from 50 to 300. Figure 3 il- lustrates the weighted average F1 /PPrecision/Reaall scores for identifying aphasia types across the dif- ferent m disfluency keywords. The performance improves as more keywords are included, but no further enhancement is observed beyond 150 key- words. However, our analysis demonstrates that us- ing 100 keywords produces better results compared to using 300 keywords. We believe that extracting important disfluency keywords will result in im- proved outcomes. In our future work, plan collaborate with clinical experts, including speech- language pathologists and neurologists, to acquire valuable clinical insights and guidance.
Model Performance
Analysis on Speech-Gesture Graph Encoder. We next explore the effectiveness of the proposed Speech-Gesture Graph Encoder to understand how the multimodal features specific to each aphasia type are helpful in performance. We first find that using multimodal features from the graph en- coder helps to improve prediction performance (FI 79.2%) compared to without graph encoder (F1 76.5%) as described in Table 4. We attribute this the strength of the graph neural network model, which can learn better representations from cross- modal relations. Also, the performance notably
0.6
Prec
0.75
* Recall
F1
50
100
150
200
250
300
The Number of Disfluency Tokens (m)
Figure 3: Performance of the model by the number of disfluency tokens (m))
Analysis on Different Gender. Earlier clinical research indicates that different gender is associ- ated with aphasia severity, including communica-

---

162 Broca's Aphasia
C
382
Broca's Aphasia
Model 9
(+)
Model 9
(5)
Ours
(b)
Ours
(b)
w/o Graph
w/o Graph
(c)
(d)
One, and
clock
strikes,
[*]
boom,
boom,
get
up
1599: Wernicke's Aphasia
C740 Wernicke's Aphasia
Model y
[a]
Model y
Ours
(b)
(b)
w/o Graph
w/o Graph
[c)
[d]
there's
[*]
ball, eventually there's
house
[*]
too.
She
SBW
[*)
Figure 4: Visualization of the crossmodal attention matrix from the V L network of the Multimodal Fusion
Encoder for the proposed model with (b) or without (c) the speech-gestie seit--eeturreee graph encoder. (a) presents the differences in the spatial position of the right wrist's 's landmark compared to the previous frame, where positive and negative values indicate upward/rightwaad and downward/leftward movements.
Table 5: Gender differences in the model performance.
6.3 Qualitative Analysis
Train
Test
Prec
Rec
Fl
In this section, we perform qualitative analysis on four representative cases involving different types of aphasia, especially Broca's (i.e., C162, C382) and Wernicke's s aphasia (i.e., C159, c740). As shown Figure 4, we compare the cross-modal attention matrix from the V + L network of the Multimodal Fusion Encoder in the proposed model with or with- out the speech-gestune graph encoder. Each atten- tion weight can be interpreted the relevance of the visual and the textual features. Note that Fig- ure in Appendix visualizes the standard deviation of pose landmarks individuals, which reveals that the Non-Fluent type (e.g.. Broca's 's aphasia) ex- hibits a broader range of activity compared to the Non- Comprehension type (e.g., Wernicke's 's apha- sia) (Lanyon and Rose, 2009).
Both
0.837
0.852
0.842
Both
Female
0.889
0.890
0.885
Male
0.795
0.820
0.805
Both
0.718
0.761
0.724
Female
Female
0.757
0.801
0.772
Male
0.668
0.727
0.683
Both
0.795
0.771
0.777
Male
Female
0.805
0.758
0.773
Male
0.794
0.782
0.785
tion impairment and lower scores on specific sub- tests (Sharma al., 2019). To validate this observa- tion, we train and validate our model using separate datasets of either male or female participants (Ta- ble 5). The model performs better when trained on the male dataset (F1 77.7%) than the female dataset (F1 72.4%). despite a larger training size from females (Table 10). These findings suggest that the linguistic impairment caused by aphasia is more pronounced in male patients, reported in previous studies (Yao al., 2015). Thus, we believe the model can learn linguistic impairment markers better when trained with the male dataset.
We find that the model without graph encoder assigns the same attention regardless of the apha- sia types when significant physical actions are dis- played. However, our proposed model can predict Broca's 's aphasia (i.c., C162.Ca52) by assigning higher attention weights if meaningful motions are cap-

---

tured. For example, 6162 points the index finger to express the 'clock' and c382 indicates upward move- ments during phrases like gel up'. This implies that PWA with Broca's aphasia leverage gestures as an additional means of communication (Preisig et al., 2018; de Kleine et al., 2023), and the pro- posed model can accurately recognize the meaning of gestures associated with the text. By contrast, in the cases of Wernicke's ' aphasia (C159, C740), where motions are commonly small, the model assigns higher attention scores to any noticeable motion change, despite the meaning of the movement.
(GDPR) (Regulation, 2018). These issues include addressing commercial purposes, handling scien- tific data, obtaining informed consent, ensuring deidentification, and maintaining a code of con- duct. Considering ethical concerns, we did not use any photographs and personal information of the participants from the AphasiaBank.
Limitations
First, traditional aphasia classification schemes, such as the WAB (Western Aphasia Bat- tery) (Kertesz, 2007), have faced criticism because patients' symptoms often do not fit into a single type, and there is overlap between the different classes (Caramazza, 1984; Swindell et al., 1984). However, the WAB still offers a means to catego- rize patients based on their most prominent symp- toms. Furthermore, datasets labeled using the WAB scheme essential for studies that rely on meth- ods requiring substantial amount of training data. Second, the previous study highlighted the po- tential of identifying repetitive mouth patterns to detect speech impairments (Einfalt et al., 2019). However, capturing mouth landmarks on the sub- jects was challenging due to the considerable dis- tance at which the videos were recorded in the AphasiaBank dataset. Considering the significant role of mouth information in human communica- tion (Busso et al., 2004), we anticipate that incor- porating mouth information in conjunction with gesture information will improve performance.
We believe the proposed model with a speech- gesture graph encoder can effectively generate aphasia type-specific multimodal features for pre- dicting aphasia types, as demonstrated in case study, Thus, the proposed model can be used for screening and identifying individuals with aphasia to prioritize early intervention for clinical support.
7 Conclusion
this paper, our approach utilizing GNNs proves effective in generating multimodal features specific to each aphasia type for predicting aphasia types. The extensive experiments demonstrated state-of- the-art results, with visual features outperforming acoustic features in predicting aphasia types. We plan to provide the codes for reproducibility, facili- tating further research in this area. The qualitative analysis offered valuable insights that can enhance clinicians' understanding of aphasia patients, lead- ing to more comprehensive assessments and timely interventions. Overall, our research contributes to the field of aphasia diagnosis, highlighting the importance of using multimodal information and graph neural networks. We believe our findings will drive advancements in speech and language therapy practices for individuals with aphasia.
Additionally, we utilized computational meth- ods to extract disfluency-relatee keywords from frequent occurrences, which were then used for constructing the graph. However, our analysis in Figure 3 demonstrated that using 100 tokens pro- duced better results compared to using 300 tokens. Consequently, we believe that extracting significant disfluency tokens guided by expert advice will re- sult in improved outcomes. In our future work, we intend to collaborate with clinical experts, includ- ing speech-languaa pathologists and neurologists, to acquire valuable clinical insights and guidance.
Ethics Statement
We sourced the dataset from AphasiaBank with Institutional Review Board (IRB) approval and strictly follow the data sharing guidelines provided by TalkBank 4, iincluding the Ground Rules for all Talk databases based on American Psycho- logical Association Code of Ethics (Association et al., 2002). Additionally, our research adheres to the five core issues outlined by TalkBank in accor- dance with the General Data Protection Regulation
Acknowledgments
This rescarch was supported by the Ministry of Education of the Republic of Korea and the Na- tional Research Foundation (NRF) of Korea (NRF- 2022$1 1 A5A8054322) and the National Research Foundation of Korea grant funded by the Korea government (MSIT) (No. 2023R I A2C2007625).
"SKKU2022-11-02 "hapss/halhammk..g//anrrcchceiit

---

References
Alfonso Caramazza. 1984, The logic of neuropsycho- logical research and the problem of patient classifica- tion in aphasia. Brain and language, 21(1):9-20.
Martin L Albert, Robert W Sparks, and Nancy A Helm. 1973. Melodic intonation therapy for aphasia. Archives of neurolog,, 1902)130--311
Gerasimos Chatzoudis, Manos Plitsis, Spyridoula Sta- mouli, uthanasia-Lida Dimou, Athanasios Katsama- nis, and Vassilis Katsouros. 2022. Zero-shot cross- lingual aphasia detection using automatic speech recognition. arXiv preprint arXiv:2204.00440
American Psychological Association et al. 2002. Ethi- cal principles of psychologists and code of conduct. American psychologist, 7(12):1060-1077
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449- 12460.
Jun Chen, Jieping Ye, Fengyi Tang, and Jiayu Zhou. 2021. Automatic detection of alzheimer's disease using spontaneous speech only. In Proceedings of Conference of the International Speech Communica- rion Association (Interspeech 2021).
Aparna Balagopalan, Benjamin Eyre, Frank Rudziez, and Jekaterina Novikova. 2020. To bert or not bert: Comparing speech and language-based approaches for alzheimer's disease detection. Proc. Interspeech 2020, pages 2167-2171.
Junyan Cheng. Iordanis Fostiropoulos, Barry Boehm, and Mohammad Soleymani. 2021. Multimodal phased transformer for sentiment analysis. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 2447- 2458.
Apara Balagopalan and Jekaterina Novikova. 2021. Comparing Acoustic-Based Approaches for Alzheimer's 's Disease Detection. In Proc. Interspeech 2021, pages 3800-3804.
Melinda Corwin. 2013. Aphasiabank english protocol pwa star corpus.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguis- tic annotation workshop and interoperability with discourse, pages 178 1866
Xia Cui, Amila Gamage, Terry Hanley, and Tingting Mu. 2021. Identifying indicators of vulnerability from short speech segments using acoustic and tex- tual features. In Interspeech, pages 1569-1573.
Marjory Day, Rupam Kumar Dey, Matthew Baucum, Eun Jin Pack, Hyejin Park, and Anahita Khojandi. 2021. Predicting severity in people with aphasia: A natural language processing and machine learning approach. In 2021 43rd Annual International Confer- ence of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 2299-2302. IEEE
Helen Bird and Sue Franklin. 1996. Cinderella revisited: A comparison of fluent and non-fluent aphasic speech, Journal of newrolinguistics 133::6872206.
Mary Boyle. 2013. Aphasiabank english protocol pwa msu corpus.
Naomi de Kleine, Miranda L Rose, Michael Weinbom, Robert Knox, and Nicolas Fay. 2023. Does gesture improve the communication success people with aphasia?: A systematic review. Aphasiology, pages 1-25.
Paul Broca et al. 1861. Remarks on the seat of the fac- ulty of articulated language, following observation of aphemia (loss of speech). Bulletin de Sociééé Anatomique, 6:330-57.
Michael Dean. 2021. Aphasiahank english protocol pwa ucl corpus.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth Narayanan, 2008 lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359,
Moritz Einfalt, Rainer Lienhart, Matthew Lee, and Lyn- don Kennedy. 2019. Detecting speech impairments from temporal visual facial features of aphasia pa- tients. In 2019 IEEE Conference Multimedia In- formatioo Processing and Retrieval (MIPR), pages 103-108, IEEE
Carlos Busso, Zhigang Deng, Serdar Yildiri,, Mur- taza Bulut, Chul Min Lee, Abe Kazemzadeh, Sung- bok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of the 6th international conference on Multimodal interfaces, pages 205-211.
Hanane El Hachioui, Evy G Visch-Brink, Lonneke ML de Lau, Mieke WME van de Koendeenenn, Femke Nouwens, Peter Koudstaal, and Diederik WJ Dippel. 2017. Screening tests for aphasia in patients with stroke: a systematic review. Journal of neurol- ogy, 264:211-2200
Katy Bynek. 2013. Aphasiabank english protocol pwa acwt corpus.
Roberta J Elman. 2011. Starting aphasia center? In Seminars in speech and language, volume 32, pages 268-272, 2 Thieme Medical Publishers.
Gilson Capilouto. 2008 Aphasiabank english controls capilouto corpus.

---

Roberta J Elman. 2016. Aphasia centers and the life par- ticipation approach to aphasia. Topics in Language Disonters, 36(2):154-167.
Susan Jackson. 2013. pwa kansas corpus.
Aphasiabank english protocol
Daniel Kempler. 2013. Aphasiabank english protocol pwa kempler corpus.
Florian Eyben, Klaus R Scherer, Bjöm W Schuller, Johan Sundberg. Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al. 2015. The geneva min- imalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE transactions on affective computing, 7(2):190-202
Andrew Kertesz. 2007. revised.
Western aphasia battery-
Jacquie Kurland. 2013. Aphasiabank english protocol pwa kurland corpus.
Florian Byben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open- source audio feature extractor. In Proceedings of the 18th ACM international conference on Mullimedia, pages 1459-1462.
Lucette Lanyon and Miranda L Rose. 2009. Do the hands have it? the facilitation effects of arm and hand gesture on word retrieval in aphasia. Aphasiology, (3(7-8):009-822
Duc Le, Keli Licata, and Emily Mower Provost. 2018. Automatic quantitative analysis of spontaneous apha- sic speech. Speech Communication,, 100:1-12.
Yasmeen Faroqi-Shab and Lisa Milman. 2018. Com- parison of animal, action and phonemic fluency in aphasia. International journal of language & com- munication disorders. 53(2):370-384
Duc Le and Emily Mower Provost. 2016. Improving automatic recognition of aphasic speech with aphasi- abank. In Interspeech, pages 2681-2685.
Margaret M Forbes, Davida Fromm, and Brian MacWhinney 2012. Aphasiabank: A resource for clinicians. In Seminars in speech and language, vol- ume 33, pages 217-222. Thieme Medical Publishers.
Zijie Lin, Bin Liang, Yunfei Long, Yixue Dang, Min Yang, Min Zhang, and Ruifeng Xu. 2022. Modeling intra- and inter-modal relations: Hierarchical graph contrastive learning for multimodal sentiment anal- ysis. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 7124- 7135, Gyeongju. Republic of Korea. International Committee on Computational Linguistics.
Kathleen C Fraser, Jed A Meltzer, Naida L Graham, Carol Leonard, Graeme Hirst, Sandra E Black, and Elizabeth Rochon. 2014. Automated classification of primary progressive aphasia subtypes from narrative speech transcripts. cortex, 55:43-60.
Julius Fridriksson. 2013. Aphasiahank english protocol pwa fridriksson corpus.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprinl arXiv: 1907.1907.1192
Kathryn Garrett. 2013. Aphasiabank english protocol pwa garrett corpus.
William L Hamilton, Rex Ying. and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings the 31st International Con- ference Neural Information Processing Systems, pages 1025-1035.
Πlya Loshchilov and Frank Hutter. 2017.
Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05101
Devamanyu Hazarika, Roger Zimmermann, and Sou- janya Poria. 2020. Misa: Modality-invarian and- specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM Interna- tional Conference on Multimedia, pages 1122-1131.
Jérôme Louradour. 2023. whisper-timessamped. https: //gi thub. com/ www/wwontooaeeeee hinto-as/whispertiine
Camillo Lugaresi, Jiuqiang Tang. Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. 2019. Mediapipe: framework for building perception pipelines. arXiv preprint arXiv:1906.08111
Nancy Helm Fata--ssabrooks. 2002. Cognition and aphasia: a discussion and a study. Journal of communication disorders, 35(2)171-1866
Fabiane Hirsch Kruse. 2013. Aphasiahank english pro- tocol pwa tueson corpus.
Brian MacWhinney. 2013. Aphasiabank english proto- col pwa corpus.
Elizabeth Hoover. 2013. Aphasiabank english protocol pwa bu corpus.
Brian MacWhinney, Davida Fromm, Margaret Forbes, and Audrey Holland. 2011. Aphasiabank: Methods for studying discourse. Aphasiology, 25(1 1286 1307.
Judy Illes. 1989. Neurolinguistic features of sponta- neous language production dissociate three forms of neurodegenerative discase: Alzheimer's, hunt- ington's, s, and parkinson's. Bruin and language, 37(4):628-642.
Denise McCall. 2013. Aphasiabank english protocol pwa scale corpus.

---

Namita Multani, Sebastiano Galantueci, Stephen M Wilson, Tal Shany-Ur Ur. Pardis Poorzand, Matthew Growdon, Jung Yun Jang, Joel H Kramer, Bruce I. Miller, Katherine P Ramkin, et al. 2017. Emotion detection deficits and changes in personality traits linked to loss of white matter integrity in primary progressive aphasia. Neurolmage: Climical, 16:447- 454.
Morency, and Ehsan Hoque, 2020a. Integrating mul- timodal information in large pretrained transform- ers. In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages 2359-2369, Online. Association for Computational Linguisties.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Ami- rAli Bagher Zadeh, Chengfeng Mao, Phiilip Morency, and Ehsan Hoque. 2020b. Integrating mul- timodal information in large pretrained transform- ers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359 2369, Online. Association for Computational Linguisties.
Maria Muñooz 2013. Aphasiabank english protocol pwa tcu corpus.
Sabiha Parveen and Siva Priya Santhanam. 2021. Speech-language pathologists' perceived competence in working with culturally and linguistically diverse clients in the united states. Communication Disor- ders Quarterly, 42(3): 16677
Amy Ramage. 2013. Aphasiabank english protocol pwa unh corpus.
Xiangdong Pei, Ke Zuo, Yuan Li, and Zhengbin Pang- 2023. A review of the application of multi-modal deep learning in medicine: Bibliometrics and future directions. International Journal of Computational Intelligence Systems, 16(1):1-20.
General Data Protection Regulation. 2018. General data protection regulation (gdpr). Intersoft Consulting, Accessed in October, 24(1).
Janet Richardson. 2008. Aphasiabank english controls richardson corpus.
Basil C Preisig. Nočmi Eggenberger, Dario Cazzoli, Thomas Nyffeler, Klemens Gutbrod. Jean-Marie An- noni, Jurka R Meichtry, Tobias Nef, and René M Müri 2018. Multimodal communication in aphasia: Perception and production of co-speech gestures dur- ing face-to-face conversation. Frontiers in human neuroscience, 12:200.
Morteza Rohanian, Julian Hough, and Matthew Purver. 2021. Alzheimer's ' Dementia Recognition Using Acoustic, Lexical, Disfluency and Specch Pause Fea- tures Robust Noisy Inputs. In Proc. Interspeech 2021, pages 3820-3824.
Saryu Sharma, Patrick M Briley, Heather Harris Wright, Jamie I. Perry, Xiangming Fang, and Charles Ellis. 2019. Gender differences in aphasia outcomes: Evi- dence from the aphasiabank. International Journal of Language & Communication Disorders, 54(5):806- 813.
Ying Qin, Tan Lee, Siyuan Feng, and Anthony Pak- Hin Kong. 2018a. Automatic speech assessment for people with aphasia using tdnn-bistm with multi-task learning. In Interspeech, pages 34 3418-3422.
Ying Qin, Tan Lee, and Anthony Pak Hin Kong. 2018b. Automatic speech assessment for aphasic patients based on syllable-level embedding and supra- segmental duration features. In 2018 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5994-5998 IEEE
Maura Silverman. 2013. Aphasiabank english protocol pwa tap corpus.
Brielle Stark. 2022. Aphasiabank english ce stark cor- pus.
Brielle Stark. 2023. Aphasiabank english aphasia neural corpus.
Ying Qin, Tan Lee, and Authony Pak Hin Kong. 2019a. Automatic assessment of speech impair- ment in cantonese-speaking people with aphasia. IEEE journal of selected topics in signal processing, 14(2)3331-45.
Carol S Swindell, Audrey L Holland, and Davida Fromm. 1984, Classification of aphasia: Wab type versus clinical impression. Clinical Aphasiology: Proceedings of the Conference 1984, pages 48-54. BRK Publishers.
Ying Qin, Tan Lee, and Anthony Pak Hin Kong. 2019b.
Combining phone posteriorgrams from strong and weak recognizers for automatic speech assessment of people with aphasia. In ICASSP 2019-2019 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pages 6420 6424. IEEE.
Muhammad Shehram Shah Syed, Zafi Sherhan Syed, Margaret Lech, and Elena Pirogova. 2020. Auto- mated screening for alzheimer's 's dementia through spontaneous speech. Proc. Interspeech 2020, pages 2222-2226.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and llya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. arXiv preprint 2r1iv:212..0333
Gretchen Szabo. 2013. Aphasiabank english protocol pwa adler corpus.
Pingie Tang, Meng Jiang, Bryan Ning Xia, Jed W Pitera, Jeffrey Welser, and Nitesh Chawla. 2020. Multi-label patent categorization with non-local
Wasifur Rahman, Md Kamrul Hasan. Sangwu Lee, Ami- rAli Bagher Zadeh, Chengfeng Mao, Louis- Phils-Phi

---

attention-basee based graph convolutional network. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 9024-9031.
Jingfan Yao, Zaizhu Han, Yanli Song, Lei Li, Yun Zhou, Weikang Chen, Yongmei Deng, Yongjun Wang, Yumei Zhang, et al. 2015. Relationship of post-stroke aphasic types with sex, age and stroke types. World Journal of Neuroscience, y:01):34.
Cynthia Thompson. 2013. Aphasiabank english proto- col pwa thompson corpus.
Jeewoo Yoon, Chaewon Kang, Seungbae Kim, and Jiny- oung Han. 2022. D-vlog: Multimodal vlog dataset for depression detection. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12226-12234.
Iván G Torre, Mónica Romero, and Aitor Álvarez. 2021. Improving aphasic speech recognition by using novel semi-supervised learning methods aphasiabank for english and spanish. Applied Sciences, 11879)8872.
Amir Zadch, Minghai Chen, Soujanya Poria, Erik Cam- bria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv: 7707.07250.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for un- aligned multimodal language sequences. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558 -6666.9.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis- Philippe Morency. 2016. Multimodal sentiment in- tensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82-4 88.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et 2019. Deep graph library: A graph- centric, highly-performant package for graph neural networks. arXiv preprint 1rXiv:1909.011
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2022. HEGEL: Hypergraph transformer for long document summarization. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 10167-10176, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Mohammad Wasay, Ismail A Khatri, and Subhash Kaul. 2014. Stroke in south asian countries. Nature reviews neurology, 1053):15--44.
Jochen Weiner, Mathis Engelbart, and Tanja Schultz. 2017. Manual and automatic transcriptions in de- mentia detection from speech. In Interspeech, pages 3117-3121.
Janet Whiteside. 2013. Aphasiabank english protocol pwa whiteside corpus.
Darlene Williamson. 2013. Aphasiabank english proto- col pwa williamson corpus.
Stephen Wilson. 2021. Aphasiabank english protocol pwa aproesa corpus.
Stephen M Wilson, Dana K Eriksson, Sarah M Sch- neck, and Jillian M Lucanie. 2018. A quick apha- sia battery for efficient, reliable, and multidimen- sional assessment of language function. PloS one, 13(2)xe0192773.
Linda Wozniak. 2013. Aphasiabank english protocol pwa wozniak corpus.
Heather Wright. 2013. Aphasiabank english protocol pwa wright corpus.
Jianing Yang. Yongxin Wang. Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadch, Soujanya Poria, and Louis-Philippe Morency. 2021. Mtag: Modal- temporal attention graph for unaligned human mul- timodal language sequences. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: man Language Technologies, pages 1009-1021.

---

A Dataset
A.2 Aphasia Types
As shown in Figure 5 in Appendix, aphasia can be classified into eight types based on the Western Aphasia Battery (WAB) (Kertesz, 2007), a stan- dard protocol for categorizing patients based on the most prominent symptoms and severity, and they are next broken down by the ability to auditory comprehension and fluency skills.
A.1 AphasiaBank Dataset
As illustrated in Table 6, we sourced our dataset from the AphasiaBank (MacWhinney et al., 2011; Forbes et al., 2012), a shared multimedia database used by clinicians for aphasia research S.
Table 6: The corpora we used from AphasiaBank
Corpus
Size
A.3 Distluency Keywords
ACWT (Byack, 2013)
Aphasia Cameer West Tous
Adler (Srabo, 2013)
Adier Aphasia Center
Table 9 shows the examples of disfluency keywords
APROCSA (Wilsen, 2021)
Vande:hith University Medical Center
(Hooves, 2013)
Boston University
ʃ.
Capiluato (Capllato, 2008)
Univenity of Kantucky
CC-Surk (Surk, 2022)
file for
Table 9: The examples of disfluency keywords J
CMU (MacWhinney, 2013)
Camagic Mellon Uriverstty
Elman (Flmon, 2011, 2016)
Aphiscia Cenier California
Dislluency Tokens
Indriksson (Fridilkoon 2013)
University of South Caroliaa
TT. 'the'. 'and", 'to.. 'um', 'she", "s'. "her", "was". 'they". "so". 'that", 'of", "cinderella", 'it', "uh', 'in', 'i', 'be", 'but', "all'. 'is', 'had', 'then', 'ball'. "go". 'with', 'prince', 'this', "one'. ''on. "you', 'two'. 'there'. 'were'. 'for', 'going". 'kingg. "be", 'know", "nut"" 'at', "no", "not", "like', 'slipper'. 'get', 'have'. 'dress', "gor". 'up". "went'. "very'. 'who', 'glass". "because", 'or', "time", "oh", "stepmotther', 'into', 'do", 'shes', "what", 'heantiful,, 'back", "its", "hume", 'girl,, "woman", "fairy', "when', 'godmother', "would", 'said', "just', 'as', "has", "little, "shoe', 'well", 'dont", "his', "mother", 'them', 'thughters', 'house', "midnight, 'stepsisters' 'by'. "men', 'bye', 'girls.. "sisters", "mice", "everything", 'fit", "came", 'other', 'gooes'''
Ganett (Ginett, 2013)
Punsbargh, PA
Kansas (Jacksnn 20133
University of Kansas Emseson College
Kempler (Kempper.
Kurland IXurland, 2013)
University of Massoclusztts, Amberst
MSU (Boyle 2013)
Mostclair State University
NEURAL (Stack, 2023)
NEURAL Research Lab, Indiasa University
Richardon (Richardson. 2008)
University
Mesico
SCALE (MoCall,, 2013)
Snyder Carrer for Aghasin liee Echancement
STAR (Corwas, 20131
Stoke Aybasia Recovery Piogram
(Sl/emieemnn, 2013)
Trinngl Aghasia Project
TCU (Matur, 3013)
Texas Christian University
Thompson (1 [Thompson. 2013)
Northwesiem University
Tucton (Hinch Krose, 2013)
Universit of Arizons
(Dean, 2021)
Univensity College London
UMD (Faroqi Stah and Milman, 2018)
University of Maryland
UNH (Ramige, 2013)
Univenitty of New Hampshize University of Central Honda Stroke Comeback IatRACT Arizara State University
Whiteside (Whicside, 2013)
Williamson /Williamson, 2013)
A.4 Analysis on Gesture Difference
Womnisk (Woniikk, 2013)
Wright (Wright, 2013)
We applied ANOVA to assess the differences gesture features among the diffcrent aphasia types groups as well as the control group, as shown in Table 11. This analysis allowed us to determine whether there are significant differences in gesture patterns between the aphasia types.
Table 7: Comparison of the number of samples in 50 Token data and 30 Token data.
Impaired
Aphasia
Sample
Labels
Flu.
Com.
Types
50 tokens (ours)
30 tokens
Cirl.
X
X
Control
1,815
3,157
Anomic
927
1,639
Hu.
X
X
Conduction
424
753
B
Experiment
Trans, Sensory
3
5
Non Com.
Wernicke
188
331
B.1 Experimental Settings
Trans. Motor
44
Non-Flu.
x
Broca
271
518
As shown in Table 10, we split the dataset into the train and test sets with a 8:2 ratio. We tune hyperparameters based on the highest Fl score ob- tained from the cross- volss-vali set for the mod- els. We use the grid search explore the dimen- sion of hidden state U3 E {64, 128, 256, 512, 768). number of LSTM layers n E 21,2,5). 5), dropout E {0.1, 0.2, 0.3, 0.4, 0.5). and initial learning rate Ir E {1e- 5, 2c 5, 30 5, 5e 5). The opti- mal hyperparameters were found to be: U2 = 768,
Total
6,447
Table 8: The Word Error Rate (WER) across Aphasia Types for all participants.
Impaired
Aphasia Types
Average WER
Labels
Flu.
Com.
Raw
Pre-proc.
Ctrl.
Cuntrol
0.332
0.18
Anomic
Flu.
0.714
0.623
Conduction
Trans. Sensory
Non-Com
0.607
0.542
Wemicke
Trans. Motor
Nom-Flu.
0.967
0.974
Table 10: Gender distribution in the dataset.
Broca
Total
0.615
0.521
Gender
Train
Test
Total
All
2.947
704
3,651
Female
1,633
327
1,960
Male
1.314
377
1,691

---

Types of Aphasia
Comprehends?
Comprehands?
Regeats?
Repeats?
Repests?
Repeatis?
Anomic
Conduction
Trans. Sensory
Wermicke
Trans. motor
Broce
Trans. mised
Global
Figure 5: Types of Aphasia
Table II: Comparison of gesture characteristics among different groups based on Aphasia types using ANOVA. The table below presents the p-values from comparing the gesture features of the top 10 tokens with the highest occurrence. (** means p-value < 0.001 / means p-value < 0.05)
Keypoints
[+]
the
and
she
her
they
NOSE
0.000)
0.000
0.000
0.0099
0.000
0,000
0.000 ** 0.000
0.000
0.2223
LEFT BETTLEYE_INNEE
0.000 * 0.000
0.000 * 0.0202
0.000 ** 0,000
0.000 +* 0.000
0.000 ** 0.3069
LEFT_EYE
0.000 * 0.000
0.000 *
0.000 **
0.016
0,000
0.000 00 0.000 . 0.000 -* 0.2201
LEFT EYE_OUTER
0,000 * 0.000
0.000 ** 0.000 **
0.0397
0,000
0.000 ** 0.000
0.000 ** 0,1535
RIGHT BYE INNER
0,2061
0.000
0.0858
0,3253
0.000 ** 0.1
0.0143 * 0.000 =* 0.000 =* 0,4163
RIGHT_EYS
0,000 * 0,000
0.0137
0.2313
0.000 ** 0.1287
0.0094
0.000 ** 0.000
0.243
LEFT_EYE, OOUTER
0.000 ** 0.000 #*
0.000 **
0.0498
0.000 TM 0.0518
0.000 201
0.000 S* 0.000 RM
0,0819
LEFT_EAR
0.000 *+ 0.000 *+ 0.0614
0.7237
0.0562
0.0481 *
0.6547
0.000 *x 0.000 AM
0,9854
LIGHT_EAR
0.000 * 0.000
0.000 . 0.000 *
0.000 * 0.000
0,000 **
0.000 ** 0.000 ** 0,000 **
MOUTIL
0.000 *+ 0.000 *+ 0.000 9 0.000 4 0.0057
0.000 4*
0.000 15
0.000 * 0.000 AM 0.1522
MOUTH RIGHT
0,1881
0.000 *+ 0.0958
0,2863
0.000 * 0.163
0,0185
0.000 ** 0.000 ** 0,4538
LEFT SHOULDER
0.000 * 0.000
0.000 ** 0.000 ** 0.3065
0.000
0,000 **
0.000 * 0.000
0.2222
RIGHT SHOULDER
0.000 #* 0.000
0.000 #* 0.000 *= 0.000 #= 0.000
0,000 201
0.000 SX
0.000 S9:
0.000 P:::
LEFT ELBOW
0.000 ** 0.000
0.000
0.000 ** 0.6054
0.000
0,000 SM
0.000 SM
0.000 S9
0.000 99
RIGHT FLBOW
0.000 *+ 0.000
0.000
0.000
0,000 391
0.000 SR
0.000 S::
0.000 P0
LEFT_ WRIST
0.000
0.000
0.000
0.000
0,2786
0.000
0.000
0.000
0.000
RIGHT WRIST
0.000
0.000
0.000
0.000
0.000
0.000
0,000
0.000
0.000
0.000
LEFT PINKY
0.000
0.000
0.000
0.000
0.7116
0.000
0.000
0.000
0.000
RIGHT PINKY
0.000
0.000
0.000
0.000
0.000
0.000
0,000
0.000
0.000
0.000
LEFT INDEX
0.000
0.000
0.000
0.000
0.6041
0.000
0.000
0.000
0.000
0.000
RIGHT INDEX
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
LEFT THUMB
0.000
0.000
0.000
0.000
0.4492
0.000
0.000
0.000
0.000
0.000
RIGIT_THIUMD
0.000 0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
n = 0 a 0.1, and Ir = - 5. We implement all the methods using PyTorch 1.12 and optimize with the mini-batch AdamW (Loshchilov and Hut- ter, 2017) with a batch size of 32. We use the Exponential Learning rate Scheduler with gamma 0.001. We train the model on GeForce RTX 3090 GPU for 50 epochs and apply early stopping with patience of 7 epochs.
involves brain neuron degeneration, lcading to over- all cognitive decline and possible language impair- ments. Although both disorders impact language and cognition, a clear understanding of their dif- ferences and similarities is required for individual diagnosis and assessment.
LR (Cui et al., 2021; Syed et al., 2020): We employed a Logistic Regression Classifier with an added L.2 penalty term, using a cost parameter c value of 0.01.
B.2 Baselines
Aphasia & Dementia Detection Baselines. Apha- sia and dementia are neurological disorders that affect language and cognition, leading to difficul- ties in various cognitive functions and communica- tion. Aphasia is caused by specific brain damage, resulting in language impairments, while dementia
SVM+Poly (Qin al., 2018b; Rohanian et al.,
2021): Each subject's text and audio embedding vector is fed a Support Vector Machine with a polynomial kernel using the cost parameter c = 0.01.

---

Table 12: Performance comparisons of the proposed model and baselines including precision, recall and Fi-score.
Cirl. (355)
Flu. (269)
No-Com (31)
No-Flu. (49)
Weighted aeg. (704)704)
Model
Proe
Ree.
Pree
Rec.
F1
Pree
Rec.
Pree
Rec.
Prec
Roc.
F1
SVM + Poly
0.504
1,000
0.670
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.254
0.504
0.338
Aphasia
0.848
0.896
0.747
0.719
0.000
0.760
Otber Disorder
0.831
0.817
0.824
0.660
0.643
0.652
0.083
0.097
0.090
0.263
0.306
0.283
0.693
0.683
0.688
Detection
IR
0.779
0.766
0.773
0.611
0.807
0.6%6
0.000
0.000
0.000
0.000
0.000
0.000
0.627
0.695
0.685
0,839
0.699
0.662
0.257
0.716
MISA
0.846
0,960
0.899
0.725
0.758
0.741
0.000
0.000
0.000
0.785
0.224
0.349
0.758
0.789
0.761
MulT
0.885
0.885
0.385
0.699
0.810
0.750
0.000
0.000
0.000
0.552
0.327
0.410
0.751
0.778
0.761
Multimodal
MAG
0.914
0.775
0.838
0.625
0.792
0.698
0.000
0.000
0.000
0.482
0.551
0.514
0.733
0.732
0.725
SP-
0.921
0.796
0.000
0.784
Ours
0.949
0.840
0.125
0.647
Random Forest (Cui et al., 2021; Balagopalan et al., 2020): The Random Forest classifier has demonstrated improved performance various speech classification tasks, including speech/non- speech discrimination and speech emotion recog- nition.
trained Transformer models for multimodal lan- guage processing. It represents nonverbal behav- ior as a vector with trajectory and magnitude, allowing to shift lexical representations within the pre-trained Transformer model.
SP- Tr-nnssformer (Cheng et al., 2021): SP- Transformer uses sampling function to create a sparse attention matrix, reducing long sequences to shorter hidden states. The model captures interactions between hidden states of different modalities at each layer. To improve efficiency, it employs Layer-wise parameter sharing and Fac- torized Co-Attention, minimizing the impact on task performance.
Decision Tree (Qin et al., 2018b; Balagopalan and Novikova, 2021): For the decision tree model, the text and audio embedding vectors of each subject are used input. The model is trained using the gini criterion.
AdaBoost (Cui et al., 2021): We set the maxi- mum number of estimators 50 and apply the boosting algorithm for AdaBoost.
Multimodal fusion Baselines. To accurately clas- sify types of aphasia, we use multimodal data con- sisting of text, video, and audio. Our aim is to demonstrate the effectiveness of our model com- pared to existing multimodal sentiment analysis baseline models, highlighting its validity in the context of aphasia.
MulT (Tsai al., 2019): The MulT model is designed for analyzing multimodal human lan- guage. It utilizes a crossmodal attention mecha- nism to fuse multimodal information by directly attending to low-level features in other modali- ties.
MISA (Hazaríka et al., 2020): MISA operates two subspaces for each modality: a modality- invariant subspace that captures commonalities and reduces modality gaps, and modality- specific subspace that captures distinctive fea- tures. These representations enable holistic view of multimodal data, which is used for fu- sion and task predictions.
MAG (Rahman et al., 2020b): MAG a method introduced for efficient finetuning of large pre-

---

Control
Fluent
Control
Conduction Aphasia
Anomic Aphasia
6.8
0.06
000
0.66
004
LerT ELBW
0.0
nowr BLeww
cary your
Rovir pebe
LSF caoge
monr, avDe Thea0
eoint 776.80
Lurr Lur_ ELEO
b0g
LEFT Paiy
Roiry, Peeer
LBVT LANEE
Hovey soaer
LIFT Theas
Rioht TRUND
LeTy ELaOW
mort ELaoo
LesT paary
noirz ParY
LEV MoDe
norr ACEE
LAFT tes
RiGHt THAB
Jon-Coopre
Non-Fluent
Wemiike's Aphasia
Trans. Sensory Aphasia
Broca's Aphasia
Trans. Motor Aphasia
0.22
582
0.2
000
0.32
LeIT (aa0
0.3
povt, Batoo
Rionr, FoMT
LeV LUNDE
Fioht, MaCE
LETy THEa
LOVE. cLAWW
Lory, Aente novey 2013 cffe THEME moirt 79000
Lavy aLeWO
raterr wwr.r.sun CLePP
LBTT pale
Lavy THUAD
HIGHT. FREa0
eovnt Paae
Figure Visualization of the standard deviation values pose key points of each aphasia type
Tox
Viay
TwV
Podtina Imbeccing
Exealo
Tel
Audio
Figure 7: Architecture of Multimodal Transformer En- coder (Rahman et al., 2020b)


Reviews:
[{"review_id": "tYj9EVTVoM", "reviewer": "Reviewer_q4Gj", "summary": "", "questions": "The authors should report ASR WER performance in the paper.\n\nThe authors should correct the line ... \"Unlike previous ASR systems focusing on transcribing clean speech, Whisper can capture filler words ...\" -> Most traditional ASR systems are able to well recognize disfluencies as long as they are trained using data that contains disfluencies. Also the claim that Whisper can capture all disfluencies should be verified.  ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments on the experiments. Based on your comments, we further conducted additional experiments. The results of these experiments are included in the answer below and will be added to the final version.\n\n## [Typo Errors]\nWe will review the paper carefully to ensure no typos exist in the final version.\n- (Figure 1) The figure size and clarity should be increased.\n- (Table 4) :  the letters T,V and A in table 4 should be explained. (T: text / V: video (Gesture in this paper), / A: Audio) \n- (L554) ~ female participants. -> ~female participants (Table 5). \n- (L557) Table 1 -> Table 8 in Appendix\n\n## [Whisper Disfluency]\nAs you mentioned earlier, Whisper is highly effective in identifying and removing disfluencies and filler words from transcriptions. In other words, it excels at detecting these linguistic elements, and you can effortlessly extract such words using the straightforward options provided by the Whisper API. \n\nFor a more accurate evaluation, we have reported the Word Error Rate (WER) results along with ASR examples below. \n\n|  Type  |                        | Average WER |                    |\n|:------:|:----------------------:|:-----------:|:------------------:|\n|        |                        |   Raw text  | Preprocessing text |\n| type 0 |         Control        |    0.332    |        0.180       |\n| type 1 |   Anomic/ Conduction   |    0.714    |        0.623       |\n| type 2 | Wernicke/ TransSensory |    0.607    |        0.542       |\n| type 3 |    Broca/ TransMotor   |    0.967    |        0.974       |\n|  total |                        |    0.615    |        0.521       |\n\n\nThe results showed that the average WER for all participants was similar to previous research, with a decrease in WER for less fluent aphasia types (e.g., type 3) [1].\n\nFurthermore, we recognized the considerable impact of preprocessing on WER (i.e., raw text vs. preprocessing text), stemming from the prevalence of fillers and disfluencies within the texts of individuals with language disorders.\n\nNevertheless, our real-world examples show that the ASR system can capture not only the overall context, but also fillers and disfluencies.\nHence, considering the implications for individuals with language disorders, exploring innovative methods for conducting disfluency ASR assessments in future research might be valuable.\n\n[1] Weiner, J., Engelbart, M., & Schultz, T. (2017, August). Dementia Detection from Speech in Manual and Automatic Transcriptions. In Interspeech (pp. 3117-3121).\n\n\n|    | Raw                                                                                                                                                      | Raw_preprocessing                                                                                | Whisper with Disfluency                                                                                                                                                  |\n|----|----------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n|  1 | &-um the [//] &-um a [/] a very wicked &-um woman and her daughters &=ges:two two of them were &-um +...                                                 | a very wicked woman and her daughters two of them were...                                        | [\\*] the, [\\*] a, [\\*] a [\\*] very wicked [\\*] woman [\\*] and [\\*] her [\\*] daughters, two of [\\*] them [\\*] were                                                                  |\n|  2 | and Cinderella she was <a &-um daughter &-um of> [//] &-um a stepdaughter . and &-um they didn\\'t like her .                                             | and Cinderella she was a stepdaughter. and they didn't like her.                                 | [\\*] Anne[\\*] Cinderella. [\\*] She [\\*] was [\\*] a [\\*] daughter [\\*] of [\\*] a [\\*] step-daughter. And [\\*] they [\\*] didn't [\\*] like her,                                         |\n|  3 | the two &-um &-um daughters and the stepmother .                                                                                                         | the two daughters and the stepmother .                                                           | the [\\*] two [\\*] daughters and[\\*] the [\\*] step-mothers.                                                                                                                   |\n|  4 | so ‡ they treat her very badly .                                                                                                                         | so they treat her very badly.                                                                    | So [\\*] they [\\*] treat her [\\*] very [\\*] badly.                                                                                                                            |\n|  5 | and she talked to the birds and trees and +...                                                                                                           | and she talked to the birds and trees and...                                                     | And [\\*] she [\\*] talked to [\\*] the [\\*] birds and trees [\\*] and                                                                                                            |\n|  6 | she said +\"/.  +\" oh ‡ <I wonder> [//] I wish I was <going to the king &-um no> [//] &-um going to the palace and all of that but I\\'ll wait .           | she said oh \"I wish I was going to the palace and all of that but I'll wait.\"                    | she, [\\*]I wonder, I wish I was [\\*] going to [\\*] the [\\*] king, [\\*] going to the palace [\\*] and [\\*] all of that, [\\*] but [\\*] I would [\\*] wait,                             |\n|  7 | okay . [+ exc]  and &-um a long &+ag &-um time ago &-um <a thing> [//] &-um a note came from the palace and it said &-um they are going to have a ball . | okay. and a long time ago a note came from the palace and it said they are going to have a ball. | okay, [\\*] and [\\*] a long [\\*] time [\\*] ago, [\\*] a [\\*] thing, [\\*] a [\\*] note came [\\*] from the [\\*] palace and [\\*] it [\\*] said[\\*] they are going to [\\*] have [\\*] a [\\*] ball |\n|  8 | and sɪndəwɛlə@u [: Cinderella] [* p:n] said +\"/.  +\" can I go ?                                                                                          | and Cinderella said \"can I go?\"                                                                  | and [\\*] Cinderella said, can I [\\*] go                                                                                                                                    |\n|  9 | and the mother said +\"/.  +\" no you can\\'t go .  +\" &=finger:wag and you are going to stay here and keen [: clean] [* p:w] the house .                   | and the mother said. \"no you can't go. and you are going to stay here and clean the house.\"      | and the [\\*] mother said, no, you [\\*] can't go and you[\\*] are going to [\\*] stay here and [\\*] clean the [\\*] house.                                                         |\n| 10 | &=sighs sɪndəwɛlə@u [: Cinderella] [* p:n] was sad .  +\" I can\\'t go to the ball &=laughs .                                                              | Cinderella was sad \"I can't go to the ball.\"                                                     | Cinderella [\\*] was sad, [\\*] I [\\*] can't go to the [\\*] ball.                                                                                                              |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Given that there is limited work on investigation of models for automatically detecting Aphasia types, this topic is relatively novel and will be of great interest to the community. \nThe proposed GNN model using multimodal information from speech and gesture data is novel. The proposed model outperforms existing state-of-the-art models by quite a margin setting a new standard for state-of-the-art models.\nThe experiments and the quality analysis of the results are thorough.  ", "reasons_to_reject": "The authors mention they used Whisper for ASR transcription. To the best of my knowledge, Whisper is well known to filter out disfluencies and filler words in the transcription or generate very limited disfluencies. In the absence of the ASR word error rate performance report, it is hard to justify the impact of disfluency keywords on Aphasia type detection. ", "questions_to_authors": "The authors should report ASR WER performance in the paper.\n\nThe authors should correct the line ... \"Unlike previous ASR systems focusing on transcribing clean speech, Whisper can capture filler words ...\" -> Most traditional ASR systems are able to well recognize disfluencies as long as they are trained using data that contains disfluencies. Also the claim that Whisper can capture all disfluencies should be verified.  ", "paper_topic_and_main_contributions": "In this paper the authors propose a graph neural network based framework that uses information from speech and gesture modalities for Aphasia type detection. The main contributions of the paper are the multimodal GNN based framework for Aphasia type detection, it's comparison with existing benchmark models on a standard dataset and the evaluation of the impact of the contributing modalities. The authors claim that this framework is the first to leverage both speech and gesture information. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The gesture depiction in Figure 1 is very hard to read for a reader and the authors should consider increasing the figure size and clarity. "}, "extra_text": {}}, {"review_id": "L2aUbscoKv", "reviewer": "Reviewer_kFrs", "summary": "", "questions": "1. It would be beneficial to see if the best performance results (Tables 2,3,4 and 5) are statistically significantly different from the rest of models you compare to.\n2. Explain the letters T,V and A in table 4\n3. Looks like Table 5 was never refered to in the paper", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your invaluable suggestions and positive comments! We plan to organize and revise the answers below so there are no unclear explanations in the final version.\n\n## [Typo Errors]\nWe will review the paper carefully to ensure no typos exist in the final version.\n- (Figure 1) The figure size and clarity should be increased.\n- (Table 4) :  the letters T,V and A in table 4 should be explained. (T: text / V: video (Gesture in this paper), / A: Audio) \n- (L554) ~ female participants. -> ~female participants (Table 5). \n- (L557) Table 1 -> Table 8 in Appendix\n\n## [Statistical significance of performance] \nResults marked with an asterisk (*) indicate statistical significance compared to MAG (𝑝 < 0.05) according to the Wilcoxon’s signed rank test. Moreover, our observations indicate statistical significance (𝑝 < 0.05) in relation to MISA and MulT, the second-best performing models, assessed by the Wilcoxon’s signed rank test for both the overall average F1 score and Control (Type 0).\n\n\n|    Model   |                | Label (F1-score) |        |       |         |         |\n|:----------:|:--------------:|:----------------:|:------:|:-----:|:-------:|:-------:|\n|            |                |       Total      |  Ctrl  |  Flu  | Non-Com | Non-Flu |\n| Aphasia &  |   SVM + Poly   |       0.338      |  0.670 | 0.000 |  0.000  |  0.000  |\n|  Dementia  |       RF       |       0.742      |  0.871 | 0.719 |  0.000  |  0.400  |\n|  Detection |       DT       |       0.688      |  0.824 | 0.652 |  0.090  |  0.283  |\n|            |       LR       |       0.655      |  0.773 | 0.696 |  0.000  |  0.000  |\n|            |    AdaBoost    |       0.719      |  0.882 | 0.662 |  0.000  |  0.303  |\n|   Fusion   |      MISA      |       0.761      |  0.899 | 0.741 |  0.000  |  0.349  |\n|            |      MulT      |       0.761      |  0.885 | 0.750 |  0.000  |  0.410  |\n|            |       MAG      |       0.725      |  0.838 | 0.698 |  0.000  |  0.514  |\n|            | SP-Transformer |       0.756      |  0.893 | 0.742 |  0.000  |  0.324  |\n|    Ours    |    Chunk 30    |       0.732      |  0.906 | 0.676 |  0.303  |  0.446  |\n|            |    Chunk 50    |      0.842*      | 0.949* | 0.84* |  0.125  |  0.530  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-written paper that presents a sound work with some interesting and useful contributions to the NLP community. The proposed model is explained in sufficient details and the corresponding code is promised to be shared with the community. Experimental work presented in the paper is technically sound and detailed - model performance is compared to multiple strong baselines, several ablation studies give a good understanding of the impact of individual model components, impact of different modalities, impact of gender, and the effectiveness of the proposed Speech-Gesture Graph Encoder. In addition to this quantitative analysis, the qualitative analysis is performed on four representative cases involving different types of aphasia. All the claims made in this work are strongly supported by empirical evidences.", "reasons_to_reject": "I see no strong reasons to reject this paper. There are several minor weaknesses that I outline in the section \"Questions for the Authors\" but I believe that the paper is strong enough in its current version already and provides substantial contributions to be accepted to the conference.", "questions_to_authors": "1. It would be beneficial to see if the best performance results (Tables 2,3,4 and 5) are statistically significantly different from the rest of models you compare to.\n2. Explain the letters T,V and A in table 4\n3. Looks like Table 5 was never refered to in the paper", "paper_topic_and_main_contributions": "In this paper, the novel graph neural network for aphasia type detection is presented. This work addresses the problem of a detailed diagnosis of aphasia type, which is important in clinical settings for proper treatment procedures. The proposed model makes use of multiple modalities, including acoustics of human speech, text in the format of transcribed speech, and corresponding gesture patterns. Multimodal features are generated with the help of a graph neural network for each aphasia type, enabling the acquisition of rich semantics across modalities. The model learns the correlation between the speech and gesture modalities for multiple aphasia types, and it helps it achieve SOTA results and outperform models relying on any single modality. \nThe method of applying a graph neural network to capture interconnections across multiple modalities and the presented novel model for aphasia type detection are two main contributions of this work.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xb8ovpRUF2", "reviewer": "Reviewer_xF2x", "summary": "", "questions": "1. In this paper, author mentioned 3651 samples were included in the dataset, and each duration was approximately 23 seconds.\nThe number of samples in the dataset is a good starting point, but it is not enough to train a complex model like the one proposed in the paper. The authors should report the number of parameters in the final model to give readers a better understanding of its complexity.\n\n2. The authors claim that gesture information outperforms acoustic information which looks fine to me, how about the text information compare to gesture information? Based on the table 4, it seems like text features outperform the gesture features, and the authors should provide more explanation for it.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. We hope provided explanations can help to clarify misunderstandings.  \n\n## [Typo Errors]\nWe will review the paper carefully to ensure no typos exist in the final version.\n- (Figure 1) The figure size and clarity should be increased.\n- (Table 4) :  the letters T,V and A in table 4 should be explained. (T: text / V: video (Gesture in this paper), / A: Audio) \n- (L554) ~ female participants. -> ~female participants (Table 5). \n- (L557) Table 1 -> Table 8 in Appendix\n\n## [Contribution]\nOur interdisciplinary research brings valuable contributions to both the fields of machine learning and speech-language pathology. When it comes to diagnosing various types of aphasia, the traditional evaluation methods used for individuals with aphasia (PWA) are not only demanding in terms of time and expenses, but they also heavily rely on the expertise of pathologists.\n\nTo tackle these difficulties, we suggest employing a graph neural network to create a model that automatically identifies different forms of aphasia. This model takes advantage of multiple sources of information, including co-speech gestures, audio, and text. Through this approach, our goal is to simplify the diagnostic process, alleviate the workload of pathologists, and simultaneously enhance the precision of detecting aphasia types.\n\nWe summarize the contributions of this work as follows. \n\n1. To the best of our knowledge, this is the first attempt that uses gesture and speech information for automatic aphasia types detection.\n2. Through extensive experimentation, our model demonstrates superior performance compared to previous methods in identifying various aphasia types. The incorporation of Graph Neural Networks (GNN) proves particularly effective in generating multimodal features by capturing the interplay between speech and gesture data, resulting in improved performance.\n3. Our analysis highlights the increased significance of gesture features over acoustic features in predicting aphasia types. This underscores the unique role of gesture expression as a distinguishing characteristic among individuals with different types of aphasia.\n4. We illustrate that the qualitative analysis facilitated by our proposed model can offer clinicians a more comprehensive understanding of aphasia patients, aiding in the delivery of timely interventions.\n5. We plan to make our code accessible to researchers within both the machine learning and pathology communities.\n\n===========================================================\n\n# Aug 28th Additional Rebuttal\n\nThank you for pointing out the settings of methods and our dataset.  We will organize and add the answers below to the final version for clarification.\n\n## [Statistical significance of Performance Result] \nResults marked with an asterisk (*) indicate statistical significance compared to MAG (𝑝 < 0.05) according to the Wilcoxon’s signed rank test. Moreover, our observations indicate statistical significance (𝑝 < 0.05) in relation to MISA and MulT, the second-best performing models, assessed by the Wilcoxon’s signed rank test for both the overall average F1 score and Control (Type 0).\n\n|    Model   |                | Label (F1-score) |        |       |         |         |\n|:----------:|:--------------:|:----------------:|:------:|:-----:|:-------:|:-------:|\n|            |                |       Total      |  Ctrl  |  Flu  | Non-Com | Non-Flu |\n| Aphasia &  |   SVM + Poly   |       0.338      |  0.670 | 0.000 |  0.000  |  0.000  |\n|  Dementia  |       RF       |       0.742      |  0.871 | 0.719 |  0.000  |  0.400  |\n|  Detection |       DT       |       0.688      |  0.824 | 0.652 |  0.090  |  0.283  |\n|            |       LR       |       0.655      |  0.773 | 0.696 |  0.000  |  0.000  |\n|            |    AdaBoost    |       0.719      |  0.882 | 0.662 |  0.000  |  0.303  |\n|   Fusion   |      MISA      |       0.761      |  0.899 | 0.741 |  0.000  |  0.349  |\n|            |      MulT      |       0.761      |  0.885 | 0.750 |  0.000  |  0.410  |\n|            |       MAG      |       0.725      |  0.838 | 0.698 |  0.000  |  0.514  |\n|            | SP-Transformer |       0.756      |  0.893 | 0.742 |  0.000  |  0.324  |\n|    Ours    |    Chunk 30    |       0.732      |  0.906 | 0.676 |  0.303  |  0.446  |\n|            |    Chunk 50    |      0.842*      | 0.949* | 0.84* |  0.125  |  0.530  |\n\n## [Data size & Model Complexity]\nWe acknowledge your concerns about the dataset's size being potentially insufficient for training a complex model. To ensure robust performance, we conducted our experiments using a Group Stratified 5-fold cross-validation approach. In the table below, we outline the complexity of our model. It's worth noting that when compared to other studies focused on detecting cognitive disorders [1-3], our dataset is notably larger—more than 2-3 times in size. Notably, the ADReSS dataset, a significant benchmark in Dementia Detection [1], comprises 156 users across the entire dataset, while our study includes 507 users sourced from various corpora (as detailed in Table 10 in the Appendix). Moreover, the performance results we presented in the table above demonstrate that our model statistically outperforms all baseline models.\n\n|             **Type**             | **Params** |\n|:--------------------------------:|:----------:|\n|              Dropout             |      0     |\n|           RobertaModel           |    124 M   |\n|          HeteroGraphConv         |   24.9 M   |\n|               LSTM               |    192 K   |\n|               LSTM               |   25.8 K   |\n|             Attention            |     138    |\n|             Attention            |     50     |\n|         MulT Transformer         |    3.3 M   |\n|              Linear              |    295 k   |\n|              Linear              |    1.5 k   |\n|         Trainable params         |    153 M   |\n|       Non-trainable params       |      0     |\n|           Total params           |    153 M   |\n| Total estimated params size (MB) |   613.536  |\n\n- [1]https://dementia.talkbank.org/ADReSS-2021/ \n- [2]  Li, C., Knopman, D., Xu, W., Cohen, T., & Pakhomov, S. (2022, May). GPT-D: Inducing Dementia-related Linguistic Anomalies by Deliberate Degradation of Artificial Neural Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1866-1877).\n- [3] Farzana, S., Deshpande, A., & Parde, N. (2022, May). How you say it matters: Measuring the impact of verbal disfluency tags on automated dementia detection. In Proceedings of the 21st Workshop on Biomedical Language Processing (pp. 37-48).\n\n\n## [Analysis on different modalities] \nWe have already provided an explanation indicating that linguistic features offer a higher level of information for the identification of language impairment disorders, which is consistent with findings from previous clinical research (L497). \n\nMoreover, our analysis in Table 4 focuses on different modalities to determine which—acoustic, gesture, or textual features—better complement the others in detecting aphasia types. While many studies in the domain of linguistic pathology typically concentrate solely on speech data, which includes linguistic and acoustic elements, we have observed that gesture information holds significant potential in identifying aphasia types. However, it is noteworthy that incorporating all three modalities yields the most optimal performance. This finding highlights that combining linguistic features with visual and acoustic attributes, as well as understanding their interrelationships, proves more effective than relying solely on one modality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is novel in that it attempts to detect aphasia types rather than simply the existence of aphasia. This is an improvement over previous methods, as it allows for more accurate diagnosis and treatment of aphasia.\n\n2. The authors compare the effectiveness of acoustic features and gesture features for aphasia detection. They find that gesture features are more effective than acoustic features, which is also a novel finding.", "reasons_to_reject": "I am removing my previous comment because it does not obey the review policy. Instead, I am adding some questions to the following sections.", "questions_to_authors": "1. In this paper, author mentioned 3651 samples were included in the dataset, and each duration was approximately 23 seconds.\nThe number of samples in the dataset is a good starting point, but it is not enough to train a complex model like the one proposed in the paper. The authors should report the number of parameters in the final model to give readers a better understanding of its complexity.\n\n2. The authors claim that gesture information outperforms acoustic information which looks fine to me, how about the text information compare to gesture information? Based on the table 4, it seems like text features outperform the gesture features, and the authors should provide more explanation for it.", "paper_topic_and_main_contributions": "In this paper, the authors use language and gesture information to detect aphasia types. They propose a new multimodal graph neural network for aphasia type detection. The authors also conduct an ablation study to assess the effectiveness of linguistic, acoustic, and gesture features. They find that gesture features outperform acoustic features. The authors also analyze the gender effect on aphasia type detection.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not found", "Typos_Grammar_Style_and_Presentation_Improvements": "Not found"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Quality:

The paper presents high-quality results, which is noted by the reviewers as follows:

-	“The proposed model outperforms existing state-of-the-art models by quite a margin setting a new standard for state-of-the-art models.”

-	“The experiments and the quality analysis of the results are thorough.”

-	“Experimental work presented in the paper is technically sound and detailed - model performance is compared to multiple strong baselines, several ablation studies give a good understanding of the impact of individual model components, impact of different modalities, impact of gender, and the effectiveness of the proposed Speech-Gesture Graph Encoder. In addition to this quantitative analysis, the qualitative analysis is performed on four representative cases involving different types of aphasia. All the claims made in this work are strongly supported by empirical evidences.”

Clarity:

The paper is clear and easy to understand: “This is a well-written paper that presents a sound work with some interesting and useful contributions to the NLP community. The proposed model is explained in sufficient details and the corresponding code is promised to be shared with the community.”

Originality:

The work is highly original in that it uses a GNN model with multimodal information from speech and gesture data to detect different types of aphasia and provides evidence on the importance of acoustic features.

Significance:

Finally, the work is significant in that it presents a multimodal study on aphasia detection, making a contribution towards a holistic approach towards detecting the condition.

The rebuttal provided by the authors in response to the questions of the reviewers is substantial.
Decision: Accept-Main