Measure Children's Mindreading Ability with Machine Reading
Authors: Yuliang Yan
Xiaohua Wang
Xiang Zhou
Xiaoqing Zheng
Xuanjing Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural Language Processing
Machine Reading Comprehension
Multimodal
Psychology
Mind-reading
TL;DR: A new framework for building automated scoring system for evaluating children's mindreading ability.

Abstract: Recently, much research in psychology has benefited from the advances in machine learning techniques. 
Some recent studies showed that it is possible to build automated scoring models for children's mindreading.
These models were trained on a set of manually-labeled question-response pairs, which were collected by asking children to answer one or two questions after a short story is told or a video clip is played.
However, existing models did not take the features of the stories and video clips into account when scoring, which obviously will reduce the accuracy of the scoring models. Furthermore, considering that different psychological tests may contain the same questions, this approach cannot be extended to other related psychological test datasets. 
In this study, we proposed a multi-modal learning framework to leverage the features extracted from the stories and videos related to the questions being asked during the children's mindreading evaluation.
Experimental results show that the scores produced by the proposed models agree well with those graded by human experts, highlighting the potential of the proposed network architecture for practical automated children's mindreading scoring systems.

Measuring Children's Mindreading Ability with Machine Comprehension
Yuliang Yan 12, Xiaohua Wang Xiang Zhou 12, Xiioqqing Zheng 12,,, Xuanjing Huang!
"hengl2.,uuanjin
SScooo of Computer Science, Fudan University, Shanghai, China Shanghai Key Laboratory of Intelligent Information Processing [ylyan21, xi xiaohuawang22)err fudan. edu. cn {zhengxa, xjhuang Jēf edu. cn
Abstract
X Sound X
Vidine for Sootan)
Vidine for Coenan
BSKE
Recently, much research in psychology has ben- efited from the advances in machine learning techniques. Some recent studies showed that it is possible to build automated scoring mod- els for children's mindreading. These mod- els were trained on a set of manually-labeled resppoosee pairs, which were collected by asking children to answer one or two ques- tions after a short story is told or a video clip is played. However, existing models did not take the features of the stories and video clips into account when scoring, which obviously will reduce the accuracy of the scoring models. Furthermore, considering that different psycho- logical tests may contain the same questions, this approach cannot be extended to other re- lated psychological test datasets. In this study, we proposed multi-modal learning framework to leverage the features extracted from the sto- ries and vidcos related to the questions being asked during the children's mindreading evalua- tion. Experimental results show that the scores produced by the proposed models agree well with those graded by human experts, highlight- ing the potential of the proposed network ar- chitecture for practical automated children's mindreading scoring systems'.
EQuetion
Watch the Videor
Rappora teastng
Assan For Queton
Same Cuoctisen How of man in dia dip?
the
Quedlet Stipunn By Vakoo Ston)
Tax QuO
Tua
Scase
Quedlaer
Figure I: Suppose there are two different psychological tests that contain some identical questions. Previous methods only considered question-answer pairs as input to the model. Therefore, when faced with question- answer pairs as shown the figure, the model failed to learn how score these two pairs since the same ques- tion and the children's 's answers were close. However, our framework can handle such situations. By incorpo- rating the background information from psychological tests into the input, the model can utilize the additional information learn how distinguish semantically similar question question-answer answer pairs, thus avoiding incorrect evaluation results. In general, our framework can be extended to multiple related datasets, requiring the train- ing of only two models: one for cases with text-only input and another for cases with video information in- cluded. These two models are unified within the UniFM framework.
1 Introduction
well-being and superior social skills. Specifically, those who perform well on mindreading assess- ments are more likely to enjoy popularity among their peers (Banerjee et al., 2011) and maintain positive relationships with their fellow classmates (Fink et al., 2015). Moreover, case-control stud- ies have revealed a higher propensity for mental health issues among individuals with impaired min- dreading abilitics (Cotter et al., 2018). Given these compelling findings, it becomes paramount iden- tiy an accurate, effective, and reliable method for evaluating the mindreading capacities of children during their middle childhood and carly adoles- cence.
In the field of psychology, the cognitive process of inferring others' mental states through the ob- servation of their actions and verbal expressions is commonly referred to as "mindreading". This intriguing phenomenon has garnered considerable attention from various disciplines, including psy- chologists, neuroscientists, economists, and com- puter scientists, over the course of many years (Hughes and Devine, 2015). Research suggests that children who exhibit exceptional mindreading abilities tend possess healthier psychological
'Our code is available at https: manic-do wwwphin//neeee waaiin/emmppp220 mmmmlp2223-nni fm-mi Tim-mmnee mindreading.

---

Presently, standardized open-ended psycholog- ical tests are available to assess children's min- dreading ability. These tests typically involve chil- dren responding to specific questions based on meticulously crafted vignettes prepared by experts (Happé, 1994; Banerjee et al., 20. Alternatiely, assessments may involve the use of short clips ex- tracted from comedic sources (Devine and Hughes, 2013) or animated content (Castelli et al., 2000).
action between the background information and the question-ansswr pairs, we secondly introduce some effective methods from machine reading compre- hension. This allows the model to not only fully utilize the background information but also seam- lessly unify different input types within a single framework.
Overall, our proposed method UniFM (Unified Framework for Measuring Children's Mindreading Ability) can effectively addresses the issues en- countered by previous models. Experimental re- sults demonstrate that our model outperforms pre- vious approaches, and the incorporation of multi- modality information significantly enhances its per- formance. Moreover, in comparison to previous methods, UniFM exhibits the capability to handle a broader spectrum of psychological testing scenar- ity. They constructed the MIND-CA CA dataset (Ko- vatchev et al., 2020), subsequently performed fine- tuning on a Transformer-based pre-trained model using the MIND-CA dataset, leading to notewor- thy advancements in automated assessment of chil- dren's mind mind-reading abilities.
However, evaluating children's 's responses to these test questions necessitates manual assessment by extensively trained experts. This process is both labor-intensive and costly. Some work (Kovatchev et al., 2020, 2021) have attempted to tackle this challenge by integrating the aforementioned psy- chological tests and developing automated scoring systems to evaluate children's 's mind-reading abil-
ios, thereby showcasing its potential for develop- ment into an automated, cost-free, online scoring system for evaluating children's 's mindreading abil- ity.
Related Work
Despite previous research has achieved promis- ing results, the methods employed still exhibit cer- tain limitations. These models only considered the test's questions and children's 's responses as input, which brings two drawbacks. Firstly, different psy- chological tests may share identical questions, us- ing this method leads to unavoidable errors during evaluation of the model, which restricts the model's s generalizability to other related psychological test datasets, limiting its applicability. Secondly, disre- garding the background information from psycho- logical tests hampers the potential improvement in the model's evaluation performance. Hence, it is crucial to acknowledge the relevance and utility of this contextual information. Figure I illustrates an example that explains why our framework has the ability extend to other related psychological test datasets.
This section provides an overview of the relevant literature that forms the foundation of our research. Firstly, we present concise introduction to the con- cept of mindreading and discuss two standardized psychological tests that were employed to construct the training dataset for automated scoring systems. Secondly, we provide brief review of studies that have explored the integration of NLP techniques with psychology, highlighting the research endeav- ors focused on developing automated scoring sys- tems. Throughout this paper. our primary objective is to compare our model with these existing sys- tems. Finally, we present overview of methods employed in the Machine Reading Comprehension task, incorporating techniques proposed in prior studies to construct our framework.
2.1 Standardized Psychological Tests for Assessing Children's 's Mindreading Ablility
To address the aforementioned issues, we firstly incorporate the psychological tests' background in- formation into our model's 's input. This background information includes the complete story text that is read to the children during the psychological tests, well the silent comedy clips that are shown to the children. In the current context, the input may involve multiple modalities, such as the combination of clips and question- question-answer answer pairs. Moreover, considering the need for effective inter-
Mindreading is a concept in psychology (Hughes and Devine, 2015) which usually is used to de- scribe someone's ' ability in understanding others' thoughts, feelings, and desires. For example, man was arguing with his wife, and finally, she said: "Well, fine." If the man is good at mindreading. he should know that his wife was still angry. Cur- rently, there are established and standardized psy- chological tests that are used to assess children's

---

mind-reading abilities. Our study involves two of these tests, will provide brief description of these two assessments. Happé (1994) orally present five brief narratives to children while dis- playing the corresponding story text on a sizable screen. These stories encompass diverse social sce- narios, such as double bluffing, cheating, misunder- standings, and lying (these stories are referred to as "Strange Story"), and each narrative concludes with a specific open- open-ended question. Children are then prompted to provide responses these questions, aiming to discern the inner mental states of the characters involved. Devine and Hughes (2013) in- volves children viewing a series of five brief silent film clips displayed on a sizable screen. These clips are carefully chosen from renowned silent comedy, portraying various social scenarios such deception and misunderstanding (these clips are referred as "Silent Film"). In alignment with the selected clips, researchers devise specific questions for children to address. Following a single view- ing of each clip, children are requested to provide written responses to the questions, which are read aloud by the researchers.
results showed that the model gained performance improvement both on the MIND-CA and the new dataset UK-MIND-20.
Despite the promising results, they only consid- ered evaluating children's mindreading ability by using question-answer pairs as input. However, in different psychological tests, the quality of chil- dren's 's responses cannot be solely evaluated based on the question-answer pairs. It is crucial in- tegrate relevant test information in order for the model to handle situations where different psycho- logical tests may contain the identical questions.
2.3 Machine Reading Comprehension
Machine reading comprehension (MRC) is a chal- lenging task in NLP field (Liu et al., 2019; Zeng et al., 2020; Zhang et al., 2019). It can be divided into four categories: cloze style, multiple-choice, span prediction, and free-form answer.
For the multiple-choice task, inspired by the human's transposition thinking process of han- dling MRC problem, Zhu et al. (2021) proposes the Dual Multi-head Co-Attention(DUMA) to cal- culate the attention score between passage and question-answer pair. It boosts the model's per- formance on the DREAM (Sun et al., 2019) and RACE (Lai al., 2017) datasets.
2.2
Natural Language Processing for Psychology Research
Recently, many works in psychology are benefited from advanced natural language processing meth- ods, including building chatbots to promote the mental health of their users (Tewari et al., 2021), making inferences about people's mental states from what they write on Facebook, Twitter, and other social media (Calvo et al., 2017), combin- ing with computational algorithms to understand a suicidal patient's 's thoughts, such as suicide notes (Pestian et al., 2010).
3 Method
In this section, we provide a detailed description of our proposed method. As mentioned carlier, our framework requires the training of two distinct models to accommodate different input scenarios. Hence model is designed to handle two sub- tasks: the Strange Stories Task and the Silent Film Task.
Different tasks corresponds to different inputs. For the Strange Stories Task, we utilize the source text of the "Strange Storie" and the corresponding question-response pairs as input to our model. On the other hand, for the Silent Film Task, employ the clips from the "Silent Films" and the related cnnestion-rsppowee pairs as input.
Similarly, NLP techniques have also been ap- plied to the research on building automated assess- ment of children's 's mind-reading abilities. Utiliz- ing the aforementioned standardized psychological tests (Happé, 1994; Devine and Hughes, 2013), Kovatchev et al. (2020) created the MIND-CA dataset which consisting 11,311 question-anse -nnwerr pairs based on the responses. They trained a se- ries of models (i.e., SVM, BiLSTM, Transformer) to build automated scoring systems, and obtained ideal results.
Given the inclusion of information from both the language modality and the visual modality in the input, we employ specific pre-trained models to encode this information. These pre-trained models serve as robust encoders that extract meaningful representations from the input data.
Following this work, Kovatchev al. (2021) adopt some data augmentation methods to enhance the performance of the automated systems. The
Subsequently, in order to facilitate the interac- tion between the representations of the text (or

---

Extracted Frames
C3D
BERT Encoder
Linear
Value
Video
Story
Decoder
Linear
Key
Linear
Linear
Query
Features
Mulli-Head
Attention
Softmax
Linear
Query
Linear
Key
Linear
Score 0
BERT
Linear
Value
Score 1
Question
Encoder
Score 2
Response
DUMA Layer
Probability Distribution
Figure 2: The architecture of our model. For inputs from different psychological test backgrounds, we need to train two models with only differences in the encoder architecture. Specifically, if the input consists of stories and question-answer pairs, employ BERT for encoding obtain hidden representations. the input includes videos and question-answer pairs, we use C3D to encode the video. Once we obtain the hidden representations of the input, we utilize dual-tower attention mechanism to interact and align these features, which a unified process. After aligning the features, concatenate and aggregate them, and finally decode them into probability distribution of scores, completing the automated assessment of children's 's mental interpretation ability.
clips) and the representations of the question- answer pairs, we employ the "dual-tower" archi- tecture (DUMA layer) that borrows from DUMA method (Zhu et al., 2021).
3.1.1 Strange Stories Task
In the Strange Stories Task, we formalize the put of the model as triplet: (S, R, where S refers to the source text of the "Strange Sto- ries" (5 in total), and each story corresponding to fixed question Q. We concatenate the question Q and children's 's response R Q@R. Morcover, for each triplet (S, Q, R), the ground truth of the children's mindreading ability score is defined as s, where C Score, and Score = {0, 1, 1,2), which describes all the possible score of children's 's min- dreading ability.
The DUMA mechanism enables our model to achieve effective alignment between different pieces of information, resulting in an integrated representation that captures the essential informa- tion of the passage (or the clips) and the associated question-answer pairs. This integrated represen- tation is subsequently mapped into probability distribution, serving measure of the model's confidence or certainty in assessing mindreading abilities. Figure provides an detailed overview of the structure of our framework.
Given N data instances {S, QiOR, 8; J.W in the train set D Dtrain, model's output is defined the p(s)S, S, Q+R). We train our model to maximize the probability: I p(s} Q R,). Toward this goal, our model must acquire the ability to accurately map the (S, Q+R) input into the corre- sponding ground truth score.
In the following subsections, we discuss in more detail our method. Section 3.1, we describe two sub-tasks, and Scction 3.2, we provide a detailed description of the model architecture.
3.1.2 Silent Film Task
Similar to the Strange Stories Task, in the Silent Film Task, we formalize the input of our model as:
3.1 Task Definition
(C, R),, where C' denotes the clips (5 total) the "Silent Film", and each clip is also correspond- ing to fixed question. Score still represents the
In this Section, we give the definition of the two sub-tasks of our model, for these two distinct sub- tasks, we design corresponding model architectures and unifiy them within our UniFM framework.
"Except for Silent Film 1, which has two questions.

---

ground truth of the score of children's mindread- ing ability. Given N data Q, s our objective aligns with the goal of the Strange Sto- ries Task, which is to maximize the probability: I1 M Ci, Q.)....
process in the DUMA layer can be found in the Appendix A.
Decoder. We use the Multi-Layer Perceptron (MLP) to decode the output of the DUMA layer into the representation named O:
3.2 Model Architecture
O = MIP(DUMA(BH) BQR))
(I)
In this section, provide comprehensive overview of the model's architecture for the two distinct tas-tasks. We discuss the design of the encoder, DUMA layer, and decoder, and elucidate the reasons behind their integration into our frame- work.
Here, O=R', / denotes the number of the score (3 in total). For each data (S, Q. R, s) in the train set, the objective probability can be calculated in the same way as the Softmax function:
uxp(O')
p(s|s, QBR)
Σ exp(O*)
3.2.1 Model for the Strange Stories Task
where Oc denotes the element's value in O that matches the ground truth score, O denotes the k-th element's value in O.
Encoder. We use the BERT (Devlin et al., 2018) en- coder generate the representations of the source text of the "Strange Story" and the source text of question-response pairs. The pre-trained BERT model is capable of providing powerful hidden representation of the input text. This greatly fa- cilitates the subsequent interaction among diverse information within the model.
Given N data (Si, Q Rs, s:) sampled in the train set, we define the loss function loss as:
lossss
Σ log p(a) Q.OR.)
Let S
[S1, S2, ... Sm] and Q&R
3.2.2 Model for the Silent Film Task
[Q,,..., Qn, Ri,.. ..., R,] represent the sequences the story and the question-responss pair respec- tively, where S₁, Q, R2 are tokens. We denote BERT(.) as the BERT encoder, the encoded rep- resentations of S and Q R are defined as Bs = BERT(S), and BQR BERT(QU respec- tively.
Encoder. Different from the Strange Stories Task, in the Silent Film Task, model's input is (C, Q, R), where C denotes a short video. For the question question-answer answer pair, we still use the BERT en- coder to obtain the output representation BQu B E HERT(QU (Q)... For the clip, we utilize 3D Con- veet (C3D) (Tran al., 2014) to extract infor- mation and obtain the hidden representation F CC). Simultaneously, we will ensure that the output hidden representations of the clip and text remain consistent across all dimensions.
DUMA Layer. We utilize the DUMA layer (Zhu et al., 2021) to capture the interaction amoung the encoded information. Compared with the vanilla multi-head attention architecture (Vaswani et al., 2017), the DUMA layer adopts the dual-tower ar- chitecture, which is inspired by the cognitive pro- cesses employed by humans during reading com- prehension. The distinctive architecture enables the model to fully understand the content of psycho- logical tests and achieve alignment between both unimodal and multimodal information. As re- sult, enhances the automated evaluation process, leading to improved performance.
There are two reasons why we do not utilize more advanced and larger models as video en- coders. Firstly, we considered that C3D is rela- tively mature and classic model with stable training performance. Secondly, our dataset contains rel- atively limited variety of videos, and the need encode highly complex information is not promi- nent. Therefore, C3D is sufficient for efficient in- formation extraction. However, in the future, if our framework necessitates application in more com- plex scenarios, we do not exclude the possibility of employing ffute-of-the-a video encoders with larger parameter sizes.
Given the representation Bs and BQR, we cal- culate the attention score in the following way: (1) Bs as Query, BQR Key and Value; (2) BQR Query, Bs as Key and Value. Here, the terms Query, Key, and Value have the same meaning as Q, K, and V in the vanilla multi-head attention respectively. We denote the output of the DUMA layer as DUMA(.). The complete computation
DUMA Layer. Besides the hidden representation Bg of the story in the input, now it becomes the hid- den representation F of the clip, the computation process in the DUMA layer remains unchanged.

---

Therefore, we unify the two tasks into one frame- work, which opens up the possibility of training the model and extending its applicability to a broader range of scenarios. Moreover, to our surprise, we found that the DUMA layer can effectively handle the interaction between multimodal information, even without adopting more advanced Video Ques- tion Answer-based methods. This validates the rationale of applying machine reading comprehen- sion techniques to address multimodal alignment issues in our task scenario.
Model
Input
val-Ace
test-Acc
baseline (reported) baseline (reported)
R only
89.00
Q+R
91.00
BERTss
Q+R
94.11
94.66 + 0.85
BERTsP
Q+R
93.63
92.01 + 1.25
UniFMgs
S+Q+R
94.76
05.49-0055
UniFMsr
C+Q+R
93.35
94. 0.88
Table The experimental results of our models and baselines. S denotes the source text of the "Strange Stories", C denotes the clips of the "Silent Film". Q. R denote the questions and responses respectively, The accuracy of the baseline models are excerpted from (Ko- vatchev et al., 2020). The subscript of the model rep- resents on which dataset the model was trained and evaluated. We select different random seeds to train and test the model 10 times. Then, we calculate the average and standard deviation of the experimental results.
Decoder, The decoding process here is completely identical to the Strange Stories Task, so we directly provide the loss function loss5p:
lossgF
Σ Cl,
(4)
In addition, considering that we divided the MIND-CA dataset into two parts for our two dif- ferent tasks, we fine-tuned two extra BERT models on the dataset Dss and DsF respectively. To be consistent with the two baselines, we only take question guestion-answer answer pairs as the model's input. These two sets of experiments eliminated the potential impact of dataset partitioning on the experimental results. We denote these two models as BERT ss and BERTsF.
4 Experiments
Firstly, we reorganized the MIND-CA dataset and obtained two datasets Dss, DsF for the Strange Stories Task and the Silent Film Task respectively. For different tasks, we train our model on the cor- responding dataset. In addition, we conduct ab- lation study explore the effect of some hyper- parameters and model's architecture. Finally, we perform a case study to test our model's perfor- mance in the real world.
4.3 Experimental Settings
In the Strange Stories Task, we train our model on the dataset Dss. we use BERT base as the encoder, and use k - layer of the DUMA Layer
4.1 Dataset
Based on the Strange Stories Task and the Silent Film Task, we divided the MIND-CA dataset into two parts: Dss and DsF respectively. The dataset Dss consist of all the question-answer pairs as- sociated with the "Strange Strories", with each question-answer pair accompanied by the corre- sponding story. Similarly, the dataset D5F consists of all the question-answer pairs associated with the "Silent Film", with each pair accompanied with the corresponding clip. The specific construction pro- cess of the MIND-CA dataset and the organization format of the dataset can be found in Appendix B.
In the Silent Film Task, We train our model on the dataset DsF. We use BERT base to encode the question-answer pairs. For each clip, we firstly utilize the OpenCVà tools to extract fixed number of frames. Typically we extract 32 frames. We implement a C3D model to encode these frames. For both tasks, we adopt Adam (Kingma and Ba, 2014) as our optimizer with the learning rate 5 x 10 5. We train our models on four NVIDIA Geforce RTX 2080Ti GPUs.
4.4 Main Results
4.2 Baseline Models
Table I shows main experimental results. Com- pared with the baselines, our models show bet- ter performance. In the Strange Stories Task, our model gains 95.19% accuracy on Dss, which out- performs the baselines and the BERT ss- In the Silent Film Task, our model gains 94.86% accu- racy on D5F, outperforms the baselines and the
Kovatchev et al. (2020) fine-tuned a DistilBERT (Sanh al., 2019) on the whole MIND-CA dataset. Their model follows the framework of the tradi- tional text classification task. Their model takes two types of data as input: 1) children's 's response only: 2) question and children's 's response. We take their model as our baselines.
Phtte//giiinn..commopeeeeropppp

---

Model
Input
test-Acc
Pair
P-Value
UniF MS
S S+Q+R R S+Q+R Q+
95.49-0.52
(UniFM BERT
0.01%
UniFMss w/o DUMA BERTsS
94.79 + 0.74 94.66 + 0.85
(UniFMss. UniEMes w/o DUMA)
0.02%
MUniFMsr. BERTsF))
0.02%
(UniFMsr. UniFMer w/o DUMA)
0.01%
Table 2: Ablation study of the proposed model on
the Strange Stories Task. S denotes a stroy from the "Strange Stories", Q and R denote the question, response respectively.
Table Paired t-test results. Each pair consists of two models that need to be compared. We conducted multiple experiments and calculated the correspond- ing p-value for each pair. A lower p-value indicates more significant performance difference between the two models that pair.
Model
Input C+ Q + R
test-Acc
UniFMsF
84.861
UniFM w/o DUMA
C+ C+Q+R
91.78 = 1.27
BERT
Q
92.01 = 1.25
Table 3: Ablation study of the proposed model the Silent Film Task. C denotes a clip from the "Silent Film", Q and R. denote the question, response respec- tively.
mance. Moreover, UniFM5s without the DUMA layer achieves better performance compared to the BERTss model trained exclusively on question- answer pairs, as discussed previously. This proves that additional background information hance the performance of the model. Moreover, the DUMA layer facilitates better understanding of the background information, leading to optimal model performance.
en-
This result gives us a strong proof that injecting rich background information from psy- chology tests into the model can indeed improve the ability of the automated scoring system.
SF
The improvement in model performance can be attributed to two factors. Firstly, the incorporation of supplementary background information from psychological tests into the input has contributed the improvement. Despite the limited diversity of stories and videos within the dataset, this in- clusion has proven beneficial for enhancing the model's performance. Secondly, the introduction of DUMA allows for effective alignment between diverse pieces of information. This enables the model to gain better understanding of psycholog- ical tests and leverage the additional information enhance its evaluation capability.
Similar to the Strange Stroies Task, the Silent Film Task, we train a UniFM model without DUMA layer on the dataset DSF- Results in the Table 3 shows that, removing the DUMA layer ex- tremely decreases the model's 'ss performance. This confirms that simply adding videos to the input does directly improve the model's performance. We also need employ effective alignment tech- niques. The DUMA layer allows the model to bet- ter understand the background information, achieve alignment between multimodal information, and attain optimal performance.
5 Ablation Study
Statistical Significance Analysis
In this section, we will dive further into the im- pact of the DUMA layer sults in the Strange Stroies Task. Additionally, will explore the significance incorporating vi- sual information in improving model performance in the Silent Film Task. We will also investigate the influence of the DUMA layer on aligning multi- modality information.
experimental re-
To confirm that our method indeed improves the model's performance, we conduct the paired t-test on the Strange Story Task and the Silent Film Task. Specifically, each pair consists of two models. We train and test these two models multiple times, each time with fixed random seed shared by both mod- els. We record the results of the models on the test set and then calculate the p-value based the sults obtained from multiple tests. Table presents the experimental results. These results demonstrate that the differences performance between each pair of models are statistically significant, which essentially confirms the effectiveness of our model.
To investigate whether DUMA laer would bene- fit the model's performance, in the Strange Stories Task, we remove the DUMA layer in our model, train and test it on the dataset Dss.
The results presented in the Table demon- strate that the removal of the DUMA layer from Ss leads to decline in the model's perfor-

---

7 Quantitative Analysis
changing the number of training epochs. Figure 4 shows the results.
7.1 Number of DUMA Layers
We found that when we trained our model for 4 epochs, our model perform the best. However, dif- ferent epochs have not brought a significant effect on our model. We believe that this is due to the lack of a sufficient number of diverse videos in the dataset.
We investigate the influence of the number of DUMA Layers. We only change the number of DUMA Layers and calculate the accuracy. The results are shown in figure In the Strange Sto- ries Task, our model show good performance both when k = and k - 4. When k 6, model's performance drop dramatically. It is not strange that increasing the number of DUMA Layers does not bring promising improvement: there are only five different types of stories in the Strange Stories Task, interacting the text of stories and question- response pairs once is enough to capture the hidden information. Stacking too many layers makes the model harder to train and may lead to the overfit- ting problem. These analyses are also applicable to the Silent Film Task.
of
Marei/Puren
Figure 4: Performance change under different hyper- parameter choices. We investigate it in the Silent Film Task. The left shows the influence of different epochs and the right shows the influence of different frames extract from one cilp.
94-
oi Shange Stenis
7.3
Case Study
We conduct a case study to verify our system's 's performance in the real world. We selected sev- eral well-educated English-speaking children aged between 10 and 11. Under the guidance of our detailed instructions, they completed the mindread- ing tests (include the Strange Stories Task and the Silent Film Task.) accompanied by their guardians. We collected their responses and hired experts to score the responses manually. These scores were used as the ground truth. Then we tested our model on these new data.
Tost
000
08
05
Num of LONWL
Figure 3: The performance change under different the number of DUMA Layers. The blue line shows the test accuracy on Dsa and the red line shows the test accuracy on Dsr.
We utilize the Pearson correlation coefficient measure the difference between the model's 's pre- dictions and the ground truth. The complete caleu- lation process can be found in Appendix C. Let r represent the Pearson correlation coefficient. In the Strange Stories Task we got r 0.83 and in the Silent Film Task we got r = 0. 0.77, which indicates that our models' evaluation have strong correlation with the experts' assessment. Therefore, our model has the potential for being applied to other related scenarios.
7.2 Number of Frames and Epochs
The silent film contains 5 clips, clips last 27.6 sec- onds on average. For each clip, we need to extract a fixed number of frames and convert the frames into tensor form. In this section, we will investigate the influence of the number of frames in the Silent Film Task. Figure 4 shows the results. We found that extracting 32 frames enables our model achieve the best performance. Due to the limitation of the video types, we think 32 frames are sufficient for the model to obtain useful information.
Conclusions
In the Silent Film Task, we have not utilized some pre-trained models while the training data is lim- ited, instead we train a C3D from scratch. However, training too much will lead to overfitting while the lack of training may weaken the advantage of C3D. this section, we seek to find the trade-off by
In this paper, we propose new framework UniFM for automated scoring children's mindreading abil- ity. We fuse additional multimodality information from mindreading tests into the model's input. Dif- ferent from previous methods, this novel frame-

---

work can be used train on more than one psy- chological test dataset. Experimental results have shown that our model achieves great performance on different tasks and outperforms the previous methods. In addition, the case study indicate that our models have the potential to be transferred in other scenarios.
Jacob Devlin, Ming-Wor Wee Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre- training of deep bidirectional transformers for language understand- ing. north american chapter of the association for computational linguistics.
Elian Fink, Sander Begeer, Candida C. Peterson, Vir- ginia Slaughter, and Marc de Rosnay. 2015. Friend- lessness and theory of mind: A prospective longitudi- nal study. British Journal of Development Psychol- ogy.
Limitations
The MIND-CA dataset only has limited question source and question-response pairs, which is not enough for training a DNN model with a large scale of parameters. It would be intriguing to see the performance of our model under larger dataset which contains variety of stories and videos.
Francesca Happé. 1994. An advanced test of theory of mind: understanding of story characters' thoughts and feelings by able autistic, mentally handicapped, and normal children and adults. Journal of Autism and Developmental Disorders.
Claire Hughes and Rory T. Devine. 2015. Individual differences in theory of mind from preschool to ado- lescence: Achievements and directions. Child Devel- opment Perspectives.
A Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by National Natural Science Foun- dation of China (No. 62076068), and Shanghai Municipal Science and Technology Project (No.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv: Learning.
Venelin Kovatchev, Phillip Smith, Mark Lee, and Rory T. Devine. 2021. Can vectors read minds better than experts? comparing data augmentation strategies for the automated scoring of children's 's mindreading abil- ity. meeting of the association for computational linguistics.
21511
References
Venelin Kovatchev, Phillip Smith, Mark Lee, Imo- gen Grumley Traynor, Irene Luque Aguilera, and Rory T. Devine. 2020. "what is on your mind?" auto- mated scoring of mindreading in childhood and early adolescence. arXiv: Computation and Language.
Robin Banerjee, Dawn Watling, and Marcella Caputi. 2011. Peer relations and the understanding of faux pas: longitudinal evidence for bidirectional associa- tions. Child Development.
Rafael A. Calvo, David Milne, M. Sazzad Hussain, and Helen Christensen. 2017. Natural language process- ing in mental health applications using non-clinical textst. Natural Language Engineering.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. empirical methods in natural language processing.
Fulvia Castelli, Francesca Happé, Uta Frith, and Chris Frith. 2000. Movement and mind: functional imag- ing study of perception and interpretation of complex intentional movement patterns. Neurolmage.
Shanshan Liu, Xin Zhang, Sheng Zhang, Hui Wang, and Weiming Zhang. 2019. Neural machine reading com- prehension: Methods and trends. Applied Sciences, (18)):669.
Jack Cotter, Kiri Granger, Rosa Backx, Mathhe Hobbs, Chung Yen Looi, and Jennifer H. Barnett. 2018. So- cial cognitive dysfunction as a clinical marker: A systematic review of meta-analyses across 30 clinical conditions. Neuroscience & Biobehavioral Reviews.
John Pestian, Henry Nasrallah, Pawel Matykiewicz, Au- rora Bennett, and Antoon Leenaars. 2010. Suicide note classification using natural language processing: A content analysis. Biomedical Informatics Insights.
Victor Sanh. Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv: Computation and Language.
Rory T. Devine and Claire Hughes. 2013. Silent films and strange stories: theory of mind, gender, and so- cial experiences in middle childhood. Child Devel- opment.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu. Yejin Choi, and Claire Cardie. 2019. Dream: A challenge data set and models for dialogue- reading compre- hension. Transactions of the Association for Compu- tational Linguistics, 7:217-231.
Rory T. Devine and Claire Hughes. 2016. Measuring theory of mind across middle childhood: Reliability and validity of the silent films and strange stories tasks. Journal of Experimental Child Psychology.

---

Abha Tewari, Amit Chhabria, Ajay Singh Khalsa, San- ket Chaudhary, and Harshita Kanal. 2021. A survey of mental health chatbots using nlp. Social Science Research Network.
B Dataset Details
To create the MIND-CA dataset, Kovatchev et al. (2020) recruited 1,066 English-speaking children aged between 7.25 and 13.53, from 46 different classrooms in 13 primary and 4 secondary schools in England between 2014 and 2019. The children took part in whole-class testing session lasting ap- proximately 1 hour, led by a trained research assis- tant using a scripted protocol (Devine and Hughes, 2016). Each child answered 11 questions five the Strange Story Task and six in the Silent Film Task. They obtained a total of 11.726 question- answer pairs. The paper test booklets were digi- talized and manually scored by two postgraduate research assistants and the test developer.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Tor- resani, and Manohar Paluri. 2014. Learning spa- tiotemporal features with 3d convolutional networks. international conference computer vision.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. neural information processing systems.
Changchang Zeng, Shaobo Li, Qin Li, Jie Hu, and Jian- jum Hu. 2020. A survey on machine reading compre- hension-tasie evaluation metries and benchmark datasets. Applied Sciences, 10621):7640.
Xin Zhang, An Yang. Sujian Li, and Yizhong Wang. 2019. Machine reading comprehension: a literature review, arXiv preprint 1907.00800
Our dataset is constructed in the format shown in Table 5. In the Dss. each data instance consists of a story a corresponding question, the child's s response to the question, and an expert-labeles score representing the child's mentalizing ability. In the DsF, each data instance consists of clip a corresponding question, the child's response to the question, and a score.
Pengfei Zhu, Zhuosheng Zhang, Hai Zhao, and Xi- aoguang Li. 2021. Duma: Reading comprehension with transposition thinking. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 30:269- 279.
A DUMA Layer
Dataset
Col-1
Col-2
Col-3
Col-4
In this section, we present the complete process of computing attention and obtaining the final output in the DUMA layer.
Der
Story
Question
Response
Score Score
Dag
Clip
Question
Response
Given the representation Bg and BQR. we cal- culate the attention score in the following way: (1) Bs as Query, BQu as Key and Value; (2) BQu
Table 5: The format of dataset Dys and DsF.
C Compute the Pearson Correlation Coefficient
Query, Bs as Key and Value.
Attention(Q. K, V) softmax( Q(K2)
)v Vdl
Let P= PP.. (ti...., fn) denote the predicted results and true labels respectively, where pt, 4 represent the children's 's mindreading ability score, n denotes the total number of the responses. We calculate the Pearson correlation coefficient in the following way:
head; Attention(QW? KW. VW."
MHA(Q, K, V) = Chead, headh,) w
MHA (1) MHA(Bs, Bon, BQR)
MILA MHA(BQR. Bs, Bs)
Σn (m- -P)(t, - 7)
DUMA(Bs, B2a) "USH(MHA() (2)
(5)
Ei (p: PPy D TT²
(6)
where dy. dr. d. denote the dimension of the Query, Key, Value. w.O CRddmodd Xda, w.K ERdmmdaa W.V eIR xde h denotes the number of the attention heads, and dmmodte denotes the fixed dimension of model. WOEB hdd xdmode! (H) and MHA(2) represent the aforemen- tioned two kinds of attention representations re- spectively. FUSE(-) refers to the concatenation operation. The output of the DUMA Layer de- noted as DUMA(.).


Reviews:
[{"review_id": "Pg31V35Rmv", "reviewer": "Reviewer_x51Z", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\nQ: This is a quite straightforward research. Because the proposed method uses more information than the baselines using only the responses, performance improvements are also trivial.\n\nR: This study endeavors to accurately assess children's ability in mindreading by analyzing their natural language responses to questions related to viewed video clips and read stories. Our aspiration is that the developed models will streamline the swift identification of children with mental health concerns, particularly in regions where access to medical professionals such as doctors and psychologists is limited. Previous investigations, such as those by Kovatchev et al. (Coling2020) and Kovatchev et al. (ACL2021), have tackled the same task. However, these studies solely relied on question-answer pairs, disregarding crucial contextual information from the video clips and stories. Several issues can be identified with their methods: 1) There is a high likelihood that identical or similar question-answer pairs accompanied by different contextual information could yield disparate and potentially contradictory evaluations. For instance, consider a question like \"How do you interpret the actions of the man in this clip?\" The answer \"He is lying\" or \"He is telling the truth\" could be equally plausible, but the accuracy depends on the clip's content, which is not provided. 2) Their methods impede the enhancement of existing models through training on diverse datasets. As the training data increases, conflicts emerge within the dataset, where similar question-answer pairs may be associated with varying evaluation scores. 3) Solely considering on the questions and answers themselves unduly impacts the assessment's accuracy.\n\nExtending current multi-modal models to reliably assess children's mindreading capacity is far from straightforward. This task significantly differs from conventional question answering or visual question answering. Unlike those tasks where a range of reasonable answers is acceptable, this study necessitates assigning precise scores to question-answer pairs linked to textual or visual content. Through the exploration of diverse model architectures and training techniques, we have designed a model inspired by the framework of machine reading comprehension. Empirical findings validate that our proposed model not only advances performance beyond that of Kovatchev et al. (ACL2021) but also accommodates multiple independently-created datasets from different sources. This adaptation enhances practicality for psychological assessment and diagnosis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Nothing.", "reasons_to_reject": "See above.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a neural network to measure a mindreading ability of a patient.\nIn addition to the patient's responses to questions, the proposed method also leverages\nvideo and story by encoding it to be fed into a neural network. As a neural network,\ndouble-headed network called DUMA is employed to finally predict the mindreading\nability of a patient from both the response and the corresponding input of video or\nstories.\n\nThis is a quite straightforward research. Because the proposed method uses more\ninformation than the baselines using only the responses, performance improvements are\nalso trivial. I could not understand where is novel or difficult for this research,\nthus I cannot believe this study is worth being published at EMNLP.\nMaybe this study should be published for psychological venues, rather than EMNLP, because\nit does not include any technical challanges that should be shared among NLP audiences.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rlVZbpesfg", "reviewer": "Reviewer_LmF8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n\nQ: The would be the lack of information regarding the number and characteristics of subjects in the case study.\n\nR: Thank you for your valuable suggestion. We will provide more information about the number of characteristics of subjects in case study in the revised version.\n\nTo address potential biases resulting from the exclusive representation of children from the United Kingdom (UK) in our dataset, we took deliberate steps. We carefully selected several child participants from regions outside the UK, both of whom were English native speakers aged between 10 and 11 years. Subsequently, we closely followed the methodology employed in prior psychological testing scenarios.\n\nOur testing protocol include both the \"Strange Stories\" and \"Silent Films\" assessments for these children. In the \"Strange Stories\" evaluation, their guardians read five distinct stories to them. For the \"Silent Films\" assessment, the children watched five silent film clips. Following exposure to the stories or videos, we posed relevant questions to the children and recorded their responses. The resulting data adhered to the formatting standards of the original dataset.\n\nTo ensure a robust evaluation process, we engaged two experts and one volunteer for human assessment. Each evaluator independently assessed the children's responses. Through thorough discussion and collaboration, a consensus was reached to establish final scores. The distribution of scores for the two participants is presented in the following table:\n\nSubject | Scores Adjudicated by Expert Evaluators\\\nChild-1 | 1 | 2 | 2 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2\\\nChild-2 | 1 | 2 | 1 | 2 | 1 | 2 | 2 | 1 | 2 | 2 | 2\n\nThe subsequent analysis reveals a strong alignment between our model's predictions and the evaluations by experts. The results convincingly demonstrate the applicability and effectiveness of our proposed models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a relatively new NLP task, Mindreading tests, which is commendable.\n- The motivation and structure of the proposed system are reasonable, and the experimental results demonstrate its effectiveness.\n- The inclusion of ablation studies and case studies enhances the credibility and reliability of the proposed system.\n- Overall, the paper is well-written and the experiments are comprehensive, making it suitable for EMNLP.", "reasons_to_reject": "- If I were to point out an area of concern, it would be the lack of information regarding the number and characteristics of subjects in the case study (section 7.3), as well as the distribution of the evaluation scores given by the human experts. Even just this experiment alone could potentially become a new paper, but addressing these issues is crucial to bolstering the reliability of the proposed system.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a system for automating the Child Mindreading Test.\nTraditional systems could only handle a single type of test.\nIn contrast, the proposed system uses different encoders for text and video and introduces a DUMA layer, enabling it to handle multiple types of tests.\nIn the experiments, the system deals with two tests: the Strange Stories Task (text) and the Silent Film Task (video), and it is confirmed to achieve higher score prediction accuracy than the baselines.\nThe main contribution of this paper is the realization of a model that can handle multiple types of tests, as described above.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uVp5NSdtUg", "reviewer": "Reviewer_J6oG", "summary": "", "questions": "Please see `Reason to Reject`", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.\n\nQ1: The task/dataset looks trivial. Overall, the paper lacks novelty. The paper directly applies the existing technique to a task without providing much technical contributions, making it less interesting to NLP community.\n\nR1: This study aims to assess children's mindreading ability by analyzing their natural language responses to viewed video clips and read stories. Our goal is to develop models that can swiftly identify children with mental health concerns, especially in regions with limited access to medical professionals. Previous research (Kovatchev et al., Coling2020; Kovatchev et al., ACL2021) addressed this task using question-answer pairs, disregarding contextual information from videos and stories. However, this approach presents issues: 1) Different contextual information for similar question-answer pairs can lead to contradictory evaluations. For example, without clip context, interpreting a man's actions as \"lying\" or \"telling the truth\" can both be valid. 2) Their methods hinder model improvement through diverse dataset training, causing conflicts and varying scores for similar question-answer pairs. As the training data increases, conflicts emerge within the dataset, where similar question-answer pairs may be associated with varying evaluation scores. 3) Relying solely on questions and answers compromises assessment accuracy.\n\nAdapting multi-modal models for reliable mindreading assessment is complex. Unlike traditional question answering, our task requires precise scoring for content-linked question-answer pairs. We propose a model inspired by machine reading comprehension, outperforming Kovatchev et al. (ACL2021) and making it possible to accommodate datasets from different sources. This adaptation enhances practicality for psychological assessment and diagnosis at a low cost.\n\nQ2: Section 7.3 doesn't include many details about the human study, e.g., how many experts and questions are involved, how much does r improve by the proposed method?\n\nR2: We appreciate your valuable suggestion. In the revised version, we will include more details about the human study.\n\nIn response to potential biases stemming from UK-exclusive dataset representation, we purposefully selected child participants beyond the United Kingdom (UK), aged 10-11 and English native speakers. Our approach adhered closely to established protocol in prior psychological testing scenarios.\n\nOur testing protocol encompassed the \"Strange Stories\" and \"Silent Films\" assessments for these participants. During the \"Strange Stories\" evaluation, guardians read five distinct stories, while for the \"Silent Films\" assessment, children viewed five silent film clips. Subsequent to exposure, relevant questions were posed, and responses recorded, maintaining alignment with the original dataset format. To ensure robust evaluation, we enlisted two experts and one volunteer for human assessment. Each evaluator assessed children's responses independently, and collaborative discussion led to consensus on final scores. Subsequent analysis revealed substantial alignment between our model's predictions and expert evaluations, validating the practicality and efficacy of our proposed models.\n\nIn comparison to the baseline, our model exhibited improved r-values: from 0.69 to 0.83 in the \"Strange Story\" task and 0.64 to 0.77 in the \"Silent Film\" task. These improvements underscore that our model's assessment results are more closely aligned with human experts on new data. We reiterate that the case study supports our model's reliability, and importantly, our framework's adaptability extends to a broader spectrum of general psychological assessment datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method improves from the baseline by 1-2% by using the latest model and taking the multi-modal context as the additional input. The model prediction also aligns well with the human experts, demonstrating its empirical value.\n2. The paper is well written and easy to follow. The technical details can help reproduce the experimental results.", "reasons_to_reject": "1. The task/dataset looks trivial. From Table 1, the answer-only and question-and-answer-only baselines already have ~90% test accuracy, showing that the task might be too simple or there might be some bias in the responses. It would be great to evaluate the proposed method on multiple datasets (if available) or a more complicated task.\n2. Overall, the paper lacks novelty. The paper directly applies the existing technique to a task without providing much technical contributions, making it less interesting to NLP community.\n3. As an application paper, I am more interested in knowing how much impact the proposed method bring to the real problem. For example, how much does the 1-2% offline improvement make a difference to the real-world evaluation? Section 7.3 doesn't include many details about the human study, e.g., how many experts and questions are involved, how much does r improve by the proposed method, etc. ", "questions_to_authors": "Please see `Reason to Reject`", "paper_topic_and_main_contributions": "This paper aims to leverage machine reading comprehension techniques and improve the automatic psychological evaluation system for children mindreading. In the existing automatic evaluation systems, the model only takes the question and answer as the input, leaving the context unused. The paper proposes a multi-modal learning framework that could take the stories and video clip into consideration in addition to the question and answer. The experiments show that the proposed system brings 1-2% gains against the baselines. A human study shows that the predicted scores agree well with the human experts.\n\n", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 281: corresponding --> corresponds\nLine 287:   score --> scores\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In general, all reviewers agreed that this is sound research.  There was mixed enthusiasm for including the work at EMNLP, since the primary objective of this research was on domain-specific task needs rather than on developing innovative NLP techniques.

**Summary of Reviewer Feedback and Discussion:**
- **Reviewer x51Z** thought that the work was sound but the research was straightforward and uncomplicated with only trivial performance improvements.  They thought the study may be more suited to non-CS venues, since the work didn't include NLP-oriented technical challenges.  In their rebuttal, the authors clarified that the technical challenges lied in extending multimodal models to the task, which differs greatly from existing multimodal task formulations (e.g., VQA).
- **Reviewer LmF8** liked that the paper introduced a new task, and thought that the authors' solution for this task was nicely motivated, reasonably structured, and convincingly evaluated.  They wished that more information was provided regarding subjects in the case study and the distribution of evaluation scores given by human subjects, and the authors provided this information in the rebuttal and promised to include it in the revised manuscript.
- **Reviewer J6oG** liked the results provided in the evaluation, but they felt that the task and dataset were trivial.  They felt that the contributions of the proposed method would be more convincingly demonstrated if it were evaluated on multiple datasets or a more complicated task.  They were also concerned that the paper's minimal technical contributions would make the paper uninteresting to NLP researchers, and wished that the paper clarified the real-world impact and potential benefits of the proposed solution.  In their rebuttal, the authors explained that the technial challenges lied in extending multimodal models for this task, included additional study information, and promised to include these details in the revised manuscript.  When responding to the rebuttal, Reviewer J6oG reiterated that the proposed method demonstrated performance improvements over the baseline, but that these results did not convincingly demonstrate task complexity or relevance to the NLP community.  In a follow-up response, the authors noted that their focus was not on innovating NLP technology, but rather on designing and improving NLP techniques for application to specific target issues in the domain of psychological assessment.
Decision: Accept-Findings