Can you Summarize my learnings? Towards Perspective-based Educational Dialogue Summarization
Authors: Raghav Jain
Tulika Saha
Jhagrut Lalwani
Sriparna Saha
Venue: EMNLP 2023 Findings (2023)
Keywords: Summarization
Text-generation
AI4Education

Abstract: The steady increase in the utilization of Virtual Tutors (VT) over recent years has allowed for a more efficient, personalized, and interactive AI-based learning experiences. A vital aspect in these educational chatbots is summarizing the conversations between the VT and the students, as it is critical in consolidating learning points and monitoring progress. However, the approach to summarization should be tailored according to the perspective. Summarization from the VTs perspective should emphasize on its teaching efficiency and potential improvements. Conversely, student-oriented summaries should distill learning points, track progress, and suggest scope for improvements. Based on this hypothesis, in this work, we propose a new task of Multi-modal Perspective based Dialogue Summarization (MM-PerSumm), demonstrated in an educational setting. Towards this aim, we introduce a novel dataset, CIMA-Summ that summarizes educational dialogues from three unique perspectives: the Student, the Tutor, and a Generic viewpoint. In addition, we propose an Image and Perspective-guided Dialogue Summarization (IP-Summ) model which is a Seq2Seq language model incorporating (i) multi-modal learning from images and (ii) a perspective-based encoder that constructs a dialogue graph capturing the intentions and actions of both the VT and the student, enabling the summarization of a dialogue from diverse perspectives. Lastly, we conduct detailed analyses of our model's performance, highlighting the aspects that could lead to optimal modeling of IP-Summ.

Can you Summarize my learnings? Towards Multi-modal Perspective-based Educational Dialogue Summarization
Raghay Jain', Tulika Saha", Jhagrut Lalwani', and Sriparna Saha'
DDept. of CSE, Indian Institute of Technology Patna, India
2 University of Liverpool, United Kingdom 3 Veermata Jijabai Technological Institute, India raghavjain106@gn il. com, saha tul ika1 50gmai com
Abstract
Cedel
bleat helen and Sitarte that bate about the fakla word for tise falhero and desio supacteely are our to which the katar ruggesia
The steady increase in the utilization of Virtual Tutors (VT) over recent years has allowed for a more efficient, personalized, and interactive Al-based learning experiences. A vital aspect these educational chatbots is summarizing the conversations between the VT and the stu- dents, as it is critical in consolidating learning points and monitoring progress. However, the approach to summarization should be tailored according to the perspective. Summarization from the VTs perspective should emphasize on its teaching efficiency and potential improve- ments. Conversely, student-oriente summaries should distill learning points, track progress, and suggest scope for improvements. Based on this hypothesis, in this work, we propose new task of Multi-modal Perspective based Dia- logue Summarization (MM-Per) (MM-PerSumm), demon- strated in an educational setting. Towards this aim, we introduce a novel dataset, CIMA-Summ that summarizes educational dialogues from three unique perspectives: the Student, the Tu- tor, and a Generic viewpoint. In addition, we propose an Image and Perspective- guiddd Dia- logue Summarization (IP- (umm)) model which is a Seq2Seq language model incorporating (i) multi-modal learning from images and (ii) perspective- based encoder that constructs a di- alogue graph capturing the intentions and ac- tions of both the VT and the student, enabling the summarization of a dialogue from diverse perspectives. Lastly, we conduct detailed anal- yses of our model's 's performance, highlighting the aspects that could lead to optimal modeling of IP (ummm..
Farer aeeeaayy hrve the dend''s Parpuetter
Tutar: Pirn ros Phess by tn
Doudar Okty Waria agait?
thr kaler mades and the correct erowor
the
aortee thes deir
Survear saton Syster
The nour cioune that Fiek mea
indechiee push as calo worls trisk the
habers and dinto regeecivily
the shadice ca lesl the adyectius follos the caaa billas
Figure I: Sample instance of MM-PerSumm task
2022), which, unlike traditional methods, are ac- cessible anyone, anywhere, anytime'. VTs are revolutionizing the education sector, breaching the barriers of geographical boundaries, and democra- tizing learning on a global scale. They offer a tai- lored learning approach, adapting to each learner's 's pace, aptitude, and preferences, thereby catering to diverse array of learning styles. While the imple- mentation of Al in education are evident, extracting valuable insights from these virtual educational in- teractions remains a challenging task.
Summarizing insights from VT carries signifi- cant implications for both the improvement of AI- based educational systems and the learning out- come of students. With growing reliance on these AI- AI-powered tools for teaching and learning, it be- comes crucial to gauge their teaching efficiency, highlight potential areas of improvement, and as- sess their overall impact on learners' progress. For learners, summaries provide a distillation of key learning points, tracking their learning journey and suggesting areas for future study, which could be pivotal in informing their learning path. While dialogue summarization techniques (Feng et al., 2022) have seen extensive development in several domains, their application to educational interac- tions, particularly with VTs, has not been thor-
1 Introduction
Artificial Intelligence (AI) is making significant strides in the field of education, paving the way for numerous innovative applications. The power of Natural Language Processing (NLP) and advanced machine learning techniques have given rise to Vir- tual Tutors (VTs) (Stasaski et al., 2020; Jain et al.,
htttp::/tehhaad.gg///ll/////////

---

oughly explored. Furthermore, the importance of summarizing these dialogues from multiple per- spectives - those of the tutor, the student, and a neutral observer- is distinct requirement in this domain (Jain et al., 2022). These perspectives can reveal diverse, valuable insights that can contribute to the evolution of virtual tutoring systems and the individual learning journey of students. Thus, this area holds vast potential for exploration and advancement in the field of AI-driven education. To address this signifieant gap in the field, the contributions of this paper are : (i) Propose a new task of Multi-modal Perspective based Dialogue Summarization (MMmP)rSumm) which is demon- strated in an educational setting; (ii) As step to- wards this goal, we extend an existing education based conversational dataset, CIMA (Stasaski et al., 2020) by summarizing the dialogues from three different perspectives: that of the student, the tutor, and a generic viewpoint to create CIMA-Summ which serves as the foundation for our research; (iii) To address the task of MM-PerSumm, we pro- Dialogue Sum-
troom Corpus (Caines al., 2022). Recently, lot of attempts have been made create a VT system utilizing these datasets (Jain et al., 2022; Macina et al., 2023).
Dialogue Understanding. In the context of the Social Good theme, there have been upsurge of research in the latest time focused on understanding dialogues. For example works in mental health con- versations (Saha et al., 2022a,b,c, 2021), disease diagnosis assistant (Tiwari al., 2022; Saha al., 2023), education (Jain et al., 2022) etc. A broad spectrum research has been done on dialoguc summarization spanning different domains. One of the notable sectors includes meeting summariza- tion, where advancements have been observed in both extractive and abstractive techniques (Wang, 2022; Rennard et al., 2023). Similarly, chat sum- marization has been focus due the explosion of messaging apps and business communication platforms (Gliwa et al., 2019; Feng et al., 2021). Email thread summarization (Carenini et al., 2007) is another domain that has seen significant research interest. The complexity of the task lies in iden- tifying the salient information buried in long and often nested conversations (Mukherjee al., 2020). The domain of customer service summarization (Liu al., 2019) has more specific goal, which to extract customer issues. Finally, medical dia- logue summarization (Joshi et al., 2020; Song et al., 2020; Enarvi et al., 2020; Wei et al., 2023) has seen substantial development, driven by the need for concise patient-doctor conversation records.
pose Image and Perspective- marization (IP-Summ) model which leverages a Seq2Seq language model with perspective- based encoder. It is designed construct dialogue graph, capturing the intentions and actions of both tutors and students, thereby enabling the summa- rization educational dialogues from diverse per- spectives. By incorporating multi-modal learning in the form of image-based instruction as a con- textual cue, our approach allows the /P-Summ to leverage both textual and visual information for more comprehensive and accurate summarization.
3 Dataset
To facilitate research in educational dialogue sum- marization, we create the which is based on an existing conversational dataset for educational tutoring.
2 Related Works
dataset,
In this section, we detail some relevant works in the context of Educational-NLPP and Dialogue Sum- marization in general.
3.1 Data Collection
Education and AL. With the advent of AI, tailored educational materials can be generated based on individual learning styles and abilities (Bhutoria, 2022). Automated feedback systems offer timely evaluations of assignments and homework (Ku- mar et al., 2022; Filighera et al., 2022). Educa- tional question generation is another area that has witnessed significant advancements (Elkins et al., 2023). In the context of virtual tutoring, advance- ments in NLG have paved the way for interactive and immersive learning experiences. This led to the creation of AI-related datasets, such as CIMA (Stasaski et al., 2020) and the Teacher-Studee Cha-
This study builds upon the existing CIMA dataset (Stasaski et al., 2020), a valuable resource the field of dialogue systems, particularly in the con- text of tutoring dialogues. This dataset was cho- sen because it is uniquely positioned in the inter- section of dialogue systems and education while also being supplemented with a rich set of features like object images and intent-action labels. CIMA dataset contains one-to-one student-tutor conversa- tions where the aim is to assist students in learning the Italian translation of an object and its character- istics. Each object discussed within the dialogue

---

is also supplemented with corresponding image. Each student utterance contains intent tags, namely, Guess, Question, Affirmation, or Other, while the Tutor's action are categorized as Question, Hint, Correction, Confirmation, and Other. However, in its original form, the CIMA dataset provides the intent-action labels only for the concluding student utterance and the corresponding gold tutor response for each dialogue. To overcome this limitation, draw upon the silver oction-intent labels from the extended- CIM dataset (Jain al., 2022).
tems. The intention was provide diverse set of examples that demonstrated range of topics and dialogue scenarios, giving the annotators an understanding of the depth and breadth of the task. Feedback sessions were arranged where the anno- tators interacted with the experienced researchers to discuss the evaluations and ways to enhance the quality of the summaries. More details on how the annotators were trained is detailed in the Appendix section. Finally, the main annotation task was con- ducted using the open-source source platform Doccano", deployed on a Heroku instance. Given the intricacy of the task and the need to ensure a high high-quality an- notation, we followed a structured schedule over a span of five days in a week (the details of which are mentioned in the Appendix section). This annota- tion process took approximately five weeks to com- plete. The final annotations demonstrated a high level of quality, with average fluency and adequacy scores of 4.98 and 4.87, respectively. Furthermore, the evaluators showed high ageeemttreeeent levels, with unanimous scores of 94.7% for fluency and 89.1 % for adequacy ratings. More details on the annotation quality assessment is detailed in the Appendix section.
3.2 Data Annotation
A detailed annotation approach was followed which is discussed below.
Annotation Guidelines. A clear set of annotation guidelines was established which were aimed at providing a structured way for annotators to distill key points from the dialogues and summarize them from three distinct perspectives: the student, the tutor, and an overall dialogue summary.
Student Perspective Annotators were tasked with crafting summary that encapsulates the key takeaways for the student from the dialogue. This includes the new knowledge gained, any miscon- ceptions corrected, and the overall progress in un- derstanding. This summary should reflect what the student has learned during the interaction.
3.3 CIMA-Summarization: : CIMA-Summ
Dataset
Tutor Perspective For the tutor perspective, annotators were instructed to focus on the teaching strategies employed by the tutor, the clarification of student doubts, and the overall guidance provided.
The CIMA-Summ dataset comprises of 1134 di- alogues with each dialogue annotated with three different perspective amounting to a total of 3402 parallel summaries accompanied with visual cues. A sample instance from the CIMA-Summ dataset is shown in Figure 2.
The summary should highlight the tutor's effort in facilitating the student's 's learning.
Overall Perspective This is aimed at present- ing a balanced view of the conversation both from the student's and tutor's perspective, highlighting key dialogic exchanges, instructional elements, and learning outcomes. More details on the annotation guidelines is presented in the Appendix section.
Qualitative Analysis. Figure 3(a) shows the distri- bution of word count for three type of summaries. The median length of the student's summary is less than the tutor's 's summary, which turn is less than the overall summary. Tutor might tend to use more technical language in the interaction to accurately convey the information, which can contribute to longer summarics compared to students, who may only focus key points. Overall summary, encom- passing both tutor's and student's perspectives. is bound to have the highest median length amongst the three. Figure 3(c) shows the distribution of similarity for pairs of summaries. It evident that the similarity between the tutor and overall sum- maries is the highest. A possible reason for this could be that the tutor provides more detailed in-
Annotation Process. We hired three annota- tors from the author's affiliation who were grad- uates in English Linguistics with substantial fa- miliarity in the educationallaccademmi background for our annotation task. The annotators were first trained to summarise dialogues using a set of pre- annotated gtad-standard samples from the CIMA- Summ dataset. These examples were annotated with three types of summaries (student perspective, tutor perspective, and overall dialogue summary) by two experienced researchers (from author's 's col- laboration) specializing in educational dialogue sys-
Pht c

---

Dialogue History
Generic Summary
Student Summary
Tutor Summary
T: Please try to fiV the blank Italian.
Tutor talls student the blenk in ittiaan The futor says that student has left fire "v" the "conigtlo" which makes masculine sunguitar.
e frante of conigl viols.
Tutor tells student to A in live blank in Italan.. Student answers "o di fronto al conighi viale which almout correct The saya the student isft word "conigho" whicn masculine and singuaar
Student ariswerz "a frante at congli viola which almost
T: Oh! You almost got it! Very good. You just left off the "o" on the word "ooniglia"" makes mesculin and singular.
Figure 2: Sample dialogue history and its perspective based summaries from the CIMA-Sume dataset
Types
Crend Connery
Cummeny
Lengt Diertrroon Smattarr
Overal
TukeeDverall
Stattoer www.rncccaceiiooooooooo
Titrr
TunnStudent
Word Count
Wond Elnerr
Cisine Similarity
(a) Distribution word count for type
(b) Histogram of Word Count for
(c) Distribution of similarity for pairs of
of summaries
type of summaries
summaries
Figure 3: Distribution statistics of the CIMA-Summ dataset
formation containing a higher level of complexity, Also, tutor and student summaries are least simi- lar, possibly because students concentrate only on the most salient ideas while taking notes. Figure 3(b) represents a histogram of word count for these three type of summaries. For 98% of the conversa- tions, tutor's summary is longer than the student's summary, reflecting a deeper understanding com- prehensive analysis of context by the tutor.
conditioned on the perspective and image.
4.1 Image and Perspective-guided Dialogue Summarization (IP-Summ) Model
This section presents the novel architecture of IP- Summ, perspective-guided. multimodal Seq2Seq model (Figure 4) that synthesizes dialogue and im- age context to enhance educational dialogue sum- marization. IP-Summ comprises of three distinct modules: (1) Global Context Encoder, (2) Perspec- tive Context Encoder, and (3) Contextual Fusion Module discussed below.
4 Proposed Methodology
this section, we discuss the problem statement and proposed methodology in detail.
4.1.1 Global Context Encoder
Problem Description. Our proposed task Multi- modal Perspective based Dialogue Sunmarization SMM-P))Summ) defined as follows:
This integral component of our model serves as the primary mechanism for understanding the overall essence and the key points of the dialogue. It pro- cesses the complete multi-turn dialogue, D along with the corresponding image, / and generates a global context representation, ZG providing high- level understanding of the conversation.
Input: The input comprises of three entries: (1) The source dialogue D {To < Aco So <
Tno >, < Ac S1 < In1 >,..., which is conversation between VT and student where T. and Ac; represents the VTs utterance and its corresponding action tag. S, and InA represents the student's s utterance and its corresponding intent tag; (2) An associated visual image which used by the tutor to facilitate the teaching process; (3) A specified perspective, P. which represents the view- point (either student, tutor, or a neutral observer) from which the dialogue is to be summarized.
Dialogue Context Encoder. Firstly, the source dialogue, D, fed into language encoder, Lene(.). This language encoder is function that maps the raw dialogue text into a gigh--imensionall space. The output of this operation is a vector representa- tion, HD of the dialogue.
Image Encoder. In parallel, also feed the associated image, I into vision encoder, Vene( This encoder is function that processes the im- age and converts into a high-dimensional vector representation, H,. Dimensionality Reduction. Following their
Output The output natural language se- quence, Y. representing the summarized dialogue from the perspective,
P. Given an instance of D, I, and P, Y can be manifested as diverse summary

---

Generated Summary
GTX Rision Marhad
Dama Laper
Decocer
BAKI
OTK Fusion Nuihle
Olde Casted
Features
Dene Log
Perapactive Encodar
Repportuiin Koral
Hilloot Space
Weie frander
BUTTEande
Perspective Dialogus Graph
Input
Parspactive
IP-Summ
TE: by
Quess
an frw tixee 2sp=e[Gunnd
Action
Ouess
Conversation
care deeoosffisnn 13:
dano" eqji
Contim.
Graph
Affim..
"ree ls "abeepllsllirt natoon)
900
Same
Tht Contec poodljttt
Diaiogue Graph
Hitt
120
Hit
Figure 4: Overall architecture of IP-Summ
generation, these text and image representations dergo dimensionality reduction procedure. This is facilitated through nonlinear transformation, succeeded by self-attention layer, as captured by the following equations:
4.1.2 Perspective Context Encoder
Depending upon the specified perspective (i.e., stu- dent, tutor, neutral observer), this encoder pro- cesses the dialogue and generates a perspective- specific context representation. This ensures that the nuances, intentions, and actions associated with (1) the chosen perspective are accurately captured and represented.
HDH HD
HD SoftMarl
Vd
Hi HH
H SoftMar
Vd
(2)
Perspective driven Dialogue Graph Construc- tion. The Dialogue Graph, denoted as G = (V, E), comprise of nodes, V representing utterances and edges, E illustrating the transition between these utterances. The methodology is intrinsically adapt- able the specified perspective, P (i.e., general, student-focused, or tutor-focused), allowing us construct graphs tailored to different perspectives discussed as follows:
Multi-modal Fusion. Following this, the two vectors are concatenated and passed through an Op- timal Transport-based Kernel Embedding (OTKE) layer (Mialon et al., 2021). This layer aims to fos- ter cross-modal interaction between the textual and visual contexts. The OTKE layer facilitates this by mapping the feature vectors into a Reproduc- ing Kernel Hilbert Space (RKHS) (Berlinet and Agnan, 2011) subsequently implement- ing a weighted pooling mechanism. This mecha- nism leverages weights that are determined by the transport plan between the set and learnable ref- erence. This operation results in single vector representation, Zg. which is given by:
Independent of the perspec- tive, P. every utterance within the dialogue, D is transformed into node in G. Every node, E V represents utterance, (S.) or (Tt). where S; and T: stand for student's and tutor's 's utterances, respec- tively.
Node Creation
Edge Creation : The formulation of edges graph, G is intrinsically tied to the chosen perspec- tive, P. Representing an edge from node to node j as e(j) each edge is attributed with a label, 4(0) reflective of the corresponding action or intent tag.
ZG = OTKEE(HH> H())
(3)
This vector, ZG encapsulates the global context, capturing both the textual and visual features the dialogue-image image set.
i) General Perspective For an edge between each consecutive pair of utterances dialogue

---

of length N, we establish: e(t) for all j such that j = i+ and i ,o. .V- 2}. The label, (11)) is assigned as per equation Addi- tional edges, E(ay), are interjected between nodes associated with identical action/intent tags.
turns. The Levi graph, L(G), with nodes denoted by V, is initially embedded using word2vee, pro- ducing node embeddings of dimension d. This graph is then subjected to several rounds of convo- lutions via a relational graph convolutional network (Schlichtkrull et al., 2017). After three convolution rounds, max-pooling graph operation is used to generate graph embedding, Hgg. In parallel, vec- representation, HD of the dialogue is generated. Following this, both HD and HG are subjected dimensionality reduction via a nonlinear transfor- mation followed by a self-attention layer, resulting in H's and HD Lastly, the OTKE layer infuses the perspective graph into the language representations of the dialogue, yielding the final vector, Zp. This vector encapsulates the dialogue's 's language repre- sentations along with perspective-specifi nuances.
ii) Student-focused Perspective When our perspective. P is focused on the student, the edge creation process remains invariant, but the label assignment function prioritizes the student's intent tags (Equation 4). We also introduce an additional edge, (ray) labeled 'same intent' for every pair of student utterances (x, y) that share the same intent. iit) Tutor-focused Perspective : When P is tutor-focused, our label assignment function priori- tizes the tutor's action tags ((Equation 4)). Under this perspective, an additional edge, {(ry) labeled 'same action' is introduced for each pair of tutor utterances (Ir, y) that share the same action tag.
4.1.3 Contextual Fusion Module
Ac;
utt. i E T &P E 'T' Gen' utt. i E S &P E ' Gen' if utt, i S &P T if utt. i T = S
The Contextual Fusion Module serves as an inte- grator, fusing the information from both the Global and Perspective Context Encoder. We utilize the OTKE fusion module to fuse the context represen- tations, establishing a synergy between the global and perspective-specific contexts. The vectors, Zg and Zp from the Global and Perspective Context Encoder, respectively, undergoes the OTKE op- eration to create a fused representation, Ziinat which ensures a more nuanced understanding of the dialogue by capturing and integrating diverse insights. The vector, ZFinol is then subjected to multi-headed self-attention mechanism to yield Z Final- This final vector, ZFimal is subsequently processed by a standard BART decoder, transform- ing the context-rich vector representation into a dialogue summary.
Ing
bi
'contezt'
'context'
(4)
where context label indicates that the utter- ance provides contextual information to the stu- s/ttor/tutor's responses.
Levi Graph Generation. Upon the comple- tion of the directed graph. G generation, the next step is to construct the corresponding Levi graph, [G)) inspired from graph theory. A Levi graph is a representation that preserves the relational struc- ture of the initial dialogue graph but further em- phasizes the linkage between utterances sharing the same intent or action. Specifically, L(G) is obtained by introducing additional vertices corre- sponding to the edges of G. Each edge, Cij in G becomes vertex, fij in L(G), where and j are utterances in the dialogue. Consequently, if two edges, eij and egl in G share a node (j- k), they are connected in L(G) by an edge enyra. Further, we preserve the context by adding edges between new vertices corresponding to consecutive turns in the dialogue. If eij and Cjk are consecutive turns in the dialogue in G, we add an edge, Gugy,s in TC)),, hence maintaining the sequential structure of the conversation. This transformation from G L(G) allows for more nuanced view of the dialogue's dynamics, highlighting both the sequen- tial and structural aspects of the conversation, as it captures not only the utterance transitions but also the shared intents or actions across different
5 Experiments and Results
This section details the experimental setup fol- lowed by results and analysis of IP-Summ.
5.1 Experimental Setup
The training was conducted for 20 epochs with a learning rate of 5 x 10-5, a batch size of 16, using the Adam optimizer, and an Adam epsilon value of x 10 8 Our proposed model, as well as all the ablated models, are built top of the BART- Base (Lewis et al., 2019) architecture. We split the 1,000 articles from CIMA Dataset into 80% for training, 10% for validation, and 10% for test- ing. The performance of the generative models was evaluated using several metrics, including aver-

---

Overall Summary
Student Summary
Summary
Model
R
B
BS
R
B
BS
R
B8
BART
0.31
0.25
0.74
0.39
0.4
0.72
0.27
0.22
0.71
TS
0.2
0.23
0.72
0.25
0.33
0.71
0.22
0.21
0.72
Baselines
DialoGPT
0.25
0.21
0.77
0.36
0.4
0.72
0.27
0.23
0.75
MultinodalBART
0.38
0.26
0.81
0.43
0.42
0.78
0.33
0.25
0.81
Summ
0.25
0.24
0.77
0.39
0.41
0.82
0.31
0.24
0.81
IP JP-Summ- - VT
0.42
0.26
0.87
0.54
0.46
0.84
0.39
0.28
0.84
Ablation Model
IP-Summ- RGCN
0.43
0.28
0.87
0.56
0.45
0.88
0.41
0.28
0.85
IP- 1Pmmamm- LGG
0.26
0.85
0.55
0.87
0.85
Proposed Model
IP-Sumum
0.46 0.3
0.91
0.59
0.49
0.93
0.43
0.9
Table I: Comparison of IP-Summ with other baselines and ablation models on automated metrics. R: ROUGE-L score, Average BLEU Score, BS: BERTScore Fl and LGG: Levi Graph Generator
age BLEU score (Papineni al., 2002), ROUGE-L score (Lin, 2004), and BERTScore Fl (Zhang et al., 2020a). Furthermore, we conduct a human eval- uation of the most effective models. We evaluate the model's 's output on the following aspects: Flu- ency, Informativeness, and Relevance. To assess these models in terms of human evaluation, three independent human users from the authors' affili- ation were asked to rate 100 simulated dialogues on a scale of (worst), (moderate), and (best) based on the above-mentioned criteria. The final reported score is the average of the human- raate scores. In our study, we used the following models baselines: (1) BART model (Lewis et al., 2019), (2) T5 model (Raffel et al., 2020), (3) DialoGPT model (Zhang al., 2020b), (4) MultimodalBART (Yu al., 2021), and (5) Con_SSumii (Liu and Chen, 2021), which is the only work to date on perspective-based conditional dialogue summariza- tion. Please refer to Appendix A.3 for details.
baseline models, BART and DialoGPT delivered relatively comparable results across BLEU and BERT-FI scores, though BART displayed a slight edge in Rouge scoring. Conversely, T5 lagged in performance across all metrics.
With the help of ablation, we obtained crucial insights into the role of individual components within the IP-Summ model. When the ViT com- ponent was omitted, there significant reduc- tion in the metric scores across all perspectives. This highlights the effectiveness of our proposed global context encoder, which is instrumental integrating visual context into the dialogue. We observed a similar trend when the RGCN and Levi Graph Generator components were excluded from IP- (P-Sum.. Without the RGCN, there was con- sistent decrease across all metrics and summaries, which underscores the importance of the dialogue graph encoder. The same trend was evident when we substituted the Levi graph with a standard graph. indicating that the Levi graph encapsulates richer information about the perspective compared to a standard dialogue graph. Additionally, a pattern emerges when we examine the scores across the student and tutor perspectives. For all models eval- uated, including IP-Summ, the scores for the stu- dent perspective are consistently higher than those for the tutor perspective. This pattern is indica- tive of the relative complexity of generating tutor summaries compared to that of student summaries.
5.2 Results and Discussion
A careful review of Table reveals that our pro- posed model, IP-Summ, excelled across all metrics and perspectives. This model embodies excellent capabilities in text summarization, as demonstrated by its impressive scores: a Rouge score of 0.46, a BLEU score of 0.3, and a BERTSccre of 0.91 when assessed from general viewpoint. IP-Summ's s strong performance remains consistent when evalu- ated from the perspectives of both the student and the tutor. This implies the model's adaptability in understanding and summarizing various perspec- tives, a feature that can be credited to the utilization the global and perspective context encoder.
Effect of different Components. In addition, we have illustrated the impact of various vision encoders, fusion mechanisms, and graph encoders on our model in Figure 5. (1) When implemented with the Vision Transformer (ViT) (Dosovitskiy et al., 2021), our model demonstrated superior performance across all perspectives compared other vision models. This highlights the efficacy of transformer-bused vision models effectively modeling image features. The subsequent most
Comparison with the Baselines. Delving into the baseline models, MultimodalBART emerged as the front-runner among this group. scored the highest across all evaluation metrics- Rouge (0.38),
BLEU (0.26), and BERTScore (0.81)- presenting a respectable benchmark for comparison. Among the

---

Performance Visien Medels
Perfernance Comperisan Fusion motheds
Performance Comgaricon
0373
Ovend
644
0.1
ea1
AM
6d4
FA1
BAT
A44
A4
641
0313
0330
Tele Seees
643
641
64 0.9
0.6
643 642 036 443 9.2 64
0403
Figure 5: Heatmap displaying variations in ROUGE scores and illustrating the impact of altering different com- ponents within the proposed model. (CASA.Contett-wwal Self Attention, CMA: Cross-Moda Attention, CON: Concatenation)
Overall Summary
Stndent Summary
Tulor Summury
feetiveness in language generation. However, the picture becomes more differentiated when consider- ing the metric of informativeness. Here, the BART model, scoring 3.6, did not perform as well its counterparts. The absence of a visual context com- ponent in BART likely to be factor, as this com- ponent plays crucial role in generating summaries rich in key information. By contrast, the /P-Summ and MultimodalBART models, incorporating a vi- sual context component, displayed enhanced per- formance in this area, achieving scores of 4.5 and 4.3, respectively. As for the metric of relevance, the IP-Sumu model showed superior performance with score of 4.3. The model's perspective encoder, which aligns the summary with the specific perspec- tive, likely plays a significant role this outcome. The scores achieved by MultimodalBART (3.9) and BART (3.3) in this area underline the impor- tance of incorporating perspective-specifee encod- ing mechanisms in models tasked with generating relevant summaries. To ensure the reliability of our human and automatic evaluations, we computed inter-annotate agreement for human judgments and statistical significance tests for metric improve- ments. For human evaluation, we calculated the rate of majority agreement across annotators on the acceptability of model responses. This yielded an inter-annotator agreement score of 72.3%, indicat- ing reliable consensus. For automatic metrics, we conducted the statistical significance Welch's 's t-test at 5% significance to ensure that the improvement of our proposed model over the baselines is reliable. All the reported results are statistically significant.
Model
FL.
IN
RE
FL
IN
RE
FL.
IN
RE
BAKT
3.6
33
12
4.1
33
38
MultimodalAART
4.3
39
4.5
38
IP-Summ
4.3
4.5
43
45
4,6
43
4.1
4.2
4.4
Table 2: Human evaluation scores of proposed model,
IP-Summ and corresponding best baselines. FL:Fluency, INInnformativeness RE:Relevance
successful vision model was CLIP (Radford et al., 2021). trailed by VGG-19 (Simonyan and Zisser- man, 2015), ResNet (He et al., 2015), and Swin (Liu et al., 2021) vision models. This sequence in- dicates the dominance of VIT and CLIP over more conventional image encoders. (2) The incorpora- tion of the OTK fusion module into our model re- sulted in a consistently higher performance across all metrics compared to other fusion techniques, thus, illustrating the superiority of the OTK fusion mechanism. Following the OTK fusion module, the next most effective mechanism was context-aware self-attention (Yang et al., 2019), which was then succeeded by cross-modal attention and, finally, concatenation. This pattern demonstrates how the utilization of more complex, sophisticated, and robust fusion mcchanism can significantly enhance the results. (3) The use of the RGCN encoder led to improved performance in comparison to other graph encoders, such as GAT (Veličković et al., 2018) and GCN (Kipf and Welling, 2017). This superior performance may be attributed to RGCN's ability to handle multi-relation graphs, making an ideal choice for our Levi graph which contains multiple relations and nodes, unlike standard dia- logue graph.
Human Evaluation. Table 2 presents the re- sults for BART, MultimodalBART, and SumSum In terms of fluency, all models demonstrated ad- mirable performance. IP-Sunm led this category with a score of 4.3, followed closely by Multi- modalBART (4.2) and BART (4). The metric's high scores across all models underscore their ef-
Qualitative Analysis of Generated Summaries. In Figure 6, we showcase two instances of our model-generated student, tutor, and generic sum- maries, alongside the ground truth summaries. The results demonstrate that the generic summaries gen- erated by our model, IP-Summ, align well with the

---

Redega Hhieery
Greand Trrth Cecete
P-Oure Genero
Groaad Trush
re-aamm Surmsay
Sroted Trubs
Suswary
7. Plaas a the
Tuler Rathuts Bl the
Tubr asta sudeetto
Tobe instreds to Fal an Tuaer trea
bakidd
5: dinon da
takee ecad for "tetinaf
aas jas ad
Stucent warts to tove nuf und
Cl Sert sortt de bntrndden Bop. lnarre
dato ler.
you lint but and but
zut nd thr cutrola
Tubl axclare
"s catoo luf,
"oukke and
bow any vry rkgii
"eugt Lai cays Tour's
tax and snitas
That "org"" "acaa"
Hiwe depauryy doggr: T.
uehar couul be wore
cacal by Tkois
glval fro
Tube intrach
Mennin talen wout Will
the liwn Babur dtakent
thared
Lirn ailleies
Per Ty
tan vnrter la bean
ther Thoc" and
por dean
babeet Tood" "yoloo"
Vota a ber
rau ueloe "Mca"
repponts
oxples
"har'l "ccaaa what giv you "olloo" do bille la uoalli paar7 F. ol okila
deae "ucataoria and
eeplain in dhacie
and yellww/
aneuers raalil
gasar.
and Splll:
lube
After
conuet the suilling
"Cooenn la r
soiling emonera
mut conves
Uie Teecoodlig
Figure 6: Sample instance presenting the gold standard summaries and the summaries generated by IP (P-Summ from multiple perspectives
Sunmery
Burnmery
Stadkett Summery
Summmary
Tutar Burrmaryy
Summrary
Plosse to fikv
Tuloe insivadt toll the fallia reed für "taineaf benine thet iss dito lor.
Tolor lruln
& howels you
alansslala houe hot in tallan the cane' means dog
Plubket weiins "hanin and
lost han ang tarind.
halla's Tuker failss "o claso inf And "bod and "Veg" "ann"""
Tulae
you that laa behina that
Baniig
how you eq tapi
"eag". Tuker eigs thoo's in
sscdeet vith
Hoo you aay digh T.
what nouid bo the tallan word cand"
slakk Jullnn shatee
twildenn tulles Stadert
The incnuats
Shobet seliwiite Taoe" dei areves glaar
boache
betine thal, "Tac" and
mockiiiing Govarias lo "pcrtoat and
so talln wodk for and lou"
"soc'la la laardia de sout give you hert ye lou" in aointe plann" T. have apo, glalo
capo lnt, "uailal and
cravarr deir la acable acveet the spellin et
and "gale"
"aaia" and
gelling orooor
"Clveri""
studon lo connec "peilr"
bur Autioo.
Figure 7: Sample instance presenting the gold standard summaries and the summaries generated by the baseline BART model from multiple perspectives
ground truth summaries in both the given exam- ples. However, there are instances where IP-Summ falters in other perspectives. For instance, in the first example, /P-Summ''s 's student summary mistak- enly incorporates information from the tutor about "giving student examples." Additionally, we ob- serve that IP-Summ's 's tutor summary covers all the core concepts but lacks fluency compared to the student summary, indicating the inherent difficulty in generating tutor summaries as opposed to stu- dent summaries. Figure 7 shows sample generated summaries (for the same instances in that of fig- ure 6) from the baseline BART model. Some more analyses of the generated summaries are reported in the Appendix section.
aspect by proposing new task of Multi-modal Perspective-bused Dialogue Summarization (MM- PerSumm) in an educational setting, which paves the way for a holistic understanding of the VT- student exchanges. We introduce a novel dataset, that features dialogue summaries from diverse perspectives, offering a multi-faccted view of the learning interaction. We further pro- pose model, Image and Perspective logue Summarization (IP-Summm) which effectively incorporates both image context and dialogue per- spectives into the summarization process.
Dia-
Acknowledgement
Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corpo- ration (formerly Media Lab Asia) for carrying out this research.
6 Conclusion
The rapid adoption of VTs has brought forth novel challenges and opportunities in enhancing the learn- ing experience amongst which dialogue summariza- tion stands out as a pivotal area warranting dedi- cated attention. Our work addresses this crucial

---

Limitations
statistics. Springer Science & Business Media.
Despite its contributions, the present study ac- knowledges several limitations. The original CIMA dataset only includes 1 1134 conversations, which re- stricts to generating summaries solely for these dialogues. This underlines the urgent need for more extensive and comprehensive datasets in a variety of educational settings. Furthermore, the specific application of our approach to language learning may not imply similar efficacy in other educational contexts, such as mathematical or science based learning. Therefore, further validation in these ar- eas is essential to confirm the wider applicability of our proposed method. In addition, this study did not incorporate the potential use of Large Language Models (LLMs), which could offer additional in- sights and improvement of educational strategies. Future research should aim to overcome these lim- itations and further explore the potential of our approach.
Aditi Bhutoria. 2022. Personalized education and ar- tificial intelligence in the united states, china, and india: A systematic review using a human-in-the- loop model. Computers and Education: Artificial Intelligence, 3:100068.
Andrew Caines, Helen Yannakoudakis. Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2022. The teacher-student chatroom corpus version 2: more lessons, annotation, automatic detection of sequence shifts. In Swedish Language Technology Conference and NLP4CALL, pages 23-35.
Giuseppe Carenini, Raymond T. Ng. and Xiaodong Zhou. 2007. Summarizing email conversations with clue words. In Proceedings of the 16th International Conference on World Wide Web, WWW '07, page 91-100, New York, NY, USA. Association for Com- puting Machinery.
Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale.
Ethical Considerations
In the course of this research, we committed to the highest standards of ethical conduct. The data used for this research was derived from pre-existing, anonymized, and publicly available dataset, en- suring the privacy and confidentiality of the indi- viduals involved. In the creation of our unique dataset, CIMA-Summ, we employed annotators who were compensated in accordance with institu- tional norms. We took great care to ensure that their work was scheduled within normal working hours, thus, promoting healthy work-life balance. More- over, the dataset was constructed in a way to ensure it does not promote or favour any demographic, race, or gender, thus, striving to mitigate the risk of potential bias in language models trained over this. The design and implementation of our model were guided by robust commitment to avoiding potential harm. The IP-Summ model is purely a tool for dialogue summarization, and it does not involve any predictive or prescriptive functionali- ties that could impact individual users negatively or unfairly. This model also doesn't require to store any personal information about students and tutors, thus, preserving the privacy of users.
Sabina Elkins, Ekaterina Kochmar, Jackie C. K. Cheung. and Julian Serban. 2023. How useful are educational questions generated by large language models?
Seppo Enarvi, Marilisa Amoia, Miguel Del-Agua Teba, Brian Delaney, Frank Diehl, Stefan Hahn, Kristina Harris, Liam McGrath, Yue Pan, Joel Pinto, Luca Ru- bini, Miguel Ruiz. Gagandeep Singh, Fabian Stem- mer. Weiyi Sun, Paul Vozila, Thomas Lin, and Ran- jani Ramamurthy. 2020. Generating medical reports from patient-doctor conversations using sequence-to- sequence models. In Proceedings of the First Work- shop on Natural Language Processing for Medical Conversations, pages 22-30, Online. Association for Computational Linguistics.
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2022. A survey on dialogue summarization: Recent ad- vances and new frontiers.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021. Language model as an annota- tor: Exploring DialoGPT for dialogue summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Pupers), pages 1499, Online. Association for Computational Linguistics.
Anna Filighera, Siddharth Parihar, Tim Steuer, Tobias Meuser, and Sebastian Ochs. 2022. Your answer is in- correct... would you like to know why? introducing bilingual short answer feedback dataset. Proceed- ings of the 60th Annual Meeting of the Association
References
Alain Berlinet and Christine Thomas-Agnan 2011. Re- producing kernel Hilbert spaces in probubility and

---

for Computational Linguistics ( (Volume Long Pa- pers), pages 2577-8591, Dublin, Ireland. Association for Computational Linguistics.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows.
Soumitra Ghosh, Asif Ekbal, and Pushpak Bhat- tacharyya. 2022. Am no good? towards de- tecting perceived burdensomeness and thwarted be- longingness from suicide notes. arXiv preprint arXiv:2206.06441.
Zhengyuan Liu and Naney Chen. 2021. Controllable neural dialogue summarization with personal named entity planning.
Jakub Macina, Nico Daheim, Lingzhi Wang. Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. Opportunitics and challenges in neural dialog tutoring.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Grégoire Mialon, Dexiong Chen, d' Aspremont, and Julien Mairal. 2021. A trainable optimal transport embedding for feature aggregation and its relationship to attention.
Alexandre
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recogni- tion.
Sudipto Mukherjee, Subhabrata Mukherjee, Marcello Hasegawa, Ahmed Hassan Awadallah, and Ryen White. 2020. Smart to-do: Automatic generation of to-do items from emails. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8680-8689, Online, Association for Computational Linguistics.
Raghav Jain, Tulika Saha, Souhitya Chakraborty, and Sripana Saha. 2022. Domain infused conversational response generation for tutoring based virtual agent. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACI. '02, page 3-1-318, USA. Association for Computational Linguisties.
Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020, Dr. summarize: Global sum- marization of medical dialogue by exploiting local structures.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Thomas N. Kipf and Max Welling, 2017. Semi- supervised classification with graph convolutional networks.
Rahul Kumar, Sandeep Mathias, Sripamn Saba, and Pushpak Bhattacharyya. 2022. Many hands make light work: Using essay traits to automatically score essays. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1485-1495, Seattle, United States. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to- to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Virgile Rennard, Guokan Shang, Julie Hunter, and Michalis Vazirgiannis. 2023. Abstractive meeting summarization: A survey.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019, Bart: De- noising sequence-tor cuuennceto-sequenee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910. 1346l.
Tulika Saha, Saraansh Chopra, Sriparna Saha, Pushpak Bhattacharyya, and Pankaj Kumar. 2021. A large- scale dataset for motivational dialogue system: An application of natural language generation to mental health. In 2027 International Joins Conference on Neural Networks (IJCN), pages 1-8, IEEE.
Chin-Yew Lin. 2004. Rouge: package for automatic evaluation of summaries. page 10.
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019. Automatic dialogue summary generation for customer service. Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery amp; Data Mining, KDD '19, page 1957-1965, New York, NY, USA. Association for Computing Machinery.
Tulika Saha, Vaibhay Gakhreja, Anindya Sundar Das, Souhitya Chakraborty, and Sriparna Saha. 2022a. To- wards motivational and empathetic response genera- tion in online mental health support. In Proceedings of the 45th international ACM SIGIR conference research and development information retrieval. pages 2650-2656.

---

Tulika Saha, Saichethan Reddy, Anindya Das, Sriparna Saha, and Pushpak Bhattacharyya. 2022b. A shoul- der to cry towards motivational virtual assistant for assuaging mental agony. In Proceedings of the 2022 conference of the North American chapter of the ussociation for computational linguisticx: Human language technologies, pages 2436-2449.
Tiezheng Yu, Wenliang Dai, Zihan Liu, and Pascale Fung. 2021. Vision guided generative pre-trained language models for multimodal abstractive summa- rization. arXiv preprint (0..::2099922220
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Tulika Saha, Saichethan Miriyala Reddy, Sriparna Saha, and Pushpak Bhattacharyya. 2022c. Mental health disorder identification from motivational conversa- tions. IEEE Transactions on Computational Social Systems.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan, 2020b. DIALOGPT Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online, Association for Computational Linguistics.
Tulika Saha, Abhishek Tiwari, and Sripama Saha. 2023. Trends and overview: The potential conversational agents in digital health. In European Conference on Information Retrieval, pages 349-356. Springer.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolu- tional networks.
Karen Simonyan and Andrew Zisserman. 2015, Very deep convolutional networks for large-scale image recognition.
Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020. Summarizing medical conversations via identifying important utterances. Proceedings of the 28th International Conference on Computational Linguis- tics, pages 717-729, Barcelona, Spain (Online). In- ternational Committce on Computational Linguistics.
Katherine Stasaski, Kimberly Kao, and Marti A. Hearst, 2020. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Work- shop on Innovative Use of NLP for Building Educa- tional Applications, pages 52-64, Seatlle, WA, USA + Online. Association for Computational Linguis- tics.
Abhisek Tiwari, Tulika Saha, Sriparna Saha, Pushpak Bhattacharyya, Shemim Begum, Minakshi Dhar, and Sarbaject Tiwari. 2022. Symptoms are known by their companies: towards association guided disease diagnosis assistant. BMC bioinformatics, 23(1):1- 23.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks.
Jun Wang. 2022. Essumm:
Extractive speech summa- rization from untranscribed meeting.
Sibo Wei, Wenpeng Lu, Xueping Peng. Shoujin Wang, Yi-Fei Wang, and Weiyu Zhang. 2023. Medical ques- tion summarization with entity-driven contrastive learning.
Baosong Yang. Jian Li, Derek Wong, Lidia S. Chao, Xing Wang, and Zhaopeng Tu. 2019. Context-aware self-attention networks.

---

A Appendix
Table 3: Annotation guidelines CIMA MuMmm dataset.
A.1 Frequently Asked Questions (FAQs)
Asmotation Ganitlins
Roorw the prostand image in indereend oontr
dialogue
* What was the reason for utilizing the BART-base model as the underlying lan- guage model for IP-Summ?
Keep the oomre iabil midt shen weinng the csmmore ac they prosite valuatee content to the dusingre
For stakest aumuerio focea key kunlllge prints
gaired misomcppion cometeel
in undereundiig
For summuries
unoteile used. clarification
suidkance
. We selected BART-Base the underlying model for proposed approach based on comprehensive experimentation. Our findings consistently demonstrated that BART-Base outperforms T5-Base across various metrics and perspectives. While both BART-Base and DialoGPT-Base exhibited comparable performance, we opted for BART-Base due to its encoder- decoder architecture, which enables the encoding of diverse information prior to generating summaries. Moreover, in or- der to accommodate our computing re- sources, we specifically focused on the Base versions of all models for our com- parative analysis.
lnghtigheng bey
dallgg eschunges insructional elements leurning
incuporaliis pansona opanions
sumuee lo.
Besure summrries are concise, intommaivee and accur refectt the diálggee contens Suuasanics sloulk aut inclade pensonal intepoaaioo us catellllruest
8
approach ensures that the CIMA-Sumin dataset of- fers rich, multi-perspective summaries that can serve as valuable resources for dialogue summa- rization in the education domain. comprehensive summary of the annotation guidelines is presented in Table
Annotation Quality Assessment. We assessed the quality of the summaries generated by the anno- tators following the approach described in Ghosh et al. (2022), which utilized two primary metrics: Fluency It focuses on the linguistic correctness of the summaries using a 5-point Likert scale where score of '1' indicated a summary riddled with grammatical errors while a score of '5" denoted a well-constructed summary free of grammatical mistakes.
* What motivated us utilize the CIMA dataset as the foundation dataset and expand it to incorporate summaries for dialogues?
Adequacy: : This measures the content of the sum- mary using S-point Likert scale. A score of '1' suggested that the summary either misrepresented or missed the intent-action labels along with the di- alogue's main points. score of "5' was given a summary that accurately captured and reflected the salient points of the dialogue and the intent-action labels.
We specifically employed the CIMA dataset to construct the dataset for our proposed task, MM-PerSumm, for sev- eral compelling reasons. In comparison to other educational dialogue datasets (Caines et al., 2022), the CIMA dataset offers wealth of contextual informa- tion pertaining to dialogues. Notably, includes valuable features such im- ages, external knowledge, action tags, and intent tags. This comprehensive and multimodal nature of the CIMA dataset makes it exceptionally well-suited for developing a multimodal dialogue sum- marization dataset, thereby presenting more relevant and impactful problem statement.
Training of Annotators. Given the specialized nature of dialogue summarization in educational context, it was critical to train annotators thor- oughly ensure the quality and consistency of the dataset. The annotators needed to understand not just the dialogue and its nuances but also the unique perspectives of the student and the tutor. Hence, a rigorous annotation training was undertaken. We employed four-phase training process to ensure the proficiency and competence of the annotators. To initiate the annotation training, set of dialogues from the CIMA-Summ dataset was pre-annotated to provide gold-standard samples. These exam- ples were annotated with three type of summaries (student perspective, tutor perspective, and overall dialogue summary) by two experienced researchers
A.2 Annotation Details
Annotation Guidelines. Annotators were guided ensure that the created summaries were concise, yet informative, capturing the essence of the dia- logue without excessive detail avoiding any per- sonal interpretation or embellishment. Th detailed

---

(from author's collaboration) specializing in educa- tional dialogue systems and summarization. Dur- ing each phase of training. the annotators were tasked with summarizing a selection of dialogues. They were reminded to follow the guidelines (Ta- ble 3) while generating the summaries, Post every phase, evaluation of the summaries" fluency and adequacy was conducted. Feedback sessions were arranged after each phase where the annotators in- teracted with the experienced researchers to discuss the evaluations and ways to enhance the quality of the summaries. These iterative feedback discus- sions facilitated the development of their annota- tion skills, gradually improving the quality of the summaries. The annotation guidelines were also refined and updated after each phase, incorporating the insights obtained from these sessions. As result of these iterative training phases, the quality of the summaries improved significantly from the first phase (fluency 3.97, adequacy 2.59) to the fourth phase (fluency = 4,91, adequacy = 4,81), in- dicating the effectiveness of our annotation training procedure.
and address any challenges they faced during the annotation process.
A.3 Baseline Models
In this section, will discuss the training process for each baseline model used in our study.
All three of these models BART, T5, and DialoGPT are unimodal, meaning they are trained handle one modality of data, in this casc, text. For the purpose of our study:
Tutor Perspective Training:
The training dataset was structured such that the input consists of a conversation, and the output is the corresponding tutor summary.
Student Perspective Training: Following similar structure, the input is the con- versation, and the output is the student's summary.
Overall Perspective Training: For holis- tic perspective, the training data again had conversations as input with the overall summary as the output.
Annotation Process. Each annotator was equipped with secure account, permitting individual annotation and tracking of progress in the open-source platform Doccano². Annotating dialogue summaries is a time-intensive task, requiring a balance between quality and efficiency. On average, our annotators needed 6-8 minutes to adequately summarize each dialogue instance, encapsulating necessary details and understanding the intent-action labels. Given the intricacy of the task, honorarium of 10 INR per summarized dialogue instance was provided. The original dataset comprises 1134 dialogue instances, all which were incorporated into the annotation process. We followed a structured schedule over a span of five days in a week as:
These models were then trained using the Maximum Likelihood Estimation (MLE) ob- jective function.
Multimodal BART: The Multimodal BART extends the original BART model by incorpo- rating both text and image modalities. The training process for each perspective is de- tailed below:
Tutor Perspective Training: The input is combination of conversation text paired with an image, and the output is the cor- responding tutor summary.
Student Perspective Training: Similar to the tutor perspective, the input pairs the conversation with image. and the out- put is the student's summary.
Day and Day 4: Each annotator was assigned 30 dialogues for summarization, split into three batches of 10 summaries, with mandatory 20-minute break between each batch to ensure focused and efficient work.
Overall Perspective Training: The holis- tic view takes the conversation and image input and provides the overall sum- mary output.
Day and Day 5: Annotators were tasked with evaluating summaries produced by their peers, based on the quality assessment criteria outlined above.
An intrinsic feature of this model is the cross- modal attention mechanism within the en- coder. This mechanism fuses the represen- tations of text and images, which is subse- quently fed into the decoder for further pro- cessing. The training objective remains the Maximum Likelihood Estimation (MLE).
Day 3: Feedback sessions were held with the annotators to discuss potential improvements

---

Dialogue History
Generic Summary
Student Summary
Tutor Summary
rese. in blank
The futor instructe student to in Itation and dictarea that Pink is rosa. The student aska the futor about the Italan for tree and behind which the tutor rapless l'albero and dietro respectival, The student's dietro l'albero?
blank
The shadent asks tutor about the Italian words for tree and behind. The student responds with cane dietro rose l'abber? which is incureet. Then the student esks correct answer
tutor dictates that is rosa in italian Tree and is behind the l'aibero e respectivety, The tutor replisa that ls prepended the following word when begins with Furthermore, the natar suggests the stodent to look the order the and telis that adjectives follow the roun in Rtalon.
Itllen.
What tree bahind Itafian agoin?
T: Tree /atboro & behwnd the is
dietro. If nol, what'' the varrect answer? l'abbro??
T: (the) is prppanded to the
is propandea to the folkowing word when it starts with vowel. The student the correct enswer suggosts look the word order agoin
following word when begins
with a
T: Look at your order words again. Agjectives (such as color wards) follow they modil Iislian.
Figure 8: Example dialogue history and its corresponding summaries from the CIMA-Summ dataset
Somm: It employs single BART model adapted for the task of generating perspective- controlled summaries from input dialogues. The model requires two-entry input: Source Input Dialogue: This is the primary conversa- tion that needs to be summarized. Perspective Token: This token indicates the desired per- spective (tutor, student, or overall) for which the summary is to be generated. Given these inputs, the output is summary corresponding to the designated perspective token.
the ground truth of the complete dataset as shown in Figure 3(a). The generated summaries have more condensed length distribution compared the ground truth of the complete dataset i.e. gener- ated summarics show less variability in summary lengths across all three categories (Student, Tu- tor, and Overall). In Figure 10, we can notice an overall trend where the number of English and cor- responding Italian word occurrences are highest in the Overall predicted summary followed by Tutor and Student generated summaries. A similar trend was noticed the word count distribution for the annotated CIMA-Summ dataset. Also, there are no occurrences of few words in Tutor's 's summaries like 'green' & it's Italian translation 'verde', 'bed' & it's Italian translation 'letto', etc. While they do appear in student summaries indicating that the student's were able to figure out these Italian words without tutor's aid and tutor only focused on improving areas were student needed help and correction.
A.4 CIMA-Summ Dataset
Figure illustrates another example instance from our proposed CIMA-Summ dataset.
A.5 Experimental Section
The models were trained on a Tyrone machine equipped with an Intel Xcon W-2155 processor and an 11 GB Nvidia 1080Ti GPU. All the models were implemented using Scikit-Leam and Py PyTocch.
Qualitative Analysis of Generated Summaries. In Figure 6, we showcase two instances of our model-generated student, tutor, and generic sum- maries, alongside the ground truth summaries. The results demonstrate that the generic summaries gen- erated by our model, IP-Summ, align well with the ground truth summaries in both the given exam- ples. However, there are instances where IP-Summ falters in other perspectives. For instance, in the first example, /P-Summ's 's student summary mistak- enly incorporates information from the tutor about "giving student examples." " Additionally, we ob- serve that IP- 's tutor summary covers all the core concepts but lacks fluency compared the student summary, indicating the inherent difficulty in generating tutor summaries as opposed to stu- dent summaries. Figure shows sample generated summaries (for the same instances in that of figure 6) from the baseline BART model.
A.6 Analysis and Discussion
Overall
Predicted Summaries
Tutor
Student
Word Count
Figure 9: Distribution of Length for 3 types of Predicted Summaries
Linguistic Analysis of Generated Summaries. Figure 9 presents the distribution of length for types of generated summaries. The generated sum- maries have relatively shorter length compared to

---

Student
50
Word 40
perk
acstoda
huny
congho
grown
top
English and Corresponding Italion words in Predicted Summaries
Figure 10: Frequency of top English and corresponding Italian words in predicted summaries


Reviews:
[{"review_id": "nbmcB7ae2y", "reviewer": "Reviewer_uLLc", "summary": "", "questions": "- For 3.3 Line 259, what cosine similarity did you use for the evaluation?\n- Please help reply and address some concerns about the reasons for the rejection.\n- The examples of the perspectives-based summaries have negligible differences, and they do not support the claim that different parts of the proposed model helped with information gathering and fusion. Some case studies would be necessary.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1: Given that the authors only obtained over 1,000 articles, the training, and evaluation split matters greatly for the final reporting results. Currently, some of those are missing.**\n\nResponse 1: Given the relatively small size of our dataset with just over 1,000 articles, we completely agree that the split matters greatly for properly evaluating the model. To clarify, we split the 1,000 articles into 80% for training (800 articles), 10% for validation (100 articles), and 10% for testing (100 articles).\n\n**Weakness 2: The student's response side seems to be rather template based and does not need that complicated graph-based approach to encode the information.**\n\nResponse 2:  To address this, we would like to highlight a few points: \n* As shown in Table 1 in our paper, adding our graph encoding method leads to clear performance gains over baseline models that do not incorporate graph structures. This indicates that the graph representations are providing useful relational information beyond just sequences. \n* In our human evaluation in Table 2, humans rated summaries generated by our graph-based model as significantly more Fluent, Informative, and Relevant as compared to the baselines. This further validates the benefits of modeling the inter-dependencies in student responses. \n* We agree that the example responses in the paper are relatively straightforward, chosen for ease of understanding. However, we have many more complicated examples in our actual dataset. For instance, here is a more complex student response from our test set:\n|Dialogue |Overall Summary |Student Summary| Tutor Summary|\n| ----------- | ----------- |-----------|-----------|\n|Tutor: Please try to fill in the blank in Italian.Student: e dentro la scatola giallo is my first guess.Tutor: Well,  \"is in front of the\" is  \"e di fronte alla\"Student: I completely forgot that. So maybe its e di fronte alla scatola giallo.Tutor: Hmm...  \"yellow\" is  \"gialla\"Student: So I just messed up the gender of yellow?Tutor: alla (\"to the\") is used when the following word (scatola) is feminine and a singular object.  It is a contraction of a (\"to\") and la (\"the\"). When the following word begins with a vowel, this is shortened to all\\' and prepended to the word.Student: Okay! My final answer is e di fronte alla scatola giallo.Tutor: Almost, gialla is the feminine form, because the noun it modifies (scatola) is feminine.'|Tutor instructs student to fill in the blank in Italian. Student answers \"e dentro la scatola giallo\" to which tutor says \"is in front of the\" is \"e di fronte alla\" and \"yellow\" is \"gialla\". Tutor explains alla (\"to the\") is used when the following word (scatola) is feminine and a singular object.  It is a contraction of a (\"to\") and la (\"the\"). When the following word begins with a vowel, this is shortened to all\\' and prepended to the word. Student gives final answers \"e di fronte alla scatola giallo\" to which tutor suggests \"gialla\" is the feminine form.|A student tries to say \"is in front of the yellow box.\" The tutor corrects their phrasing and explains the gender agreement in Italian, emphasizing the need to match adjectives with the gender of the nouns they modify. The student struggles to apply the rule correctly throughout the conversation.|The tutor helps the student with Italian phrasing and gender agreement. They explain the correct way to say \"is in front of the\" and emphasize the importance of matching adjectives with the gender of the nouns they modify. Despite the explanations, the student struggles to apply the rule correctly.|\n\n**Weakness 3: The dialogue of language learning would be restricted under a small number of turns, have the authors tried simply tuning the model with the perspective tokens and reporting the results?**\n\nResponse 3: Thank you for the suggestion to try fine-tuning the model with perspective tokens as a baseline. We agree this is an important baseline, and we actually already included results using perspective tokens in Table 1 of our paper (Con_Summ row). As you can see, our proposed model outperforms this model across all metrics.\n\n**Weakness 4: The authors compared baselines dated back to 2021. Were there any improvements in 2022 but not included in the paper? There have been works and discussions on role-related dialogue summarization, which is close to perspective-based but more complicated. [1].**\n\nResponse 4: Thank you for pointing out this important recent work [1] on role-related dialogue summarization. We apologize for missing this highly relevant baseline in our initial submission. You are absolutely right that modeling role-specific perspectives is an important extension beyond just conditioning on single speaker roles, and more accurately captures the complexity in perspectives during dialogues. We are currently working on adapting the role-modeling approach from [1] as a baseline for our dataset, by incorporating their proposed entity-aware role graphs into our framework. This is a very relevant technique we should compare against. If the program committee finds our work worthy of acceptance, we will add results using this improved role-related baseline in the final camera-ready version of our paper.\n\n**Question 1: For 3.3 Line 259, what cosine similarity did you use for the evaluation?**\n\nResponse 1: Here is the cosine similarity that we used:\t\n\n    def calculate_cosine_similarity(text1, text2):\n\n    word_tokens_text1 = word_tokenize(text1.lower())\n    filtered_sentence1 = ' '.join([w for w in word_tokens_text1 if not w in stop_words])\n\n    word_tokens_text2 = word_tokenize(text2.lower())\n    filtered_sentence2 = ' '.join([w for w in word_tokens_text2 if not w in stop_words])\n\n    # Create a TfidfVectorizer object\n    vectorizer = TfidfVectorizer()\n\n    # Fit and transform the text data\n    tfidf_matrix = vectorizer.fit_transform([filtered_sentence1, filtered_sentence2])\n\n    # Compute the cosine similarity between the two vectors\n    cosine_sim = cosine_similarity(tfidf_matrix[0], tfidf_matrix[1])\n\n    return cosine_sim[0][0]\n\n**Question 2: Please help reply and address some concerns about the reasons for the rejection.**\n\nResponse: Answered above\n\n**Question 3: The examples of the perspectives-based summaries have negligible differences, and they do not support the claim that different parts of the proposed model helped with information gathering and fusion. Some case studies would be necessary.**\n\nResponse:  We already presented a casestudy in appendix section A.5 .As illustrated in Figures 9 and 10 of the appendix, our proposed IP-Summ model generates higher quality summaries compared to the baseline BART model. A few key trends demonstrate IP-Summ's advantages: \n\n* IP-Summ produces more informative and relevant generic summaries. This is likely due to its use of multimodal fusion to incorporate visual information and perspective dialogue graphs to maintain conversational flow. \n* BART summaries tend to be vague and miss key details that IP-Summ captures through its richer encoding mechanisms. \n* IP-Summ better targets perspective-relevant information through its graph-based modeling. BART lacks this perspective grounding. \n* IP-Summ summaries are more coherent and complete in covering all salient information. BART shows fragmentation across turns.\n\nIn summary, the case study analysis highlights IP-Summ's ability to generate improved summaries by leveraging both visual and conversational context through dedicated modeling components. This allows it to overcome limitations of a text-only summarizer like BART.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The annotated dataset quality is under good moderation.\n- The perspective-driven module seems to contribute a lot to the final summarization quality.", "reasons_to_reject": "- Given that the authors only obtained over 1,000 articles, the training, and evaluation split matters greatly for the final reporting results. Currently, some of those are missing.\n\n- The student's response side seems to be rather template based and does not need that complicated graph-based approach to encode the information.\n\n- The dialogue of language learning would be restricted under a small number of turns, have the authors tried simply tuning the model with the perspective tokens and reporting the results? \n\n- The authors compared to baselines dated back to 2021. Were there any improvements in 2022 but not included in the paper? There have been works and discussions on role-related dialogue summarization, which is close to perspective-based but more complicated. [1].\n\nReference:\n\n[1] [Other Roles Matter! Enhancing Role-Oriented Dialogue Summarization via Role Interactions](https://aclanthology.org/2022.acl-long.182) (Lin et al., ACL 2022)", "questions_to_authors": "- For 3.3 Line 259, what cosine similarity did you use for the evaluation?\n- Please help reply and address some concerns about the reasons for the rejection.\n- The examples of the perspectives-based summaries have negligible differences, and they do not support the claim that different parts of the proposed model helped with information gathering and fusion. Some case studies would be necessary.", "paper_topic_and_main_contributions": "This paper contributed an annotated Multi-modal perspective-based dialogue summarization corpus dedicated to Educational Virtual Tutors. They then proposed a novel multi-modal model that accepts image and text as inputs and demonstrated better performance than baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gvUuNv2XFV", "reviewer": "Reviewer_gdPt", "summary": "", "questions": "* Did you consider conducting similarity among pairs of summaries in the same category?  Since some tutoring utterances in CIMA dialogues were constructed using the VT, I wonder if this would result in similar summaries.  I would also be curious in these results for all categories to ground the results presented in 3C.\n* As CIMA contains both an image and related vocabulary and grammar information, did you consider whether to represent the student’s problem using the image or text representation?\n* It’s unclear to me how the baseline models were trained for the task.  Could you please provide these details?  Did you fine tune these models on the task?  This can be added into the appendix, but is necessary information\n* Did you conduct any inter-annotator agreement or significance testing for results reported in Table 2?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable review.\n\n\n**Weakness 1: While I love the premise of this task, I wish the authors had taken the tutor summaries a step further. Instead of stating factually what the tutor has done (as shown in examples), why not include summaries of the educational techniques being deployed? Since the introduction motivates tutor summaries being used to examine VT effectiveness, I feel the dataset falls short of this.**\n\nResponse 1: You make an excellent point - the current factual tutor summaries limit the utility for analyzing pedagogical techniques and virtual tutoring effectiveness. Augmenting the summaries with more explicit educational details would be valuable future work to better meet the intended goals of understanding VT strategies. However, you are right that our primary aim with this initial dataset was to create a benchmark for the task of perspective-based virtual tutoring summarization. So while the summaries do fall short for pedagogical analysis purposes, they do provide a basis for advancing summarization capabilities from varied perspectives. You raise a fair critique that we should expand the depth of the summaries to enable deeper analysis of effectiveness as future work. But developing the core summarization task itself was our primary focus when constructing this first iteration of the dataset. Thank you again for the thoughtful feedback - it gives us good direction for enhancing the dataset in future iterations, while recognizing our current objective of establishing a summarization benchmark. Please feel free to suggest any other ways we could build on this limitation in follow-on efforts.\n\n**Question 1: Did you consider conducting similarity among pairs of summaries in the same category? Since some tutoring utterances in CIMA dialogues were constructed using the VT, I wonder if this would result in similar summaries. I would also be curious in these results for all categories to ground the results presented in 3C.**\n\nResoonse 1: To examine potential repetition within summary categories, we performed an intra-category similarity assessment as follows:\n\n* Data Preprocessing: We filtered our dataframe to only include rows corresponding to a particular category. Each category was examined individually. \n* Computing Similarity: We then employed a TF-IDF Vectorizer to transform the summaries into a sparse matrix of TF-IDF features. Cosine similarity was subsequently used to determine the similarity between every pair of summaries within the category. \n* Aggregation: For each summary, we calculated its average similarity with all other summaries in the same category. \n* Reporting Results: On analyzing the average similarity values, we observed that the overall average similarity across all summaries in the tutor category was 0.294.\n\nThese findings indicate that while there might be some overlap, the summaries are distinct enough to be considered non-similar. An average similarity value of 0.294 suggests that even though some tutoring utterances were constructed using the VT, the resulting summaries have considerable variations and unique aspects.\n\nSimilarly we obtain the similarity of 0.155 and 0.291 for student and overall summary categories respectively. We assure the reviewers that we will add a corresponding plot for visualization in camera ready version.\n\n**Question 2:  As CIMA contains both an image and related vocabulary and grammar information, did you consider whether to represent the student’s problem using the image or text representation?**\n\nResponse 2: Thank you for raising this important question about our use of image and text representations in the CIMA dataset. You make an excellent point that with both modalities available, it was a key modeling decision how to effectively leverage them. To analyze this, we compared both unimodal baselines (T5, BART, DialoGPT using text only) and multimodal baselines (MultimodalBART using images and text) in our experiments. As you observed, our proposed model incorporates both representations. Critically, we found that the multimodal baselines outperformed the unimodal ones by a sizable margin as shown in Table 1, demonstrating the importance of utilizing both image and text information for this dataset. The image provides helpful visual context, while the text conveys key linguistic knowledge. You raise a very insightful point, and our analyses confirm that leveraging both modalities leads to superior performance compared to text or image alone. Please let us know if you would like us to include any additional discussion in the paper about our rationale and experiments around utilizing both representations.\n\n**Question 3: It’s unclear to me how the baseline models were trained for the task. Could you please provide these details? Did you fine tune these models on the task? This can be added into the appendix, but is necessary information**\n\nResponse 3: Thank you for pointing out that details on the baseline model training were not fully clear. Let me explain how each was trained on the summarization task:\n\n* All three of these models — BART, T5, and DialoGPT — are unimodal, meaning they are trained to handle one modality of data, in this case, text. For the purpose of our study: \n    * Tutor Perspective Training: The training dataset was structured such that the input consists of a conversation, and the output is the corresponding tutor summary. \n    * Student Perspective Training: Following a similar structure, the input is the conversation, and the output is the student's summary.\n    * Overall Perspective Training: For a holistic perspective, the training data once again had conversations as input with the overall summary as the output. \n\nThese models were then trained using the Maximum Likelihood Estimation (MLE) objective function.\n* Multimodal BART: The Multimodal BART extends the original BART model by incorporating both text and image modalities. The training process for each perspective is detailed below: \n    * Tutor Perspective Training: The input is a combination of conversation text paired with an image, and the output is the corresponding tutor summary. \n    * Student Perspective Training: Similar to the tutor perspective, the input pairs the conversation with an image, and the output is the student's summary. \n    * Overall Perspective Training: The holistic view takes the conversation and image as input and provides the overall summary as output. \n\nAn intrinsic feature of this model is the cross-modal attention mechanism within the encoder. This mechanism fuses the representations of text and images, which is subsequently fed into the decoder for further processing. The training objective remains the Maximum Likelihood Estimation (MLE). \n\n* Con_Summ: It employs a single BART model adapted for the task of generating perspective-controlled summaries from input dialogues. The model requires a two-entry input: Source Input Dialogue: This is the primary conversation that needs to be summarized. Perspective Token: This token indicates the desired perspective (tutor, student, or overall) for which the summary is to be generated. Given these inputs, the output is a summary corresponding to the designated perspective token. \n\nWe hope this detailed explanation sheds light on the training mechanisms of our models. If further clarification or details are required, please do not hesitate to ask.\nWe assure the reviewers that we will add these baseline setups in detail in camera ready paper if accepted.\n\n**Question 4: Did you conduct any inter-annotator agreement or significance testing for results reported in Table 2?**\n\nResponse 4: \n\nTo ensure the reliability of our human and automatic evaluations, we computed inter-annotator agreement for human judgments and statistical significance tests for metric improvements. For human evaluation, we calculated the rate of majority agreement across annotators on the acceptability of model responses. This yielded an inter-annotator agreement score of 72.3%, indicating reliable consensus. For automatic metrics, we conducted the statistical significance Welch’s t-test at 5% significance to ensure that the improvement of our proposed model over the baselines is reliable. All the reported results are statistically significant.  We will provide full details on the inter-annotator agreement computation and statistical significance testing in the camera ready version to demonstrate the rigor of our evaluation methodology.\n\nWelch BL. The generalization of student’s’ problem when several different population variances are involved. Biometrika. 1947;34(1/2):28–35.\n\n**Typos Grammar Style And Presentation Improvements:**\n**How were fluency and adequacy ratings measured (line 235-238)? Even if this is included in the appendix, it is good to include a minimal explanation if you are including the result in the main paper**\n\nResponse 1: We would like to assure the reviewer that we will move the section from appendix to main paper in the final camera ready version.\n\n** 933: Typo: the and 400: backwards ‘ **\n\nResponse 2: We will correct all these typos in final camera ready version", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Authors propose an important, novel task which aims to summarize Student-Virtual tutor conversations\n* Authors augment an existing dataset with multiple summaries from different perspectives\n* Authors propose interesting model to approach this task, which includes a perspective-based dialogue graph taking dialogue actions into account.  Authors ablate the different components of their approach and produce a comprehensive analysis.", "reasons_to_reject": "* While I love the premise of this task, I wish the authors had taken the tutor summaries a step further.  Instead of stating factually what the tutor has done (as shown in examples), why not include summaries of the educational techniques being deployed?  Since the introduction motivates tutor summaries being used to examine VT effectiveness, I feel the dataset falls short of this.\n", "questions_to_authors": "* Did you consider conducting similarity among pairs of summaries in the same category?  Since some tutoring utterances in CIMA dialogues were constructed using the VT, I wonder if this would result in similar summaries.  I would also be curious in these results for all categories to ground the results presented in 3C.\n* As CIMA contains both an image and related vocabulary and grammar information, did you consider whether to represent the student’s problem using the image or text representation?\n* It’s unclear to me how the baseline models were trained for the task.  Could you please provide these details?  Did you fine tune these models on the task?  This can be added into the appendix, but is necessary information\n* Did you conduct any inter-annotator agreement or significance testing for results reported in Table 2?\n", "paper_topic_and_main_contributions": "This paper proposes the new task of Multi-Modal Perspective-based Educational Dialogue Summarization.  The task involves summarizing a student-virtual tutor dialogue with 3 different perspectives: student, tutor, and generic.  Summarizing from the student’s perspective focuses on the student’s progress and outcomes.  From the tutor’s perspective focuses on the tutoring strategy and efficacy.  The authors propose a model to approach this task and analyze performance.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* How were fluency and adequacy ratings measured (line 235-238)?  Even if this is included in the appendix, it is good to include a minimal explanation if you are including the result in the main paper\n* 933: Typo: the\n* 400: backwards ‘\n"}}, {"review_id": "rBPA8XZUgm", "reviewer": "Reviewer_q3T3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable review.\n\n\n**Weakness 1: Can the author provide some case studies in experiments to better demonstrate the effect of the model.**\n\nResponse 1:  We do already present a comprehensive qualitative analysis in Appendix A.5 of the paper, including: Figure 9 showing generated summaries from our model vs gold summaries for a sample instance, across all perspectives. This illustrates our model's ability to produce varied, nuanced summaries capturing key details from different viewpoints. Figure 10 contrasting our model's outputs against a baseline model, for the same instance. A few key trends demonstrate IP-Summ's advantages: \n\n* IP-Summ produces more informative and relevant generic summaries. This is likely due to its use of multimodal fusion to incorporate visual information and perspective dialogue graphs to maintain conversational flow. \n* BART summaries tend to be vague and miss key details that IP-Summ captures through its richer encoding mechanisms. \n* IP-Summ better targets perspective-relevant information through its graph-based modeling. BART lacks this perspective grounding. \n* IP-Summ summaries are more coherent and complete in covering all salient information. BART shows fragmentation across turns.\n\n In summary, the case study analysis highlights IP-Summ's ability to generate improved summaries by leveraging both visual and conversational context through dedicated modeling components.\n\nLet us know if you would like us to include any additional details or case examples to further demonstrate the qualitative improvements from our approach. We are happy to expand this analysis in the paper based on your feedback.\n\n**Weakness 2: I didn't particularly feel the benefits of summarizing the action content of different characters in the educational dialogue. Instead, I felt that the data became discontinuous and I couldn't see the logical relationship of the summary content. Should the other party's behavior be weakened in it, but not completely deleted?**\n\nResponse 2: Thank you for the thoughtful feedback. You raise a fair point - when summarizing from different perspectives, there is a risk of losing continuity if the content is fragmented too much across roles. However, I want to reassure you that our approach does maintain a complete, logical flow in the generated summaries. The key here is the Perspective Context Encoder module, which uses a perspective-focused graph to model relationships between all dialogue turns. This helps preserve continuity even when summarizing from particular viewpoints. To clarify further - we do not completely delete or ignore the other party's contributions when generating role-specific summaries. Rather, the graph encoding retains the full context to keep the logical flow intact. The perspective simply changes which details are focused on more in the summary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a new and innovative task of MM-PerSumm in an educational setting. This task involves summarizing educational dialogues from three unique perspectives: the Student, the Tutor, and a Generic viewpoint. By considering different viewpoints, the summarization process can provide diverse insights into the virtual tutoring interactions and contribute to the improvement of AI-based educational systems.\n\nThe paper extends an existing educational conversational dataset, CIMA, by summarizing the dialogues from three different perspectives (Student, Tutor, and Generic viewpoint) to create CIMA-Summ. \n\nImage and Perspective-guided Dialogue Summarization (IP-Summ) Model: The paper proposes the IP-Summ model, which is a SeqSeq language model incorporating multi-modal learning from images and a perspective-based encoder. \n", "reasons_to_reject": "1）Can the author provide some case studies in experiments to better demonstrate the effect of the model.\n2）I didn't particularly feel the benefits of summarizing the action content of different characters in the educational dialogue. Instead, I felt that the data became discontinuous and I couldn't see the logical relationship of the summary content. Should the other party's behavior be weakened in it, but not completely deleted?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses the increasing use of Virtual Tutors (VT) in recent years, which has enabled more efficient, personalized, and interactive AI-based learning experiences. A crucial aspect of educational chatbots is summarizing the conversations between VT and students, as it helps consolidate learning points and monitor progress. The paper proposes a new task called Multi-modal Perspective-based Dialogue Summarization (MM-PerSumm) in an educational setting. They introduce a novel dataset, CIMA-Summ, which contains educational dialogues summarized from three unique perspectives: Student, Tutor, and Generic.\n\nTo address the MM-PerSumm task, the paper presents the Image and Perspective-guided Dialogue Summarization (IP-Summ) model. This model is a SeqSeq language model that incorporates multi-modal learning from images and a perspective-based encoder. The perspective-based encoder constructs a dialogue graph, capturing the intentions and actions of both the VT and the student. By leveraging both textual and visual information, the IP-Summ model allows for more comprehensive and accurate summarization of educational dialogues from diverse perspectives.\n\nThe paper emphasizes the importance of summarizing dialogues from multiple perspectives to gain valuable insights for improving AI-based educational systems and enhancing students' learning outcomes. Summaries can provide essential feedback for the VT's teaching efficiency and suggest areas of improvement. For learners, summaries distill key learning points, track their progress, and inform their future study paths. The proposed MM-PerSumm task and IP-Summ model hold significant potential in advancing AI-driven education by revealing diverse insights and contributing to the evolution of virtual tutoring systems.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new task called Multi-Modal Perspective-based Educational Dialogue Summarization (MM-PerSumm), which focuses on summarizing a student-virtual tutor dialogue from three different perspectives – that of a student, a tutor, and generic. For example, a summary from the student’s perspective may focus on the student’s progress and outcomes, while that from the tutor’s perspective should focus on the tutoring strategy and efficacy. To that end, the authors introduce a novel dataset, CIMA-Summ, and build a novel multi-modal Image and Perspective-guided Dialogue Summarization (IP-Summ) model that accepts image and text as inputs and constructs a dialogue graph, capturing the intentions and actions of both the virtual tutor and the student. Experiments suggest that the model performs better than the baselines and holds potential in advancing AI-driven education.

The reviewers have identified the following **contributions and strengths** of this paper:
1. *Novelty of the task*: Reviewers gdPt and q3T3 consider that the authors propose an innovative and important task, and that summarizing student-virtual tutor conversations from different perspectives is beneficial.
2. *Dataset*: All reviewers find the extended CIMA dataset, augmented with multiple summaries from different perspectives, promising and potentially useful.
3. *Model*: Finally, all reviewers also agree that the proposed model is interesting and seems to contribute to the summarization quality. Reviewer gdPt also mentions that the authors run ablation studies and provide comprehensive analysis.

At the same time, all reviewers have identified **weaknesses and areas for further improvement**. Specifically:
1. *Technical rigor*: Reviewer uLLc points out that with the small dataset of 1,000 articles the training and evaluation splits matter greatly when reporting results, but the analysis of this seems to be missing from the paper. Given a relatively small number of articles, running experiments using cross-validation would be more appropriate than a single data split.
2. *Limitations of the dataset*: Reviewer uLLc mentions that the student's response side appears to be rather template-based, which might not warrant the complicated graph-based approach to encode the information. Moreover, the dialogues included in the data are restricted in terms of the number of turns. Reviewer gdPt expresses further concerns regarding the dataset – for those, please refer to the original review.
3. *Limited comparison of the results*: Reviewer uLLc highlights that the baselines considered in this paper date back to 2021 and more recent work has not been taken into account (the missing references are provided in the review).
4. *Further analysis of the model and the results* is needed, as reviews, in particular from reviewers q3T3 and gdPt, suggest.

It is also clear from the reviewers' questions that many important technical details are missing in the paper. The authors did a thorough job answering the questions and even running additional experiments and analysis – all these need to be included in the revised version of the paper.
Decision: Accept-Findings