Towards Online Multimodal Social Interaction Understanding
Authors: Xinpeng Li
Shijian Deng
Bolin Lai
Weiguo Pian
James Matthew Rehg
Yapeng Tian
Venue: Accepted by TMLR ()

Abstract: In this paper, we introduce a new problem, Online-MMSI, where the model must perform multimodal social interaction understanding (MMSI) using only historical information. Given a recorded video and a multi-party dialogue, the AI assistant is required to immediately identify the speaker’s referent, which is critical for real-world human-AI interaction. Without access to future conversational context, both humans and models experience substantial performance degradation when moving from offline to online settings. To tackle the challenges, we propose Online-MMSI-VLM, a novel framework based on multimodal large language models. The core innovations of our approach lie in two components: (1) multi-party conversation forecasting, which predicts upcoming speaker turns and utterances in a coarse-to-fine manner; and (2) socially-aware visual prompting, which highlights salient social cues in each video frame using bounding boxes and body keypoints. Our model achieves state-of-the-art results on three tasks across two datasets, significantly outperforming the baseline and demonstrating the effectiveness of Online-MMSI-VLM.

Published in Transactions on Machine Learning Research Alarch/20200
Towards Online Multimodal Social Interaction Understanding
Xinpeng Li University of Teras at Dallas
sinpeng.li@utdadd edu
Shijian Deng University of Teros at Dallas
shijion.deng@@uttallas....
Bolin Lai
bolin. holin.lai@gatech.oo edu
Georgia Institute of Technology
Weiguo Pian University of Texas Dallas
weiguo. Satiguo..pn@@udalllsss edu
James Matthew Rehg
jrehgtrilinois.co edu
University of Mlioois Urbona- Champoign
Yapeng Tian
yapeng, tian Waw.nnttaaaaaann edu
University of Texos Dallas
Reviewed on OpenReview: https: / wwtp:/vee...eeee..ooeoe net/ formm? VP7YD
Abstract
In this paper, we introduce problem, Onlino-MMSI, where the model must per- form multimodal social interaction understanding (MMSI) using only historical information. Given recorded video and multi-party dialogue, the AT assistant is required to immedi- ately identify the speaker's referent, which is critical for real-world human-Al interaction. Without access to future conversational context, both humans and models experience sub- stantial performance degradation when moving from offline to online settings. To tackle the challenges, we propose Online-MLMSI-VLNN a novel framework based on multimodal large language models. The core innovations of our approach lie in two components: (1) multi- multi-party conversation forecasting, which predicts upcoming speaker turns and utterances in coarse-to-fine manner; and (2) socially-aware visual prompting, which highlights salient social cues in each video frame using bounding boxes and body keypoints. Our model achieves state-of-the tett---r-----ttttto results on three tasks across two datasets, significantly outperform- ing the baseline and demonstrating the effectiveness Online-MMSIA VLM. Project page: -ee.iinnneeeeiiiuubbeeeee wowonnioooennnneeee mmsi paoeeeit--oeettppappp
1 Introduction
Multimodal Social Interaction Understanding (MMSI) plays critical role in advancing human-Al social interaction, aiming to interpret social behaviors by jointly leveraging verbal and non-verbal cues (Lee et al. 2024b Lai et al. 2023 Loe et al. 2024a). Recent work has explored tasks such as speaking target identi- lication, pronoun coreference resolution, and mentioned player prediction. One common goal is to identify a speaker's referent among multiple participants using recorded dialogues and video streams. For example, as illustrated in Figure [ (1), when a user asks Which player the speaker referring to? at moment, the model analyzes the multimodal input and responds, "Player0." MMSI can enable applications such AI Al-powered assistants for social support and collaborative tasks. Smart AR glasses, for instance, help

---

Published in Transactions on Machine Learning Research [Mrchh2020g
Player
DOffin BOnline
73.3 66.6
59.1
38.47
36.16
Lee et al. (2024a)
GPT-4o Speaker Target Identification (%)
Human
Gemini 2.5 Pro
[Player2]:1 robbed with you.
[Play er You rebled
Past
Fanure
100
73.3
80
63.4
Which player is the ypaker Player2 reforring so?
Time
40
30.34
40
26.67
20
Offline MMSI
Delayed Response
Playerô.
Lee et al. (2024a)
Human
GPT-4o
Gemini 2.5 Pro
Post signals
information
Pronoun Coreference Resolution (%)
80
73.3
Online MASI
Immediate Reply C
50
47.3
42,79
Port signals
PlayerO.
40
23.33 16.66
Finall Prompting
Seciall--nare
Onlle-MMSS--VL.
Consersation Farveasting
Multiparty
Lee et al. (2024a)
Human
GPT-4o
Gemini 2.5 Pro
Mentioned Player Prediction (%)
(2)
Figure/ (1) Existing MMSI studies depend both past and future context (offline setting), limiting their practicality in real-world scenarios that require immediate interpretation and response. To bridge the gap, we introduce a new problem, Online-MMSI, where the model must perform MMSI using only historical information (online setting). To address the challenge, we propose Online-MMSI-VII multimodal large language model- based framework that integrates multi-party conversation forecasting and aociall--waare visual prompting.
a novel
(2) Performance drops siguificantly when both models and humans shift from offline online settings across three social tasks on the YouTube dataset (Lai et al. 2023)
autistic individuals better understand social cues (Hbbrr et al. assistants actively participate in social settings [Brazzeal et al.
2020
Elsherbini et al.
2023
while intelligent
2016
As a result, MMSI has attracted
growing interest from the social AI research community [Lee et al.
2024b Li et al.
2024a Feng et al. 2025
Prior MMST studies focused on offline setting, conducting referent identification by leveraging both past and future context at a given moment (Lee al. 2024a).. They rely on extended transcripts and video data, producing responses with inherent delays. However, real-world AI assistants are expected respond in real time, interpreting and responding to social dynamics without access to future information (i.e., online sctting). To bridge this gap, introduce OnlIne-MMSI, a new problem formulation where models must perform MMSI tasks using only historical information. This setting is crucial for building responsive, real-time intelligent systems that are practical and deployable in real-world social environments.
Yet, Onlino-MMSII presents substantial and unique challenges compared with traditional offline settings. As shown in Figure (2), performance drops significantly when both models and humans shift from oflline online settings across three social tasks. First, online models lack access to delayed and explicit cues, such future transcripts or the referent's responses. For example, the referent may respond directly to the initial speaker, other participants might clarify reiterate the referent in later turns. Traditional offline MMSI model (Lee et al. 2024a) rely on seeing the future social cues directly, making it not an ideal solution for real- time scenarios where such cues are unavailable. Second, online models must rely entirely on immediate and past signals, where subtle social cucs-such as pointing gestures and posture, head turns, or body orientation- are crucial for resolving referents. Figure (1) illustrates such a case: the speaker gesturing toward the referent with the index finger, serving as critical disambiguating contexts for referent resolution. Since social scenes often involve multiple participants engaged in dynamic intsractious, it is difficult to detect such subtle yet informative social signals from raw RGB video frames alone without explicit guidance.
novel Vision-Languae Model (VLM)-basod framework (hat inlegrates multi-parly conversation forecasting and socially-aware visual prompting, For the first challenge, cognitive studies suggest that humans improve online interpretation by anticipating upcoming social interactions /Epperlcin et al.) 2022: Ma al. 2024 Hadley Culling 2022). This motivates us leverage multimodal large language models to anticipate futurr conversations enrich social context. Our coarse-to-fine strategy first predicts the identity of the upcoming speaker, followed by generating their likely
To address these challenges, we propose Online-MMSI-VII
2

---

Published in Transactions on Machine Learning Research (March/2020)
utterance. For the second challenge, propose to enhance the representation of past video by using socially-aware visual prompting, which employs off-the-shelf detectors to extract and highlight the positions, postures, and gestures of participants. This facilitates the model to attentively interpret subtle and complex social interactions in the previous video. By jointly forccasting future dialogue and enriching past visual context, our framework can effectively tackle the challenges of Online-MMSI. We evaluate our approach using two VLM models on three referent identification tasks within two social datasets (Lai et al. 2023). The results indicate that OnLine-MMSI benefits from multi-party conversation forecasting and socially-aware visual prompting, and Online- VLM achieves otate-of-the-art performance.
In summary, our contributions are threefold: 1)
We propose a new task, Online-MMSI, where the AI assistant must interpret multimodal social interactions and provide immediate feedback using only historical dialogues and videos. 2) We propose Online-MMSIL MLMI a multimodal large language model based framework that integrates multi-party conversation forecasting and socially-aware visual prompting. To the best of our knowledge, is the first work to use multimodal large language models for advancing MMSI. 3) Extensive experiments, for three tasks on two social datasets, demonstrate the eveectiveness of our approach and show Online-MMSI-VIL VLM establishes strong benchmark for Online-MME
2
Related Works
2.1 Multimodal Social Interaction Understanding
Multimodal Social Interaction Understanding (MMS1) aims to interpret complex interactions among multiple participants by leveraging both verbal and ner-verbal cues, which has attracted increasing attention in the machine learning and social AI communities. For non-verbal social cues, some studies have attempted to perceive social meaning from visual behaviors such as body gestures (Benitez-Garcia et al. 2021 Liu et al.
2021
Chen et al. 2023a
Kapitanov et al. 2024)), gaze patterns (Jindal & Manduchi, 2023 Chong et al.
2020
Grauman al. 2022
Tafasca et al. (202)). and facial expressions (Zhang et al. [022 Savchenko 2023
Zhao et al.] 2024 Li et al.
2024b). For verbal social cues, prior works have explored social understanding
from linguistic signals such as game-theorotic agents (Feng et al. 2025), speaker intent (Malhotra al. 2022
Chen el al. 2023c), and dialogue sentiment (Feng el al. 2021). For multimodal social cues, several studies
integrate verbal and non-verbal modalities to holistically interpret social interactions, such as recognizing
emotions (Cheng ct al.
2024b Lian et al.) 2025 2024), conversational dynamics (Raman et al. 2022 Ryan
et al. 2023 Hou et al.
2024), and social situalions (Hyun et al. 2023 Guo et al. 2025
Recently, Li al. (0244) integrate the perception capability of vision models and the reasoning capability of
LLMs for social relation recognition. Gupta et al.] (2024) introduce novel framework to jointly predict the gaze target and social gaze label for all people in the scene. Lee et al. (2024b) conduct comprehensive survey, multimodal social cues, multi-party dynamics, and beliefs. Lai et al.] (2023) introduce a multimodal dataset for modeling persuasion behaviors. CCao et al. (2025) introduce a large-scale dataset for multi-person gesture analysis. Lee et al. tasks to model the fine-grained dynamics between multiple people: speaking target identification, pronoun corcference resolution, and mentioned player prodiction. To address these tasks, Loe et al. (2024)) introduce a densely-aligned multimodal framework to capture social cues from transcript and video.
identifying three keys for effective social understanding:
(2024a) introduce three
Despite these advances, current research typically focuses on the offinee setting, where the model can have access to entire vidco frames and dialogues to make predictions. However, this setup does not align well with real-time requirements in interactive AI assistant systems, where the model can only leverage histor- ical information (i.e., the online setting). Although methods from recent research (Lee et al. 2024a) can thcoretically be adapted to online scttings through data rearrangement, they do not adequately account for the challenge of missing future social cues and exploiting historical information in the online setting. In our work, we investigate the online setting for MMST and propose novel methods to address the challenge.
3

---

Published in Transactions on Machine Learning Research (March/20P)
2.2 Multimodal Large Language Models
Multimodal large language models (MLLMs) have demonstrated outstanding performance in vision-language
tasks (Liu et al.) 2023 Team et al., 2024; Team
2025 Zhang et al. 2023a Zhu et al. 2023 Zhang et al.
2023b; Chen et a 2023b; Lin et al.
2023;
Chen et al 2024b Cheng et al. 2024c Zhang et al. 2024b;
Thawakar et al. 2025 Zhang et al.
2024c
This success is attributed to their strong perception and
reasoning capabilities, which makes them useful for dynamic multimodal social interaction understanding.
Online scenarios attract growing attention (Wang et al.
Girdhar & While several online MLLMs have been developed for training and
2021
Zhao & KKähennühl
2022
Grauman 2021 Zhao et al. 2023
deployment CCen et al. 2024a Liu et al. 2024 Fu et al.
2025 our work aims to conduct Online-MMST.
Conversation forecasting has been explored for conversation agents (Hassan et al. 2024 El Hattami
2019), and harmful behaviors (Sicilia & Alikhani 2024). These studies mainly focus on conversation between agents and users. Instead, we focus
let al. 2023). predicting derailment (Chang
on understanding multi-party conversations of multimodal social events.
Visual prompting has also proven effective in enhancing MLLM performance (Wu et al. 2025 Cai et al, 2024
Wu et al. 2024 Lei et al. 2024 Yang et al. 2023 Xu et al. 2024). Some studies show that attentively
leveraging the history is crucial for improving online video understanding in noisy and large-volume visual
data Yao et al. 2025 Zhang et al., 2024a: Huang et al. 2024 Chandrasegaran et. al. 2024 Patil et al.,
2024 In this work, we propose to leverage visual prompting to enhance historical video for Online-MMSI.
3 Problem Formulation and Challenges
Following Lee et al. (2024a), we study three multimodal social interaction understanding tasks: Speak- ing Target Identification (STI), Pronoun Coreference Resolution (PCR), and Mentioned Player Prediction (MPP); using recorded video and its corresponding multi-party dialogue transcripts. As shown in Fig. 5 STI aims to identify who the current speaker is talking to when the utterance contains sccond-person reference, e.g. "you" and "your"; PCR focuses resolving which participant third-person pronoun refers to, e.g., "he", "she", "him", "her" and "his"; MPP requires predicting which participant is referred to by name when the participant name in the current utterance is masked, e.g,, "So we vote for [MASK]".
The online lask formulation is as follows: At timestep I, the system receives:
User prompt (Pt): a query related to the specific task. The prompts for the three social tasks are "Identify which player the speaker talking to?", *Determine which player a pronoun refers to?", and "Predict which player is mentioned by name?", respectively.
Historical dialogues (P(t-)a):: the most recent time steps of dialogue history. Each entry consists both the speaker identity and the corresponding utterance. E.g., "[PlayerO]: So, you're a mason? [Player0]: I'm the troublemaker. [Player3]: Did you swap me with anybody?"
Recorded video frames (V(z-ja):: the most recent d time steps of vidco frames involving multiple participants. These frames include dynamic non-verbal social cues, such as gestures and gazes.
The Al assistant is required to produce an immediate reply:
Response (Rt): the predicted referent identity to the referent identification query, which is ex- pressed as a categorical label from a closed set of participants, ie.. "PlayerO", ,..., "PlayerN".
Given Pt, D(e-ajas and Vir-dja, the objective is to optimize the model to generate an accurate RG. A prediction is considered correct if the predicted identity exactly matches the ground-truth referent.
In the offiine sctting, the model has accoss to both past and future data-P,. D(s-d):(+d)) and ((t-a) -and and generates a response Riy4a) at timestep t+d. In contrast, the online setting restricts access to only historical video and dialogue, cutting off any future context. To quantify the impact of this constraint, Figure 1 (2)
4

---

Published in Transactions on Machine Learning Research (March/2026)
Past
Future
[Player]]: Yeah. Villager and werewolf.
[Player2]:1 I think we're voting to the left. [Player4]: We think there are two werewolves in the middle?
[Player]]: All right.
[Player2]: Yeah.
[Player4]: be ready to vote if you guys are.
[Player0]:Amd you saw villager and werewolf when you were seer, right?
[Player0]: I'm just, honestly, I'm worried about [Player3).
Online output: Player2 x
Offline output: Playerl o
Speaking Target Identiftcation
[Player3]: No, I'm not.
[Player2]: That makes belleve [Player0]
[Player0]: I think [Player3] is the tanner.
[Player(]:: How does it feel to be lying about being the v villager right now.
[Player4]: Feels good.
[Player2]: I think [Player3] the tanner as well.
[Player0]: the Insomniac. This problem. He's lying.
[Player0]: I think (Player3), this good gambit be the tanner.
Online output: Player4 x
Pronoun Coveference Resolution
Offline output: Player3 /
[Playerd]: Rip.
[Player4]: Fascinating.
[Player0]: Rip [Player4].
[Player0]: I'm the troublemaker and switched [Player3] with somehody.
[Playerd]] So what are we doing?
[Player2]: That's true, yeah.
[Player2]: think we're voting to the left.
[Player0]:And And then you were the last to say robber. So we kill [MASK].
Online output: Player3 x
Offline output: Player4 /
Mentioned Player Prediction
Figure Illustration of the challenge of missing future information when transitioning from the offline setting to the online setting. In the online setting, only immodiate (blue font) and past data (black) is available, whereas the oflline setting additionally includes useful future context (green), such as the referent's responses. As a result, the online model may be confused and provide an incorrect answer.
compares performance across the three social lasks under both settings using the YouTube dataset (Lai et al. 2023). We evaluate the previous method (Lee et al.) 2024a), human participants, GPT-4o (OpenAl, 2024), and Gemini 2.5 Pro (Comanici et al. 2025). To ensure fairness the user study, cach participant first viewed the online input, followed by the offiine version. This design mitigates potential bias by preventing participants' online judgments from being influenced by future context seen in the offline setting.
As shown in the results, there is considerable drop in accuracy across all three tasks when both humans and models when transitioning from offline to online scenarios. The prior model (Lee et al. 2024a) suffers 13.6% in speaking target identification, 2.5% in pronoun coreference resolution, and 11.5% in mentioned player prediction. The average accuracy of humans, GPT-40, and Gemini 2.5 Pro drops by around 7%, 5%, and 10%, respectively, confirming that the performance gap stems from task difficulty. The evaluation results of GPT-4o and Gemini 2.5 Pro underscore that current LLMs still struggle with understanding multimodal multi-party social interaction compared with humans [Inoue et al. 2025 Tan et al. 2023)
This performance degradation arises from two fundamental limitations. First, online models lack access to delayed and explicit cues- as future transcripts or the referent's responses -hat that are available in offline scttings. The referent froquently replics to the speaker or is clarified by other participants in subsequent turns. For instance, as seen in F in speaking larget identification, after the speaker says "[PlayerO]: And you saw villager and werewolf when you were seer, right?"", the referent later replies "[Player1]: Yeah. Villager and werewolf." Offline models can leverage such useful future context to correctly identify "Player0" as the referent. In contrast, online models must rely solely on historical information, which may be ambiguous. For example, in mentioned player prediction, before the speaker says "[PlayerO]: And then you were the last to say robber. So we kill [MASK].", "" a previous utterance states "[PlayerO]: I'm the troublemaker and switched [Player3] with somebody." The online model may be confused and incorrectly answer "Player3".
Second, online models must resolve social references based entirely on immediate and past signals, where subtle social cucs-such as pointing gestures, posture, head turns, and body orientation- -ocomee essential. However, in complex interactions involving multiple participants, such signals are often too subtle to interpret from raw RGB video alone, especially without structured visual guidance. Experimental results, shown in Figure 3. indicate that VLMs fail to capture detailed and accurate social cues from raw RGB video. When prompted to describe the social interaction cues in the video, the VLM typically returns only coarse attributes such as the relative positions and appearances sopearances-using phrases like "on the far right" " and "shaved head and a mustache," " and occasionally produces hallucinations, such as incorrectly stating "four people."
Discussion. OnLIne-MMSI differs fundamentally from history- egocentric Al tasks (Grauman et al.
2022 Yang et al. 2025 Li et al. 2025 Zhou et al. 2025 Hong et al. 2025 Ye et al. 2024 Cheng et al.] 2024a
5

---

Published in Transactions on Machine Learning Research (Macch2026)
Raw RGB fromis
Plaverd
Player!
Playera Player Paverrrrrr Playyer
and women, sitting around table. The player on the far left is a woman with red hair. The player next to her is man with a bald head and beard. The player next to him is man with shaved head and mustache. The player on the far right is man with beard.
The scene shows four people,
VLM's deseription the social interaction cues in the video.
Illustration of the challenge in relying solely immediate and past context when transitioning from offline to online settings, where subtle social cues become increasingly critical. In complex multi-party interactions, such cues are often too muanced to be reliably inferred from raw RGB video alone, especially in the absence of explicit visual prompting. When asked to describe social interactions, the VLM frequently produces coarse spatial references (green), generic appearance descriptions (yellow), or even hallucinated content (red), while failing to capture the subtle and intricate social signals multi-person dynamics.
Figure 3:
in its modeling objective. Online-MMMI focuses on high-level social interaction understanding, requiring the model to infer who is being referred to in the current utterance by reasoning over verbal and non-verbal social cues. In contrast, retrospective egocentric tasks are typically formulated as event retrieval or grounding problems, answering questions such when, where, or what happened. Moreover, Online-MMSI emphasizes short-horizon situation modeling, as referential intent is usually resolved through local and rapidly evolving social context, whereas egocentric tasks rely on long-horizon memory retrieval over extended video histories. Finally, Online-MMSI explicitly requires consistent multi-person identity tracking, since the output is a specific participant identity, an assumption that is well-defined in fixed-view multi-party interactions but generally ill-posed in first-person egocentric videos with frequent viewpoint changes.
4 Methodology
To address these challenges, we introduce Online-MMSI-VLLNN a framework designed to respond to social tasks using only historical input. As illustrated in Figure + the model takes as input the user prompt, historical utterances, and video frames, and produces real-time response under the online setting. To compensate for the lack of future information and enhance the quality of historical signals, we propose two key components: (i) multi-party conversation forecasting, which anticipates future dialogue turns; and (ii) socially-auen aware visual prompting, which guides the model to attend to socially relevant visual cues.
4.1 Multi Conversation Forecasting
To enable forecasting multi-parly conversations, append forecasting query after the lask prompt, which is: "Predict the upcoming speakers' turns and then predict the upcoming utterance of each speaker." After answering the OnLine-MMSI task, the model generates the next few speaker labels and utterances. Directly predicting the full utterances of each future speaker in a single step can be overly complex and may lead to sub-par generation. Instead, we propose a coarse-to-fine approach with the following two phases:
The model first generates a sequence of four upcoming speaker identi- tics, c.g., "The upcoming speakers' turns: Playcr0, Player4, PlayerO, Player3."
Speaker-turn prediction:
Utterance refinement: For each predicted speaker, the model then produces fine-grained utter- ance. For instance: "The upcoming utterances: [PlayerO]: I didn't swap you. [Player4]: was the Insomniac. did not wake up as myself. So I... [Player0]: Yes. [Player3]: Who did you swap?"
This strategy mimics human conversation patterns, where we first anticipate who might speak and in what order, and then try to guess what they would say. The above process can be formulated as:
S4414K = (DLMM(D(LaL) U.41 fo (S44144K, D(u-jaa).
(1)
(2)
where S denotes the predicted sequence of speakers over the next K turns, and Ü represents the generated sequence of corresponding utterances for each speaker.

---

Published in Transactions on Machine Learning Research (March/2026)
Historical Input
Training Target
Response Player0.
Multi-party Conversation Forecasting
The turn: Player0, Player4, Playert, Player3. The utterances: [Pluyer0]: didn't swap you [Player4]: was Insommiac. did not wake up myself. So I.. [Player(]]: Yes. [Player3]: Who did you swap?
Biscot
Multimodal Large Language Model
LoRA
Video
[PlayerO]: She just went sleep villager and woke as alien.
Identify which player the Player3 is talking to?
[Player0]: you're mason?
Predict the upcomping speakers' turns and then predier the upcoming ufterance of each speaker,
[Player2]: I'm mason, P I'm troublensker. [Player3); Did you swap with anybody? Dialogues
Socially-aware Visual Prompting
Video
Dialogues
Prompt
The training pipeline of Online-MMSI-VLM. The model takes the user prompt, historical di- alogues, and recorded video as input and generates immediate response. To tackle the challenges of Online-MMST, we introduce two techniques: (i) mulli-party conversation forecasting to enrich language con- text, and (ii) socially-aware visual prompting to facilitate historical social cues modeling.
Figure 4:
4.2 Socially-aware Visual Prompting
Since online models must rely on immediate and past information, capturing subtle visual cues is crucial for resolving referents. Social scenes often involve multiple participants engaged in dynamic non-verbal inter- actions, making it difficult to localize and interpret key visual cues such posture and gestures. Without explicit guidance, the model may fail to attend to the socially salient regions in raw RGB image input, espe- cially in cluttered situations. To address this, we propose socially-aware visual prompting, which explicitly highlights important social cues in the historical video frames. Specifically, we annotate all visible partici- pants with: (a) speaker labels, aligning players with their spoken utterances; (b) bounding boxes, indicating each participant's spatial position; and (c) upper body keypoints, capturing posture, gaze direction, facial expressions, and gestures. These annotations are generated using AlphaPose (Fang et al. 2022)
As shown in Figure 1 integrate these annotations by overlaying them directly onto the video frames. To distinguish between individuals, we assign a unique color to each person's annotation and include system prompt specifying the mapping: "The red, blue, green, yellow, purple, and orange colors correspond to PlayerO, Player1, Player2, Player3, Player4, and Player5, respectively." The annotated frames Vir are led into the visual encoder of the VLM, providing a more semantic representation of the social scene. This representation is then concatenated with the tokenized historical dialogues Dit-dy:t and the user prompt Pt. The final multimodal embedding is processed by the language head to produce response RG;
Re fy
.VLM (D D(t-dja. P.l).
(3)
where R₁ represents the predicted speaking target, the referent of a pronoun, or the player mentioned by name. By visually highlighting key social signals, this prompting approach helps the model focus on the most relevant regions, improving its ability to interpret past visual context in the absence of future cues.
4.3 Supervised Instruction Tuning
To enhance the VLM's performance on specific social tasks, we first construct onstruction-output pairs using existing social datasets (Lee al.) (2024a), incorporating future utterances from transcripts into these pairs. We then fine-tune the model via supervised learning with two objectives: (a) task-specific objective: the

---

Published in Transactions on Machine Learning Research (March /2026)
model learns to produce correct answers for specific Online-MMSI task; (b) forecasting objective: the model learns to anticipate future dialogue turns S4+14+K and corresponding utterance content U++1++K-
Let Lymsi and Liorecast denote the loss for Online- and forecasting, Our training objective is:
Liotal = -mmsi
Liorecast
(4)
This multi-task formulation enables the model to not only answer current queries but also to anticipate future interactions, leading to better context. understanding and improved performance in online social scenarios.
5
Experiments
5.1
Implementation
We select LLaMA-3.2 Vision-11B (Dubey et al. 2024) and Qwen2.5-VI-7B Team 2025 as our multi- modal large language models, both of which are representative state-of-the-art models. For Qwen2.5-VL VL-7B, we dynamically sample video at a rate of 1 frame per second (fps) as visual input. For LLaMA-3.. 3-2- Vision- 11B, we sample six frames from each video clip evenly and arrange them into single 3 grid-like image, following the suggestion from prior work (Kim al.) 2024). We apply instruction tuning with LoRA (Hu) et al. 2022), targeting the query and value projection layers, to further specialize these models for our tasks. Specifically, we use an alpha value of 16, a dropout rate of 0.05, and rank of 512 for the LoRA configuration. We train all new or adapted parameters with unified cross-entropy loss. The weights of two losses are set as 1 by default. The learning rate is set to 1 x 10 empirically for the speaker target identification and mentioned player prediction lasks, and 1 x 10-3 for the pronoun coreference resolution task. The batch size is set 1, with gradient accumulation steps of 4. Each model trained for 5 epochs on all three tasks. The historical window length d is set to 10 dialogue turns, and the forocasting horizon K is set to 4 utterance turns. The average duration of each dialogue turn is approximately 2.8 seconds. The tracking information and keypoints for prompting are generated by AlphaPose (Fang et al. 2022) and curated using the reference frame from Lee et al. (2024a). More experiment implementations are provided in Appendix section.
5.2 Datasets
Experiments are conducted on the Werewolf Among Us dataset, which comprises two subsets (You'Tube and Ego4D) of social deduction games (Lai et al. 2023 We follow the dataset and task setup of Lee et al. (2024a). while cutting the future video and transcript in each sample for the online setting.
YouTube contains 151 games of One Night Ultimate Werewolf, which corresponds to 151 separate videos with 14.8 hours and transcripts comprising 20,832 utterances. It has 3,255 samples for speaking target identification, 2,679 for pronoun coreference resolution, and 3,360 for mentioned player prediction.
Ego4D has 40 games of One Night Ultimate Werewolf and 8 games of The Resistance: Avalon. It contains 101 separate videos with 7.3 hours and transcripts containing 5,815 utterances, with 832 samples for speaking target identification, 503 for pronoun coreference resolution, and 472 for mentioned player prediction.
Evaluation. We follow Lee al. 2024a) to report the overall accuracy of the predicted referent.
5.3 Performance Comparison
Table I presents the performance of our proposed Online-MMSI-VLLN. framework with two VLM backbones Qwen2.5-Vi LL and LLaMA-3.22 --oompreeed to wide range of baselines across three social understanding tasks: speaking target identification, pronoun coreference resolution, and mentioned player prediction. Eval- nations are conducted on both the YouTube and Ego4D subsets. We compare against capable VLMs such as InternVL3 (Zhu et al.) 20225), Qwen2.5-VI and LLaMA-3.2-V under vanilla finetuning. We also include results from Lee et al. Lee et al. (2024a) trained offline / online and tested online. Both variants of Online- MMSI-VLM consistently outperform the basclines across all tasks and datascts. Notably, the LLaMA-bascd model achieves the highest average accuracy on YouTube (60.6%) and Ego4D (56.1%). We report the zero- shot performance of Gemini 2.5 Pro and GPT-4o. Interestingly, their average zero-shot accuracy Ego44D
8

---

Published in Transactions on Machine Learning Research (March/2026)
Speaking Target Identification
Pronoun Coreference Resolution
Mentioned Player Prediction
Video and dislogue streams
Video and dialogue streams
Video and dislugue streams
Playeri Plaver Player Plaaeerd
Identify whit wich playee the speaker is talking to?
Playerd Player1
Playes 2 Playeee y
promoum refers to?
Determinn which player a
Playerd Player] Plavera Plaver Plaver Palver
Predlet which player meationed by name?
[Paayer0]: that you or that, is it that... This is you. (To [Mask])
[1aaeer3]: He like, what's [MASK] (ovigiaally-bb)) microwaving?
[Player0]: You saw [MASK]'s (wwigioally-Pllyyeddp
Lee et al. prediction: Player4.
X
Lee et al. prediction: Player0.
X
Lee al. prediction: Player4.
X
Ours prediction: Player3.
Ours prediction: Player4.
Ours prediction: Player2.
Figure 5: Qualitative results of Ouline-MMSSI- on three social tasks. Compared the prior work (Lee) et al. 2024a), method provides immediate and accurate referents based historical video and dialogue. YouTube Ego4D across three MMSI tasks: Speaking Target Identification (STT), Pronoun Coreference Resolution (PCR), and Mentioned
Table 1: Performance comparison of baselines and Maline-MMSI-VLLI
Player Prediction (MPP). Results are reported as Accuracy (%) on YouTube / Ego4D.
Model
STI
PCR
MPP
Average Accuracy
Lee al.
2024a
(Ofline)
60.0
52.8
63.5
41.1
43.2
32.1
55.6
42.0
Lee et al.
2024a
(Online)
59.1
59.6
63.4
55.4
47.3
41.0
56.6
52.0
Vanilla InternVL
62.4
52.8
65.1
55.2
47.0 / 36.8
58.2
48.2
Vanilla Qwen
60.8
57.7
62.9
48.2
45.8 / 32.9
56.5
46.3
Vanilla LLaMA
63.2 / 60.2
65.4
54.5
46.6 / 38.2
58.4
/ 51.0
Gemini 2.5 Pro (Zero-shot)
23.3 / 30.5
26.7
29.4
16.7
/ 30.2
22.3 / 30.0
GPT-4o (Zero-shot)
38.5
44.0
35.4
50.9
37.9
31.6
37.3
42.2
Online-MMSI- (Qwen)
64.6
61.7
68.8
60.7
47.2
39.5
60.2
/ 53.9
(LLaMA)
64.9 65.1
68.5
59.8
48.4
43.4
60.6
/ 56.1
is higher than on YouTube, while the trained models show the opposite trend. This can be explained by the fact that YouTube contains substantially more training samples than Ego4D, which boosts the performance of fine-tuned models but does not affect zero-shot results. The streaming video understanding models, such
Chen et al. (2024a) and Liu et al. (2024), are not directly suitable for comparison due to fundamen- tal differences in objectives. Our work locuses on Online-MMSI, which requires reasoning over multi-parly social dynamics and explicitly handling the absence of future context. But streaming video understanding modols are mainly designed for efficient processing in gencral-purpose video tasks and are built upon similar backbones as ours (i.e. a LLaMA-3-8B language model). They don't have mechanism to explicilly capture the social interaction cues (e-g. gestures and postures) nor deal with the challenge of missing future context..
Across tasks, observe that pronoun coreference resolution generally achieves the highest accuracies (up to 68.8%), followed by speaking target identification, while mentioned player prediction remains the most challenging, with accuracies typically below 50%. This pattern can be explained by the differences how each task depends on context. Both pronoun coreference and speaking target identification often involve referents located in adjacent utterances either the preceding or following turn-making them more amenable forccasting and short-range historical modeling. In contrast, we find mentioned player prediction typically requires tracking entities introduced several turns earlier, the referent is often not mentioned in the immediate context. This temporal gap increases the difficulty of reasoning mentioned player.
Figure 5 illustrates qualitative comparison examples of the three social tasks in streaming video, demonstrat- ing how our method leverages only past dialogue turns and annotated frames to accurately identify social references. For user queries such as "Predict which player is mentioned by name," "Determine which player a pronoun refers to," and "Identify which player the speaker is talking to, our model consistently provides immediate and accurate referents based solely on prior context. In contrast, the prior method (Lee et al. 2024a) often struggles in online settings due to lack of tailored design for these challenges.
9

---

Published in Transactions on Machine Learning Research MMacch2022
Table 2: Ablation study of proposed components for three social tasks on the YouTube and Ego4D datasets. Both socially-aware visual prompting and multi-party conversation forecasting consistently improve perfor- mance over the baseline. Their combination yiclds the best results across all three tasks and both datascts.
Speaking Target
Pronoun Coreference
Mentioned Player
Model
Forecasting
Prompting
Identification
Resolution
Prediction
YouTTbe
Ego4D
You'll Tube
Ego4D
YouTube
Ego4D
Qwen
60.76
57.71
62.88
48,21
45.83
32.89
Qwen wen
61.45
58.10
64.43
58.93
46.24
35.52
Qwen
63.96
60.00
65.20
54.46
46.52
36.84
Qwen
64.58
61.71
68.83
60.71
47.20
39.47
LLaMA
63.20
60.20
65.39
54.46
46.61
38.15
LLaMA
64.02
64.00
67.87
55.35
47.33
42.10
LLaMA
64.43
61.71
67.30
55.35
47.34
40.78
LLaMA
64.88
14.14
68.45
59.82
48.43
43.42
Table 3: Performance impact of forecasting components on three social tasks using Qwen-based model on the YouTube dataset. Detailed futurr utterance generation consistently enhances results, while adding speaker- turn prediction yields further improvement, demonstrating the strength of coarse-to-fiie strategy.
Speaker
Detailed
Speaking Target Identification
Pronoun Corcference
Mentioned Player
Turns
Utterances
Resolution
Prediction
60.76
62.88
45.83
60.31
64.63
46.39
V
62.69
64.92
45.93
63.96
65.20
46.52
5.4 Effects of Conversation Forecasting
Table 2 shows the effects of multi-party conversation forecasting, where forecasting consistently improves performance across all tasks. For example, in the pronoun coreference resolution task on the YouTuue dataset, applying forecasting boosts Qwen-based's accuracy from 62.88% to 65.20%. Similarly, for speaking larget identification, forecasting improves Qwen-based's performance from 60.76% to 63.96%. These results confirm that anticipating future conversational turns and utterances helps the model compensate for the lack of future context in the online setting. The improvement for mentioned player prediction is relatively smaller, possibly because this task, mentioning someone in a dialogue, relies less on future context.
Table 3 further analyzes the impact of the two components in our coarse-to-fine forecasting framework: speaker-turn prediction and detailed utterance generation. Results indicate that enabling both components yields the highest overall performance. In particular, direct detailed utterance generation consistently en- hances accuracy by enriching the context. Moreover, integrating speaker-turn prediction provides additional gains, suggesting that predicting turn structure first facilitates the generation of relevant utterances.
Table explores the effects of different forecasting turn lengths (i.e. 2, 4, or 8 future turns). Empirically, 4-turn forecasting length provides the most reliable performance gains across the three social tasks on both datasets. From the observation in 5.6 multi-party conversation forecasting is challenging due to its uncertainty nature. Extending the forecasting length further increases the risk of hallueinated, question- irrelevant content and adds computational complexity, while contributing little additional useful context. Thus, longer forocasts do not improve performance lincarly; instcad, they exhibit diminishing returns. For these reasons, forecasting 4 turns strikes the most practical balance bet ween accuracy and complexity.
5.5 Effects of Visual Prompting
Table [ also shows the effects of socially-aware visual prompting, where prompting further enhances model performance by effectively enhancing past information. For example, in the pronoun coreference resolution fask, adding visual prompting raises Qwen-based's accuracy from 62.88% to 64.43% and I.LaMA-basee
10

---

Published in Transactions on Machine Learning Research (March/202))
Table 4: Impact of forecasting length on three social tasks using Qwen-based model across YouTube and
Ego4D datasots. Forecasting 4 future turns typically yiclds the highest performance gains across all tasks.
Dataset
YouTube (%)
Ego4D (%)
Forecast Length (Turns)
2
4
8
2
4
8
Speaking Target Identification
62.44
64.58
61.98
62.03
61.71
59.43
Pronoun Coreference Resolution
68.26
68.83
67.11
56.07
60.71
55.54
Mentioned Player Prediction
45.98
47.20
46.25
34.26
39.47
39.47
Table 5: Effects of different visual prompting modules on three social tasks. We adopt the Qwen-based model and evaluate on the YouTube dataset. Incorporating bounding boxes (BB) or upper-body keypoints (KP) improves average accuracy over raw RGB frames, and combining them yields the best results.
Speaking Target Identification
Pronoun Coreference
Mentioned Player
Visual Cues
Avg. Accuracy
Resolution
Prediction
Raw RGB frames
63.96
65.20
46.52
58.56
RGB + BB
64.13
68.13
46.71
59.66
RGB + KP
63.96
65.79
47.10
59.28
RGB + BB + KP
64.58
68.83
47.20
60.20
from 65.39% to 67.87% on YouTube. This result verifies that visual prompting can serve as highlighting the crucial areas in the video, enabling VLMs to capture subtle and informative social cues in historical video.
We investigate the impact of incorporating different types of visual annotations namely, bounding boxes and upper body keypoints model performance. In the baseline setting, raw RGB frames are annotated only with speaker identities to align visual inpul. with the corresponding utterances. As shown in Table 5 incorporating bounding boxes and keypoints improves average accuracy over raw RGB frames by 1.1% and 0.72% respectively, and further combining bounding boxes and keypoints yields the highest gain of 1.64%. It indicates that bounding boxes and keypoints stress complementary social cues at different levels of granularity. Bounding boxes primarily highlight coarse-level information, such as participants' spatial location and body orientation. In contrast, upper-body keypoints emphasize finer- finerngraieed signals, including gaze direction and hand gestures. These improvements demonstrate that explicitly highlighting players' spatial positions and gestures enables the model to better interpret social dynamics within the scene.
The tracking and keypoints are provided by Lee et al. (2024a), where annotations are generated by Alpha- Pose (Fang et al.) 2022) followed by human curation with the reference frame. To further investigate model robustness to inaccurate tracking and keypoints, we conducted a preliminary study for visual prompting. During inference, we explored 20% dropout to keypoints or 10% dropout to tracking, simulating keypoint jitter (motion/oce) /occlusion) and short-term tracking loss. Our Qwen-based approach maintains strong perfor- mance the speaker target identification task, with 64.5% on Youtube and 61.7% on Ego4D.
Furthermore, as shown in Table 2 combining socially-aware visual prompting and multi-party conversation forecasting consistently yields the best performance across all three tasks and both datasets. This demon- strates the complementary strengths of forecasting future conversations and enriching visual understanding.
5.6 Evaluation of Generated Conversations
Figure ? presents two examples of generated future speaker turns and utterances, where green text indicates accurate and consistent conversations with the ground truth and yellow denotes different but reasonable utterances. Although the predicted turns and wording may differ slightly from the ground truth, the model successfully produces plansible speakers and content that align with the surrounding context and human social reasoning. This helps the model build a more coherent understanding of the ongoing dialogue while missing future context. In the first example, the generated utterances reinforce the exchange between Player1
11

---

Published in Transactions on Machine Learning Research (March/200 (
Previous
Previous
[Player5]: For real.
[Player2]: He could be drunk.
[Player4]: Ah, that's a convenient card.
[Player]]: Which ones did you look again? can't remember.
[Player]]: I was the Insomniac.
[Player0]: looked at just card.
[Player5]: What you now?
[Player1]: Just his card.
Ground Truth
Ground Truth
[:layer]]: The Insomniac.
[Player0]: Yes.
[Player5]: That's funny because I'm the Insomniac.
[Player]]: And it said
werewolf?
[Player1]: That's funny because I'm the Insommiac.
[Player2]: suddenly don't think you 're a seer.
[Player4]: They have the same story.
[PlayerI]: |: His said werewolf?
Forecast
Forecast
[Player]]: J: was the Insomniac.
[Player0]: Yeah.
[Player5]: So you're saying he's lying?
[PlayerI]: Okay. What did you see?
[: Yeah.
[Player0]: saw werewolf.
[Player5]: And you're saying you're the Insomniac?
And
[Player2]: That's little suspicious.
Samples of ground truth future conversations and generated forecasts. Although the predicted turns and wording may differ slightly from the ground truth (yellow), the model successfully produces plausible speakers and content that align with the surrounding context and human social reasoning (green).
Figure 6:
History
Ground Truth Future
Forecast
[Player0]: [Player4] could.
[Player0]: These two. Yeah. [Player1]: Yeah. [Player0]: So we're nice, happy town after that last yelling around.
[Player4]: Yeah.
[Player4]: was.
[Player2]: Okay. So have two people who were the robber.
[Player2]: Okay. You
confirmed your story. You
[Player4]: was the robber.
switched her and him.
Figure 7: Failure case of the forecasting module on the STT task. The predictions align with historical context but deviate from the ground truth, illustrating the challenge of anticipating uncertain social dynamics. Notably, the goal of forecasting is enhance social understanding, rather than exact future prediction.
and Player5 regarding the Insomniac role. In the second, the forecasting captures a multi-party interaction among PlaycrO, Player1, and Player2 concerning the obscrvation of another playor's card.
To further measure the quality of the generated conversations, we employ Macro F1 (Pedregosa. el. al. 2011) and BERTScore (Zhang al. 2019) as evaluation metrics. Macro F1 measures the accuracy of predicted speaker turns, while BERTScore evaluates the semantic similarity between generated and ground- truth utterances. We exclude BLEU (Papineni exact n-gram matches, which makes it less reliable for assessing short and diverse conversational responses, where minor paraphrasing or changes in word order can disproportionately penalize the score. Our approach achioves a Macro F1 of 0.68 and BERTScore of 0.86, indicating accurate upcoming speaker prediction and strong semantic alignment. with the ground truth. These results validate the effectiveness of our proposed conversation forecasting in enhancing linguistic context for Online MMSI.
al., 2002) from our evaluation due to its reliance
Figure 7shows failure case of forecasting where forecasts consistent with history but diverge from ground truth, highlighting the challenge of anticipating uncertain social dynamics. Through these experiments and evaluation, we observe that multi-party conversation forecasting remains challenging task due to the inherent unpredictability of real-world dialogue and the current limitations of LLMs in complex social reasoning. Nevertheless, our goal is not to achieve precise forecasting, but to use the process as a means of enhancing social understanding. Interestingly, we observe that even inaccurate forecasts can improve performance-pont because forecasting encourages the model to role-play the historical dialogue, thereby deepening its understanding of social context and interaction dynamics.
12

---

Published in Transactions on Machine Learning Research (March/200 (
Raw RGB francs
Plasri
Plawrd
Plaser Playord
The person the far left is gesturing with their hands, suggesting they actively contributing the conversation. The person second from the left is holding drink, while the person second from the right is holding phone, possibly taking photo or recording the moment. The person on the far right is looking at the person on the far left, indicating active engagement in the conversation.
Plaveri
Playerd
The person in the center appears to be speaking, as indicated by their open mouth and the direction of their gaze towards the other participants. The other individuals are looking at the speaker, indicating that they are listening attentively.
syntodlay Aport + savee Coniin
Flayert Maagert
Playse2
Plavee Playere
The person in the red box is speaking, while the other two individuals listening attentively. The person in the green box appears to be nodding and smiling, indicating agreement understanding, The person in the yellow box is also smiling and looking at the speaker, suggesting that they are engaged in the conversation.
VLM-generated descriptions under different visual prompting settings. The results show that using raw RGB frames yields generic descriptions with limited reference social interactions. In contrast, when participants are annotated with bounding boxes and body keypoints, the VLM generates more accurate and detailed accounts of social dynamics, such as "nodding and smiling" and "looking at the speaker".
Figure 8:
5.7 Further Analysis of Visual Prompting
We explored how the model understands the social interaction in the historical video under different visual prompting. We provide Qwen2.5 VL with videos containing different visual promptings and the query: "De- scribe the social interaction cucs in the video." The generated descriptions are presented in Figure X where green text indicates accurate social descriptions, red highlights misleading or incorrect descriptions, and black denotes general, non-specific deseriptions. The results demonstrate that, by annotating each partici- pant with bounding boxes and body keypoints, the VLM produces more accurate and detailed descriptions of social interactions. For example, the model generates the description: "The person the yellow box is also smiling and looking at the speaker." Compared to other prompting's generation, such description has more details about body movement and facial expression. This suggests that socially-aware visual prompting effectively emphasizes critical regions of interest, thereby improving the model's ability to interpret subtle social dynamics in noisy and large historical video. We also noticed that, in raw RGB frames input, the description not only fails to capture detailed social interaction cues but also contains hallucinations: "The person second from the left is holding a drink, while the person second from the right is holding a phone." The visual prompting shows capability of suppressing inaccurate description.
5.8 Efficiency Evaluation
To assess Online-MMSI-VLMN real-time applicability, we evaluate the model's latency and throughput. Experiments show the model achieves median latency of 302 ms and throughput of 1.99 samples per second when running on a single A6000 GPU, supporting deployment in natural conversational scenarios.
For completeness, we additionally conducted efficiency evaluations in terms of inference latency and GPU memory consumption on representative baselines. Table D reports the inference latency of different methods. We observe that Online-MMSIIVT.M consistently maintains sub-second latency across both backbones (Qwen and LLaMA), with inference times of 0.302 s and 0.554 s, respectivcly. This efficiency enables immediato responses in ouline scenarios. In contrast, API-based methods such as GPT-4o and Gemini 2.5 Pro exhibit higher response latency, which limits their suitability for real-time deployment. Moreover, the substantially higher latency Lee et al. (2024a) primarily stems from its offline processing pipeline, which relies on future conversational context and delayed processing, making it inherently unsuitable for online inference.
We additionally evaluate GPU memory consumption during inference for open-source vision-language models, as summarized Table 7 Onlino-MMSI-VI.. requires approximately 20 GB of GPU memory, which is comparable to its vanilla counterparts. Importantly, this memory footprint is well within the capacity of widely adopted embedded computing platforms such as Jetson AGX Orin, which provides 32 GB unified
13

---

Published in Transactions on Machine Learning Research MMarch/2020)
Table 6: Inference latency comparison across different models. Online-MMSIIVLLM maintains sub-second
latency, supporting immediate feedback in online scttings.
Model
Latency (s)
Lee (2024a)
14.010
GPT-40
3.846
Gemini 2.5 Pro
2.207
Vanilla Intern VL
2.953
Vanilla Qwen
0.268
Vanilla Llama
0.533
Onlivc-MMSI-IIMM (Qwen)
0.302
Ouline-MMSIIVIM (Llama)
0.554
Table 7: GPU memory usage during inference for open-source vision-language models. Online-MMSI-VLM
requires approximately 20 GB GPU memory, enabling stable real-time inference Jetson platforms.
Model
GPU Memory (GB)
Vanilla Intern VI.
21.31
Vanilla Qwen
17.26
Vanilla Llama
21.49
MnlIne-MMSI-VT.M (Qwen)
17.39
OMline-MMSI-VLM (Llama)
21.52
memory. This leaves sufficient headroom for runtime overhead and system-level processes, enabling stable and real-time online inference in practical deployment settings, including robotics and embodied Al applications.
We also evaluated the end-to-end latency of whole system, processing speech and generating forecasts and responses, and the result. is 343 ms, demonstrating leasibility for real-time online use. Specifically, in the social game scenario, participants sit together, each equipped with a headset microphone. The streaming audio from cach speaker is transcribed by Whisper (Radford al. 2023), which processes audio at speed of 41 ms per second. Because transcription is incremental, this latency remains nearly constant regardless of utterance length. Combined with forecasting and generating answer, the end-to-end latency is 343 ms.
5.9 Common Failure Cases
Fig. illustrates representative failure cases across the three social interaction tasks. We observe that the model tends to fail in scenarios involving overlapping social dynamics. For example, in the first. STI case, two participants (Player2 and Player3) simultaneously address different interlocutors, creating ambiguity addressee resolution. Similarly, in the second PCR case, the target utterance contains multiple potential referents (e.g., she and him), increasing coreference uncertainty. In addition, the model struggles when social interaction cues weak indirect. For instance, in the third MPP case, Player3 talks to Player4 and refers to PlayerO with the statement "PlayerO is werewolf", while the supporting verbal context ("PlayerO: think we kill [Player3].") provides only a weak and indirect cue. In such cases, the lack of strong verbal or nerbverbal signals makes correct referent identification particularly challenging.
6 Conclusion
We introduce & new task, Online Multimodal Social Interaction Understanding, which requires models to interpret social interactions using only historical information. This setting better aligns with real-world human-AT interaction scenarios, where the AT assistants are required to reply immodiately, Since future con- text unavailable, the performance of existing models, human, and advanced AI tools degrade significantly. To address this challenge, propose Online-MMMSI-VLLM. novel framework built upon recent. advanced
14

---

Published in Transactions on Machine Learning Research (March/200) /2026)
Speaking Target Identification
PlayerO Player1 Player2 Player3 Playerd Players
[Player3]: But said 100%.
[Player2]: [Player5] that's proving anything.
[Player4]: Who's carde did you switch?
[Player1]: 100%.
[Player5]: Them two, [Player1] [PlayerO]'s.
[Player4]: Okay so [Player1] and [Player0] switched, you were so means you're Werewolf you have his card.
Seer you could
[Player3]: But he wasn't- but he's-
Daa
[Player0]: Yeah if he was lying about being a Villager originally.
Prediction: Player3
X
[Player2]: J: was robber?
[Player3]: 3::
saying
my Werewolf Troublemaker, who the Robber? [MASK])
Villager
why him.
Ground Truth: PlayerS
[Player2]: You
Pronoun Coreference Resolution
Playord Playert
Player2
Player3 I Playerd
[Player0]: believe You Insomniac like right away. you up Insomnisc?
[Player2]: woke up the Insomniac.
[Player0]: believe this hundred percent that I don't even want to take time overthinking it.
I'm ready to vote to the left.
[Player4]:
There werewolf?
[Player0]:
just working together. [Player1]'s trying [PlayerS] out.
[Player4]: you warewolf?
[Player0]: He's robber now.
[Player2]: Not pick on you, but to pick on you. Sorry. Was anyone able confirm your story or just that...
20U6 Daw light Arrocaat
Prediction: Player1
X
[Player0]: [Player4] could.
Ground Truth: Player3
[Player4]: was.
[Playe12]: Okay. You confirmed story. switched her and [MASK Horignellyyynnn
Mentioned Player Prediction
Player Playar1
Player2
Playará Playark
[Player4]: Yes.
[Player0]: Yes?
[Player1]: Who'd you steal from?
[Player3]: Can you confirm story unconfirm it?
[Player4]: confirming Audio:
remsining.
[Player0]: Guys,
minute. This isn't villagers win. We're hiding behind
information.
[Player4]: want to know what role you want.
[Player0]: Ithink we [Player3]. the only one hasn't given out any information
Prediction: Player1
X
[Player2]: That literally
Ground Truth: Player0
[Player3]: [MASK] fariginally (Playarr) is the werewolf, the Tanner took
Figure 9:
Common failure cases across three social tasks. We observe that the modol is likely to fail when social dynamics overlap or when social interaction cues are weak or ambiguuus.
MLLMs, integrating techniques: (1) mulli-porly conversation forecasting, which anticipates future dialogue turns to enrich linguistic context, and (2) socially-awaa visual prompting, which highlights critical visual cues for more accurate social reasoning. Extensive experiments across multiple benchmarks demonstrate the effectiveness of our method in online social understanding tasks. Further ablation studies validate the individual contributions of each proposed component. We believe this work represents an important step toward developing practical and robust social Al systems capable of functioning in realistic environments. Nevertheless, such systems could also be repurposed for surveillance of social interactions, raising significant privacy and ethical concerns. These considerations underscore the need for responsible deployment.
Acknowledgements
We thank Teng Wang for carly-stage inspiration that shaped this line of work. We also thank our colleagues and peers for their valuable feedback and suggestions this paper.
15

---

Published in Transactions on Machine Learning Research MMarch/2020)
References
Gibran Benitez-Garcia, Josus Olivares-Mercado Gabriel Sanchez-Perez, and Keiji Yanai. Ipn hand: A video dataset and benchmark for real-time continuous hand gesture recognition. In 2020 25th international conference pollern recognition. (TCPR), pp. 1340-4347. IREF, 2021.
Cyuthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda. Social robotics. Springer handbook of robotics, pp. 1935-1972, 2016.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12914-12923, 2024.
Xu Cao, Pranav Virupaksha, Wenqi Jia, Bolin Lai, Fiona Ryan, Sangmin Lee, and James M Rehg. So- cialgesture: Delving into multi-person gesture understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19509-19519, 2025.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimmming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour lideu-language understanding. Aduances in Newral Information Processing Systems, 37:53168- 53197, 2024.
Jonathan P Chang and Cristian Danescu-Nicnlescu--e Mizil. Trouble on the horizon: Forecasting the derailment of online conversations they develop. arXiv preprint XrXi::1909.0 01362, 2019.
Haoyu Chen, Henglin Shi, Xin Liu, Xiaobai Li, and Guoying Zhao Smg: A micro-gesture datasel towards spontaneous body gestures for emotional stress state analysis. International Journal of Computer Vision, 131(6))1346-136, 2023a.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18407-18418, 2024a.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoor- thi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09 78, 2023b.
Wei Chen, Zhiwei Li, Hongyi Fang, Qianyuan Yao, Cheng Zhong, Jianye Hao, Qi Zhang, Xuanjing Huang, Jiajie Peng, and Zhongyu Wei. A benchmark for automatic medical consultation system: frameworks, tasks and datasets. Bioinformatics, 39(1):btac817, 2023c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the TEEF/CVE conference on computer vision and pattern recognition. pp. 24185 24198, 2024b.
Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, and Yang Liu. Videgothink: Assessing egocentric video understanding capabilities for embodied ai. arXiv preprint arXiv:24 10.1 11628, 2024a.
Zebang Cheng, Zhi-Qi Cheng, Jun- Yan He, Kai Wang, Yuxiang Lin, Zhong Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion -lama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805-110852 2024b.
Zesen Chong, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Vidcollama 2: Advancing peatial-temporal modeling and audio understanding in video-Ilms. Xiv preprint or arXio:2406.07//00 2024c.
16

---

Published in Transactions on Machine Learning Research (March/20P)
Eunji Chong, Yongxin Wang, Nataniel Ruiz, and James M Rehg. Detecting attended visual targets in video. In Proceedings of the IEEE/CVP conference. on compuler vision and pallern recognilion, pp- 5396-5400, 2020.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXio preprint arXiv:2507.06261 2025.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-eflciint exact attention with io-awareness. Advonces in neural information processing systems, 35:16344-16359 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguislies: haman language technologies, volume 1 (long and short papers), pp. 4171 4186, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.211 00224077.178, 2024.
Amine El Hattami, Issam H. Laradji, Stefania Raimondo, David Vazquez, Pau Rodriguez, and Christopher Pal. Workflow Discovery from Dialogues in the Low Data Regime. Transactions Machine Learning Research (TMLR), 2023.
MM Elsherbini, Ola Mohammed Aly, Donia Alhussien, Ohamed Amr, Moataz Fahmy, Mahmoud Ahmed, Mohamed Adel, Mohamed Fetian, Mahmoud Hatem, Mayar Khaled, et al. Towards a novel prototype for superpower glass for autistic kids. International Journal of Industry and Sustainable Developmend, 4(1): 10-24, 2023.
Theresa Epperlein, Gyula Kovacs, Linda S Oña, Federica Amici, and Juliane Bräuer. Context and prediction matter for the interpretation of social interactions across species. Plos one, 17(12))c0277783, 2022.
Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE transactions on pottern analysis and machine intelligence, 45(6):715777711 2022.
Shutong Feng, Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Michael Heck, Carel van Niekerk, and Milica Gasié. Emowoz: A large-scale corpus and labelling scheme for emotion recognition in task-oricntod dialogue systems. arXie preprint or Xix:22109. 04919, 2021.
Xiachong Feng, Longxu Dou, Minzhi Li, Qinghao Wang, Haochuan Wang, Yu Guo, Chang Ma, and Lingpeng Kong. A Survey on Large Language Model-Based Social Agents Game-Themree Scenarios. Transactions on Machine Learning Research (TMLR), 2025.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint XxXi:2501.00955 2025.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVE international conference on compuler vision, pp. 13505 13515, 2021.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 18995-19011 2022.
17

---

Published in Transactions on Machine Learning Research MMarch/2020)
Hongcheng Guo, Shaosheng Cao, Boyang Wang, Lei Li, Liang Chen, Xinze Lyu, Zhe Xu, Yao Hu, Zhoujun Li, et al. Sns-bench: Defining, building, and assessing capabilities of large language models in social networking services. In Forty-second International Conference on Machine Learning, 2025.
Anshul Gupta, Samy Tafasca, Arya Farkhondch, Pierre Vuillecard, and Jcan-Mare Odobez. Mtgs: novel framework for multi-person temporal gaze following and social gaze prediction. Advonces in Neural Infor- mation Processing Systems, 97:15646-116665 2024.
Nick Haber, Catalin Voss, and Dennis Wall. Making emotions transparent: Google glass helps autistic kids understand facial expressions through augmented-reaitty therapy. IEEE Spectrum, 57(4):46 52, 2020.
Lauren V Hadley and John F Culling. Timing of head turns to upcoming talkers in triadic conversation:
Evidence for prediction of turn ends and interruptions. Frontiers in Psychology, 13:1061582, 2022.
Md. Mahadi Hassan, Alex Knipper, and Shubhra Kanti Karmaker Santu. Introducing 'Forecast Utterance" for Conversational Data Science. Transactions on Machine Learning Research (TMLR), 2024.
Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. In Proceedings of the Compuler Vision and Pattern Recognition Conference, pp. 5344 5354, 2025.
Yuqi Hou, Zhongqun Zhang, Nora Horanyi, Jaewon Moon, Yihua Cheng, and Hyung Jin Chang. Multi- modal gaze following in conversational scenarios. In Proceedings of the IEEE/CVF Winter Conference on Ayplicalions of Compuler Vision, pp. 1186 1195, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 12):3, 2022.
Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: A comprehensive benchmark and memory-augmented method. arXiv preprint ar Xrx::255010008 2004.008444 2024.
Lee Hyun, Kim Sung-Bit Bin, Seungju Han, Youngjac Yu, and Tac-Hyun Oh. Smile: Multimodal dataset for understanding laughter in video with language models. arXiv preprinl wxXim:2312.0981 2023.
Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara. An Ilm benchmark for addressee recognition in multi-modal multi-party dialogue. arXie preprint or arXi:22011.16443 2025.
Swati Jindal and Roberto Manduchi. Contrastive representation learning for gaze estimation. In Gaze Meets Machine Learning Workshop, pp. 37-49. PMLR, 2023.
Alexander Kapitanov, Karina Kvanchiani, Alexander Nagaev, Roman Kraynov, and Andrei Makhliarchuk. Hagrid-hand gesture recognition image dataset. In Proceedings of the IEEE/CVF Winter Conference on Ayplications of Computer Vision, pp. 1572-4587 2024.
Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth video: Zero-shot video question answering using a vlm. IEEE Access, 2024.
Bolin Lai, Hongxin Zhang, Miao Liu, Aryan Pariani, Fiona Ryan, Wenqi Jia, Shirley Amugrah Hayati, James Rehg, and Diyi Yang. Wercwolf among us: Multimodal resources for modeling persuasion behaviors in social deduction games. Association for Computational Linguislies: ACL 2023, 2023.
Sangmin Lee, Bolin Lai, Fiona Ryan, Bikram Boote, and James M Rehg. Modeling multimodal social interactions: New challenges and baselines with densely aligned representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14585-14595, 2024a.
Sangmin Lee, Minzhi Li, Bolin Lai, Wenqi Jia, Fiona Ryan, Xu Cao, Ozgur Kara, Bikram Boote, Weiyan Shi, Diyi Yang, al. Towards social ai: A survey on understanding social interactions. arXiv preprint arXiv:2409.1533 2024b.
18

---

Published in Transactions on Machine Learning Research (March/200 /0026)
Yiming Lei, Jingqi Li, Zilong Li, Yuan Cao, and Hongming Shan. Prompt learning in computer vision: survey. Fronliers of Information Technology & Electronic Engineering, 25(1): 0-442-33--- 2024.
Wanhua Li, Zibin Meng, Jiawei Zhou, Donglai Wei, Chuang Gan, and Hanspeter Pfister. Socialgpt: Prompt- ing Ilms for social relation reasoning via greedy segment optimization. Advances in Neural Information Processing Systems, 37:2267 2291, 2024a.
Xinpeng Li, Teng Wang, Jian Zhao, Shuyi Mao, Jinbao Wang, Feng Zheng, Xiaojiang Peng, and Xuelong Li. Two in one go: Singlo-stage emotion recognition with decoupled subject-context transformer. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 9340 9349, 2024b.
Yanjun Li, Yugian Fu, Tianwen Qian, Qiaoo Xu, Silong Dai, Danda Pani Paudel, Luc Van Gool, and Xiaoling Wang. Egocross: Benchmarking multimodal large language models for cross-domain egocentric video question answering. arXiv preprint Xii::2508.10790 2025.
Dongze Lian, Zehao Yu, and Shenghua Gao. Believe it or not, we know what you are looking at! In Asian Conference Computer Vision, pp. 35-50. Springer, 2018.
Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, et al. Ov-mer: Towards open-vocabulary multimodal emotion recognition. 1CML 2025, 2024.
Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. ICML 2025, 2025.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiw:2311.1019, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing syslems, 36:34892 34916, 2023.
Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, and Jose M
Alvarc. Streamchat: Chatting with strcaming video. ar preprint 2xXi:2442.0800 2024.
Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, and Guoying Zhao. imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10631-10642, 2021.
Ning Ma, Norihiro Harasawa, Kenichi Ueno, Kang Cheng, and Hiroyuki Nakahara. Decision-making with predictions of others' likely and unlikely choices in the human brain. Journal of Newroscience, 44(37), 2024.
Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar. Yolo-pose: Enhancing yolo for multi person pose cstimation using object keypoint similarity loss. In Proceedings of the IEEE/CVF conference on compuler vision and pollern recognition, pp. 2637 2646, 2022.
Ganeshan Malhotra, Abdul Waheed, Aseem Srivastava, Md Shad Akhtar, and Tanmoy Chakraborty, Speaker and timo-aware joint contextual learning for dialoguc-act classification in counselling conversations. Proceedings of the fifeenth ACM international conference on web search and data mining, pp. 735 745, 2022.
AL Gpt-4o, 2024. URL.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
19

---

Published in Transactions on Machine Learning Research (March/202)
Vaidehi Patil, Yi-Lin Sung, Peter Hase, Jie Peng, Tianlong Chen, and Mohit Bansal. Unlearning Sensitive Information in Multimodal L.Mss: Benchmark and Attack-Defense Evaluation. Transactions on Machine Learning Research (TMLR), 2024.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Maith Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, (2(85):2822 2830, 2011. URL http:httt///it http/ppprr//v22/pedeeeggaaa html
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya. Sutskever. Robust specch recognition via large-scale weak supervision. In International conference on machine learning, pp. 28492 28518. PMLR, 2023.
Chirag Raman, Jose Vargas Quiros, Stephanie Tan, Ashraful Islam, Ekin Gedik, and Hayley Hung. Conflab: A data collection concept, dataset, and benchmark for machine analysis of freo-standing social interactions in the wild. Advances in Newral Information Processing Systems, 35:23701 -2:3711--23715, 2022.
Fiona Ryan, Hao Jiang, Abhinav Shukla, James M Rehg, and Vansi Krishna Ithap.. Egocentric auditory attention localization in conversations. Tn Proceedings of the IEFE/ /CF Conference on Computer Vision and Pattern Recognition, pp. 14663 14674, 2023.
Andrey Savchenko. Facial expression recognition with adaptive frame rate based on multiple testing correc- tion. In International Conference on Machine Learning, pp. 1011993 PMLR, 2023.
Anthony Sicilia and Malihe Alikhani. Eliciting uncertainty in chain-of-thought to mitigate bias against forecasting harmful user behaviors. arXiv preprint arXiv:2410.14744, 2024.
Samy Tafasca, Anshul Gupta, and Jean-Mare Odobez. Childplay: A new benchmark for understanding children's s gaze behaviour. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10935-20944 2023.
Chao-Hong Tan, Jia- Chen Gu, and Zhen- Ling. Is chatgpt good multi-party conversation solver? arXiv preprint ar 23v: 16301, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Qwen Team. Qwen2.5-vl, January 2025. URL https://quenln httpps///ggennngggttu...././///
5-vl1/
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-ol: Rethinking step-by-step visual reasoning in Ilms. arXiv preprint 2xX::2501006666, 2025.
Xiang Wang, Shiwei Zhang, Zhiwu Qing, Yuanjie Shao, Zhengrong Zuo, Changxin Gao, and Nong Sang. Oadtr: Online action detection with transformers. In Proceedings of the IEEE/CVF International Con-
ference on Computer Vision, pp. 7565-755 7575, 2021.
Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, al. Visual prompting in multimodal large language models: A survey, arXiv preprint X0:: 15310, 2024.
Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Lú, Oucheng Huang, Gen Luo, Hao Fei, Guannan Jiang, Xiaoshuai Sun, and Rongrong Ji. Controlmllm: Training- free visual prompt learning for multimodal large language models. Advonces in Neural Information Processing Systems, 37:45206 45234, 2025.
Jiarui Xu, Yossi Gandelsman, Amir Bar, Jianwei Yang, Jianfeng Gao, Trevor Darrell, and Xiaolong Wang. IMProv: Inpainting- based Multimodal Prompting for Computer Vision Tasks. Transactions on Machine Learning Research (TMLR), 2024.
20

---

Published in Transactions on Machine Learning Research MMarch/2020)
Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In Proceedings of the Computer Vision and Pottern Recognition Conference, pp. 28885 28900, 2025.
Lingfeng Yang, Yueze Wang. Xiang Li, Xinlong Wang, and Jian Yang. Fino-grained visual prompting. Advances in Newral Information Processing Systems, 36:24993 25006, 2023.
Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, el al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. arXiv preprint trXit:2504.110 17343, 2025.
Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal Iims for video qa. ar preprint xrXo::4400.07177, 2024.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8807 8817, 2019.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuna audio-visual language model for video understanding. arXiv preprint arXiv:2306.0888 2023a.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprinl or Xiv:2406.08065 2024a.
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internim-xcomppsee:: vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiw:2300.1511, 2023b.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Interaln--cooppser22 A versatile large vision language model supporting long-contextual imput and output. arXie preprint orXiv:2407.03320 2024b.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXic:1904.0000 09675, 2019.
Yuhang Zhang, Chengrui Wang, and Weihong Deng. Relative uncertainty learning for facial expression recognition. Advances in Neural Information Processing Systems, 34:17616 17627, 2021.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal Chain-of- Thought Reasoning in Language Models. Transactions on Machine Learning Research (TMLR), 2024c.
Chenxi Zhao, Jinglei Shi, Liqiang Nie, and Jufeng Yang. To err like human: Affective bias-inspired measures for visual emotion recognition evaluation. Advances in Neural Information Processing Systems, 37:134747- 134769, 2024.
Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-term action anticipation from videos? arXiv preprint ar 20v: 16368, 2023.
Yue Zhao and Philipp Krähenbühl. Real-time online video detection with temporal smoothing transformers. In European Conference on Computer Vision, pp. 185-502. Springer, 2022.
Wenqi Zhou, Kai Cao, Hao Zheng, Yunze Liu, Xinyi Zheng, Miao Liu, Per Ola Kristensson, Walterio Mayol- Cuevas, Fan Zhang, Weizhe Lin, et al. X-lebench: A benchmark for extremely long egocentric vidco understanding. arXie preprint or arXiv:2501.06833 1(3), 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamod Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. Xix preprint ar arXiv:2304. 10592, 2023.
21

---

Published in Transactions on Machine Learning Research MMarch2020
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. Xiviv preprint arX4o:2504.0447, 2025.
22

---

Published in Transactions on Machine Learning Research (March/200)
EAME (KNGAAINIAARR)
M1361
Sectire
Draert) and Dhavend mited frw
(Ploett:: wa thatt
(Faind): Bas l. lo Scadual bare la Frier sel kurnl sillel the not lmatee lave nardii aoat
(Pound)
[Ploaet0]] Yook anu sioild Benoace love infrrmiion Oatl aare and, stick wax cuperoo and me tasser wermool tae oenter. the (a0e3))
selaed Lyvidh be inferstte
Flryer
Player
AIR
1. deniin the rnl, Vlen sella ule and uoaaye silec reperd Fleed, Fleed, Flevsh Flaerk Paank
Higmss copostoniyy
Royer
To auciile
Itend: Hene intiretter Mot
(hareti
Shaweh: Dravwr
(hawd)) fe figm wit
Dtavett:: 2000 van icil Iflines".
Pham
tance vccovott Cto
Stawedi just ci.....
Joacity plawe spedir saliae escicce vitk lesitic
Miserd
Plaert
Piaet
Paner
Pinerr
Pauert
Dvslasdas
Figure 10: Mustration of human study on online setting and then offline.
A Appendix
A.1
Implementation Details
LLaMA- 3.2-11B-Vision (Dubey et al. 2024) architecture with image splitting disabled. The model operates with bfloat16 precision and employs FlashAttention-2 Dao et al. 2022) for efficient memory usage. For Qwen2.5- VL-7B-Instruct, the vidcos are sampled at 1.0 FPS and a resolution constraint 36 42 10 pixels per frame; for LLaM-3.2- 1---------BB--- L-i-vvinn. we transfer the video into a 3 grid-like image and set the maximum image size as 1120 1120 pixels. The max new tokens sot as 1024 in both models. LoRA-based (Hu) et al. 2022) fine-tuning strategy implemented, with LoRA alpha set 16, dropout rate of 0.05, rank of 512, and modifications applied to the query and value projection layers. The training configuration includes
The training of our video-based model is built upon the Qwen2.5-VI-7B-Imen Instruue Team 2025
httpp///wweeaeeeeeeee///eee/eeee hhtts///uuggggaeeccco//mtt/////a Vision
23

---

Published in Transactions on Machine Learning Research Mlarch/2020)
ROOSEVELT
Examples of curated clips from the new evaluation dataset, SIQ. The clips are primarily drawn from talk shows with multiple participants, representing domain that differs from social deduction games.
Figure 11:
batch size of 1 per device, gradient accumulation steps of 4, and total of 5 epochs. The optimizer used is AdamW with fused implementation, and the learning rate follows a linear decreasing scheduler strategy. The regularization weight decay of 0.01 and gradient clipping threshold of 0.3. Please note that we adopt different Icarning rates for pronoun coreference resolution because tasks differ in the number of available training samples and annotation density (3,255 for speaker target identification, 3,360 for mentioned player prediction, and 2,679 for pronoun coreference resolution). In practice, we empirically observed that assigning learning rate of 10-3 to pronoun coreference resolution leads to more stable optimization and improved performance across tasks compared to using uniform learning rate of 1 x 10-4.
A.2 Human Study
We select 30 samples randomly for each participant and show the online input followed by its offiie version, as shown in Figure 10 In this way, the participant gets rid of being affected by the prior offline information.
A.3 Limitations
While our method achieves strong performance practical Online-MMSI, several limitations remain. First, the approach relies on external preprocessing steps as previous approaches (Lee et al. 2024a), such visual tracking and speech transcription, which may introduce errors and affect overall performance. Second, the capabilitics of the model are bounded by the social reasoning abilitics of current LLMs, which may struggle with complex or subtle social dynamics. Future work may explore end-to-end solutions and more advanced perception and reasoning mechanisms further enhance online social interaction understanding.
A.4 Auxiliary Signals
To explore more auxiliary signals for visual prompting, we also extracted gaze signal from vidcos. However, integrating gaze through visual prompting results in performance drops, i.e,, 0.8%, 0.3%, and 0.5% on speaker target identification, pronounce coreference recognition, and mentioned player prediction, respectively. It might be due to inaccurate gaze estimations which are derived from GazeFollowing (Lian et al., 2018). Besides, some studies show existing LLMs might struggle to incorporate gaze information within the context of multi-party dialogues (Tnoue et al. 2025). indicating more sophisticated methods are needed.
24

---

Published in Transactions on Machine Learning Research (March/202))
Table 8: Cross-datasct transfer on speaker target identification (accuracy gain). These results indicate
that the model fine-tuned on social deduction games exhibits robust generalization ability.
Train Test
YubTube
Ego4D
SIQ
YouTube
64.58 (+43.82)
63.71 (+44.85)
68.80 (+26.8)
Ego4D 47.18 26.42)
61.71 (+422885)
47.20 (+5.2)
Table 9: tero-lest results on speaker larget identification. We can see SIQ appears less challenging than the social game datasets. This is consistent with the domain characteristics: while talk shows involve meaningful host-guest exchanges, the interactions are typically centralized and sequential. In contrast, social games distribute turn-taking among multiple participants, leading more frequent, multi-directional interactions.
Test Datasct
YouTube
Ego4D
SIQ
Accuracy
20.76
18.86
42.00
A.5 Cross Dataset and Domain Generalization
To assess the generalization capability of Onlnne-MMSI--LMM, we conducted both cross-datasct and cross- domain evaluations. Specifically, we trained the Qwen-based model on either YouTube Ego4D and tested it on YouTube, Ego4D, and SIQ, respectively. Currently, YouTube and Ego4D subsets are only publicly available datasets (Lee et al. 2024a) designed for MMST tasks, we curated a new evaluation dataset, SIQ, from the talk show domain, which provides setting distinct from social deduction games.
The Table g reports the accuracy and relative performance gain for speaker target identification under Ouline-MMSI-VLM (Qwen-based). Here, the reported gain measured against the zero-shot baseline. As shown, when fine-tuned on YouTube, the model achieves substantial improvements (over 25%) when transferred to Ego44 and STQ, demonstrating strong generalization across datasets and domains. When fine-tuned on Ego4D, the model's improvement suffers degradation when transferred to YouTube and S1Q, This degradation can be attributed to the limited size of Ego4D (832 samples) compared to YouTube (3,255 samples), which increases the risk of overfitting and weakens generalization. Overall, these results indicate that the model fine-tuned on social deduction games exhibits robust generalization ability.
We provide additional details on the curated SIQ dataset. We selected videos leaturing multiparty social in- teractions from Social-IQ 1.0 (Zadeh al. 2019). Although Social-IQ 1.0 contains 1,115 videos, most are TV shows with only two participants or lack genuine social interaction. After careful screening, we constructed 125 samples for the speaker target identification task, following the procedure of Lee et al. (2024a). The curated subset has a total duration of 53 minutes and contains 953 utterances. The selected clips, as shown in Figure II are primarily drawn from talk shows with multiple participants, representing a domain that differs from social deduction games. The Table shows the zcro-shot results, indicating SIQ appears less challenging than the social game datasets. This is consistent with the domain characteristics: while talk shows involve meaningful host host-guest exchanges, the interactions are typically centralized and sequential. In contrast, so- cial games distribute turn-taking among multiple participants, leading to more frequent, multi-directional interactions. All videos are publicly available at https://www. www.ww.aagl..om... wwt.gle....m//aee httt:/aattsmmaamcs/s/s/saaa daaaaaatt/maatuurnnnnnn httpt/aaaaaceeeeoeceeeoooo /social-iq.
A.6 Weights of Lmmsi and Lrorecast
The weights of Lumsi and Liorecast are set to by default, since tokens for MMSI and forecast are treated equally in loss computation. To study the effect of varying their ratio, we fixed the weight of Limsi at 1 and tuned the weight of Ctorecast to 0.2, 0.5, 1, 1.2, 1.5, and 1.8, respectively. Experiments were conducted on the YouTube subset for speaker target identification using Online-MMSI-VL.. (Qwen). As shown Table I0, weight ratios of 1, 1.2, and 1.5 achieve the best results, while both too low and too high ratios lead to degraded accuracy. These results indicate that our default setting of 1 is near-optimal and robust..
25

---

Published in Transactions on Machine Learning Research MMarch/2020
Table 10: Impact of loss weight ratio on speaker target identification (YouTube subset). These results indicate that our dcfault setting of for two loss weights is near-optimal and robust.
weight for Liorecast
0.2
0.5
1.0
1.2
1.5
1.8
Accuracy (%)
63.5
64.3
64.6
64.6
64.6
63.1
GAME: NOT NERV?
Prev
Next
Sa
4,90
Var
Player
Player
alpha
Selp
Playar
Original
Annoisted
Mode:
video, lastt, video__ast
frane
KPaa Binose, 1: LettLovee Leve, 2iright, ave, 5: tettuuuuueer 6 Brrlght_shoullder Teleft_etbow, Biright_elbow, telett 28: right rightt tarristt File:
Cnw.hightuule www.wttittttttttoooooooo LeLovool (PGRERLLGHI WWW.EEEEEELLLL _aae Frame: 1924
Text:
PCK (vo 180, 0% 19/9), thrm23.37ppc nore-223. 48px
Halluc vab N/A
POK is normalized by maxtm the boow only counts kecyolints valld both prediction and annotztion.
Click annctatee keypainns order (een XPc right). Click marics kmypoint (ve2) U Undea (S) Mark mosing (160)). AD Previouajneal samp
Cusput: ko_ www_verrry_suupppree http://wtppttt/motttttsssssssss
Figure 12: Illustration of web-based keypoint verification tool for sampled vidco frames.
A.7 Societal Impacts and Concerns
Although Online-MMSI has the potential support assistive and collaborative applications, it also intro- duces societal risks. In particular, fine-grained recognition of gaze, gesture, and conversational dynamics could be repurposed for surveillance of social interactions, workplace monitoring, targeted manipulation. Such uses raise privacy and ethical concerns, especially if deployed without consent or oversight. Morcover, errors in speech recognition, tracking, or gaze estimation may disproportionately affect certain groups, ampli- fying bias and inequity. These considerations highlight the importance of responsible deployment, including transparency, privacy-preservee design, and appropriate governance mechanisms.
A.8 Human Validation on Annotations
To facilitate the socially-aware visual prompting, we reuse the AlphaPosc-basod annotations released by Lec et al. (2024a), including identity-tracked bounding boxes and upper-body keypoints, rather than generating new annotations in this work. These annotations are produced by AlphaPose and further curated using human-verified reference frames to align verbal and non-verbal cues. Specifically, AlphaPose outputs may occasionally suffer from missing detections, for example when participant is temporarily out of the camera view. In such cases, we leverage human-verified reference player positions provided by Lee al. (2024a) correct missing player locations and maintain consistent identity tracking.
To further assess annotation quality, conducted additional lightweight human validation on per-joint pose accuracy. Using web-based relabeling tool shown in Fig. 12 randomly sampled 30 video clips and manually reviewed all upper-body keypoints (cyes, nose, shoulders, clbows, and wrists) for every detected player in the last frame. This resulted in 129 player entries and a total of 1,161 keypoints. For Percentage of Correct Keypoints (PCK) at a = 0.05
visible keypoints, we obtain a bounding- boo-booooiiie of 0.912. Keypoints marked as not visible are excluded from the PCK computation, as they lack uniquely verifiable ground-trute truth locations. Overall, this human validation indicates that the automated annotations are generally accurate for observable joints and suitable for socially-aware visual prompting.
26

---

Published in Transactions on Machine Learning Research Mlarch/2020)
Table 11: Ablation study on different pose estimation models evaluated on the YouTube subset. Results are reported accuracy (%). Our method is robust to the specific pose estimation backbone.
Pose Model
STI
PCR
MPP
Avg. Accuracy
AlphaPosc
64.58
68.83
47.20
60.20
YOLO--ose 64.13
68.54
47.61
60.09
A.9 Details on Instruction-Tume Tuning Data Construction
We construct instruction-luning data specifically for the Online-MMSSI setting, where models are required to perform multimodal social reasoning using only historical information. Online-MMSI emphasizes short- horizon social dynamics, which a speaker's referent is typically resolved through recent conversational turns. Each instructiontuuiin instance is constructed using a fixed-length sliding window over the most recent 10 dialogue turns. This window corresponds to average temporal span of approximately 28 seconds and is sufficient to capture the local conversational and social cues required for referent resolution in online settings. Each training instance is formatted as a standard user-assistant pair. As shown in Fig. 13 the user input concatenates the historical multimodal context, including annotated video frames and dialogue history, with the task-specific instruction. The assistant output, as shown in Fig. 14 contains both (i) the task answer (i.e., the resolved referent) and (ii) the multi-party conversation forecasting targets, which include upcoming speaker turns and their corresponding utterances.
User
"In the video, the red, blue, green, ycllow, purple, and orange colors correspond to PlayerO, Player1, Player2, Player3, Playerd, and Player5, respectively. The transcripts are: [Player2]: All right. What was your point, if she was a Werewolf? [Playero]: Well, she would want everybody to kill her if she was Tanner, but she doesn't want anybody kill her. So, T think she's werewolf. [Player2]: Okay. [Player0]: Because she's trying say that you guys the. [Player2]: Okay. [PlayerO]: And she put the Tanner out there. [Player2]: Okay. [Player0]: So, I think she's trying to deflect and calls everybody
for her. [Player]]: Because, [MASK]'s saying he was Robber. Determine which player pronoun refers in the position of [MASK]. Predict the upcoming speakers' turns. Predict the upcoming conversations.
Figure 13: An example of user input prompt for Online- MMST instruction.
Assistant
"Player3. PlayerO, Player1, Player3, Playerl. [Player0]: mean, calls everybody to for you guys. [Player]]: Let's say he robbed you and the Insomniac. You're not disagreeing with him, but you would've looked your card after he robbed you. the question is, were you Robber when... [Player3]: The question is, was lying were you lying? [Playerl]: Were you lying? One of you is liar, or you're n'r both Werewolves."
Figure 14: An example of assistant output for Online- MMSI instruction.
A.10 Ablation Studies on Pose Estimation Models
To evaluate the influence of different pose estimation models, we replace AlphaPose with YOLO-Pose Maji
et al. 2022 and conduct an ablation study. Specifically, we reuse the tracking results and bounding boxes released by Lee et al. (2024)), and estimate upper-body keypoints for each participant using YOLO--Osee. As shown in Table [1] the choice of pose estimator has minimal impact on performance. The average accuracy differs by only 0.11%, indicating that our method is robust the specific pose estimation backbone.
A.11 Ablation Studies on Larger Backbone Models
We further evaluate the impact of scaling backbone model size by conducting experiments with larger -asiunn models, Qwen2.5-VTI-32B and Qwen3-VT the YouTube subset across three social
27

---

Published in Transactions on Machine Learning Research MMarch/2020
Table 12: Ablation study on larger backbone models evaluated on the YouTube subset. Results are reported
accuracy (%). Larger backbone models consistently outperform their smaller counterparts, indicating that increased model capacity benefits online multi-modal social reasoning.
Model
STI
PCR
MPP
Avg. Accuracy
Owen2.5-VLL-7B
64.58
68.83
47.20
60.20
Qwen2.5-VI.-327
66.26
69.41
49.93
61.87
Qwen3- VL-32B
Qwen3-VL-32B
67.63
67.68
49.65
61.65
interaction tasks. As shown in Table 12 larger backbone models consistently outperform their smaller counterparts, indicating that increased model capacity benefits Online-MMSI. Specifically, Swen2.5-VL-322 improves the average accuracy by 1.67% over Qwen2.5-VL-7B, while Qwen3-VLL VL-32B achieves a 1.45% gain.
A.12
Additional Qualitative Comparison
Fig. 15 presents additional qualitative comparisons between our method and representative baselines (i.e., Lee ct al. (2024a) and Vanilla Qwen) across the three social interaction tasks. Overall, our approach more reliably infers referents from ongoing conversation turns, which altribute to the effects of multi-party conversation forecasting and socially-aware visual prompting in capturing complex social dynamics. In the first STT example, when Player4 turns his head to ask Player3, "Are you werewolf?", our method successfully captures the head motion and correctly identifies the speaking target. In the second PCR example, after Player2 states "I was a seer. Oh yeah." and the discussion continues for several subsequent turns, our model correctly infers that Playerl is referring to Player2, whereas the baseline methods fail to maintain this referential consistency. In the third MPP example, Playerl asks multiple participants the same question, "Who are you voting for?"; our approach correctly predicts that the mentioned player is Player4 by jointly leveraging gaze direction and conversational context. We further observe that Lee al. (2024a) incorrectly predicts the speaker as the referent in both the second and third examples. This behavior is likely due to limitations of its language backbone (BERT (Devlin et al. 2019 ), which struggles to disentangle speaker identity from referential targets when referent identification becomes challenging.
28

---

Published in Transactions on Machine Learning Research (March /202)
Speaking Target Identification
PlayerO
Player1
Player2 Player3 Playerd [Player0]: Yes. Yeah. And yeah, you' re the robber.
[Player4]: All right.
[PlayerO]: So do we kill [Player3] cause he hasn't said anything else?
[Player4]: What about [Player2]?
[Player2]: This is like a little suspicious to me that this is all just falling into place.
[Player0]: believe it. Did you wake up the Insomniac?
[Player2]: woke up as the Insomniac.
Beofarknn G01 EhIEE
[Player0]: I believe this hundred percent that don't even want to take time
Lee et al. Prediction: Player0
overthinking it. I'm ready to vote to the left.
Vanilla Qwen Prediction: Player0
[Player4]: Vote
left? There [Player0]: We're all just working together. [Player1]'s I's trying to get [Player3] out.
werewolf?
Qurs (Owen) Prediction: Plaver3
[Player4]: Are werewolf? (To [MASK])
Pronoun Coreference Resolution
Pleyerd
Playert
Player2
Player3
[Player2]: So, here's 's my thing is that hers wild card.
[Player0]: Okay.
[Player3]: agree with this, but you
[Player2]: was seer. Oh yeah.
[Player3]: A hundred percent seer? was-
[Player2]: Did forget look? Yes.
[Player1]: honestly think we're good this game.
[Player0]: Ithink
Lee et al. Prediction: Player1
[Player1]: What makes you trust her?
[Player0]: That looks sketchy.
Vanilla Qwen Prediction: Player0
[Player1]: What makes me trust her is that she came out and was confident
Ours (Qwan) Prediction: Plaver2
that [MASK] (eriginelly one) was herself even after said troublemaker.
Mentioned Player Prediction
PlayerO Player1 Player2 Player3 Player4
[Player3]: Okay, so...
[Player2]: Who are you going to vote for?
[Player3]: Hold on. He saw werewolf
[Player2]: I'll tell (Player3] at the last second.
[Player3]: ... in the center.
[Player2]: That's really like rude.
[Player3]: No, no, no.
[Playert]: Who are you voting for?
Lee et al. Prediction: Player1
X
[Player2]: don't know. need some help.
Vanilla Qwen Prediction: Player3
[Player3]: Yeah, yeah, yeah. We're in trouble.
[Player1]: [MASK] (originally (Player4H) who are you voting for?
Qurs (Qwen) Prediction: Playard
Figure 15: Qualitative comparison illustrating the advantages multi-party conversation forecasting and socially-aware visual prompting for online social interaction understanding tasks.
29


Reviews:
[{"review_id": "55LbDBGQE8", "reviewer": "Reviewer_uLVX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nBased on Multimodal Social Interaction Understanding (MMSI), this paper introduces Online-MMS, which aims to identify social cues (speaking target, coreference, and mentioned player) using only historical information, thereby prohibiting access to future context, defined as an “online” setting. The authors trimmed future frames and transcripts from the existing Werewolf Among Us dataset to build this task. The authors also propose Online-MMSI-VLM, which leverages AlphaPose-generated bounding boxes and keypoints to construct instructions to tune MLLMs. The proposed method is compared against baselines like InternVL, Qwen, and LLaMA, achieving SoTA performance in this specific online setting.\n\n**Strengths**\n\n1.\tThe idea of this paper is intuitive and easy to understand.\n2.\tDeveloping a real-time AI assistant for MMSI or Egocentric (i.e., AR Glasses) is a challenge and critical research direction.\n3.\tIt is interesting to evaluate models in a streaming-like setup, which is more aligned with real-world deployment than static benchmarks.\n\n\n**Weaknesses**\n\n1.\tOverlap with existing egocentric tasks. The core contribution of leveraging historical context for online understanding overlaps significantly with existing research in Egocentric AI Assistants [1, 2, 3]. Tasks such as EgoLife’s EventRecall [1] and Ego4D’s NLQ [2] already utilize historical video context to answer user questions or provide assistance in real-time. Specifically, works like EgoLife [1] have already explored using historical information for instruction-tuning GPT models for long-context video QA, which the authors overlook. This lack of comparison makes me concern the novelty and necessity of defining \"Online-MMSI\" as a distinct new task.\n\n2.\tUnicentric task definitions. The definition of the subtasks in Section 3 is overly simplistic and lacks self-containment. The authors merely list the tasks (i.e., \"speaking target identification,\" \"pronoun coreference resolution,\" and \"mentioned player prediction\") without defining the specific inputs, outputs, or success criteria, effectively forcing the reader to consult external sources [4] to understand the basic premise. As a submission to a journal, the paper should be self-contained and clearly define the problem space within the text.\n\n3.\tLimited dataset contribution. The proposed dataset appears to be merely a subset of an existing dataset with truncated timestamps, which limits its novelty. Furthermore, the new annotations rely on AlphaPose to generate bounding boxes and keypoints automatically. The paper lacks a detailed description of quality control, human verification of these automated annotations, or metrics assessing their accuracy in this specific domain. This raises me concern the data quality and its ultimate value to the community.\n\n4.\tLack of technology contribution. The primary technical contribution is the application of SFT with LoRA on an existing MLLM. As this is a standard setting in current research, I am concerned that the technical contribution may not be sufficient for the TMLR community without further methodological innovation.\n\n5.\tMissing efficiency verification for the “Online” Claims. The paper frames the work as an \"Online\" task aiming for \"immediate feedback,\" implying a requirement for real-time application. However, the authors overlook a critical efficiency analysis. Efficiency is a major challenge for online tasks, yet there is no comparison of latency or memory consumption against API-based MLLMs or other open-source baselines. I recommend including a table comparing the inference speed and resource usage of the proposed method against baselines to verify that \"immediate feedback\" is achievable.\n\n6.\tLack of details on instruction tuning. The paper lacks sufficient detail regarding the construction of the instruction tuning data. It is unclear how the historical context was sampled or formatted for the SFT process. It is a recommendation to proposed sample data in your supplementary.\n\n7.\tInsufficient ablation studies. For example, What is the necessity and impact of the specific modules in Socially-aware Visual Prompting (e.g., performance without bounding boxes vs. without upper body keypoints)? What is the influence of using different pose estimation models other than AlphaPose? What is the performance of larger backbone models (e.g., Qwen3-VL-32B)?\n\nReferences\n\n[1] Grauman K, Westbury A, Byrne E, et al. Ego4d: Around the world in 3,000 hours of egocentric video[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 18995-19012.\n\n[1] Yang J, Liu S, Guo H, et al. Egolife: Towards egocentric life assistant[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 28885-28900.\n\n[2] Damen D, Doughty H, Farinella G M, et al. The epic-kitchens dataset: Collection, challenges and baselines[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(11): 4125-4141.\n\n[4]  Lee S, Lai B, Ryan F, et al. Modeling multimodal social interactions: new challenges and baselines with densely aligned representations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14585-14595.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission required more evidence to support it claims, particularly regarding the \"Online/Real-time\" nature of the task and the quality of the dataset. Specifically,\n\n1. Missing evidence for \"Online\" capabilities. The paper frames the method as an \"Online\" assistant for immediate feedback, but fails to provide any efficiency analysis (e.g., latency, memory consumption) to prove it can operate in a real-time streaming setup (see Weakness 5). Without this, the claim of \"online\" applicability is unsupported.\n\n2. Dataset reliability. The proposed dataset relies heavily on automated annotations (AlphaPose) without any reported quality control or human verification. There is no metric provided to validate the accuracy of these ground truths, which weakens the empirical results (see W.3).\n\n3. Ambiguous Definitions. The subtasks are listed without formal inputs/outputs or success criteria, making the evaluation protocol unclear and difficult to reproduce (see W.2).\n\n4. Insufficient Ablation. Critical design choices (e.g., specific modules in visual prompting) lack ablation studies to justify their necessity (see W.7).", "audience_yn": "Yes", "audience_explanation": "The general direction of the paper is relevant to the community. Specifically,\n\n1. Relevance. Developing real-time/streaming AI assistants and evaluating them in a dynamic setup (as opposed to static benchmarks) is a challenging and practical problem that aligns with real-world deployment needs (see Strengths 2 & 3).\n\n2. Caveat. While the topic is interesting, the audience's interest might be dampened by the significant overlap with existing egocentric tasks like EgoLife and Ego4D. The authors need to better clarify the unique value of \"Online-MMSI\" to distinguish it from these well-known retrospective tasks (see W.1). However, the streaming evaluation perspective itself holds value.", "requested_changes": "Major:\n\n1.\tClarify the novelty and contribution relative to overlapping history-based Egocentric AI tasks (i.e., Ego4D’s NLQ and Egolife’s EventRecall). \n\n2.\tProvide self-contained definitions for the three subtasks in Section 3. \n\n3.\tAdd quantitative efficiency experiments (latency, memory) to verify the \"online\" and \"immediate feedback\" claims against baselines. \n4.\tProvide validation or human evaluation for the automated annotations to ensure dataset quality.\n\n5.\tProvide more details on the instruction tuning data construction, specifically the sampling and formatting of historical context. \n\n6.\tExpand ablation studies. Please refer to W.7.\n\nMinor:\n\n1.\tIt is recommended to provide more qualitative experiments to compare the baseline models.\n\n2.\tIt is recommended to provide common failure cases", "broader_impact_concerns": "None", "additional_comments": "Please refer to the Weakness section.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uqp4GCARJh", "reviewer": "Reviewer_QsZG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the Online-MMSI for multimodal social interaction understanding based on predictive modeling characterizing speaker changes (speaker turns) and utterances (content) as well as modeling accounting for visual cues in a time-causal (online) setting (i.e., only past and current data used to predict future social interaction). The approach uses Llama and Qwen with instruction tuning based on LoRa whereas AlphaPose is used to extract visual annotations. Data and evaluation design is taken from Lee et al 2024a and the contribution of the paper is thus to extend this framework to the online setting. The results show that the proposed approach is superior to the prior work of Lee et al. 2024 in the online setting (which lee et al. 2024a is not trained to handle) and model ablations confirm the benefits of the various introduced design choices including both MMSI and forecasting in training, conversation forecasting length and visual prompting employed.\n\nStrenghts:\n* The paper advances multimodal social interaction modeling to the online setting whereas past work (Lee et al 2024a) were designed for the offline setting.\n* Results appear to improve upon alternative existing modeling strategies.\n\nWeaknesses:\n* It appears the comparison to Lee et al 2024a is for models trained in the off-line setting and not online setting.\n* The methodological innovations are limited mainly based on combining existing tools.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The approach is contrasted Lee et al 2024a, however, it appears this is designed to only be trained in a non-causal setting (not online setting) and a fair comparison would require Lee et al. 2024a be adapted to an online setting (which is described in the introduction as feasible).  It is very unclear if this is the case but if trained in the traditional non-online manner and compared in the online setting this would not be a very fair comparison. It would then be relevant to see how well the proposed method compare in online performance to Lee et al. 2024a also trained in an online setting (as discussed at the end of section 2.1). It is also unclear why the results in the paper are not compared to GPT-4o and Gemini2.5Pro in Table 1 as provided results for in Figure 1. It would be nice to include such comparisons in the table although these methods provide subpar performance.\n\nThat being said the experimental design is sound and valid and the approach provides good results with reasonable systematic ablations evidencing the benefit of various design choices.", "audience_yn": "Yes", "audience_explanation": "Whereas prior work has considered multimodal social interaction modeling from an off-line setting (i.e., Lee et al 2024a) the current paper advances such modeling to the online setting which is a valid and relevant contribution. The methods contribution is somewhat incremental utilizing and combining existing tools such as Llama and Qwen instruction tuned by LoRa and with AlphaPose for visual annotations extraction. Data and evaluation design are largely also taken from Lee et al 2024a but adopted to the online setting. Overall this makes the nature of the paper somewhat incremental.", "requested_changes": "Compare to Gemini2.5Pro and GPT-4o for the results in Table 1 and include Lee et al 2024a both trained in an off-line and on-line setting for comparison as discussed possible end of section 2.1. \n\nIn Figure 1 why is Player2 denoted Player2Player3 in the top left figure?\n-\tThe description of what  “Prior model” in Figure 1 (2) refers to should also be clarified, i.e. Lee et al. 2024a?\n\nHow are the two objectives L_MMSI and L_forecast balanced and should there here be weights tuned to optimally combine the two terms?\n\nPlease clarify why the different step sizes respectively for speaker identification and mentioned player as opposed to pronoun coreference tasks are used and not kept the same step-size for all three learning tasks.\n\nThe paper would benefit from a bit of careful proof-reading:\nThe VLM abbreviation is never introduced (Visual Language Model) – please explain this abbreviation at first occurrence.\nwhich benefit addressing dynamic multimodal > which makes them useful for dynamic multimodal\nFurthure analysis -> Further analysis", "broader_impact_concerns": "While the paper describes positive aspects of social aware AI assistants for social support and collaborative tasks, the paper should also discuss negative societal impacts and concerns such as algorithms for surveillance purposes of social interactions etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MBqmOYnMAZ", "reviewer": "Reviewer_ERFD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new problem **Online Multimodal Social Interaction Understanding (Online-MMSI)**, which requires AI models to interpret social interactions using only historical video and dialogue context, without access to future information. \n\nThe authors propose **Online-MMSI-VLM,** which using Conversation forecasting and Visual Prompting techniques to enrich contextual understanding and focus on relevant visual context. Results show that Online-MMSI-VLM outperforms baselines and even humans in online settings, demonstrating the effectiveness of the proposed components.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The experimental design is extensive, with multiple tasks, two datasets, and extensive ablation studies.\n- The latency of model is reported, strengthening the paper's claims about its suitability for real-time deployment", "audience_yn": "Yes", "audience_explanation": "This work propose a new problem which is valuable for MMSI and real-time AI applications. \n\nThe MMSI community will be interested in this paper. And the proposed question is valuable in practice.", "requested_changes": "- Generalization. The VLM is trained and tested on same datset with the same domain (social deduction games). It would be valid to show the generationzation between datasets and between domain (like online meetings) .\n- Limited Baselines. Only one main baseline is considered in this work.  it would be beneficial to compare against other capable MLLMs or streaming video understanding models.\n- End-to-end latency The authors only report the latency of generation. What is the whol latency including processing speech, do forecating and generating answer for a 11B model? This would offer a more complete picture of the system's feasibility in a true online scenario.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)