EgoBrain: Synergizing Minds and Eyes For Human Action Understanding
Authors: Nie Lin
Yansen Wang
Dongqi Han
Weibang Jiang
Jingyuan Li
Ryosuke Furuta
Yoichi Sato
Dongsheng Li
Venue: ICLR 2026 Poster (2026)
Keywords: Electroencephalography (EEG)
First Person Vision(Egocentric Vision)
Human Action Understanding
Primary area: applications to neuroscience & cognitive science

Abstract: The integration of brain-computer interfaces (BCIs), in particular electroencephalography (EEG), with artificial intelligence (AI) has shown tremendous promise in decoding human cognition and behavior from neural signals. In particular, the rise of multimodal AI models have brought new possibilities that have never been imagined before. Here, we present \data --the world's first large-scale, temporally aligned multimodal dataset that synchronizes first-person (egocentric) vision and EEG of human brain over extended periods of time, establishing a new paradigm for human-centered behavior analysis. This dataset comprises 61 hours of synchronized 32-channel EEG recordings and first-person video from 40 participants engaged in 29 categories of daily activities. We then developed a muiltimodal learning framework to fuse EEG and vision for action understanding, validated across both cross-subject and cross-environment challenges, achieving an action recognition accuracy of 66.70\%. EgoBrain paves the way toward a unified framework for multimodal and egocentric brain–computer interfaces, bridging neural signals and first-person perception. Our dataset and code are publicly available at: https://huggingface.co/datasets/ut-vision/EgoBrain and https://github.com/ut-vision/EgoBrain.

Published as a conference paper at ICLR 2026
EGOBRAIN: SYNERGIZING MInDS AND EYES FOR HUMAN ACTION UNDERSTANDING
Nie Lin Yansen Wang", Dongqi Han?, Weibang Jiang? Jingyuan Li Ryosuke Furuta', Yoichi Sato & Dongsheng Li
'The University of Tokyo, MMicrosoft Research Asia
(nielin, furuta, ysato] ) @iis. iiissu-tooyo. (yansenwang, donggihan, dongsli (ttgssii/mmccroooofii /ewcr/emot.cocoooooooooooooo (v-nielin, v-weibangjiang, v-jingyuanlil @mi crosoft.com.
ABSTRACT
The integration of brain-computer interfaces (BCIs), in partieular electroen- cephalography (EEG), with artificial intelligence (AI) has shown tremendous promise in decoding human cognition and behavior from neural signals. In par- ticular, the rise of multimodal Al models have brought new possibilities that have never been imagined before. Here, we present EgoBrain-the world's first large-scale, temporally aligned multimodal dataset that synchronizes first-person (egocentric) vision and EEG of human brain over extended periods of time, es- tablishing new paradigm for human-centered behavior analysis. This dataset comprises hours of synchronized 32-channel EEG recordings and first-person video from 40 participants engaged in 29 categories of daily activities. We then developed a muiltimodal learning framework to fuse EEG and vision for action understanding, validated across both cross-subject and cross- environnvirn chal- lenges, achieving an action recognition accuracy of 66.70%. EgoBrain paves the way toward a unified framework for multimodal and egocentric brain computr interfaces, bridging neural signals and first-person perception. Our dataset and code are publicly available at: hugging fegggnngaac//// weagffac//ttuue seeeeutvvise a Saggo/ooaaa in and
github/ ut wtttuuubttiiiioo/ooooo
INTRODUCTION
The explosive growth of artificial intelligence has greatly advanced the field of Brain-computer in- terfaces (BCI), with massive research efforts to understand brain functions from neural recordings. Among various neural signals, non-invasive systems such as scalp electroenccehallgraphh (EEG)
more scalable, cost-effective, and safer for large-scale adoption (Willett et al., 2021 [ Anumanchipall
ett al.
2019
Sivasakthivel et al.) 2025 Metzger et al., 2023) Bai et al.] [0033] Li et al. [025 Lan
et al. 2023 thus appealing increasing interest to connect EEG with human perceptions and in-
tentions. Boosted by deep learning techniques. booming breakthroughs have been seen in recent years to decode visual and acoustic stimuli in controlled laboratory settings. For example, recent works achieved accuracies of 15.6% in a 200-way zero-shot task on the EEG-image dataset (Song eet al.) 2023) and 21.9% in a 9-way task on the EEG-video dataset (Liu et al. 2024b). However, the visual stimuli existing studies were merely presented on screens and the informative envi- ronmental background was ignored. Moreover, the active interactions between the subjects and the environment are less explored due to the passive settings in the experiments.
To better capture real-world human perceptions and actions, we introduce egocentric vision complementary modality to EEG. The egocentric vision has emerged as a powerful paradigm for modeling human interactions and perceptual processes in real-world settings, with representative large-scale datasets such as SPICCKITCHENS(Damme et al., (022)), and Ego4D(Grauman et al.) 2022). These datasets primarily capture observable outcomes from human-liks perspective, yield- ing valuable analysis of human behavior such as action recognition (Zhag et al.) (024] 2025). hand
pose estimation(Fan al. 2024 Lin al. 2025) and spatial reasoning(Feng al. 2025)
Work done during an internship Microsoft Research. Co-corresponding authors.

---

Published as a conference paper at ICLR 2026
However, despite the rapid progress in both EEG decoding and egocentric vision, these two research lines remain fundamentally disconnected. Existing egocentric datasets capture what people do but not what they internally perceive, while traditional EEG studies reveal these internal processes but lack the richness and ecological validity of real-world human-environment interaction. As a result, current benchmarks can only reflect either the external visual outcomes the internal cognitive sponses, but never the interplay between the two. This motivates us to introduce EEG into egocentric vision research, with the goal of filling this critical scientific gap. By pairing real-world egocentric video with simultaneously recorded brain activity, we aim to enable deeper understanding of how perception and cognition jointly shape human actions.
Interestingly. EEG and egocentric vision provide mutually reinforcing information. While the first- person-view video offers objective information about scenes and actions, the sensorimotor experi- ences, intentions, and other forms of implicit knowledge remain largely unobservable. The missing pieces can be seamlessly complemented by EEG signals which reveal the latent cognitive signals related to attention, motor planning, decision-making and intention. Given the complementary na- ture of egocentric vision and EEG, three fundamental questions arise. First, can their combination lead to deeper understanding of human behavior? Second, when does this integration outperform unimodal approaches? Third, what technical methodologies can effectively handle the fusion?
To seek the answer and advance EgoBrain, a large multimodal dataset that synchronously captures EEG and egocentric video from 40 participants engaged in natural daily activities. With sophisticated design of 29 actions and diverse environmental conditions in the test sets, EgoBrain offers the first benchmark for multimodal action recognition from synchronized EEG and egocentric video. It lays the foundation for multimodal and egocentric brain-computer interfaces that integrate neural activity with first-person vision.
multimodal research, we start from introducing
Similar to other multimodal tasks with synchronized timeline, it's crucial to handle the shared tem- poral structure carefully and fuse information from modalities for downstream prediction. Upon our EgoBrain dataset, we present an adaptive Brain- Time Interval Machine (Brain-TIM) -TMM model, inspired from (Chalk et al. 2024) to integrate synchronized visual and EEG signals and capture rich multi- modal information for action understanding. Each modality is processed through modality-spechee embedding layers and merged to the aggregated global context, while the shared temporal structure explicitly modeled using the Time Interval MLP (TIM) module. We then conducted experiments with our Brain-TIM to evaluate both the standalone effectiveness of individual modalities and their synergy, and the highlighted results confirmed that the fusion of EEG and vision consistently out- performs unimodal approaches across multiple experiments. Further visualization provide deeper insight into the complementary roles of egocentric vision and EEG signals.
In summary, the main contribution of this paper is threefold:
1) We introduce EgoBrain, the first large synchronized EEG dataset designed for egocentric vision research. Featuring data from 40 participants engaged in real-world activities such as tool use and daily tasks (in total 61 hours), this dataset sets a benchmark for cross-modal action understanding and advances the application of BCI technologies in real-life settings.
2) To lay the groundwork, we provide standardized preprocessing pipelines for vision-brain syn- chronization data, along with benchmark evaluations and our proposed Brain-TIM model. These resources ensure experimental reproducibility and offer a unified comparative benchmark for future research based on EgoBrain.
3) We conduct ablation studies to assess the individual and combined contributions of different modalities. Our findings offer valuable insights into designing cross-modal learning frameworks for egocentric vision and brain signal integration.
RELATED WORK
EEG & Vision Integration: In recent years, combining electroenceepallgraahy (EEG) with visual data has emerged central theme in brain-computer interface (BCI) research, elucidating cog-
nitive processes and motor intentions intentions(Mushhag et al. 2024; Gutttmann-Flury et al. 2025 Bertoni et al. 2025; Dreyer et al.] 2023) Kaya al. 2018)). EEG's 's high temporal resolution and portabil- ity enable real-time monitoring of brain states, yet most work examines resting-state responses to static visual stimuli, neglecting neural dynamics during natural movement Yang et al. 2025 Liu
2

---

Published as a conference paper at ICLR 2026
et al. 2025: Ma et al., 2022 2020 Liu et al. 2024a). A few studies have recorded EEG during active locomotion for example, assessing cognitive load while walking in lower-limb exoskele- ton(Ortiz al.) 2023) and virtual-reality tasks like supemumerary thumb control via motor im- agery(AAluraa) et al., 2024). However, these efforts target prosthetic control and lack a systematic exploration of real-world, first-person multimodal interactions in unconstrained movement. The recently introduced ToMCAT et al. (2023) dataset includes diverse tasks such as image rat- ing, virtual table tennis, and Minecraft gameplay. However, unlike EgoBrain, primarily focuses on screen- based task paradigms and does not capture synchronized egocentric, real-world interactions, leaving the integration of EEG with natural first-person perception largely unexplored.
Egocentric Vision Datasets: Recent egocentric video corpora have advanced human
human-object object in-
teraction modeling through varied contexts and annotations Damen et al.) [2022] Grauman et al.
2022
Wang et al., 2023: Darkhalil et al. 2022.
Kwon et al., 2021: Liu et al., 2022; Ragusa et al.,
2021
Sener et al.) 2022 Ohkawa et al. 2023
Zhang et al. 2022 Grauman et al.
2024] Huang
ct al. 2024). SPIC-KITCHENS(Dumme
et al.
2022) offers detailed kitchen-activity labels, while
Egr4D(Grauman et al. 2022) provides the largest in-the-wild egocentric set for 3D perception and
social analysis. HoloAssist Wang al., 2023) enables multi-user task completion, and Assembly-
series(Sener et al. 2022 Ohkawa et al. 2023) and H2O(Kwon et al.) (021) cover procedural and
et al. (024) and Ego- EGruDaGraumma et al., (024) deliver asynchronous and dual-perspective recordings of skilled ac- tivities. While several egocentric datasets provide multimodal annotations (e.g., audio. IMU, gaze, multi-view footage), none include human-centered internal neural signals such as EEG. As a re- sult, existing resources cannot capture the coupling between brain activity and first-person visual experience, which is the central focus of our work.
two-hand manipulations. More recent datasets like LgooExoLeeam((unng
Overall, existing research overlooks the synchronization of egocentric visual data and brain activity during dynamic human interactions in daily life.
EGOBRAIN DATASET
We first introduce the data acquisition setup and collection pipeline, followed by the semantic action design and statistical overview, with the corresponding supplementary details in Appendix B
Environment & Acquisition System: Fig Da illustrates our data capture environment. The setup incorporates lighting and a modular workstation containing objects (books, food, etc.) for controlled interactions. The right panel of Fig. M. illustrates the configuration of our portable recording appa- ratus. The setup includes helmet-mounted GoPro HERO12 camera (1080P/30Hz) for capturing high-quality egocentrie video and a 32-channel wireless EEG headset (Emotiv FLEX 2 Gel System, 256Hz sampling rate) compliant with the international 10-20 electrode placement standard.
Throughout the session, the subject remains seated to reduce excessive lower-limb movement that may otherwise introduce artifacts into the EEG signals, and the GoPro camera is carefully aligned to the participant's visual horizon to ensure a natural first-person perspective. The subject is asked to conduct some everyday interaction with the objects illustrated in Fig, [... Meanwhile, the data acqui- sition system captures two key modalities: EEG signals, with example shown in Fig. ПТp. Recordings were collected from 40 volunteer participants (see Appendix T for subject details).
egocentric video recordings and 32-channel
Acquisition Pipeline: Fig. Td presents a detailed visualization of our standardized action execu- tion pipeline. A session consists of a predefined yet randomly shuffled sequence of 29 actions, with "Consume" "oonsume--relted -related actions repeated for three times (narrated in the next section). At the begin- ning of each action, a large display screen presents a task prompt (e-g., "Play Cube"). The prompt instructs the subject to identify the relevant object placed on the table, initiate the corresponding hand-object interaction. The completion of a task is marked by the subject successfully performing the interaction and manually confirming it via mouse click. This human-initiated confirmation ensures the intentional execution and completeness of each action, and naturally results in varying action durations across different tasks. After the subject explicitly confirms completion by clicking the "Confirm" " button, the system proceeds to the next predefined action. This process continues until the subject has completed the entire set of preprogrammed tasks.
3

---

Published as a conference paper at ICLR 2026
Video
GoPro Camera
Emotiv Headsets
Collection Environment
EEG Data
C
Action
Video
Read Book
Action
Screen Prompt
Subject Execute
Click
Action
EgoBrain Dataset
Annotation
Synchromize
EgoBrain Subject
Time Line
The EgoBrain dataset and experimental setup. (Left) Acoustic isolation chamber with adjustable lighting and modular workstation containing standardized interaction objects. (Right) Portable apparatus configuration showing helmet-mounted GoPro camera and Emotive FLEX 2 Gel EEG headset. b High-fidelity egocentric video recording hand-object interactions and 32-channel EEG signals. e Subject performing ("Read book") action following on-screen textual prompts. From command display ("Play Cube") to object interaction and completion confirmation.
Figure 1:
Table 1: Overview of the four high-level activity classes in the EgoBrain dataset.
High-Level
Description Examples
(1) Work
Operating office software such as Word, PowerPoint, Excel, and Paint, etc. Engaging in screen-based games, object-based puzzles, and mobile games, etc. Performing writing tasks, reading various materials, and drawing activities, etc. Eating different types of snacks and drinking multiple beverages, etc.
(2) Play
(3) Learn
(4) Consume
The EgoBrain dataset covers a broad spectrum of daily activities, consisting of 29 action classes organized under 10 verbs. We illustrate the design of these semantics in Tab. [ These four top-level categories offer a coarse yet meaningful structure over the action space, en- suring clear distinctions in cognitive demand, motor behavior, and real-world context. Specifically, "Work" includes productivity-orient ical entertainment activities, "Learn" captures reading and writing behaviors commonly observed in academic environments, and "Consume" reflects everyday eating and drinking actions. To more faithfully capture the diverse cognitive demands and motor behaviors inherent in different forms of "Play", we further subdivide the "Play" category into three subtypes (see Appendix B.2 for details).
Category Design:
computer operations, "Play" contains both digital and phys-
Evaluation Settings: To systematically assess model generalization, we define two evaluation set- tings with increasing difficulty: Cross-subject-one and Cross-subject & Cross-scene. The Cross- subject-only evaluates subject-level generalization under a shared physical environment and object configuration, whereas the Cross-subject & Cross- scene further introduces previously unseen vironment with different object and background contexts, posing a more challenging cross-domain generalization scenario. Detailed statistics and split configurations provided in Appendix B.3
4

---

Published as a conference paper at ICLR 2026
Duration of Actions in EgoBrain Dataset
1) Work
2) Play
3) Learn
Consume
Hours
Operate Word
Operato PowerPaint
Operats Excel
Operats Paint
Watch Computer
Wateli Snartphone
Playfi Spider Soltaire
Playf) 30 Pintall
Playll)) Toya
Play(t) Cards
Play(n) Putzle
Playtti) Cute
Play Fruit stnjo
Play(ih)) Subway Surfera
Playill)) Angry Birds
Play(t)) Chose
Write Copybook
Wiits Notoo
Reau Textbook
Road Conts
Read Reaarch
Draw Traco
Draw Picturee
Eat Snack
Eat Spicy shine
Eat Sugars
Drinni Buter Ground Juice
Drinkl Water
Drink Cala
Figure 2: The EgoBrain statistics. The total duration per category is presented, highlighting the longest duration (Play puzzle: 4.29 hours) and the shortest duration (Drink Bitter Juice: 0.49 hours)
Dataset Statisties: These activities span a wide range of temporal scales, with individual task durations ranging from 1,753 seconds (approximately 0.49 hours) 15,441 seconds (approximately 4.29 hours), reflecting diversity in task complexity. We visualize the cumulative time per activity (in hours) in the Fig. 2 A more comprehensive breakdown, including subject-level distributions and category- temporal analysis, is provided in Appendix B.4
4 METHODS
After constructing the EgoBrain dataset, we build an effective framework, namely Brain-TIM, to model these multimodal temporal inputs for action understanding, with additional implementation details provided in Appendix
4.1 TASK DEFINITION
We consider a time- sime-synchronized pair of raw data: the egocentric video stream and the EEG signal sequence, both sharing common timeline T [0, T). The video stream represented Viow (ve E x Tt=0 sampled recorded at yo Hz, where C is the number of channels. The target of action recognition task can be formulated finding the best mapping from input to the action and verb categories
frame rate f and the EEG signal as Brow {b. E RO) T
fV V, B) {1...., N₂}² where N₂ equals to 10 for verb classification or 29 for action categories, and Q is number consecutive queries which evenly divides the whole time interval [0, T), i.e, the i-th query corresponds to the action within time [( /TT/Q.iT/Q)
4.2 OVERVIEW OF BRAIN-TIM
An overview of Brain-TIM is presented in Fig. We first extract feature representations for each modality using pre-trained backbone networks Tong et al. 2022 Jiang et al. 2024) into ø" and &². These features then projected into shared embedding space via layers: g" and g². The embeddings from different modalities are concatenated to form a unified input sequence for the Transformer module the right side of the Fig. Eventually, the Transformer
embedding
5

---

Published as a conference paper at ICLR 2026
Sequence
Video Frames
Features Block
Tokens Block
Time
361
Visual Encoder (²(-)
n
WIL
Embedding Layer g'(()
Transforer Encoder
Multi-Head Attention
X
Add & Norm
Embedding Layer (-)-)
TTM
Feed Forward
Brain Encoder (c)-)
Time
A
241
3At
Add & Norm
EEG Signals
Head A
C
Head A
Figure 3: The overall architecture of Brain-TIM. The model processes synchronized visual and EEG signals using modality-speciftt encoders, followed by embedding layers to obtain token se- quences. The shared temporal axis is concurrently encoded by the TIM module. A modality-aware CLS token is appended to the sequence to capture global semantics. The resulting tokens are fed into a Transformer encoder for downstream action classification.
encoder models the temporal dependencies and cross-modal interactions within the sequence and linear classifier maps the encoded features to the final action category predictions.
4.3 FEATURE EXTRACTION
Before performing action recognition, we first extract features from all modalities using pre- trained encoders. EEG signals are processed through LaBraMgJiang et al. 2024), a model pre- trained on 2,500 hours of masked EEG data that generates 2000-dimensional features per channel. Video frames are encoded using VideoMAB(Tong et al.) 2022), pre-trained on EPIC-KITCHENS. 100(Damen et al. (022), which outputs d024-dime features per segment. During feature extraction, we adopt the parameter settings for EEG features as suggested (Jang et al. 2024), while the parameters for video features follow those introduced in (Chalk et al. 2024).
Sliding Window Mechanism: To extract aligned segments, Brain-TIM apply a sliding window mechanism with duration of At and step size öt. Each window contains N fr At video frames and N6 f AL EEG samples. The raw data are divided into segments aligned with T-At + 1, represented as V = {v; E NN x L st
N =
Ji=1 and B = (b, E RN*xO LN Jir
Below, we detail how to extract feature representations from V and B.
The sliding-window mechanism further mitigates the impact of sub-second temporal jitter in a struc- tural manner. In this design, the window stride öt is always smaller than the potential misalignment, and adjacent windows exhibit substantial temporal overlap. As a result, each moment in the se- quence is covered by multiple windows, creating natural temporal redundancy. Moreover, both modalities are segmented using identical window indices, ensuring that their relative temporal struc- ture remains consistent even under slight timestamp shifts.
Visual Features: Within each window in V. we uniformly down-sample K frames from their cor- responding segment, denoted as (uf...., ). The superscript refers to the i-th window, and 4; is the starting timestamp of the i-th window, corresponding to the time interval [4i, 4 + At). The timestamps for each sampled frame are denoted as [rí:..., T J. where of t/ + 2k-1 At, with 2K
E (1,..,, Kj. This formula ensures that the sampled frames are evenly distributed within the time window and are centered within the window. Each frame is resized to 224 x 224 and normalized using ImageNet(Deng et al. 2009) statistics. These frames are then passed through a frozen, pre- trained visual encoder ø" produce window-level feature vector e" Rdd where d' represents the feature dimension. Combining the sliding stride öt, the full video is encoded into a sequence of window-level feature vectors -d .e.,..., e% where each ef E Rd"

---

Published as a conference paper at ICLR 2026
Brain Features: We adopt the same mechanism extract neural features from B. For the i-th time window, the EEG signal is denoted as b, For the EEG signal within the window, we first apply a band-pass filter with a range of 0.5-50Hz, followed by downsampling to for Hz. The input b; is fed into a frozen pre-trained encoder <d to obtain its feature representation e ) E RCxxxxx"" where d is the EEG feature dimension. The features are aggregated via channel-wise average pooling as e = + (b)..... by) E |Rtxa", and temporal pooling is applied when necessary. This produces an aligned sequence of window-level features, denoted as ed =
,.y), where each of Rdd and the number of windows N kept consistent with the visual modality.
Token Preparation: After the features from both modalities are obtained, the learnable embedding layers g g²(+) and g²) are applied E" and & respectively, map modality-speciife features into a shared D-dimensional space, As a result, we obtain the visual feature tokens &" = {ēe E RP J.N and the EEG feature tokens Eh = {e E RDIN without dimension misalignment for further fusion.
To enable cross-modal interaction and support classification for the Q queries, 2Q Iearnable clas- sification tokens (els tokens) {cī RD) feature tokens are introduced for vision and EEG modality, respectively.
==1 and {c) R012 of the same dimension with the
4.4 SEQUENCE CONCATENATION
Temporal & Modality-Aware Token: We enrich all feature tokens o¹ o and CLS tokens cjj, of with lightweight time-aware embeddings generated by the Time-Interval MLP (TIM), which computes interval-based embeddings such as e and e) from their corresponding temporal ranges.
To distinguish tokens of different modality, we introduced the modality-specifie embedding, repre- sented by two learnable vectors, m" R2D and m2 E RR2D, store shared vision and EEG modality information, respectively. The modality-specific embeddings are directly added to the tokens of the corresponding modality, with detailed formulations provided in Appendix C.I and Appendix C.2
Sequence Concatenation:
The input sequence to the transformer encoder is obtained as follows: (éllle/ + 1nmm" m²) {cjlle) + m 1==1 -= (cyllef + m 1==1);
X = Concat {e'lle/ + "
visuna feature block
bruin feuture block
where each element is constructed by concatenating the original token with its temporal embedding,
added to the modality-specific embedding. This final input sequence X E R(2N+2Q) x2D is formed by orderly concatenating all processed feature representations and CLS tokens. After constructing the final input sequence X, the tokens are processed by a standard Transformer encoder (as detailed in Appendix C.3), followed by linear classification head (described in Appendix C.4).
This design of Brain-TIM offers three key advantages: 1) it ensures cross-modal time-aware align- ment through shared temporal encodings; 2) preserves modality-specinn characteristics by utiliz- ing independent modality embeddings; and 3) facilitates cross-modal interaction and query-specific classification by implementing symmetric handling of CLS tokens.
5 EXPERIMENTS
We rephrase the research questions proposed in the introduction (Sec.
D here:
RQ1:
Does combination of egocentric video and EEG enable more comprehensive understand- ing of human behavior?
RQ2:
Is proposed method effective for this multimodal action recognition task?
RQ3:
When does this integration outperform unimodal approaches?
We designed comprehensive experiments answer these research questions in this section.
5.1 ACTION CLASSIFICATION RESULTS ON EGOBRAIN
We evaluate Brain-TIM on test sets of the EgoBrain dataset (Tab. 2 to answer RQ1. Note that all experimental results presented in the tables Mean + STD across five different random seeds.
7

---

Published as a conference paper at ICLR 2026
Table Action recognition results on the EgoBrain set. We systematically evaluate unimodal (Brain only, Visual only) and multimodal (Visual+Brain models under two protocols: cross-subject only and cross-subject & cross-scene. The table reports the parameter scale (Params) of each model and the mean + standard deviation across five random seeds to ensure statistical reliability. The primary evaluation metric Top-1 accuracy (%), with the best results highlighted bold.
Protocol
Modality
Encoder
Params
Verb Acc.
Action Acc.% %
LaBraM
Brain only
5.8M
21.53 0.99
8.44 + 2.25
Cross-subject only
Jiang et.
2024
VIdcOMAE
Visual only
305.0M
88.95 0.80
78.44 + 0.71
Tong ct all
2022
VideoMAB
LaBraM
Visual Brain
310.8M
90.11 1.10
80.16 + 1.67
Tong 61 al
2022
Jang et
2024
LaBraM
Brain only
5.8M
19.41 = 1.57
9.36 + 0.52
Cross-subject
Jiang et al.
2021
Cross-scene
VIdcoMAF:
Visual only
305.0M
81.67
1.89
63.40 + 0.95
Tong et al.
2022
VideoMAE
LaBraM
Visual Brain
310.8M
83.43
0.41
66.70 + 0.83
Tong
Bang 2024
Unimodal Comparison: As well-studied computer vision problem, the visual modality demon- strates significantly strong performance, achieving 88.95% Top-1 accuracy for verb classification and 78.44 % for action classification in the cross-subject setting. Thanks to its superior spatial res- olution and contextual richness, egocentrie visual input provides fine-gra fine-grained cues that are critical for distinguishing actions and achieved the performances of 81.67% and 63.40% for verb and action classification even under the challenging cross-subject and cross-scene settings.
As for the EEG modality, the model achieves relatively low yet significantly better performance than chance level for both settings. While EEG data contains certain cognitive information, its relatively low sampling rate and limited feature dimensionality restrict its effectiveness in complex real-world scenarios applied individually. Due to space constraints, we provide the details of the mandom-baseligg results in the Appendix for further reference.
Multmodal Comparison: As shown in Tab. 2 fusing EEG with visual inputs boosts accuracies across both evaluation protocols. Under the cross-subject only setting. the vision-only baseline achieves 78.44% Top-1 accuracy for action classification, while Brain-TIM reaches 80. 16%, giv- ing 1.72% improvement. Specifically, under the most difficult cross-scene setting, the vision- only baseline achieves 63.40% Top-1 accuracy for action classification, while Brain-TIM with both modalities reaches 56.70% yielding a 3.30% absolute improvement on the 29-class task. This per- formance gain answers RQ1 and further highlights the semantic complementarity between the two modalities: while the visual stream captures external manifestations of action, EEG encodes neural signatures of motor intention and implicit knowledge that are not observable from eyes alone.
Importantly, the performance gains of EEG are not attributable to differences in model capacity, but rather to its unique cognitive value. Our EEG encoder is extremely lightweight, with only 5.8M parameters (approximately 1/52 of the visual backbone), yet it still delivers statistically significant improvements. This demonstrates that EEG provides indispensable complementary information in cases of visual ambiguity or occlusion, thereby rendering visual understanding more complete.
5.2 ABLATION STUDY
To answer RQ2 and see whether all the proposed techniques are positively contributing to the de- coding task, we removed some components from Brain-TIM and conducted ablation study.
Table 3 presents the performance under three modality settings: Brain Only, Visual Only, and Vi- sual & Brain. For each setting, we evaluate different combinations of three key components: the embedding layer g (v.b), the time interval MLP T(v.b), and the modality embedding (".)))
Overall, each component improves performance in both the pure EEG and multimodal settings. The embedding layer strengthens feature representations. The interintrvval MLP helps encode temporal information. The modality embedding preserves modality-specific cues.
8

---

Published as a conference paper at ICLR 2026
Table Ablation results of Brain-TIM. These results provide a detailed view of key module con- tributes to performance under the Brain-Only, Visual-Only, and combined Visual & Brain settings.
Brain Only
Visual Only
Visual & Brain
Finhedinng Layer gfsA)
X
X
/
X
V
K
/
X
X
/
Time Interval MLP f(sd)
X
X
X
/
X
Modality Eihheriii (m-)))
Action Acc. % @ Top-1 Moan
7.44
7,54
6,15
9.36
64.94
64. 64.67
64.01
63.40
65,71
65.81
66 36.39
66,18
66.70
Action Acc.9t Top-1
0.39
0.06
0.52
3.64
1.75
6.34
0.95
0.09
2.22
0.83
Confusion Matrix (Visual)
b
Confusion Matrix (Visual Brain)
1.0
0.E7
0:67
0:50
0.00 0.22
6:90
0:24
0.60
3.00
0.00
0.03
0.00
WA
0:09
0.50
0.34
0.00
0.41
0.00
9:00
2.96
0.00
WA
5.28
0.54
9:13
0.01
2:19
0.00
9.85
0.50
P1-
0:26
0.03
0.16
0.00
8.30
0.00
0.23
0.00
3.24
0.00
0.10
0.30
3.24
0.00
3.01
0.00
0.03
0.00
0.15
5.00
Ground truth
P2
0.00
0.00
0.00
0.79
0.92
0.00
9.00
0,00
0.18
0.00
P2
0.80
0.00
9.00
0.82
3.00
0,.00
0.00
0.00
0.88
0
P3
0.00
5.85
0.00
0.00
P3
0.00
0.37
8.30
0.00
2.03
0.02
0.00
0.02
0.00
WR
WR
0.4
8:50
0.00
9:04
8:38
0.00
0.00
0.00
0.60
0.2
ET
0.00
0.50
0.00
9.20
0.00
0.00
9.00 0.62 9.02
0.00
0.23
0.00
DK
0.00
0.03
6.60
00
6.00
0.00
0.00
0.00
3.22
0.14
DK
6.00
0.00
6:00 0.00 3.00 0.00
0.03
0.00
0.08
6.41
0.0
6P
Predicted
Predicted
Figure 4: Confusion matrix for verb classification, Visual-only. b Visual Brain fusion. Owing to space constraints, verb names are abbreviated; the full abbreviation table is provided in Table
Interestingly, these additional designs reduce performance in the vision-only only setting. We suspect that the visual modality alone already sufficient for accurate predictions, and the additional parameters introduce unnecessary complexity that interferes with the training process.
In an additional ablation, we compare Brain-TIM with spatial- fusion variant and confirm that temporal fusion achieves better performance complex action recognition; full details and results are provided the Appendix
5.3 DETAILED ANALYSIS
To answer RQ3, we further conduct more detailed analysis of specific categories and representative cases to clarify when the multimodal framework outperforms the unimodal approaches.
Confusion Matrix of Classification: We present in Fig. 4 the confusion matrices comparing uni- modal and multimodal models for verb classification. Comparing Fig. 4 (Visual-only) and Fig. 4p (Visual + Brain) reveals that EEG integration does not uniformly improve all categories. No- table improvements are seen in "Play(I)" (0.46 + 0.64), suggesting EEG complements cognitively demanding actions. The "Drink" category benefits from EEG under visual occlusion (0.87 0.94).
However, "Write" accuracy decreases (0.83 + 0.77). likely due to kinematic redundancy. where EEG introduces noise in cases of clear visual motion patterns. These results indicate EEG's com- pensatory effect is task-dependent, offering marginal gains when visual cues are strong. Due to space limitations, the complete confusion matrices for all 29 action classes are provided in the Appendix
As shown in Fig 5h above, when subject is drawing in notebook (e.g., Santa Claus), the visual model misclassifies the action as "Writing" due to the high visual similarity between the two tasks. However, after incorporating EEG data, the model correctly clas- sifies the action as "Drawing". This suggests that EEG signals may capture neural patterns related to task intent and offer additional discriminative cues. It indicates that EEG reflects distinct neural activations associated with visuospatial motor planning, as opposed to language-relas tasks a distinction that has been well documented in prior neuroscience studies Taag et al. 2024).
Benefits of Integrating EEG:

---

Published as a conference paper at ICLR 2026
Ground Truth: pictures
Cross-aubject Only
Visual:
X
Write notes
Visual EEG: pictures
Ground Truth: drink bitter ground juice
Cross-subject scene
Visual: drink bitter ground juice
Visual EEG:
Ground Truth: operate power point
Cross-subject Only
Visual: operate power point
Visual EEG: X
operate paint
Figure 5: Success and Failure Cases for Unimodal (Visual) and Multimodal (Visual EEG) Models. (a) Multimodal model correctly recognizes actions that the visual-only model misses, aided by EEG. (b) EEG causes misclassification. possibly due to overlapping cognitive strategies.
Another example Fig 5h involves the action of "Drink- bitter ground juice.. Due occlusion, the subject's 's hand and the cup are not visible, and the visual model misclassifies the action "Snack" based on nearby contextual visual cues (e.g., a bag chips). With EEG integration, model not only correctly identifies the verb "Drink" but also the object "bitter melon juice". This improvement likely stems from EEG's s ability to reflect orofacial motor patterns and anticipatory neural activity re- lated to swallowing, which differ from those associated with chewing(Saite al. (022)). The results underscore EEG's s value disambiguating semantically similar actions when vision limited.
Limitations of Integrating EEG: Despite these advantages, EEG does not always lead to improved recognition. Fig presents failure cases of the multimodal model. The subject actually operating PowerPoint, but the model incorrectly identifies the task "Draw Pictures". Possible reason is that the subject is creating multiple rectangles, evoking visuomotor activity patterns that resemble those during freehand drawing. Prior studies have shown that such overlapping cognitive strategies lead to similar EEG Signatures(Dvoral et al. 2018), making semantic discrimination harder.
CONCLUSION AND DISCUSSION
We draw on the metaphor that "the eyes are the windows to the mind" to argue that egocentric video can illuminate neural states that EEG alone cannot fully capture. Despite its promise, no dataset or systematic study has yet explored EEG-vision synergy in real-world tasks. To address this gap, we construct EgoBrain, the first action understanding dataset that simultaneously captures first-person video and EEG signals, aiming to advance research on vision-braii signal integration. We further develop Brain-TIM as the first multimodal research baseline on EgoBrain. Experimental results show that combining EEG and visual modalities significantly outperforms single-modale approaches, highlighting the potential of multimodal modeling in complex cognitive scenarios.
Our work lays the foundation for applying multimodal nitive tasks by introducing a new visuo-neume neural dataset and an efficient benchmark model. We believe that these contributions will open up possibilities for brain-vision multimodal learning, and anticipate future work to actively explore the underlying interaction mechanisms between visual and neural modalities, thereby further inspiring the discovery of new research tasks and directions.
interfaces to high-level cog-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Number JP24K02956 and JST SPRING, Grant Number JPMISP2108. This work was also supported in part by Microsoft Research.
REFERENCES
H. Alsuradi, J. Hong, A. Sarmadi, R. Volcic, H. Salam, S. F. Atashzar, F. Khorrami, and M. Eid. Neural signatures motor imagery for a supernumerary thumb in vr: an eeg analysis. Scientific Reports, 14:21558. 2024. doi: 10.1038/4415988-244-23333
G. K. Anumanchipalli, J. Chartier, and E. Chang. Speech synthesis from neural decoding of spoken sentences. Nature, 568:493 498, 2019. doi: 10. 10.1038/s41586- 019-1119-1 1119-1.
Y. Bai, X. Wang. Y. p. Cao, Y. Ge, C. Yuan, and Y. Shan. Dreamdiffusion: Generating high-quality images from brain eeg signals. arXiv preprint arXiv:2306. 16934, 2023.
T. Bertoni, J. P. Noel, M. Bockbrader, C. Foglia, S. Colachis, B. Orset, N. Evans, B. Herbelin, A. Rezai, S. Panzeri, C. Becchio, Blanke, and A. Serino. Pre-movement sensorimotor oscil- lations shape the sense of agency by gating cortical connectivity. Nature Communications, 16: 3594, 2025. doi: 10.1038/41447--25--- 46700225-8---225--86------9
J. Chalk, J. Huh, E. Kazakos, A. Zisserman, and D. Damen. TIM: A Time Interval Machine for Audio-Visuu Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18153-18163 2024.
D. Damen, H. Doughty, G. M. Farinella, A. Furnari, Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Rescaling egocentric vision: Collection, pipeline and chal- lenges for spic-kitchens- 100. International Journal of Computer Vision (IJCV), 130(1):33-55, 2022.
A. Darkhalil, D. Shan, B. Zhu, J. Ma, A. Kar, R. E. L. Higgins, S. Fidler, D. Fouhey, and D. Damen. EPIC- EPICCKITCHENS VISOR benchmark: Video segmentations and object relations. Proceedings of the Neural Information Processing Systems (NeurIPS) Track Datasets and Benchmarks, 2022.
J. Deng, W. Dong, R. Socher, L.JJ. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
P. Dreyer, A. Roc, L. Pillette, S. Rimbert, and F. Lotte. A large eeg database with users' profile information for motor imagery brain-computer interface research. Scientific Data, 10:580, 2023. doi: 10. 10.10388/41597-023-044555
D. Dvorak, A. Shang. S. Abdel-Baki, W. Suzuki, and A. A. Fenton. Cognitive behavior classification from scalp eeg signals. IEEE Transactions Neural Systems and Rehabilitation Engineering, 26(4):729-73) 739, 2018.
Z. Fan, T. Ohkawa, L. Yang, N. Lin, Z. Zhou, S. Zhou, J. Liang, Z. Gao, X. Zhang, X. Zhang, F. Li, Zheng, F. Lu, K. A. Zeid, B. Leibe, On. Baek, A. Prakash, S. Gupta, K. He, Y. Sato, O. Hilliges, H. J. Chang, and A. Yao. Benchmarks and challenges in pose estimation for egocentric hand interactions with objects. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 428 448, 2024.
Z. Feng, Z. Kang, Q Wang, Z. Du, J. Yan, S. Shi, C. Yuan, H. Liang, Deng, Q. Li, et al. Seeing across views: Benchmarking spatial reasoning of vision-language models in robotic scenes. arXiv preprint ar arXiv:2510.1940 2025.
K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu. M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M.g Xu, Zhongcong Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q Fu, C. Fuegen, A. Gebreselasie,
11

---

Published as a conference paper at ICLR 2026
C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini. C. Li, Y. Li, Z. Li, K. Mangalam. R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M.
Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. Soo Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, Lo Torresani, M.i Yan, and J. Malik. Ego4D: Around the world in 3, 000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18973- 18990, 2022.
K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, E. Byrne Z. Chavis, J. Chen, E. Cheng, F.-. Chu, S. Crane, A. Dasgupta, Dong, M. Escobar, C. Forigua, A. Gebreselasie, S. Haresh, J. Huang, M. M. Islam, S. D. Jain, R. Khirodkar, D. Kukreja, K. J. Liang, J.-W. Liu, S. Majumder, Y. Mao, M. Martin, E. Mavroudi, Su, Z. Xue, E. Zhang, J. Zhang, A. Castillo, C. Chen, X. Fu, R. Furuta, C. González, P. Gupta, J. Hu, Y. Huang, Y. Huang, W. Khoo, and et al. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 19383-19400, 2024.
T. Nagarajan, F. Ragusa, S. K. Ramakrishnan, L. Seminara, A. Somayazulu, Y. Song.
E. Guttmann -lury, X. Sheng, and X. Zhu. Dataset combining eeg, eye-tracking, and high-speed video for ocular activity analysis across bei paradigms. Scientific Data, 12:587, 2025, doi: 10. 04861-9.
Y. Huang, G. Chen, Xu, M. Zhang, Yang, B. Pei, H. Zhang, D. Lu, Y. Wang, L. Wang, and Y. Qiao. Egocxolearn: A dataset for bridging asynchronous ego- and exo-centric view of proce- dural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22072-22083 2024.
W. Jiang, L. Zhao, and B. Lu. Large brain model for learning generic representations with tremen- dous EEG data in BCI. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2024.
M. Kaya, M. Binli, E. Ozbay, H. Yanar, and Y. Mishchenko. large electroencephalograhicc motor imagery dataset for electreencephalographhee brain computer interfaces. Scientific Data, 5:180211, 2018. doi: 10. 1018///ata.220188885 2018..11...
T. Kwon, B. Tekin, J. Stühmer, F. Bogo, and M. Pollefeys. H2O: two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10118-10128, 2021.
Y.-T. Lan, K. Ren, Y. Wang, W.-L. Zheng, D. Li, B.-L. Lu, and L. Qiu. Seeing through the brain: Image reconstruction of visual perception from human brain signals. arXiv preprint arXiv:2308. 02510, 2023.
J. Li, Y. Wang. N. Lin, and D. Li. Translating mental imaginations into characters with code- books and dynamics-enhanced decoding. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IREE, 2025.
N. Lin, T. Ohkawa, Y. Huang, M. Zhang, M. Cai, M. Li, R. Furuta, and Y. Sato. SiMHand: Min- ing similar hands for large-scale 3d hand pose pre-training In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
H. Liu, P. Wei, H. Wang, X. Lv, W. Duan, M. Li, Y. Zhao, Q. Wang, X. Chen, G. Shi, B. Han, and J. Hao. An eeg motor imagery dataset for brain computer interface in acute stroke patients. Scientific Data, 11:131, 2024a. doi: 10. 10.1008//4155--0--07777
X. Liu, Y. Liu, Y. Wang. K. Ren, H. Shi, Z. Wang. D. Li, B. Lu, and W. Zheng. Eeg2video: Towards decoding dynamic visual perception from eeg signals. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 72245-72273, 2024b.
12

---

Published as a conference paper at ICLR 2026
Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang. and L. Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21013-21022, 2022.
Y. Liu, Z. Gui, D. Yan, Z. Wang, R. Gao, N. Han, J. Chen, J. Wu, and D. Ming. Lower limb motor imagery eeg dataset based on the multi-paradigm and longitudinal-traning of stroke patients. Scientific Data, 12:314, 2025. doi: 10. 10.1038/s41597-022 -22141161872---81444444444444444
J. Ma, B. Yang, W. Qiu, Y. Li, S. Gao, and X. Xia. A large eeg dataset for studying cross-session variability in motor imagery brain-computer interface. Scientific Data, 9:531, 2022. doi: 10.1038/ 641597-022-01647-
X. Ma. S. Qiu, and H. He. Multi-channel eeg recording during motor imagery of different joints from the same limb. Scientific Data, 7:191, 2020. doi: 10. 1008//411577000--53..
S. L. Metzger, K. T. Littlejohn, A. B. Silva, D. A. Moses, M. P. Seaton, Wang, M. E. Dougherty, J. R. Liu, P. Wu, M. A. Berger. I. Zhuravleva, A. Tu-Chan, K. Ganguly, G. K. Anumanchipalli, and E. F. Chang. A nigh-perffrmance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976):
2023.
F. Mushtaq, D. Welke, A. Gallagher, Y. G. Pavlov, L. Kouara, J. Bosch-Bayard, J. J. F. van den Bosch, M. Arvaneh, A. R. Bland, M. Chaumon, C. Borck, X. He, S. J. Luck, M. G. Machizawa, C. Pemet, A. Puce, S. J. Segalowitz, C. Rogers, M. Awais, C. Babiloni. N. W. Bailey, S. Baillet, N. A. Busch, A. Calzada- Reyes, A. Chatard, P. E. Clayson, M. X. Cohen, J. Cole, M. Constant, A. Corneyllie, D. Coyle, D. Cruse, I. Delis, A. Delorme, D. Fair, T. Falk, M. Gamer, G. Ganis, K. Gloy, S. Gregory, C. D. Hassall, K. E. Hiley, R. B. Ivry, K. Jerbi, M. Jenkins, J. Kaiser, A. Keil, R. T. Knight, S. Kochen, B. Kotchoubey, O. E. Krigolson, N. Langer, H. R. Liesefeld, S. Lippé, R. E. London, A. MacNamara, S. Makeig, W. Marinovic, E. Martinez-Montes, A. A. Marzuki, R. K. Mathew, C. Michel, J. d. R. Millán, M. Moll-iiliamss, L. Morales-Chacóne R. Naar, G. Nilsonne, G. Niso, E. Nyhus, R. Oostenveld, K. Paul, W. Paulus, D. M. Pfabigan, G. Pourtois, S. Rampp, M. Rausch, K. Robbins, P. M. Rossini, M. Ruzzoli, B. Schmidt, M. Senderecka, N. Srinivasan, Y. Stegmann, P. M. Thompson, M. Valdes-Sosa, M. J. W. van der Molen, D. Veniero, E. Verona, B. Voytek, D. Yao, A. C. Evans, and P. Valdes- Sosa. One hundred years of eeg for brain and behaviour research. Nature Human Behaviour, 1:443-1444, 2024. doi:
R. C. A. Bendall, D. Brady, M. L. Bringas-
T. Ohkawa, K. He, F. Sener, T. Hodan, L. Tran, and C. Keskin. AssembiyHands: Towards egocentric activity understanding via 3D hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19999-13008, 2023.
M. Ortiz, L. de la Ossa, J. Juan, E. láñez, D. Torricelli, J. Tornro,, and J. M. Azorín. An eeg database for the cognitive assessment motor imagery during walking with a lower-limb exoskeleton. Scientific Data, 10:343, 2023. doi: 10.
A. Pyarelal, E. Duong, C. Shibu, P. Soares, S. Boyd, P. Khosla, V. A. Pfeifer, D. Zhang, E. An- drews, R. Champlin, V. Raymond, M. Krishnaswamy, C. T. Morrison, E. Butler, and K. Barnard. The tomeat dataset. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 40872-40923, 2023.
F. Ragusa, A. Furnari, S. Livatino, and G. M. Farinella. The MECCANO dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1568-1577, 2021.
N. Saito, Т. Ogawa, N. Shiraishi, R. Koide, H. Komine, M. Yokoyama, S. Hanawa, and K. Sasaki. Difference in the electromyographic behavior of the masticatory and swallowing muscles dur- ing cued versus spontaneous swallowing. Dysphagia, 39(3):398 406, 2024. doi: 10.1007/ 0044550024-1003111
F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101. A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21096-21106 2022.
13

---

Published as a conference paper at ICLR 2026
R. Sivasakthivel, M. Rajagopal, G. Anitha, K. Loganathan, M. Abbas, A. Ksibi, and K. S. Rao. Simulating online and offline tasks using hybrid cheetah optimization algorithm for patients affected by neurodegenerative diseases. Scientific Reports, 15:8951, 2025. doi: 10.1038/ 14158-025599047-2
Y. Song, B. Liu, X. Li, N. Shi, Y. Wang, and X. Gao. Decoding natural images from eeg for object recognition. arXiv preprint arXiv:2308. 13234, 2023.
Tang, Yang. Q. Zhao, Y. Ding, Zhang, Song, and Kong. Visual guided dual-spatial inter- action network for fine-grained brain semantic decoding. IEEE Transactions Instrumentation and Measurement, 2024, doi: 10. 109/TTM..222.3400222..
Z. Tong, Y. Song, J. Wang, and L. Wang. VideoMAE: Masked autoencoders are data-efficient learn- for self-supervised video pre-training. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 10078-10093, 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Proceedings of the Advances Neural Information Process- ing Systems (NeurIPS), 2017.
X. Wang, T. Kwon, M. Rad, B. Pan, Chakraborty, S. Andrist, D. Bohus, A. Feniello. B. Tekin, F. Vieira Frujeri, N. Joshi, and M. Pollefeys. Holoassist: an egocentric human interaction dataset for interactive Al assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20213-202244 2023.
F. R. Willett, D. T. Avansino, L. R. Hochberg, J. M. Henderson, and K. V. Shenoy. High- performance brain-to-text communication via handwriting. Nature, 593:249-254, 2021. doi: 10. 01088//4158-00110335000
B. Yang, F. Rong, Y. Xie, D. Li, Zhang, F. Li, G. Shi, and X. Gao. A multi-day and high-quality ceg dataset for motor imagery brain-computer interface. Scientific Data, 12(1))488, 2025. doi:
11.1038/411597-05---44533
M. Zhang, Y. Huang, R. Liu, and Y. Sato. Masked video and body-worn imu autoencoder for egocentric action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 312-330, 2024.
M. Zhang, R. Yonetani, Y. Huang, Ouyang, R. Liu, and Y. Sato. Egocentric action-awan aware inertial localization in point clouds with vision-language guidance. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 97209-27219, 2025.
Zhang, Q. Ma, Y. Zhang, Z. Qian, Kwon, M. Pollefeys, Bogo, and Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 180-200, 2022.
14

---

Published as a conference paper at ICLR 2026
A EEG PREPROCESSING PIPELINE
Following LaBraM(Jiang et al. 2024) 's preprocessing pipeline to ensure compatibility across modalities, we first applied a band-pass filter of 0.1-75 Hz suppress low-frequency noise, fol- lowed by a 50 Hz notch filter to eliminate power-line interference. The signals were then resampled 200 Hz and normalized by scaling the EEG amplitudes from their raw range (-0.1 mV to 0.1 mV) to approximately -1 to 1, with 0.1 mV set as the unit. Unused or noisy channels were removed to further improve signal quality. Finally, raw EEG recordings (e.g., in edf format) were converted into HDF 5 files to facilitate efficient storage and training.
B SUPPLEMENTARY DESCRIPTION OF THE EGOBRAIN DATASET
To ensure sufficient demographic diversity and enhance the generalizability of multimodal human- centric models, the EgoBrain dataset was collected from broad and diverse pool of volunteer participants. This section provides overview of the recruited subjects, the dataset split protocol, finer-grained taxonomy design, and additional visualizations.
B.1 SUBJECTS
The dataset includes recordings from a total of 40 participants, with a gender ratio of 27 male to 13 female subjects. All subjects were informed of the experimental process and signed informed consent forms before the experiment. This study was approved by the ethical committee of local Institutional Review Board for Human Research Protections.
B.2
TAXONOMY OF PLAY
As briefly introduced in the main text, the "Play" category is further decomposed into three finer- grained subtypes to better reflect differences in cognitive load and behavioral patterns:
Play I: Screen-based games such "Spider Solitaire" and "3D Pinball", involving mini- mal physical movement.
Play Object-based activities such as "Toys", "Cards", "Puzzles", and "Cubes", requiring moderate motor activity and hand-object interaction.
Play III: Fast-reaction grategy-oriente mobile games such "Fruit Ninja", "Subway Surfers", and "Chess", requiring rapid responses or cognitive planning.
Although all three subtypes fall under the same high-level semantic verb "Play", they differ substan- tially in visual appearance, cognitive load, and behavioral patterns. Introducing this finer taxonomy makes the dataset more rigorous. It also helps reduce long-tail effects during data collection and annotation, particularly when modeling multimodal signals involving both vision and EEG.
B.3
DATA SPLIT
We divide the dataset into training, validation, and test sets following the standard data split proto- cal. To increase the evaluation challenge of the EgoBrain dataset, we design two splits of different difficulty gradient, namely the Cross-subject-onf, split and the Cross-subject & Cross-scent split. The entire training pipeline strictly follows the standard procedure selecting the best-performing model on the validation set before conducting the final evaluation on the test set, ensuring fairness and reproducibility in the results.
For the Cross-subject ronment and object arrangement. These sessions are divided into a train set of 22 subjects (32.96 hours), a validation set of 6 subjects (7.75 hours), and test set of subjects (9.08 hours).
setting, collected 34 different subjects under the same physical envi-
For the Cross- Subsect & Cross-Scene split, we additionally collected 6 new sessions from entirely new subjects in different environment. These sessions follow the same data collection protocol, but use completely different object arrangement and take place a distinct background setting. The train set and validation set consists of 28 subjects (40.71 hours) and 6 subjects (9.08 hours), and
15

---

Published as a conference paper at ICLR 2026
the 6 more sessions (11.28 hours) in the new environment constitutes the test set, assuring that the new environment never seen during the training and validation stage.
B.4 TEMPORAL DISTRIBUTION OF ACTIONS
From a temporal standpoint, the longest-duration activities predominantly fall within the "Play" and "Work" categories. For instance, "Play Puzzles" demands sustained attention and intricate hand movements, while "Watch Computer videos" or Pllay Games" can span extended periods. In contrast, actions within the "Consume" category typically brief and episodic. To mitigate under-repressntatioo of such short-duration behaviors, we introduced randomized repetition and each "Consume" -related action was performed three times during collection.
B.5 VISUALIZATION ACROSS SUBJECTS
To illustrate the inter-subject diversity inherent in EgoBrain, present representative initial ego- centric video frames from all 40 participants. Fig. 6 and 7 showcase subjects P0001-P0020 and R0021-PP04D respectively. These visualizations highlight substantial cross-subject variability in appearance, posture, and interaction style, reflecting the richness of the collected participant pool and supporting robust generalization in downstream multimodal modeling.
B.6 VISUALIZATION ACROSS ACTION CATEGORIES
To further demonstrate the breadth of daily activities captured in EgoBrain, we provide visualiza- tions from all 29 annotated action categories. Figures 12 present representative egocentric frames across major action types such operating computers, reading, writing, playing games, and con- suming food or beverages. These examples reveal the diversity of visuomotor patterns and con- textual scenes within each action class, offering valuable insights into the multimodal dynamics of real-world human behavior.
Overall, the EgoBrain combines demographic diversity, multi-environment robustness, and compre- hensive visual recordings, ultimately forming a highly reliable resource for advancing multimodal brain-computer interface research and enabling deeper exploration of real-world human behavior.
C SUPPLEMENTARY DESCRIPTION OF THE BRAIN-TIM MODEL
This section provides additional technical details of the Brain-TIM model. Following the same order as described in Sec. we first detail the Intervat MILP (TIM) for temporal encoding, then the coddllty-sppeciff embeddings, followed by the Transformer encoder architecture, and finally the classification heads and training objective.
C.I TIME-AWARE TOKEN EMBEDDING
To incorporate the time-specific information to the tokens, we explicitly add it by introducing the Time-Interval MLP (TIM), I(.) R2 RD, consisting of three linear layers with ReLU activations and LayerNorm operation. This TIM module takes the start and end time (t,, te) of the correspond- ing interval as input and generate the temporal embedding which can be further appended to the specific token time-aware token embedding.
For the feature tokens è and e/, the time interval is determined by the i-th window [t,, t + At), also named feature time in (Chalk et al. 2024). The temporal embedding is thus calculated as ef [t,, 4 + At) RD, As for the CLS tokens сj and cy the time interval corresponds the j-th query [[j HTT///jj//Q) also known query time. And the temporal embedding is similarly obtained as e! = [((j - 1T//Q,jI/QQ RD.
C.2 MODALITY-SPECITE EMBEDDING
To distinguish tokens of different modality, we further introduced the modality-specinee embedding, represented by two learnable vectors, m E R2D and m² E R2D to store shared vision-modalie modality
16

---

Published as a conference paper at ICLR 2026
and EEG-modality information, respectively. The modality- embeddings are directly added to the tokens of corresponding modality.
C.3 TRANSFORMER ENCODER
As shown in the Transformer encoder the right side of Fig. the input sequence X is processed by stack of Transformer encoder layers. Each layer consists of a self-attention mechanism fol- lowed by feedforward network, following the architecture proposed in (Vaswani al.) (017). The self-attention mechanism enables the model to capture long-range dependencies across different po- sitions in the sequence, while the feedforward network applies non-linear transformations to the input. Each layer uses residual connections and layer normalization to facilitate gradient flow, with the output passed to the next layer to refine the input sequence X representations.
C.4 LINEAR CLASSIFICATION
Following the Transformer Encoder, as shown the bottom right of Fig. 3 we extract the modality- specific and query-specific CLS tokens from the output sequence. These tokens are fed into their respective classification heads, h" and h², which consist of linear layers followed by softmax to produce class probabilities. The two modality branches are trained and evaluated independently, without merging their predicted probabilities into single unified prediction.
The model is supervised using modality-specifff cross- enosoyy losses, denoted as L, and C, for the visual and EEG branches, respectively. The total loss function L defined as the sum of the visual modality loss fop and weighted term for the EEG modality loss La. scaled by a hyperparameter A:
C= Cv X- Lo.
It is worth highlighting that the high-level semantic structure (e.g.. "Work", "Play", "Learn", "Con- sume"" intentionally not used during training. The three-tier hierarchy ("high-level category verb - action")) was introduced primarily to conceptually abstract and organize human daily activ- ities. As such, the highest level serves merely as a semantic scaffold for dataset users and is not supervisory signal for model optimization.
suitable to function
In contrast, our classification design explicitly incorporates the two semantic levels used during training-verbe and actions. These labels are fully leveraged in our training pipeline: the embed- dings produced by the Transformer encoder are passed into two separate classification heads-one for verbs and one for actions-each equipped with its own loss function. This design enables the model learn behavior representations at multiple levels semantic granularity, while avoiding potential biases that may arise from enforcing overly coarse, concept-driven category supervision.
17

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
To provide a clearer sense of the lower bound performance on the EgoBrain benchmark, report several standard random baselines for both verb and action classi- fication. These baselines help contextualize the difficulty of the task and offer reference points for interpreting the multimodal models.
Random-baseline Result:
We evaluate three commonly used forms of random performance: (1) uniform chance level, (2) prior-based random sampling that follows the empirical fres-ffy distribution, and (3) the majority-class baseline. The results are summarized in Table
Table 4: Random baselines for EgoBrain verb and action classification.
Metric
Verb Classes
Action Classes
Chance Level
10.00% 11.77%
3.45%
Prior-based Random
4.02%
Majority Class
18.59 %
7.02 %
Among these, the majority-class classifter yields the strongest random performance, with 7.02% accuracy for action classification. To contextualize the Brain-only model under the challenging Cross-Subject & Cross-Scene protocol, we compare its performance against this upper-bound ran- dom baseline. The Brain-only model achieves 9.36 + 0.52, corresponding to 33.3% relative im- provement over the majority-class baseline.
These baselines demonstrate that, despite the substantial inter-subject variability and the inherent difficulty EEG-based prediction in real-world scenarios, the Brain-only model consistently sur- passes all random and prior-driven strategies. At the same time, the gap between these baselines and the Brain-onk) performance indicates meaningful room for future progress on EgoBrain, inviting further exploration from the research community.
Interestingly, the Brain-only model shows slightly higher action accuracy under the cross-subject & cross-scene protocol. This arises because EEG captures head-centered neural dynamics that remain largely invariant across environments, so increasing the training set from 22 28 subjects directly enhances its discriminability. In contrast, VidcoMAE depends on visual appearance and motion cues that shift significantly with background, lighting, and object changes, leading strong degradation under scene variation. Consequently, EEG benefits from richer cross- cross-subject diversity, whereas VideoMAE suffers from ceoss-scene domain shift.
Robustness Comparison:
18

---

Published as a conference paper at ICLR 2026
E TEMPORAL FUSION VS. SPATIAL FUSION
Our Brain-TIM adopts a temporal fusion strategy to integrate visual and EEG modalities when con- structing the multimodal input sequence X. In contrast, a simpler alternative way is to concatenate modality features along the spatial dimension while keeping the fusion module active but effectively removing one modality's tokens. We refer this baseline as spatial fusion. Under this formulation, the input X is formed as:
X = Concat (o D
(c]le]]le]) E R(N+Q)x3D
fenture black
CLS token block
The key difference between the two fusion paradigms lies in how modality-specific structure preserved. Brain-TIM uses dedicated modality embeddings Ey and Eg to explicitly encode visual and EEG token identities before temporal fusion. contrast, the spatial-fusion baseline directly concatenates the visual and EEG representations:
Xfusion Concat(Xv,, X E RX(++
and treats the concatenated tensor as a single fused modality without distinguishing token types.
Table 5: Comparison of temporal fusion (Brain-TIM) and spatial fusion. Temporal fusion clearly
improves performance for the more challenging action recognition task.
Setting
Verb Acc.
Action Acc.
Vision Only
81.67 1.89
63.40 0.95
Visual & Brain (Spatial fusion)
83.74 + 0.62
64.81 + 1.04
Visual Brain (Brain-TIMe TIM)
83.43 + 0.41
66.70 0.83
To ensure consistency with our main experimental protocol, all results were averaged across five dif- ferent random seeds. The comparison between the fusion strategies is summarized in Table 5 While both methods achieve comparable performance on verb classification, Brain-TIM, with tem- poral fusion substantially outperforms spatial fusion on the more challenging 29-way action recog- nition task. This demonstrates that although simple feature concatenation can suffice for coarse semantic discrimination, explicitly modeling temporal dependencies across time steps leads to more robust and discriminative multimodal world action recognition.
beneficial for fine-grained, real-
19

---

Published as a conference paper at ICLR 2026
F CONFUSION MATRIX OF ACTION
Table 6: Verb-level and action-level abbreviations used in the EgoBrain dataset. Each verb is assigned two-letter code ("VVerb code"). Each fine-grained action is assigned two-letter "Action code", and the final label formed as Verb_Act ion.
Verb
Verb code
Action
Abbreviation
Operate Word
OP_WD
Operate PowerPoint
OP PP
Operate
OP
Operate Excel
OP EX
Operate Paint
OP_P1
Watch Computer
WA CP
Watch
WA
Watch Smartphone
WA SP
Play(I) Spider Solitaire
P1 P1_S$ SS
Play (I)
P1
Play(I) 3D Pinball
P1_PB
Play(II) Toys
P2_TY
Play(II) Cards
P2_CD
Play (II)
P2
Play(II) Puzzle
P2_P2
PIay(II) Cube
P2_CB
PIly(III) Fruit Ninja
P3_FN
Play(III) Subway Surfers
P3_SS
Play (III)
P3
PIIy(III) Angry Birds
P3_AB
Play(III) Chess
P3_CH
Write Copybook
WR_CB
Write
WR
Write Notes
WR_NT
Read Textbook
RD_TB
Read
RD
Read Comic
RD_CM
Read Research Paper
RD_RP
Draw Trace
DR_TR
Draw
DR
Draw Pictures
DR_PC
Eat Snack
ET SN
Fat Spicy strips
ET_SS
Eat Sugars
ET_SG
Drink Water
DK_WI
Drink
DK
Drink Cola
DK_CL
Drink Bitter gourd juice
DK_BG
To maintain clarity in the main paper. omit the full 29-way action confusion matrices due their size and the limited space available. These matrices, however, offer useful insights into fine-grained model behavior and characteristic supplementary material, we adopt the verb-action abbreviation scheme in Tab. where each action encoded using a concise two-level code. This scheme preserves semantic interpretability while enabling a cleaner and readable visualization of the dense 29 x 29 matrices. In Fig. 13 and Fig. [4] we provide the complete confusion matrices for the visual-only and visual-brain models, offering more comprehensive view of their respective error distributions and class- caas--eepaattin behavior.
patterns. To present them compactly and coherently in the
Taken together, the two confusion matrices offer detailed view of how visual and multimodal models behave in fine-grained egocentric action recognition. The Visual-only model tends to strug- gle with actions that share similar hand trajectories, object appearances, desktop-level contexts, which leads to noticeable clusters of confusion in several verb groups. When EEG is incorporated, the Visual+Brain model shows a general trend toward stronger diagonal concentration and reduced cross-class ambiguity, suggesting that neural signals may provide complementary ferentiate visually similar actions. Overall, these results indicate that multimodal integration can improve recognition in scenarios where vision alone limited ambiguous.
that help dif-
20

---

Published as a conference paper at ICLR 2026
GO
ree
P0001
PO002
P0003
P0004
SRA
P0005
PO006
P0007
PO008
PO009
P0010
P0011
P0012
P0013
P0014
P0015
P0016
P0017
P0018
PO019
PO020
Figure 6: We provide visualizations of initial video frames from participants: P0001 to P0020.
21

---

Published as a conference paper at ICLR 2026
WHER
P0021
PO022
P0023
PO024
P0025
PO026
PO027
P0028
PO029
P0030
P0031
P0032
P0033
P0034
P0035
P0036
tuu
P0037
PO038
PO039
PO040
Figure 7: We provide visualizations of initial video frames from participants: P0021 to P0040.
22

---

Published as a conference paper at ICLR 2026
1) Operate Word
2) Operate PowerPoint
BSUNS
3) Operate Excel
Operate Paint
Watch Computer
6) Watch Smartphone
Figure 8: Visualization of selected action categories including "Operate" and "Watch". The egocentric perspective in each sequence offers intuitive insight into the subject's ongoing motor behavior.
23

---

Published as a conference paper at ICLR 2026
7) Play(I) Spider Solitaire
8) Play()) 3D Pinball
9) PIay(II) Toys
10) Play(II) Cards
11) Play(I)) Puzzle
12) Play(II) Cube
Figure 9: Visualization of selected action categories including "Play(I)" and "Play(II)". The egocentric perspective in each sequence offers intuitive insight into the subject's ongoing motor behavior.
24

---

Published as a conference paper at ICLR 2026
13) Play(III) Fruit Ninja
14) Play(III) Subway Surfers
15) Play(III) Angry Birds
16) Play(III) Chess
17) Write Copybook
18) Write Notes
Figure 10: Visualization of selected action categories including "Play(IlI)" and "Write"' The egocentric perspective in each sequence offers intuitive insight into the subject's ongoing motor behavior.
25

---

Published as a conference paper at ICLR 2026
19) Read Textbook
20) Read Comic book
21) Read Research
22) Draw Trace
23) Draw Pictures
Figure 11: Visualization of selected action categories including "Read" and "Draw". The ego- centric perspective in each sequence offers intuitive insight into the subject's ongoing motor behay- ior.
26

---

Published as a conference paper at ICLR 2026
24) Eat Snack
25) Eat Spicy strips
26) Eat Sugars
27) Drink Water
28) Drink Cola
29) Drink Bitter Gourd Juice
Figure 12: Visualization of selected action categories including "Eat" and "Drink", The egocen- tric perspective in each sequence offers intuitive insight into the subject's ongoing motor behavior.
27

---

Published as a conference paper at ICLR 2026
1.0
Confusion Matrix (Visual)
DP_ PP 20--0.22.000 220.300220
210.150.09
0.01
0.01
91.0000.000
600:66
69_ -7-5
0.00.55
-3.05.003
0.180 44
0.260.02
0.02
0.05
0.8
F1555 00.02
0 010.1800144
0.48
0.17
020.06
0.250
F2 TY
P2_GD -0.02
01
030.360.03
0.15
0.82
0.170.010.19
0.02
P2_PZ-
P2 CB
02
96
0.6
Ground truth
0.98
0.01
P3.55 -0.01
0.01
444...01..09
P3,AB
0.04
3.94
0.02
WR.CB -0.02
0.01
0.699.06
0.17
0.09
0.540.03
0.4
RD_TB-
0.30
RD,CM
0.005.0
0.04
JPP
DR. TR
0.01
0:02
0.07
979
01.0 01
0.020.01
8T.5%
0.03
0.01
0:60
0.000.334
ET
0.01
0s 9.70
0.2
ET
80
0.383.000.7
DK WT
0.020.13
0.03
0.04
70.000000
DE
25
0.27
OP WD
99
Ex
oP PT
WA SP
53
P2.T7
p2.co
P2.F2
cz
P2. m
22.55
23.40
P3 cH
WR. ca
hr
RD.1 T
CM
no AP
DR Pc
67.0 2..
C7 50
WT
d
DK PG
UP
HA.
P2.
DR
D%
Predicted
Figure 13: 29-way action confusion matrix of the Visual-only model.
28

---

Published as a conference paper at ICLR 2026
1.0
Confusion Matrix (Visual Brain)
OP_ PP -..20.133 390.110.12
0.01
0.01
0.01
0.01
2.070.010
0.01
69_ -2.5
-3.40.03
0.01 50
0.35
0.04
0.04
0.8
F1,55 -0.01
0.010.21
0.65
0.04
0.08
010
0.270
F2 TY
01
0.63
0.01
0.34
0.03
P2_PZ-
P2 PB.CB
0.04
91
0.05
0.6
Ground truth
0.15
P3.5S
9442.060.03
P3,AB
3.98
0.01
WR.CB
0.330.370.00
0.162LC20.150. 01
0.01
0.85
0.4
RD_
0.74
0.25
RD,CM
3-96
0.01
JPP
0.070.670.01.33
DR. TR 01
0.1
0.01
03
0.010 01
-
0.030.01
0.090.01
8T.5%
0.02
0.740.00..2
00
0.2
ET
DK WT
0.01
0.02
0.01
0,30
3034550.2
DK
0.28
0.020
0.3g
OP WD
9
DR Ex
PT
SP
PLPa
P2.T7
p2.c0
P2.P2
c.
P2. m
22.5
P3 cH
WR. ca
hr
RD.1 m
CM
no AP
DR Pc
ET.35
ET 17.50
d
DK PG
0P.
H4.
P2.
23
DR
DK.
DK
Predicted
Figure 14: 29-way action confusion matrix of the Visual+Brain (multimodal) model.
29


Reviews:
[{"review_id": "6ri7w81U4U", "reviewer": "Reviewer_Lp6o", "summary": "The paper introduces EgoBrain, a large-scale, multimodal dataset of 61 hours of synchronized egocentric video and 32-channel EEG signals. This data was collected from 40 participants performing 29 different daily activities such as work, play, learn, and consume within a controlled laboratory setting. The authors also propose a model, Brain-TIM which fuses the visual and EEG signals data using a temporal-aware embedding mechanism and modality-specific encodings for action understanding. The paper evaluates this model on action and verb classification tasks for cross-subject and cross-subject & cross-scene settings. The results indicate that the multimodal model of using both vision and EEG modality achieves a 66.70% accuracy, representing a 3.30% absolute improvement over a visual-only baseline in the cross-scene setting.", "questions": "Since the dataset has been collected in a lab setup with isolation chamber with limited limb movement, can the authors discuss the robustness of the EEG signal? How much noise can the EEG signal have and still be able to give better action recognition results than just using the visual modality?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper contributes a new dataset, EgoBrain which has synchronized video and EEG signals which can be valuable for computer vision research.  \n2. The paper shows that that EEG signals can be a useful modality for tasks such as action recognition when the visual modality is occluded.\n3. The paper shows analysis on cross-subject and cross-subject & cross-scene analysis which is a challenging benchmark to evaluate the model generalization.", "weaknesses": "1. The architecture method of Brain-TIM seems to be incremental when compared to TIM [1]. The architecture presented in the paper of modality-specific encoders, embedding layers, Time-Interval MLP, and a Transformer encoder seems to be a direct application of the existing TIM framework to a new pair of modalities. Can the authors clarify the differences between TIM and Brain-TIM? Is Brain-TIM just an extension of TIM to multiple modalities?\n2. While the idea of using EEG signal to understand egocentric actions is well-motivated, can the authors discuss the application of their proposed approach to real-world scenarios where collecting EEG signals can be hard and require specific hardware?\n\n[1]. Chalk, Jacob, et al. \"Tim: A time interval machine for audio-visual action recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dhc8urdXxn", "reviewer": "Reviewer_vYK1", "summary": "This paper introduces a multi-modal dataset for Action Classification, including synchronized egocentric videos and 32-channel encephalography recordings. Additionally, this paper proposes a baseline method that achieves an overall accuracy of 80.16% in action classification, utilizing both data modalities. This paper empirically demonstrates that the information extracted from encephalography may be complementary to visual data.", "questions": "1.\tWill the source code and pretrained models be released to support reproducibility? If so, what is the reason for not including them in the supplementary material?\n2.\tWhat is the motivation for including encephalography data?\n3.\tGiven the already high reported accuracy, how does the paper address concerns about task saturation?\n4.\tWhy was the decision made to keep data acquisition simple, and how does this impact the task's difficulty and generalizability?\n5.\tOn what basis is the dataset described as “large-scale,” and how does its size compare to existing benchmarks?\n6.\tThe paper states that prior datasets only use visual data, but most include multiple modalities. Can this claim be clarified?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This review evaluates the paper's quality based on the following criteria: task relevance, related work, technical novelty, technical correctness, experimental validation, writing and presentation, and reproducibility. Each aspect is discussed and highlighted as a strength or a weakness in the sections below.\n-    **Dataset Contribution and Reproducibility:** This paper contributes to the community a dataset of synchronized egocentric videos and 32-channel encephalography recordings. However, it is not explicitly indicated whether the source code will be released, and it is not included as part of the submission.\n-    **Writing and presentation:** Overall, this paper is easy to read and well-written.", "weaknesses": "-    **Relevance of the task and Experimental Validation:** Even though Action Classification from egocentric videos and encephalography recordings may be a relevant problem for the ICLR community. The motivation behind including this novel data type modality is not well stated in the paper's introduction. This paper already reports high performance for the proposed task, so it may probably saturate fast. Considering these results, what are the reasons to keep the data acquisition as simple as possible to not make the task harder?\n-    **Technical Correctness and Related Work:** This paper overclaims about contributing a “large-scale” dataset when its size is not comparable to current state-of-the-art benchmarks for human action recognition from egocentric vision data. Moreover, it states that", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCXpiAui1R", "reviewer": "Reviewer_UTCn", "summary": "This paper introduces EgoBrain, the first large-scale, temporally aligned multimodal dataset combining egocentric video and EEG signals for real-world human action understanding. It includes 61 hours of synchronized EEG and first-person video from 40 participants performing 29 categories of daily activities. The authors also propose a Brain-TIM model (Brain-Time Interval Machine), a multimodal transformer-based framework that fuses EEG and vision representations via time-interval MLPs and modality-specific embeddings to capture temporal and cross-modal dependencies. Experiments demonstrate that multimodal fusion improves action recognition performance over unimodal baselines, particularly in cross-subject and cross-scene generalization.", "questions": "Please refer to the Weaknesses section for detailed questions and suggestions to the authors.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tFirst-of-its-kind dataset integrating real-world egocentric vision with EEG; extensive and ethically curated.\n2.\tClear methodological design with well-justified architecture choices (temporal embeddings, modality-aware tokens).\n3.\tInsightful qualitative results showing when EEG signals complement vision (e.g., occlusion or intent disambiguation).", "weaknesses": "1.\tThe synchronization precision is stated as <1s jitter. This is a relatively large jitter for fast-changing neural signals and short actions. This level of jitter could potentially limit the precise time-locking necessary for analyzing rapid neural correlates of action initiation or error. A discussion on how the Brain-TIM model's windowing strategy mitigates the impact of this 1s jitter is needed.\n\n2.\tThe model’s novelty is limited, Brain-TIM primarily applies existing time-embedding concepts to a new modality pair. Moreover, the framework relies solely on two pre-trained models for feature extraction and classification, which restricts the scope of analysis. How would other models or architectures perform on this dataset? Additional comparative experiments are essential to validate the dataset’s generality and utility.\n\n3.\tThe reported accuracy improvements (≈1–3%) may be statistically marginal, raising questions about their practical significance. The authors should provide statistical validation (e.g., p-values or confidence intervals) to confirm whether these gains are significant rather than due to random variation.\n\n4.\tThe ablation study (Table 2) demonstrates the contribution of the temporal/modality embeddings, but it doesn't directly test the core hypothesis of fusion effectiveness by removing one entire modality from the multimodal setting. Specifically, in the \"Visual & Brain\" section, the base case uses both the Visual and Brain encoders. It would be insightful to see an ablation where the fusion mechanism is active, but one modality's tokens are zeroed out or entirely removed to isolate the Transformer's fusion contribution, separate from the unimodal encoders' raw output strength.\n\n5.\tThe Cross-Subject & Cross-Scene task is expected to be more challenging than the Cross-Subject-Only task; however, the “Brain Only” model surprisingly achieves higher action-classification accuracy in this setting. In contrast, VideoMAE experiences a large accuracy drop, while LaBraM shows only a minor decrease. The paper should clarify why EEG-based performance remains stable (or improves) across scenes, despite EEG being more sensitive to noise, and why VideoMAE—supposedly a stronger zero-shot model—shows a significant decline.\n\n6.\tIt is unclear whether the dataset includes cross-session experiments, i.e., whether the same subjects were recorded again after a time interval performing the same tasks. Including or clarifying this aspect would be valuable for evaluating intra-subject consistency and longitudinal generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MtEgzokZwp", "reviewer": "Reviewer_qdr1", "summary": "In this paper, a new egocentric dataset is proposed which includes the modality of EEG recordings. Unique to this dataset, participants wore an EEG device whilst recording the videos so that their intentions could be better modelled and improve performance. The dataset includes ~61 hours of recording across 4 major classes which are split into 10 verbs which are split into 29 actions. An extension of the Time Interval Machine which instead of audio uses the EEG modality is presented as a baseline and results show that whilst visual only gives a strong baseline, the inclusion of EEG recordings is complementary, providing a boost in overall performance.", "questions": "1. What are the differences between the different play labels, i.e. play(I), play(II), play(III), is this referring to computer, physical, mobile? Why were they split up this way, to make sure that the distribution wasn't too long-tailed/dominated by a majority class? (This becomes clear after looking at the appendix but isn't mentioned in the main paper)\n2. For the Cross-Subject&Cross-Scene split, it is not explicitly mentioned, but I assume the 6 new sessions in the test set have different subjects to the 28 in training?\n3. It would be good to know the random performance of the new dataset for Table 1 to get a sense of the lower bound. The Brain-only results are very low in comparison to the visual only and visual + brain for example and it would be interesting to know the relative performance of this model compared to random.\n4. From the dataset construction, there is a 3 stage hierarchy, including the high level classes, the verbs, and the actions themselves, was this considered at all for the method/dataset splits?\n5. Why was the verb confusion matrix split into two, just for space reasons? And why were the confusion matrices of the actions excluded?\n6. Given the imbalanced nature of the dataset, was an investigation conducted into whether the models are biased towards certain classes would be interesting to see, i.e. looking into per-class accuracy metrics or similar? The confusion matrices shows this in some fashion for verbs, but doesn't include the entire matrix and doesn't include actions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The dataset represents the first dataset of its kind combining both egocentric visual data and EEG recordings, thus opening up a new research area.\n* The dataset includes a good variety of actions across the ~61 hours.\n* The results showcase the benefit of the EEG modality when combined with the vision modality.\n* The paper is generally well-written and easy to follow, there are very few issues with spelling/grammar/presentation.", "weaknesses": "During the description of the dataset, it is not clear within the main paper what the difference is between the verbs Play(I), Play(II), and Play(III) are. This becomes evident within the appendix that it is based on device, but should be included earlier.\n\nThe 3 level hierarchy of high-level class, verb, and action is presented but it is unclear whether this hierarchy/taxonomy is utilised in some way during training or whether this was considered at all.\n\nThe benchmarking of the dataset could be improved in a few different ways:\n* Firstly, there is no random performance given, the gap between the visual modality and the EEG modality is quite large, with the latter reaching only 21% and 10% on the dataset. It would be good to include these numbers to get a sense of the lower bound and how far the EEG only performance is.\n* Secondly, whilst the confusion matrices are provided for the verb classification, these are across two 'clusters' of verbs, and do not show all the misclassifications (for example the operate row only adds up to 0.94). Including full confusion matrices for both actions and verbs (potentially in the appendix) would be interesting to look at to check the biases of the model/data and/or providing per class accuracy metrics would be interesting to see how the performance differs. Given that the dataset is imbalanced in terms of length (though seems uniform from a class frequency perspective) it would be good to know if this impacts model training.\n* Finally, on the cross-subject only results, the method is already achieving 90% and 80% on the verb/action classification respectively, showcasing that this split is almost saturated already, some discussion regarding this and the more challenging cross-subject and cross-scene setting would be interesting to see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)