Go Beyond Earth: Understanding Human Actions and Scenes in Microgravity Environments
Authors: Di Wen
Lei Qi
Kunyu Peng
Kailun Yang
Fei Teng
Ao Luo
Jia Fu
Yufan Chen
Ruiping Liu
Yitian Shi
M. Saquib Sarfraz
Rainer Stiefelhagen
Venue: ICLR 2026 Poster (2026)
Keywords: Microgravity
Action Recognition
Vision-Language Understanding
Primary area: datasets and benchmarks
TL;DR: MicroG-4M is the first benchmark for human action and scene understanding in microgravity, offering clips, captions, and QA pairs to reveal domain gaps and guide robust vision-language models.

Abstract: Despite substantial progress in video understanding, most existing datasets are limited to Earth’s gravitational conditions. However, microgravity alters human motion, interactions, and visual semantics, revealing a critical gap for real-world vision systems. This presents a challenge for domain-robust video understanding in safety-critical space applications.
To address this, we introduce MicroG-4M, the first benchmark for spatio-temporal and semantic understanding of human activities in microgravity. Constructed from real-world space missions and cinematic simulations, the dataset includes $4{,}759$ clips with $13{,}261$ action annotations covering $50$ actions, $1{,}238$ context-rich captions, and over $7{,}000$ question–answer pairs on astronaut activities and scene understanding. MicroG-4M aims to support three core tasks: fine-grained multi-label action recognition, temporal video captioning, and visual question answering, thereby enabling a comprehensive evaluation of both spatial localization and semantic reasoning in microgravity contexts. We establish baselines using state-of-the-art models. All data, annotations, and code are available at https://github.com/lei-qi-233/MicroG-4M.

Published as a conference paper at ICLR 2026
Go BEYOND EARTH: UNDERSTANDING HUMAN AC- TIONS AND SCENES IN MICROGRAVITY ENVIRON- MENTS
Di Wen Lei Qi Kunyu Peng
Kailun Yang² Fei Teng²
Ao Luo
Jia Fu
Yufan Chen-
Ruiping Liul
Yitian Shi!
M. Saquib Sarfraz
Rainer Stiefelhagen!
"arlsruhhe Institute of Technology *TTH Royal Institute of Technology
"Hunan University
"aseed University
RRSE Research Institutes of Sweden
ABSTRACT
Despite substantial progress in video understanding, most existing datasets are limited Earth's gravitational conditions. However, microgravity alters human motion, interactions, and visual semantics, revealing a critical gap for real-world vision systems. This presents challenge for domain-robust video understanding in safety-critical space applications. To address this, we introduce MicroG-4M, the first benchmark for spatio-temporal and semantic understanding of human activities in microgravity. Constructed from real-world space missions and cinematic simu- lations, the dataset includes 4,759 clips with 13,261 action annotations covering 50 actions, 1,238 context-rich captions, and over 7,000 dreestionaasweee pairs on astronaut activities and scene understanding. MicroG-4M aims to support three core tasks: fine-grained multi-label action recognition, temporal video captioning, and visual question answering, thereby enabling a comprehensive evaluation of both spatial localization and semantic reasoning in microgravity contexts. We establish baselines using state-of-the-art models. All data, annotations, and code are available at ht ttttpps//eet //tp://gttttt//t/ com/ lei mei-- -qii-ii--- -ii-i2220 (ail-23/M00 Mroor------- 4M.
INTRODUCTION
Yuri Gagarin's historic flight in 1961 marked the beginning of human space exploration. Since then, sigmificant milestones have been achieved, including crewed lunar landings, the continuous operation of the International Space Station (ISS) for over 25 years, and the participation of more than 650 individuals in space missions Moskowitz & Wolf (2025). With numerous planned crewed missions in the near future, the frequency and complexity of human activities in space are expected to increase substantially. In this context, ensuring the safety, enhancing the operational efficiency of space missions, and safeguarding the health and well-being of astronauts are of paramount importance. With the rapid advancement of artificial intelligence, the integration of robotic systems aboard spacecraft anticipated in the near future. These systems will assist astronauts in routine and mission-critio tasks. Consequently, there is growing demand for the development of deep learning-basee scene understanding and action recognition methods tailored specifically for the unique challenges posed by the microgravity environment.
Human action recognition Wang et al. (2023a); Feichtenhofer (2020); Feichtenhofer et al. (2019); Fu et al. (2019), video captioning Chen et al. (2024), and Visual Question Answering (VQA) Hu et al. (2023) are essential for intelligent human-robot collaboration, particularly in space, where precise perception and understanding of astronauts' actions and their surrounding context are crucial for ensuring operational safety, efficiency, and autonomous assistance under constrained conditions. This capability is crucial for ensuring mission efficiency, enhancing astronaut safety, and providing autonomous assistance in the confined and complex conditions of space habitats. Numerous human action recognition datasets and video caption datasets have been developed, playing an important role in various research areas within computer vision and in industrial applications Camarena et al. (2023);
Equal contribution.
Corresponding author; kunyu.peng@ kit.edu

---

Published as a conference paper at ICLR 2026
MicroG -M
Real Footage
Space Film
VIDEO CAPTIONING
VIDEO QA
HUMAN ACTION RECOGNITION
C estronauts
and Ricky Amild
polo lihhiiiicating inside
Present:
"Q": "What gesture does
Left person:
person: stand
Node International space Statin, trott are boomn mash not and commuterr vartatation, packages drvting g the backgrrunc. modkilala
Ye Guangfu make towaards camera?",
walk,
grab (a person), watch person) Right person: stand, open (o.g. door)
lous
"A": "Ye Guangfa waves to the camera."
completal covared with equipment, misorio gail and contral pareds holding a small
Right person: stand, carry/oldd
The scane opans
Abvent:
pesition. while Risky foats upright hgdding a ping pong pedele Drew execures forward swing to a tennie bell. ther catches an returs him...
tannis mecket while fleeting in
"a": "What is the time of day during video?:
turn (e.g. screwdriver)
"A": mentioned."
person moverent object manipulation parson Interaction
Figure 1: An illustration of the MicroG-4M, containing videos from real and simulated microgravity environments (e.g., movies). The dataset supports benchmarks for three tasks: (1) video captioning. (2) video question answering, and (3) fine-grained human action recognition under microgravity.
Pareek Thakkar (2021); Al-Faris et al. (2020); Le et al. (2022). However, most of the existing datasets for video captioning and action recognition are recorded on Earth without microgravity settings, e.g.. Kinetics400 Kay et al. (2017). AVA Gu et al. (2018), FineGym Shao et al. (2020), EgoCross Li et al. (2025), and ActivityNet Captions Chen et al. (2019).
Human actions in space depart markedly from their terrestrial counterparts because microgravity removes gravity-aligned orientation and reliable support surfaces. Basic behaviors bttaviins--ttaddngg. loco- motion, eating, and manual manipulation-
different kinematics and contact patterns Hagio al. (2022). Standing becomes orientation-invariann, often maintained via foot restraints or hand- holds rather than ground support; locomotion is achieved by drifting or pulling along structures rather than gait; and manipulation frequently involves releasing or catching free-floating objects instead of placing or picking from surface. These shifts violate terrestrial orientation, support/contact, and object-dynamics priors, which helps explain the degradation of Farth-trained Human Action Recognition (HAR) models in orbit and motivates microgravity-spedcee benchmark.
To address this gap. we introduce MicroG-4M, a new video benchmark specifically designed for spatio-temporal and semantic understanding of human activities in microgravity. The name "4M" reflects four characteristies: Multi-source (Real mission footage and physically plausible film), Multimodal (RGB + text annotations), Multi-task (HAR, captioning. VQA), and Microgravity. MicroG-4M comprises 4,759 threc-second video clips drawn from public YouTube footage of real space missions and carefully selected, realistic space- theee films to augment scenario diversity and coverage. The clips span scenes inside the ISS, the Tiangong Space Station, crewed spacecraft cabins, and extravehicular activities. The corpus contains more than 390,000 annotated frames with bounding boxes for each visible individual and 13, 13,000+ action labels covering 50 distinct actions. In addition, it includes 1,238 descriptive clip captions created by human annotators and 7,000+ open- ended question-answer pairs aimed at assessing factual, causal, and counterfactual understanding of scenes in microgravity. Both captions and QA annotations were carefully curated through multiple refinement rounds and data selection stages, ensuring high semantic fidelity, contextual accuracy, and relevance to the microgravity setting. Together, these resources enable multi- spatio-temporal detection, fine- fine-grained action recognition, caption generation, and VQA within single dataset. To evaluate existing methods, we build the first comprehensive benchmark for these tasks, dubbed MicroG-Bench. We evaluate representative video encoders (SlowFast Feichtenhofer et al. (2019), X3D Feichtenhofer (2020), and MViT Li et al. (2022)) for human action recognition and leading vision-language models (e.g., Intern Video Wang al. (2022), Gemini 1.5 Pro Reid et al. (2024), GPT-4o Hurst et al. (2024)) for captioning and VQA tasks. Across all evaluated subtasks, state-of- the-art terrestrial models experience significant performance degradation, underscoring the unique
2

---

Published as a conference paper at ICLR 2026
challenges posed by microgravity environments, such as arbitrary orientations, floating objects, and confined spacecraft interiors. These results highlight the necessity of dedicated benchmarks to facilitate the advancement of more robust and generalizable AI systems tailored for space applications. MicroG- Bennch,, therefore, provides a unified, fair yardstick that will guide the development of more robust and generalizable perception and language systems for astronaut assistance and autonomous mission operations.
We summarize our contributions as follows. We introduce MicroG-4M and MicroG-Bench, es- tablishing the first comprehensive benchmark for microgravity video understanding. Beyond providing 4,759 richly- richly-annotated clips across 50 actions. we define a rigorous evaluation protocol with standardized splits for HAR, Captioning, and VQA. Crucially, our in-depth empirical analysis provides the core insight: by benchmarking strong baselines, we quantify a persistent performance gap and uncover characteristic failure modes unique to weightlessness. These findings expose the lim- itations of terrestrial models, positioning MieroG-4M as the foundational standard for developing future microgravity- architectures.
2 RELATED WORK
Vision-Based Understanding in Microgravity and Space Environments. Microgravity challenges terrestrial vision assumptions, requiring perception models for space habitats. Early work showed traditional SLAM to be unreliable, leading to robust alternatives with visual-inertial fusion, semantic mapping, CAD-informed constraints Soussan et (2022); Mao al. (2024); Miller et al. (2022); Tweddle et al. (2015), and validated on platforms like Astrobee Kang et al. (2024). Real-time vision has also enabled dynamic tasks such as target tracking and scene change detection Oestreich et al. (2021); Dinkel et al. (2024). Research has shifted toward interaction, including astronaut pose recovery Gan et al. (2023); Ouyang et al. (2025). gesture recognition Lingyun et al. (2020); Gao et al. (2020), and EMG-based input Assad et al. (2013). Yet, high-level semantic understanding of astronaut actions and intent remains limited. Current assistants like CIMON Eisenberg et al. (2024) provide basic perception but lack contextual reasoning. To address this, we propose unified benchmark for action recognition, video captioning, and visual question answering in space operations.
Datasets for Action Detection, Video Captioning, and VQA. Progress in video understanding has been largely enabled by the introduction of benchmark datasets across action detection, cap- tioning, and VQA. For action detection, early datasets such as UCF101 Soomro et al. (2012) and HMDB51 Kuchne et al. (2011) provided trimmed classification tasks, later extended by Kinetics Kay et al. (2017) and Net Caba Heilbron et al. (2015) to large-scale, untrimmed settings with diverse action categories. AVA Gu et al. (2018) further introduced spatio-temporal annotations for atomic actions, enabling fine- -ne-grained multi-label detection in realistic scenes. Video captioning evolved from MSVD Chen & Dolan (2011) and MSR-VTT Xu et al. (2016), which paired short clips with multiple sentences, to dense captioning in untrimmed videos via ActivityNet Captions Krishna al. (2017) and procedural datasets like YouCook2 Zhou al. (2018). Multilingual benchmarks such as VATEX Wang et al. (2019) expanded the task to cross- looss-lingual settings with high-quality parallel annotations. In VQA, static image datasets like VQA v2.0 Goyal et al. (2017) and CLEVR Johnson et al. (2017) laid the foundation for compositional reasoning, while TGIF-QA Jang et al. (2017), MovicQA Tapaswi et al. (2016), and TVQA Lei et al. (2018) introduced temporal and multimodal reasoning in video. Recent efforts such as NEXT-QA Xiao et al. (2021) and CLEVRER Yi et al. (2020) focus on causal inference and counterfactual reasoning. Existing datasets focus on terres- trial settings and lack the unique complexity of space activities. To address this, we introduce the first microgravity benchmark for video captioning and VQA, enabling evaluation in long-horizon, space-relevant scenarios.
Fine-Grained Human Action Recognition. Fine-grained video-based action recognition Gritsenko et al. (2023); Chung et al. (2021) aims to detect fine-gr -nniiiri indivisible human actions in both single- and multi-person videos. Unlike standard clip-level recognition, atomic action localization requires frame-level, multi-label classification with spatio-temporal bounding box predictions. To tackle this, CNN-based Wang et al. (2023a); Feichtenhofer (2020); Feichtenhofer et al. (2019) and transformer-bant based Ryali et al. (2023); Wang al. (2023a); Li et al. (2022); Wang et al. (2023b; 2022); Peng et al. (2022); Gritsenko et al. (2023) models have been adapted by adding multi-label heads, bounding box regressors, and region-of-intere modules. Notably, Ryali et at Ryali al.
3

---

Published as a conference paper at ICLR 2026
Adaylhv Aeputimntt
Trukkg
Redie
Cit Areciding
0
0
Carrion Vible
Arcosbos
VOS Selection
Figure 2: Overview of the MicroG-4M collection and annotation pipeline. Blue, orange, and purple modules denote automated, human, and MLLM-assisted stages. Raw footage is trimmed into 3-second clips, automatically filtered, and manually verified to retain authentic microgravity scenes, Motion-driven detection produces person-leved boxes for HAR, while a real- footage protocol constrains captioning and VQA to genuine microgravity content. AVA-based action labels, along with caption and VQA generation and validation, produce the final annotations for action recognition, captioning, and VQA.
(2023) introduced a hierarchical vision transformer balancing accuracy and efficiency, while Wang er al. Wang et al. (2023a) proposed dual-masked autoencoder for improved video pretraining. In this work, we evaluate such HAR methods for the first time in microgravity scenarios.
3 COLLECTION METHODOLOGY
We aim to construct dataset comprising authentic footage recorded aboard the International Space Station and other spacecraft, as well as films that realistically depict microgravity conditions. In the following, we introduce our comprehensive pipeline for collecting, assembling, filtering, and annotating the MicroG-4M dataset, which supports multi-label spatio-temporal action detection, video captioning, and video question answering tasks from Internet-soureed vidcos. An overview of this collection and annotation pipeline is shown in Figure 2.
Raw Video Information Collection. Video sources primarily include genuine microgravity footage from actual spacecraft missions and selected cinematic clips known for their realistic depictions of weightlessness. Authentic spacecraft videos were mainly retrieved from online video platforms, while cinematic clips manually chosen based their fidelity real microgravity conditions. Videos were downloaded with a target resolution of 480p whenever available, and otherwise at the highest available resolution below 480p. The final dataset comprises approximately 5, 000 three-second clips, predominantly composed of authentic space station footage.
Dataset Assembly Pipeline. We established automated pipeline preprocess and structure raw videos, enabling consistent and accurate downstream annotation. The pipeline consists of three main stages: video segmentation, filtering, and automated bounding-box annotation.
Raw Video Trimming. Raw videos are trimmed into uniform three-second clips at 30 fps, discarding shorter segments to maintain temporal consistency.
Filtering. Video clips undergo automatic filtering based on person detection and ssene-transition analysis. Specifically, we employed YOLOv11 Khanam & Hussain (2024) for human detection and DySceneDetect Castellano to identify abrupt scene changes, discarding clips with insufficient human-action content or disrupted temporal continuity.
Bounding-box Annotation. Person bounding boxes are automatically annotated using YOLOv11 11 Khanam & Hussain (2024) detection combined with BoT-SORT tracking Aharon et al. (2022). To enhance annotation accuracy, adaptive strategies were employed by assessing video motion intensity using sparse optical flow methods Jeannin & Divakaran (2001); Ali (2013); Szeliski (2010). Annotation parameters were dynamically adjusted accordingly to optimize computational efficiency and annotation precision. The resulting structured annotations include bounding-bov coordinates, unique identities, and detection confidence scores.
4

---

Published as a conference paper at ICLR 2026
Manual Video Screening. Following automated preprocessing, an additional manual verification step was performed to ensure dataset purity and environmental consistency. Specifically, each generated video clip was individually reviewed to exclude terrestrial scenes, such ground-based footage or pre-launch preparations, thereby retaining only those segments clearly depicting human activities under authentic microgravity conditions. This rigorous screening process ensured semantic precision and enhanced the overall reliability of the dataset.
Action Label Annotation. In this phase, we derive a microgravity--aiiored action taxonomy from AVA's 80 atomic actions Gu et al. (2018). To ensure environmental applicability and semantic conti- nuity, we (i) exclude actions that are physically inapplicable in space (e.g., water- or ground-speeifie). (ii) merge near-duplicates into unified categories, and (iii) introduce context-aware semantic adjust- ments. To disambiguate visually or semantically similar actions, we define explicit differentiation criteria that standardize annotation decisions. Each three-second clip is treated as self-contained unit: for every detected individual, annotators assign up to five visible or inferable action labels per clip. The verified annotations are exported structured CSV files for downstream training and evaluation. Retaining AVA class names while re-grounding their semantics in microgravity enables fair Earth Earth->space comparisons without altering the label space.
Caption and VQA Annotation. For caption and VQA annotation, annotators consulted publicly available, authoritative resources, including official crew rosters and astronaut biography databases from major space agencies (e.g.. NASA, CNSA, ESA), historical spacecraft module layout diagrams, equipment specification documents, mission operation reports and task logs, and onboard video tran- scripts and archival footage. These domain resources are systematically used to resolve ambiguities in astronaut identity, cabin context, and equipment usage, and enforce consistent naming conventions for modules and objects.
For caption annotation, captions are written by human annotators based on frame-by-fran visual inspection of each 3-second clip at 30 fps. Beyond the core action and object description, captions explicitly encode astronaut identity, cabin location and background spatial layout, fine-grained hand-body-obije interactions, equipment appearance and usage, and posture, orientation, and other distinctive visual traits (e.g., helmet and suit state). All factual content cross-checked against the domain resources above to ensure semantic accuracy. Multimodal Large Language Models (MLLMs) are applied a subsequent refinement step to improve grammatical fluency, clarity, lexical diversity, and to simplify overly complex sentences into a standard captioning style; all edited captions are re- re-validated by annotators. These high-quality captions provide strong semantic grounding to support downstream tasks such as action recognition and context-aware retrieval.
For Video Question Answering (VQA), we adopt a caption- -rriiongroouned,, two-stage generation and filtering pipeline guided by Heilman & Smith (2009). Given each refined caption, candidate ques- tion-answer pairs are generated via MLLMs using prompt templates designed to elicit variation across multiple reasoning dimensions, including standard Wh-forms, foreground versus background elements, coarse versus fine-grained actions, identity, location and equipment, and temporal or causal relations. One deliberately unanswerable question per clip is optionally included. A refinement pipeline then filters out candidates that rely on sound, latent intent, or non-visual cues. The remain- ing QA pairs are ranked by the MILM based on logical consistency, linguistic fluency, semantic relevance, and information value. Annotators review the top-ranked pairs, remove any hallucinated or fabricated content, revise prompts if necessary, and verify that all retained questions are visually grounded or explicitly marked as "Not mentioned" when unanswerable. To ensure comprehensive coverage from broad contextual understanding to detailed actions while maintaining visual fidelity and controlled reasoning scope, a final set of six diverse QA pairs is retained for each clip.
A Annotation Quality Control. During the annotation phase, a team of 9 annotators collaboratively worked on labeling the fine-grained human actions, generating video captions, and crafting VQA pairs. To ensure high annotation quality, a comprehensive cross-verification protocol was implemented throughout the process, supported by group discussions to resolve disagreements and reach consensus. For the captioning and VQA tasks, all entries were further subjected to a semantic consistency check utilizing LLMs, followed by iterative human review to enhance both linguistic clarity and factual accuracy. This layered validation pipeline, combining automated and manual strategies, ensured the reliability and coherence of the annotations across all modalities. As a result, we introduce the MicroG-4M dataset, an extensively curated benchmark explicitly designed for fine- aineigriied, multi- label, spatiotemporal action recognition, captioning, and VQA under microgravity conditions. This
5

---

Published as a conference paper at ICLR 2026
Space
hand
olue
GOUL Dranto
wear
ingri
sible
nand
NASA
S
black
hand Space
NASA
seen
ideo
Biter
Anside
mentioned
PEAN CAPK
ment lentiona astrooall
floating
Hadtiele
tistronaa
a en Bear Ing.
sstronau
Idar Leatting
weaa Staaiinnnwhihh white
st station Spa module
(a) Caption
(b) Question
(6) Answer
Video Quterrtvaen
Hod8c
Action label: Stand, Carry/Hold. Talk Lift/Pick up
Videet
Caption: Inside the US Lob NASA ustronaut Steven Swanson wears charcoal groy T-shlrt and dark blue utility trousers. He drifts backward between avionies and experiment racks while talking. iPad shows
300
exterior view the iSS floats in front of him. He solzes the rablet with both hands and turns hils head to the right.
X000
003
Steven Swanson wearing Ac" AA charooal gras T-shir and dark Liue ufilily trousers."
US
QM
P1
P64
(1) Label Statisties
(e) Num Persons
"How rioe Strven Swarson Interact with the floating liFa??
Ac" 'He seiees the tablet with both hands."
Figure 3: An illustration of the statistics of the dataset and the annotation samples. Word clouds of the (a) Caption, (b) Question, and (c) Answer from our MicroG-4M dataset are provided. The label statistics of the fine-grained human action recognition
provided in (d), which showcases the annotation number per action group (i.e., Object Manipulation (OM), Person Interaction (PI), Person Movement (PM)). The distribution of person counts per video clip is visualized in (e). On the bottom right, one annotation sample from MicroG-4M is provided.
dataset sets new foundation for advancing robust scene understanding and human activity analysis in space- based environments.
4 DATASET COMPOSITION
We release the MicroG-4M dataset, comprising the following subsets: Fine-Grained Human Action Recognition Subset: Contains fine-grained, multi-lahel annotations for spatiotemporal human action detection, comprising 4,759 manually annotated three-second video clips from authentic microgravity environments. Each clip includes annotations for up to five distinct actions per detected individual. The 50 action labels are organized into three categories: Object Manipulation (4,986 annotations, 37. 37.60%) Person Interaction (4,288 annotations, 32.34%), and Person Movement (3,987 annotations, 30.07%). Object Manipulation includes actions involving physical objects or equipment, Person In- teraction involves social or physical engagement with others, and Person Movement covers individual postural or locomotor motion. These categories are derived from AVA's Gu al. (2018) atomic action labels and regrouped for semantic coherence under microgravity conditions, following macro-level groupings similar to those adopted in AVA-based fine-grained action recognition benchmarks Peng et al. (2024). Figure 3(d) illustrates the distribution of these categories in both real and simulated video clips, while Figure 3(e) shows the distribution of person counts per clip, highlighting the diver- sity of social configurations. Overall, the subset contains 13,261 action annotations, including 9,610 from real footage, 3,651 from simulated sources, and 390, 390,000 0,000 bounding box annotations. Video Caption Subset: Comprises 1,238 detailed, semantically rich descriptions validated against official aerospace agency documentation for astronaut identities, spacecraft locations, actions, appearances, and interactions. Each caption corresponds uniquely one video clip. The word cloud visualiza- tion in Figure 3(a) provides insights into frequently mentioned terms and concepts, illustrating the thematic and semantic distribution of the captions. Visual Question Answering (VQA) Subset: Includes 7, 428 structured QA pairs, systematically generated and refined via MILMS to ensure linguistic fluency, semantic relevance, and comprehensive coverage of detailed actions and broader context, with each of the 1, 238 video clips associated with up to six diverse QA pairs. Figure 3(b) and (c) show word clouds for questions and answers separately, revealing prevalent inquiry types and common semantic patterns within the dataset. Figure presents representative video clips from the dataset, demonstrating typical annotation examples and highlighting the visual and semantic diversity of the dataset.

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS VALIDATING
5.1 BENCIMARR PROTOCOL
Data Split. We partition the dataset into training, validation, and test subsets in a 7:1:2 ratio. Row- level distribution as follows: Training set: 9,273 records (69.93% of 13.261): Validation set: 330 records (10.03% of 13,261); Test set: 2,658 records (20.04% of 13.261). Video-level distribution as follows: Training set: 3,331 videos (69.99% of 4, 4,75;; Validation set: 475 videos (9.98% of 4, 4,759); Test set: 953 videos (20.03% of 4,759).
Baselines for Fine-Grained HAR. For fine-grained action recognition, we evaluate well- established baselines from video-based human action recognition, including transformer-basee models (MViTvl Fan et al. (2021), MViTv2 Li et al. (2022)) and CNN-based models (I3D Carreira & Zisserman (2017), SlowFast Feichtenhofer et al. (2019), X3D Feichtenhofer (2020), C2D Feichten- hofer et al. (2019)). These widely used architectures cover both paradigms and allow us to assess generalization to the unique spatiotemporal dynamics of microgravity. All models initialized with Kinetics400 Kay et al. (2017) pretrained weights for better convergence. HAR experiments were conducted on NVIDIA RTX 2080 Ti GPUs and an AMD Ryzen Threadripper 2990WX CPU.
Baseline methods for Video Captioning and QA. For video captioning and question answering in microgravity, we evaluate strong baselines, including open-source models (VideoCharGPPT Li et al. (2023), mPLUG-OW1-3 Ye et al. (2025), LLaVe-Next Li et al. (2024), VideoLLa Lin et al. (2024), Qwen-2.5-VL Bai al. (2025), InternVideo Wang et al. (2022)) and closed-source models (GPT-4o Hurst et al. (2024), Gemini 1.5 Pro Reid et al. (2024), Gemini 2.5 Pro Comanici et al. (2025)). Open-source models offer reproducibility via public weights and code, while closed-source models serve upper-bound references. This mix enables both transparent analysis and comprehensive evaluation of video-language understanding microgravity. All evaluations were conducted using NVIDIA A100-40 GPUs and Intel Xcon Platinum 8368 CPUs.
Cross-doman transfer protocol. To quantify the microgravity-induee domain gap, we use a fixed transfer setup: models pretrained on Kinetics are fine-tuned on AVA Gu et al. (2018) with matched settings, then evaluated zero-shot on MicroG-4M. For terrestrial contrast, we test on JHMDB (Split 1) Jhuang al. (2013) with the overlapping action set and standard protocol. This isolates domain effects (microgravity vs. Earth) from implementation choices; evaluation details in Sec. 5.
Our evaluation metrics include mAP@0.5, F1 score, recall, and AUROC, all calculated using the macro method. Among these, mAP@0.5 is the primary metrie for measuring average detection accuracy per category and thus comprehensively evaluating the model's action recognition performance in a microgravity environment. Per-class threshold sweeps provided in Appendix.
Evaluation Metries for Fine-Grained HAR.
Evaluation Metrics for Video Caption and QA. We adopt standard automatic evaluation metrics, including CIDEr Vedantam et al. (2015), BLEU-4 Papineni al. (2002), ROUGE-L Lin (2004), METEOR Banerjee & Lavie (2005), and BERTScore (FI) Zhang et al. (2020), all rescaled to a 0~ 0~~100 range for consistency. For semantic similarity, we report S-BERT Reimers Gurevych (2019a) and S-VQA Pathak et al. (2023) scores, both computed as cosine similarity between Sentence- BERT Reimers& & Gurevych (2019b) embeddings of predicted and reference texts. S-VQA used specifically for answer evaluation in generative visual question answering settings, capturing semantic equivalence beyond lexical overlap. More details of the implementations are delivered in Appendix.
5.2 RESULT ANALYSIS FOR
HUMAN ACTION RECOGNITION
Quantitative Analysis. We evaluate several representative models on MicroG-4M, all pretrained on Kinetics400 Kay al. (2017) and fine-tuned on our dataset. As shown in Table 1, results plateau around 47% test mAP, indicating a substantial gap to Earth-trained regimes. The ranking also inverts common trends on Kinetics/AVA, with CNNs plus non-local modules Feichtenhofer et al. (2019); Carreira & Zisserman (2017) leading mAP/AUROC and Slow (4x 16) Feichtenhofer et al. (2019) yielding the best F1, suggesting that local spatial encoding and structured receptive fields remain ad advantageous when motion lacks gravitational consistency. Longer temporal windows further help, highlighting the need for domain domain-adapted temporal reasoning. Under the matched transfer protocol, AVA MicroG--MMM underperforms AVA-JHMMD (Table 3), isolating physies-driven
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance of models fine-tuned on MicroG-4M, evaluated on the validation and test sets.
Model
Valakaiin
Test
Arch
Bicihees
Warare
(AAP:S)
Flscan((S)
(5c
.A.RIC (5)
Recall (56)
Feidanlede C3D MLN Fiscochelli at (D11))
te (2116)
331.7
91:22
131.3
67:11
15:30
24.30
100 Carstre & 21 120 7y
Ba5
R53 al (2010)
27:33
01.00
10:15
03.08
$5.05
48:41
29:33
22:36
56-11
Skee Foshholafer cill (2309)
6xd
R59 Bout :2010)
01.34
$1.83
21.3
18.06
84.00
45.10
38 15.15
22.19
55.23
Saw Erahurbeti aial (2204)
4035
RS318
42.7
18:73
85:45
15:27
24.73
33.88
:201l
Swedie Tnichwebsfien andl (3019)
15:10
RS3 la (2/10)
11.75
10
17.70
18:00
84.00
13:01
11
31.36
K7:51
ctel
B-CONV
14,03
MWiT Lieaal (3122)
38.37
18:33
0.80
15.11
X:n bacterabuiir (yta)
1755
16.33
5.83
14.77
5.77
4.50
T5.31
Note: All models have been pretrained on Kinetics400 cataset Kay et al. (2017) and continually trained on
MicroG-4M. TC denotes the temporal configuration (frame length sampling rate). #Params indicates the number of parameters (in millions, M).
Table 2: Zero-shot performance on MicroG-4M test set for models pretrained on Kinetics and fine- tuned on AVA Gu et al. (2018).
Model
Teit Reaull
(19)
(96)
Slow or at 12099
4x16
R30 iss
12000)
Kincias 400 Kay stal. (201))
AWA 122 Ga
(2015)
13:74
6.35
3:35
12.53
slaw Frodecabaeer (2009)
12005)
Kinstis 400 st.. (2017)
x2.1 Ge
(2015)
16:34
2.57
1201))
Shaffiet bekhheebwwrarr d. (909))
608
Ribi Neara (201
Kineive 600 ca al (20n5)
ADA Ge (Tmin
1495
1.90
1:33
T1.30
Stadfoe feittrerberrrr ai (209))
Ribi te (206)
Kinereettn Cor al (7818)
33:8
6:35
Note: All metrics are macro- aavero-deergee over action classes. mAP measured at IoU
0.5.
Fl and AUROC are
computed per class and then averaged. TC denotes the temporal configuration (frame length sampling rate).
Green: Identical Ground Truth
Red: Incorrect Action
Blue: Missed Action
Sinnd
Will, Comy/Holl Tall to, Point to Microti 4M: Malk,, Carry/Hold, Talk to Point to AVA: Malk, Carry/Hlod,, Talk Point to. Bend/Bon,
CT: Stand, Talk to, Litt
GT: Stand, Cany//Hold Pull MicraG-MM: Stand, Carry/Holl, Pull AVA: Stard, Pull, Carry/Hold
CT: Stand
Microt:-aM: Stand, Talk to, Carry/Hold
Microt: 4M: Stand,
MicroG44M:: Stand
Talk LiFt/Pick
up, Carry/Hold AVA: Stard, Talk to, Lift/Pickup
AVa: Stand, Talk to
AVA: Sät
Figure 4: Qualitative results for fine- fine-grained action recognition in microgravity. GT denotes ground truth; MicroG-4M and AVA refer to models fine-tuned on MicroG-4M and AVA, respectively. The MicroG-4M model yields more accurate predictions.
gap beyond conventional terrestrial shifts. Comparing to AVA-finetuned models (Table 2), observe a sharp drop when transferring directly from AVA Gu et al. (2018) to MicroG-4M despite identical pretraining/backbori
e.g., SlowFast 32x2 reaches only 23.81% % mAP on MicroG-4M, far below its MicroG-tuned counterpart. This indicates that Earth-trained assumptions about orientation, support/contact, and object dynamics are fragile in microgravity and are not resolved by naive fine-tuning alone. Together. these findings position MicroG-4M as diagnostic benchmark that surfaces gravity-dependen failure modes and motivates methods for robust, apace-ddapted video understanding. Our dataset thus provides a rigorous testbed for evaluating and advancing space- adapted video understanding models, especially in the context of astronaut assistance and autonomous system development.
Qualitative Analysis. Figure 4 presents qualitative comparisons between models trained on AVA Gu et al. (2018) and those fine-tuned on MicroG-4M, across 5 representative video clips captured inside and outside spacecraft cabins. The MicroG-4M- trained model demonstrates high alignment with ground truth labels for core actions such as "Stand", "Walk", and "Talk-to", while the AVA Gu et al. (2018)-based counterpart consistently misinterprets floating or inverted postures as "Bend/Bow or "Sit", revealing its reliance Farth-centric gravitational priors. A representative example in the fifth
8

---

Published as a conference paper at ICLR 2026
Table 3: Cross-domain transfer under matched AVA fine-tuning (zero-shot evaluation).
Model
TC
Backhone
Set
(A.)
(1))CC(()
SlowFaas Jast Feichscnhofer et (2019)
a2x2
RI01
(2016)
JHMDB Jhuang (203))
FlowFFast Feichsccholer et al. (2019)
32x2
RJ01 He et (2016)
MicroG-IM
23.81
77.83
Slow Feichsenhufer et al. (2019)
8
R50 He al: (2016)
JHMDB Thuang al. (2013)
34.24
76.96
Slow Foichtenhofer et (2019)
8x8
R50 He (2016)
MicroG-4M
16.24
73,85
Table 4: Comparison of video captioning performance across open source and closed-source models on the MicroG-4M benchmark. "#f" denotes the number of input frames used during model inference.
Model
CIDEr BLEU-
Rouge-L
Meteor
S-BERT
BERTScore
Open-Source
Chdee-C Manz et (2024)
0.12
39.61
85.40
mPLUG-OW13 Ye ct. (2025)
0.16
0.40
11.87
5.88
47.45
85.91
LLaVA-NeXT Zhang et al. (2024)
0.30
1.88
16.32
14.45
54.16
84.98
Video-LLaVA Lin et al, (2024)
0.03
0.07
9.29
4.12
42.97
84,89
Qwen2.5-VL. Yang et al. (2024)
9
0.03
1.34
13.75
15.67
56.46
84.01
Recsie22-Recap-78 Yuan et al. (2025)
16
0.04
0.03
0.17
0,12
51.35
84,53
Video Wang et al. (2022)
90
0.77
2.60
16.57
15.18
55.28
85.41
Soured-SSurce
GPT-4o Hurst (2024)
1.74
2.65
16.46
11.27
62.18
86.75
Gemini 1.5 Pro Reid al. (2024)
16
3,52
3.28
17.34
15.19
63.38
86.25
Gemini 2.5 Pro Comanici et (2025)
3.27
3.10
17.92
13.08
62.82
86.95
column shows the AVA Gu et al. 2008))finetuned model misclassifying a floating astronaut as "Sit" while the MicroG-44-ttrained model correctly predicts "Stand" reflecting a common correction of gravity-induced biases. Models trained on MicroG-4M also demonstrate improved robustness in distinguishing passive object drift from intentional manipulation, though semantic ambiguity remains, egg., predicting "Carry/Hold" when a tool drifts near the astronaut's hand without actual interaction. These results indicate that MicroG-4M mitigates terrestrial biases, enhances sensitivity body-object dynamics, and better captures domain-specific actions, supporting future work on temporal coherence and intentionality modeling.
5.3 EVALUATION OF VIDEO CAPTIONING MODELS
The results in Table 4 reveal how video-languee models perform on the MicroG-4M benchmark, highlighting key challenges introduced by microgravity-specife content. Lexical metries such as CIDEr and BLEU-4 show low overall values, especially among open-source models, suggesting a significant distributional shift between MicroG-4M and typical pretraining data. The dataset's domain-specific vocabulary, visually compositional scenes, and semantically dense annotations likely reduce surface-level overlap, which these metrics are sensitive to. In contrast, semantic similarity metrics such S-BERT and BERTScore remain relatively higher and more consistent, indicating that several models capture the underlying intent even without lexical alignment. This underscores the semantic richness of MicroG-4M, where alternative phrasings and scientific terminology often convey similar meanings. Performance differences further reveal that input frame density and pretraining modality play key roles. Intern Video Wang al. (2022), which processes 90 frames sampled within a 3s window, consistently outperforms other open-source models. This suggests that dense sampling, coupled with andeo-specific pretraining, enhances the model's ability to capture subtle spatial patterns features that are particularly important in microgravity scenarios, where visual cues are often atypical or physically ambiguous. flosed-source models, i.e., GPT- 4o Hurst et al. (2024), Gemini 1.5 Pro Reid et al. (2024) and Gemini 2.5 Pro Comanici et al. (2025) achieve better scores on both lexical and semantic metries, likely due to broader data exposure, larger capacity, or more advanced cross-modal fusion strategies. However, their relatively small performance gains further validate the challenge posed by MicroG-4M. In general, these findings position MicroG-4M as demanding benchmark for evaluating multimodal models under domain change, highlighting the need for robust spatial reasoning, domain adaptation, and semantically aware generation strategies in unconventional environments.
and object-scene relationships -eattus

---

Published as a conference paper at ICLR 2026
Table Experiments of Visual Question Answering (VQA) models on the MicroG-4M benchmark.
Model
CIDEr
BLEU-
Rouge-L
Meteor
S-VQA
BERTScore
Open-Source
LLVVA-NeTT Zhang al. (2024)
24.00
22.14
15.56
12.40
38.08
87.15
Yidoo-LLaVA Lin et al, (2024)
25.70
28:47
15.90
10.71
35.39
87.13
Qwen2.5 5- VL Yang et al. (2024)
2.99
0.65
8.35
8.47
40.65
84,80
Tarsier2 Recap-7B Yuan
(2025)
85.30
Closed-Soures
Gemini 1.5 Pro Reid et al. (2024)
8.78
1.33
13.03
12.54
43.15
86.41
Gemini 2.5 Pro Comanici et al. (2025)
0.67
0.48
7.58
9.12
44.75
84,89
GPT-4o Hurst et (2024)
33.98
3.76
18.11
15.89
87.81
5.4 EVALUATION OF VISUAL. QUESTION ANSWERING MODELS
The results in Table demonstrate that MicroG-4M presents distinct challenges for visual question answering. Notably, there is significant divergence between lexical and semantic evaluation metrics, visible in both open- and closed-source models. For example, Qwen2.5-VL Bai et al. (2025) yields a BLEU-4 of only 0.65 and a CIDEr score of 2.99, yet achieves the highest S-VQA score in its category (40.65). similar pattern appears for Gemini 2.5 Pro Comanici al. (2025), which attains the highest S S-QQA score among closed-source models (44.75) while exhibiting very low CIDEr and BLEU-4 scores, largely because it tends to answer simple questions with long, explanatory responses. This contrast suggests that MicroG-4M questions often admit multiple semantically valid answers that differ lexically, such as paraphrased actions, scientific terms, or object references adapted to microgravity settings. This characteristic does not reflect inconsistency in evaluation, but rather underscores the conceptual and linguistic diversity embedded in the dataset. In addition, the moderate absolute scores of even the top-performing closed-source models, such as GPT-40 Hurst et al. (2024) (CIDEr 33.98, S-VQA 44.56), reveal the difficulty of reasoning over visual content in microgravity. Unlike conventional VQA datasets, MicroG-4M includes visually ambiguous cues, e.g. floating objects, unusual body orientations, and tool manipulations under microgravity that challenge models trained primarily on terrestrial data. This suggests that current pretraining corpora lack sufficient coverage of such scenarios, and that purely scaling model capacity is insufficient for reliable generalization. Interestingly, increasing the number of input frames within the fixed 3s window does not consistently yield better performance. For example, Gemini 1.5 Pro Reid et al. (2024) processes 16 frames but performs worse than GPT-4o Hurst et al. (2024), which uses only 6 frames, and Gemini 2.5 Pro Comanici et al. (2025) achieves the best S-VQA score with just 3 frames. This indicates that dense frame sampling alone is insufficient. Instead, performance depends more eritically on the model's ability extract semantically salient cues, e-g., astronaut posture, object manipulation, and spatial configurations, from visually subtle or low-motion segments. In microgravity environments, where conventional motion dynamics and object affordances are altered, effective spatial reasoning and cross-modal alignment appear to be more decisive than temporal redundancy. In summary, MicroG-4M reveals key limitations of current VQA systems in addressing domain-specifie challenges, particularly those involving spatial complexity. ambiguous motion, and semantically flexible queries inherent microgravity. Its comprehensive and specialized design establishes it as a valuable testbed for probing the robustness, adaptability, and generalization capabilities of multimodal models well beyond the scope of conventional barth-based benchmarks.
CONCLUSION
In this work, we present MicroG-4M, the first large-scale dataset specifically curated for human action recognition and vision-language understanding in microgravity environments. The dataset features 4, 759 annotated video clips with over 390, 000 bounding boxes and 13, 000+ action labels across 50 unique action classes. It also includes human-written captions and over 7, 000 VQA pairs, enabling rich semantic understanding and reasoning. We introduce MicroG-Bench. a benchmark for evaluating state-of-thh--a models in fine- -raingrined action recognition, video captioning. and question answering. Results show significant performance degradation in space- like settings, highlighting the need for domain-speciin benchmarks and adaptation. MicroG-4M advances robust, generalizable Al for astronaut support and autonomous space operations.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The project funded by the Deutsche Forschungsgemeinscaatt (DFG, German Research Foundation) -SFB-1577 -476687386 This work supported in part by the SmartAge project sponsored by the Carl Zeiss Stiftung (P2019-01-0033 2021-2026), This work is also supported in part by the National Natural Science Foundation of China under Grant No. 62473139 in part by the Hunan Provincial Research and Development Project (Grant No. 2025QK3019), and in part by the State Key Laboratory of Autonomous Intelligent Unmanned Systems (the opening project number ZZKF2025-2- research was partially funded by the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).
This
ETHICS STATEMENT
This work constructs benchmark from publicly available videos released by official space agencies and publicly distributed films. No private recordings, surveillance footage, or restricted-acee -cccess materials are included. For public release, we distribute annotations and metadata only, and require users to obtain source videos independently through legal channels. This avoids redistribution of copyrighted audiovisual content and supports compliance with institutional and regional regulations.
The dataset depicts professional activities of publicly visible astronauts and actors in occupational settings. While publicly disseminated material typically does not require additional individual consent for research use, downstream users are instructed to follow their institutions' IRB/ethics guidelines and applicable privacy regulations. No personally identifying information beyond what is inherently visible in the source videos is collected or released, and metadata exclude personal identifiers.
The dataset does not include satellite imagery or sensitive geolocation data. Film clips are screened for physical plausibility and safety consistency, and implausible scenes removed. For vision-languee language components, large language models are used to generate and rank candidate questions under in- terrogative diversity, with human verification of final questions and answers. Unanswerable "Not mentioned" cases are retained to discourage hallucination and overconfident speculation.
To mitigate misuse, the resource is released for research purposes under terms that prohibit re- identification attempts, surveillance applications, or violations of platform policies and content owners' rights. We will honor takedown requests from rightsholders and provide mechanism to remove or amend entries if concerns arise. Overall, the release policy combines metadata-only distribution and clear usage terms, which aims to balance openness and reproducibility with privacy, safety, and copyright considerations.
REPRODUCIBILITY STATEMENT
All data annotations, metadata, and code are publicly available at https: /tt::///gtttbt com/ lei -qi -233 /Mi croG-4M, We provide scripts for data preprocessing, benchmark construction, and evaluation, along with documentation for reproducing the reported results.
REFERENCES
Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky, BoT-SORT: Robust associations multi-pedesttian tracking. arXiv preprint arXiv:2206. /465/, 2022.
Mahmoud Al-Faris, John Chiverton, David Ndzi, and Ahmed Isam Ahmed. A review on computer vision-based methods for human action recognition. Journal of Imaging, 2020.
Saad Ali. Measuring flow complexity in videos. In ICCV, 2013.
Christopher Assad, Michael Wolf, Adrian Stoica, Theodoros Theodoridis, and Kyrre Glette. BioSleeve: A natural EMG-based interface for HRI. In HRI, 2013.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li. Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng,
11

---

Published as a conference paper at ICLR 2026
Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. arXiv preprint drXiv:2502.13923, 2025.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In IEEvaluation @ @ACL, 2005.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niehles. ActivityNet: A large-scale video benchmark for human activity understanding. CVPR, 2013.
Fernando Camarena, Miguel Gonzalez-Mendozz, Leonardo Chang, and Ricardo Cuevas- An overview of the vision-based human action recognition field. MCA, 2023.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600, arXiv preprint arXiv:1808.01344 2018.
Brandon Castellano. PySceneDetect.
PySceneDetect.
https://githi: cow.ttaaa///raaaoouuggg////
David L. Chen and William B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Lin Bin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. ShareGPT4Video Improving video understanding and generation with better captions. NeurIPS, 2024.
Shizhe Chen, Yuqing Song, Yida Zhao, Qin Jin, Zhaoyang Zeng, Bei Liu, Jianlong Fu, and Alexan- der G. Hauptmann. Activitynet 2019 task 3: Exploring contexts for dense captioning events in videos. arXiv preprint urXiv:1907.05092 2019.
Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang, Yu-Wing Tai, and Chi-Keung Tang. HAA500: Human-centric atomic action dataset with curated videos. In ICCV, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626// 2025.
Holly Dinkel, Julia Di, Jamie Santos, Keenan Albee, Paulo VK Borges, Marina Moreira, Ryan Soussan, Oleg Alexandrov, Brian Coltin, and Trey Smith. AstrobeeCD: Change detection in microgravity with free-flying robots. Acta Astronautica, 2024.
Till Eisenberg, Gerhard Reichert, Ralf Christe, and Judith Irina Buchheim. Assistant in space. Aerospace Psychology and Human Factors: Applied Methods and Techniques, 2024.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In CVPR, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In ICCV, 2019.
Yuqian Fu, Chengrong Wang, Yanwei Fu, Yu-Xiong Wang, Cong Bai, Xiangyang Xue, and Yu-Gang Jiang. Embodied one-shot video recognition: Learning from actions of a virtual embodied agent. In MM, 2019.
Shuwei Gan, Xiaohu Zhang, Sheng Zhuge, Chenghao Ning, Lijun Zhong, and You Li. A multi-view vision system for astronaut postural reconstruction with self-calibration. Aerospace, 2023.
Qing Gao, Jinguo Liu, and Zhaojie Ju. Robust real-time hand detection and localization for space human-roban interaction based on deep learning. Neurocomputing, 2020.
12

---

Published as a conference paper at ICLR 2026
Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Alexey Gritsenko, Xuehan Xiong, Josip Djolonga, Mostafa Dehghani, Chen Sun, Mario Lučié, Cordelia Schmid, and Anurag Arnab. End-to-end spatio-temporal action localisation with video transformers. arXiv preprint arXiv:2304.1211 121600 2023.
Chunhui Gu, Chen Sun, David Ross, Carl Vondrick, Caroline Pantofaru. Yeqing Li. Sudheendra Vijayanaraimmhan George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: video dataset of spatio-temporally localized atomie visual actions. In CVPR, 2018.
Shota Hagio, Akihiko Ishihara, Masahiro Terada, Hiroko Tanabe, Benio Kibushi, Akira Higashibata, Shin Yamada, Satoshi Furukawa, Chiaki Mukai, Noriaki Ishioka, and Motoki Kouzaki. Muscle synergies of multidirectional postural control in astronauts on earth after a long-term stay in space. Journal of Neurophysiology, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Michael Heilman and Noah A. Smith. Question generation via overgenerating transformations and ranking. DTIC Document, 2009.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. PromptCap: Prompt-guided image captioning for VQA with GPT-3. ICCV, 2023.
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker Whitcomb, Alex Beutel, Alex Borzumov, Alex Careey, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Back, Angela Jiang, An- toine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig. Beth Hoover, Blake Samie, Bob McGrew, Bobby Spero. Bogo Giertler, Bowen Cheng. Brad Lightcap. Brandon Walkin, Brendan Quinn, Brian Guarraci. Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll I.. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. GPT-4o system card. arXiv preprint 2rXiv:2410.21226, 2024.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR, 2017.
Sylvie Jeannin and Ajay Divakaran. MPEG-7 visual motion descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 2001.
Jhuang, J Gall, Zuffi, C Schmid, and MJ Black. Joint-annotated human motion data base, 2013.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Suyoung Kang, Ryan Soussan, Daekyeong Lee, Brian Coltin, Andres Mora Vargas, Marina Moreira, Katie Browne, Ruben Garcia, Maria Bualat, Trey Smith, Jonathan Barlow, Jose Benavides, Eunju Jeong, and Pyojin Kim. Astrobee ISS free-flyer datasets for space vetrulve robot navigation research. RA-I. 2024.
13

---

Published as a conference paper at ICLR 2026
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan. Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv: 2017.
Rahima Khanam and Muhammad Hussain. YOLOv11: An overview of the key architectural enhancements. arXiv preprini arXiv:2410. 17725, 2024.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. ICCV, 2017.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In ICCV, 2011.
Viet-Tuan Le, Kiet Tran-Trung, and Vinh Truong Hoang. A comprehensive review of recent deep learning techniques for human activity recognition. CIN, 2022.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. TVQA: Localized, compositional video question answering. In EMNLP, 2018
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaVe-NexT-Ineereaae Tackling multi-image video, and 3D in large multimodal models. arXiv preprint arXiv:2407.07895. 2024.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. VideoChat: Chat-centric video understanding. arXiv preprint arXiv:2305.0635. 2023.
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. MViTv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022.
Yanjun Li, Yuqian Fu, Tianwen Qian, Qi'ao ao Xu, Silong Dai, Danda Pani Paudel, Lue Van Gool, and Xiaoling Wang. EgoCross: Benchmarking multimodal large language models for cross-domain egocentric video question answering. arXiv preprint arXiv:2508. 10729, 2025.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In EMNLP, 2024.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 2004.
Gu Lingyun, Zhang Lin, and Wang Zhaokui. Hierarchical attention-based astronaut gesture recogni- tion: A dataset and enn model. IEEE Access, 2020.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In ACL, 2024.
Luisa Mao, Ryan Soussan, Brian Coltin, Trey Smith, and Joydeep Biswas. Semantic masking and visual feature matching for robust localization. In iSpaRo, 2024.
Ian Miller, Ryan Soussan, Brian Coltin, Trey Smith, and Vijay Kumar. Robust semantic mapping free-flying robot in microgravity. In ICRA, 2022.
and localization
Clara Moskowitz and Zane Wolf. The astronaut club. Scientific American, 332(2):88, February 2025.
doi: 10. 11l.0cccciiiciiicaeo22222////aa0
Charles Oestreich, Antonio Terán Espinoza, Jessica Todd, Keenan Albee, and Richard Linares. On-orbit inspection of unknown, tumbling target using NASA's astrobee robotic free-flyers. In CVPR, 2021.
Jingxuan Ouyang, Wentao Xie, Guangsheng Xu, Chujun Li, Shuwei Gan, Xiaohu Zhang, Xia Yang, Changhua Jiang, et al. SpacesuitPose: Deep learning-based spacesuit pose estimation in extravehicular activities from monocular images. Acta Astronautica, 2025.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACI, 2002.
14

---

Published as a conference paper at ICLR 2026
Preksha Pareek and Ankit Thakkar. A survey video-based human action recognition: recent updates, datasets, challenges, and applications. AIR, 2021.
Sanchit Pathak, Garima Singh, Ashish Anand, and Prithwijit Guha. S-VQA: Sentence- visual question answering. In ICVGIP, 2023.
Kunyu Peng, Alina Roitherg, Kailun Yang, Jiaming Zhang, and Rainer Stiefelhagen. Transdarc: Transformer-based driver activity recognition with latent space feature calibration. In IROS, 2022.
Kunyu Peng, Jia Fu, Kailun Yang. Di Wen, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiaming Zhang, M. Saquib Sarfraz, Rainer Stiefelhagen, and Alina Roitberg. Referring atomic video action recognition. In 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrae, Radu Soricut, Angeliki Lazaridou. Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mía Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0555 2024.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. In EMNLP/JJCNLP, 2019a.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: /0084, 2019b.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po- Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury. Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: hierarchical vision transformer without the bells-and- whistles. In ICML, 2023.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. FineGym: A hierarchical video dataset for fine-grained action understanding. In CVPR, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1rXhv:1212.0402 2012.
Ryan Soussan, Varsha Kumar, Brian Coltin, and Trey Smith. Astroloc: An efficient and robust localizer for a free-ffying robot. In ICRA, 2022.
Richard Szeliski. Computer Vision: Algorithms and Applications. Springer-Verlag, Berlin, Heidelherg, 1st edition, 2010. ISBN 1848829345.
Makarand Tapaswi, Yukun Zhu, Rainer Stiffelaggn, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding stories in movies through
In CVPR, 2016.
Brent E. Tweddle, Alvar Saenz-Otero, John J. Leonard, and David W. Miller. Factor graph modeling of rigid-body dynamics for localization, mapping, and parameter estimation of spinning object in space. JFR, 2015.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-basee image description evaluation. In CVPR, 2015.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang. Yali Wang, and Yu Qiao. VidcoMAE V2: Scaling video masked autoencoders with dual masking. In CVPR, 2023a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR, 2023b.
15

---

Published as a conference paper at ICLR 2026
Xin Wang. Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VaTeX: A large-scale, hugh-quality multilingual dataset for video-and and-language research. In ICCV, 2019.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Intern Video: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NEXT-QA: Next phase of question- answering to explaining temporal actions. In CVPR, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In MM, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, 2016.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 11Xiv:2412.1511, 2024.
Hui Yang, Lekha Chaisorn, Yunlong Zhao, Shi- Shi-Yong Neo, and Tat-Seng Chua. VideoQA: question answering on news video. In MM, 2003.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mPLUG-OWI3: Towards long image-sequence understanding in multi-modal large language models. In ICLR, 2025.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: Collision events for video representation and reasoning. In ICIR, 2020.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu. Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet- QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding. arXiv preprint arXiv:2501.0788; 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In ICLR, 2020.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.027/33 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional vidcos. In AAAI, 2018
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMS two limited ways. During dataset construction, an LLM proposes candidate VQA questions per 3 s clip; humans filter, verify, and finalize both questions and answers, and captions human written. For benchmarking, several LLM-based systems are evaluated as baselines under a unified inference setup. LLMs do not assign action labels alter evaluation protocols. We also used an LLM for language polishing of the manuscript, and all technical content and conclusions were written and verified by the authors.
B TECHNICAL LIMITATIONS
MicroG-4M introduces the first unified benchmark for high-level video understanding in micrograv- ity environments, but several technical limitations remain that could guide future refinement and expansion.
One challenge lies in the ambiguity and subjectivity of annotations. Interpreting actions and generat- ing captions can vary between annotators, especially in visually ambiguous frames or when inference required. Even with vallsvaaii protocols in place, some degree of subjectivity may persist, introducing annotation noise.
Another limitation the restricted temporal context. Each video clip spans only three seconds, which may hinder the ability to model long-term dependencies. This limitation is particularly relevant when modeling multi-step operations that are common in space missions.
The dataset is currently limited to RGB visual input, which constrains the potential for multimodal understanding. Future versions could benefit from the inclusion of additional modalities such as audio signals communication transcripts to support more comprehensive reasoning.
There also an inherent domain bias introduced by the inclusion of cinematic footage. While these clips visually high-fidelity and physically plausible, they may differ in visual style and narrative framing from real operational recordings. This discrepancy can affect the generalizability of models trained on the dataset.
Finally, the scale and annotation coverage of the dataset present further constraints. The current release offers carefully annotated subset of the full video collection, suitable for benchmarking, but smaller than many large-scale web datasets. Continued annotation efforts are underway to expand the dataset, covering wider range of clips, actions, and scene types support more diverse downstream applications, including temporal reasoning and sequence-level inference.
C POTENTIAL SOCIAL IMPACTS
MicroG-4M introduces the first benchmark specifically designed for video understanding and vision- language reasoning in microgravity environments. It comprises 4,759 curated clips supporting fine-grained action recognition, video captioning, and visual question answering. These tasks collectively provide testbed for evaluating models under the unique motion dynamics and spatial ambiguities posed by microgravity.
While MicroG-4M is not intended for deployment, it may inform downstream research in areas such as astronaut behavior modeling, procedural understanding, or human-robot collaboration. Its vision-language annotations also facilitate studies video summarization and temporal grounding, with potential implications for future human-Al Al interfaces in space-based or analog settings.
MicroG-4M enables the analysis of model limitations under microgravity-speciie challenges, such as sensitivity to gravitational priors and orientation ambiguity, offering a foundation for research on robustness and generalization. Although developed for space-based contexts, the dataset's motion and interaction patterns may inspire comparative studies in gravity- analog environments, such underwater settings.
17

---

Published as a conference paper at ICLR 2026
Furthermore, the semantic complexity of the captioning and QA tasks highlights challenges including hallucination and semantic inconsistency, positioning MicroG-4M as a testbed for evaluating the reliability and grounding of multimodal models in physically unfamiliar conditions.
D SAFETY AND ETHICAL DISCUSSION
MicroG-4M is developed as a research-oriented benchmark for video understanding in microgravity environments, While ethical standards were followed throughout its construction, several considera- tions are noted to promote responsible and informed use.
All real-wwele world videos sourced from publicly available materials released by official space agencies and educational institutions, containing no private or sensitive content. Astronauts are shown exclusively in professional contexts. Simulated cinematic content, while enriching visual diversity, may introduce stylistic bias that differs from real operational footage.
All captions and QA annotations created manually by annotators with domain guidance. LLMs were employed exclusively for grammatical correction and fluency enhancement, without contributing semantic generation. Nonetheless, users should remain aware of any residual stylistic biases introduced during this refinement process.
MicroG-4M intended solely for non-commercial academic research. It is validated for real- world deployment, particularly in sensitive domains such as surveillance or defense. Users are advised evaluate generalization carefully and avoid overextension of model outputs.
We encourage community feedback on potential biases, content issues, or safety risks. Future versions will include expanded validation and filtering to enhance transparency and data quality.
E
ADDITIONAL QUALITATIVE ANALYSIS
E FINE-GRAINED HUMAN ACTION RECOGNITION
provide qualitative comparisons using representative video samples to illustrate the fine-grained action recognition performance of different models in microgravity scenarios (Figure 5). These examples reveal two consistent signatures that drive the domain gap between Earth-trainch and micd models.
First, gravity-dependent person-movement classes are fragile. When the vertical reference is am- biguous decoupled from posture, the AVA-finetunel model projects microgravity poses onto terrestrial categories such as bend/bow, crouch/kneel, sit, or stand. Representative cases include: hatch crossing labeled as bend/bow (upper panel, column I); free- rree-floatng or foot-restrained standing either mislabeled as sit (upper panel, column 3; lower panel, column 1) or not detected at all (upper panel, column 4; lower panel, column 5); sideways crouch/kneel missed entirely (lower panel, last column); and microgravity sit mapped to lie/sleep due to orientation mismatch (upper panel, column 5). These observations indicate that person-movement classes that implicitly assume an upright axis are unstable in orbit.
Second, transient force direction and short-contact object manipulation are difficult. Short episodes of lift/pick up and put down are detected inconsistently or absorbed into background motion (upper panel, column 2; lower panel, column 2). The action hit (an object) receives low confidence and is suppressed by the detection threshold, yielding no action label (lower panel, rightmost column). In contrast, device-centric actions with clear visual anchors and sustained contact, most notably carry/hold (an object), are comparatively stable and predicted consistently across models in the same examples (upper panel, column 2; lower panel, columns 2-4).
E.2 VIDEO CAPTIONING
We provide qualitative examples to emphasize the unique challenges posed by space-related scenarios and the limitations of existing ofate-of-the-ar multimodal models (mPLUG-OW13, LLaVA-NeXT. GPT-4o, Gemini 1.5 Pro) accurately capturing specialized details inherent to space station environ- ments.
18

---

Published as a conference paper at ICLR 2026
Key Frame
stand, carry/hold talk to, put down put or/off clothing
Ground Truth
walk, enter,
touch talk
carry/hold
stand
operate spaceship
walk louch enter, talk to bend/bow
stand, carry/hold put put on/off clothing
carry/hold
sit operate spaceship lle/sleep, carry/hold
AVA
smoke
stand, carry/hold to, put down put orvoff clothing stand, carryhold talk put down put or/off clothing lift pitk/ickup
sit operate spaceship carryhold
MicroG-4M SLOW
walk, touch enter,
carry/hold
talk to
MicroG-4M I3D NLN
walk, enter,
touch talk to
carry/hold talk to
stand
sit operate spaceship
Key Frame
Ground Truth
watch
stand, carryhold talk to, pick up
stand carrytoold talk to
stand carry/hold
stand carrythold
crouchAkneel carryhold hill
stand, carryhold sit
stand carryholld talk to
stand carryhhold bend/boww
crouch/kneel carryhhoid
SLOW
stand carryholld
carryholld
MicroG SLOW
stand, parryholld talk pick
stand carryholl talk to
stand carryhoold
stand carryfholl
crouch/kneel carryhold hrt stand crouch/kncol carryhold hil staind
stand
MicroG-4M 120 NLN
watch
stand, carry/hold falk pick
stand carryhold
stand carryholld
stand
carryhhold
Figure 5: Qualitative results for fine-graine finengrained human action recognition in microgravity. Below the frame samples, the first row presents the ground truth labels of the actions. The second row presents the predictions of the Slow architecture fine-tuned predictions of the Slow architecture fine-tuned on the MicroG-4M dataset. The last row shows the predictions of the I3D Non-Local Network (NLN) architecture fine-tuned on MicroG-4M. For both the Slow and 13D NLN architectures, the AVA and MicroG-4M models were trained under the same configuration: a ResNet-50 backbone with 8x8 input (frame length sampling rate), pre-trane pre-trained on Kinetics-400. The 13D NLN model fine-tuned on MicroG-4M achieved the highest mAP among our baselines. Gray text denotes missed detections, while red text denotes false detections.
the AVA dataset. The third row shows the
In the first scenario (Figure 6, left), all models demonstrate significant shortcomings in capturing critical, Astronaut-specinn information and precise operational context, underscoring the difficulty in accurately describing space-station activities without access specialized annotations.
The second scenario (Figure 6, right) represents a relatively simpler context, yet models still lack precise details and fail to fully leverage the specialized information present in our ground-trute truth anno- tations. These examples illustrate the inherent difficulty in accurately modeling highly specialized and contextually rich scenarios typical of space environments, highlighting the necessity and distinctive value our carefully annotated space-oriented dataset.
Overall, these qualitative results demonstrate the critical importance of domain Nomain-specific annotation for effectively capturing the nuanced details and specialized context essential in space exploration scenarios.
19

---

Published as a conference paper at ICLR 2026
length
Length:
NASA astronaut Steven Swarson waaas- charcoo grag Tahinn, dons olue uiiltty quarter his right. the doonsay's blue handle with his left hand, and points his iFac holding sgin hand insidc while ta ng. & wuuck banes visblic in background on ovorhoadd EXCS ai snsen aaoca block ESs.
the video, NASA astronaut Suni Williams stacioned Node module, wearing backk Tubhit and andlenente She is sec and the teadmil uring aiatine hurmess systemm Iestur ng alantic cords and tncking chaing counteree bar.brrkgggeeee
mikrnngiatly
army ppequ pontrol
elong coils amphweinng the onk highteeee operaricnal anvirommen
loans
$2000 Di1
mPLUG DwtS
Caption Length 38
PPLUG
Coptfon Length:
A person seep footing in space rhclen seh iing homees and mav ng around In zecoograayyyy envinomment
oxting spacce
Interacting with
LLWV NaNT
Caption Longth: N67
Caption Length: 183
ongaging physical activily with confired that appoars bo ammuisece control od e'wnremmnn, possiay
space station, The and vidual is wearing a hamness and is seen unn ng a treaomil
appears
internationa Shat (155) The man weeeinn
shirt and blue pants, and he standing in frant a conticl of panel with vano as buttinn and switcches He seerma
with mix of suggessing ng focus card ovascular lor exerdso. sotting is possibly solent bil
propuning process ope shing cantrels, eosaityy y scjuting somatring scresens indicaating
toch nolc ical contort. The pe sonts attile cong of black too and bi sho and they are borefoce whtch is cocomoon gam settings comfort and safeey. The correct engle from movement perspectbe hamesss the ovecel strrogpeen is chniiii and controlled which is typical ford space designed for yelca toining or research.
and technolog cally advamned ment The man's posture suggesss ne is focused and enggeged in his workk The background nd filldd with the tyotcal intertor spacecraft, with onitt walls and punels, there funciioonal lyy The lighting la bright, capicall of soace anw ronment, and the zverall at almoshher hu sione of confentttaionn and purposss The man leck to and the sotting suppest that perfor ming tusk is cruc the oporation the
urruuing percon's profile they
empohesting
lighting
and
CPT
Caption Lengthi
video shiws entroman comut ranning or a treastril invilee a apacce st attern They are virammmnnt d. The astronaut enarciing sing wi with coudial for maintaninn hea tn
moy
4o
Coption Length:
dari up spsta miss ons The sumbu xings ara lildd o thangupmmant i nd rnnoumenns [usical of spavrrsationn
The viteoo shown an secroneut maade space stacion mocule, He interacting with
sege compann timeni, opening ng and inapecting emvioomment Dled with carefully liaeyy
Camini 1.5 Pro
Caption Length 151
spilying conditions
treadmal
onal
Space Station (5S) The astro haut wearing ing a blckk srint, blue shorts and a white har noss that atae icc thom to the treadmill i evenntt them from flecting away in crog lavlly crvinon ronnt The tradd sppears
Camini Pro
Caption Longth:
in relathedy narcow with various preces of couipment and wnd ling virilee A folded up vitt olject. possibly bedking n can be scen the background Blue 24 pupport str actures are frame: astronout running Thier contiaaation suggests
The video nows a man inside what ooka like a oot He wears a dark gray lower Hol
the
onay comoo ession dievvss his ooons outward He 2lo the door parbway, then further od usts som ething indiao nearthe eocning A computerrn monhor visible in the background behind he rightt phoulder various procos coulsmonn visille thouuh out the
t
blue shorts, podded, box
steady
engaged a prysical cal evercise roucine
Figure 6: Qualitative examples illustrating ground-truth captions and outputs from four state-of-the art multimodal models. The left example represents challenging scenario, in which all models fail to accurately capture detailed and precise information. The right example demonstrates a relatively simpler scenario, where model-generan captions exhibit closer alignment to the ground truth. Each caption includes the corresponding caption length (in words), with key details highlighted in the ground-truth captions.
E.3 VIDEO QUESTION ANSWERING
We present qualitative examples illustrating the performance of tfate-of-thh-art multimodal models (Gemini 1.5 Pro, GPT-4o, Video-LLaVA) on challenging Video Question-Anseen Annwerinn (VQA) tasks specifically related to space environments.
The provided examples (Figure 7) showcase diverse and challenging scenarios uniquely associated with space-based contexts. The top-left example demonstrates the complexity of accurately inter- preting multiple astronauts' actions within confined spaces, especially when occlusions occur. The bottom-left example emphasizes challenges in accurately identifying specific locations unique to space environments. The top-right scenario tests models' abilities to describe various specialized objects within densely detailed backgrounds, typical of spacecraft interiors. Lastly, the bottom-right example addresses the models' propensity for hallucination by evaluating their capacity to accurately infer details not explicitly mentioned in the visual content.
Overall, these qualitative analyses highlight significant limitations of current multimodal models in handling space-specific scenarios, underscoring the critical need for detailed and specialized annotations provided by our dataset to enhance performance and robustness in space-related VQA tasks.
F DATA FORMAT CONVERSION
To simplify frame-level annotation processing, we adapted our dataset to match the AVA format, with one modification: using frame stamps rather than timestamps. This allows direct frame indexing aligning better with our annotation scheme and reducing unnecessary conversion overhead.
20

---

Published as a conference paper at ICLR 2026
What www.erntnttnniin........
Question
Question
members doing behind Williams?
Whhtt the backgrours sctting the
CT: They are seated a posture suggesting conversation,
GT: The green sleeping bag Velcroed inside the cubicle, the white nalls, and the tool-Laden warkibnchh of Node 2.
but the content 15 not mentioned.
Gemini1.5 PRO
Gemini PRO
hooting thei back appprpppyyyyyy.. saing pio of walions theoter enuipmrntt
station You can verous couperent, weea and cummsed quaters typecall loclage
vodie belandltle aul noden
space
The two onn membor berind Willimm acp taingg pldd
The hesiground settig the vide the ste of spece station
-40
GP1-40
Haltao mantars bet an Willomee ve laster
beckgroor and dacvaloii apeoe whaze caring and holding toothbrahh
Question
Question
What module is the astronaut inside during the demonstration?
What is the purpose of the worit exercise shown in the video?
GT: International Space Japanese Experinent
Hodule kibbo'
nentioned.
Gemini11.5 PRO
namaatr tndicatia
used seartss
Gemini PRO
amsut in Dentiry moduls Uniieel Ststes
improee muude emenggs stove apectioall, namator mao shobs of a prokicidd akd bation program
wobs These
conribrr shoum
coed developing murk
40
The is mnde space statio's eebock theingthe
The wist everiber shown t dee designed het aetronouts mave shrnghh filundelll car keng dunkkon
Vitro-LLVA
Vintrr LL
Figure 7: Qualitative examples illustrating Ground-Truth (GT) answers and corresponding responses generated by three state-of-the-ard multimodal models.
F.1 DATASET METADATA
Our dataset metadata structured hierarchically into lete-ories-vvddo--evell and frame-level-to facilitate efficient retrieval and flexible querying.
Video-Level Metadata: summarizes high-level annotations and semantic context:
Video ID: Unique identifier for each video clip.
Person ID: Unique identifier for annotated individuals within videos.
Action Labels: Fine-grained human action labels.
Caption: Detailed textual description of the video content.
Question-Answer Pairs: Structured questions and answers related to video content.
Frame-Level Metadata: provides detailed annotations at individual frame granularity:
Video ID: Corresponding identifier linking video-level metadata.
Frame stamp: Temporal location of annotated frames within videos.
Person ID: Unique identifier for each individual per frame.
21

---

Published as a conference paper at ICLR 2026
Bounding Box: Pixel coordinates (xmin, ymin, xmax, ymax) for each annotated person.
Clip Type: Indicates real-world cinematic origin of video clips.
Video naming follows the "[source identifier]_ number]" format, where the identifier is the YouTube ID or film name, and the sequence number indicates its temporal position. This structured metadata approach ensures efficient integration between frame-level and vidco-level annotations, effectively supporting downstream vision-language research in microgravity environments. For comprehensive documentation of the dataset metadata, please refer the resources provided on our GitHub and Hugging Face repositories.
G SUPPLEMENTARY FOR HUMAN ACTION RECOGNITION (HAR) TASK
G.1 DATASET PARTITIONING
The dataset was partitioned at the video level following two primary criteria: (1) ensuring coverage of all action classes through greedy selection, and (2) proportional random splitting of remaining videos into training. validation, and test subsets 17:200:20 ratio). This approach avoids label leakage by maintaining video-level annotation consistency within subsets. Table 6 provides a summary of the sample-level and video-level distributions across splits.
Table 6: Train/val/test split statistics for the HAR task in MicroG-4M. Each cell reports both the absolute count and its corresponding percentage (%).
Split
Samples (# %)
Video Clips (# / %)
Train
9,273 (69.93%)
3,331 (69.99%)
Val
1,330 (10.03%)
475 (9. 99.9%)
Test
2,658 (20.04%)
953 (20.03%)
Total
13,261 (100.00%)
4,759 (100.00%)
G.2 DATASET STATISTICS
Table 7 summarizes the distribution of three broad action types, namely Object Manipulation, Person Interaction, and Person Movement, across the entire dataset, cinematic subset, and real video subset. Object Manipulation constitutes the largest category, particularly within real videos (39.42%). followed by Person Interaction and Person Movement categories.
Table 8 presents statistics regarding the number of persons annotated per video. Single-person videos dominate the dataset, particularly in real footage (92.41%) while multi-person annotations are comparatively more frequent in cinematic sources.
Additionally, Table provides a detailed breakdown of 50 fine-grained action classes. The most frequent actions across both subsets include 'stand', 'carry/hold object', and 'talk to self/person', whereas actions such as 'climb', 'take photo', and 'kiss person' are comparatively rare. Notably, the distribution of action labels exhibits pronounced long-tail pattern, consistent with Zipf's law commonly observed naturally occurring datasets, indicating the realistic and representative nature of the collected action annotations.
Table 7: Label type distribution across the full dataset, cinematic subset, and real video subset. Each cell shows the number of labels followed by its proportion (%).
Label Type
All Videos
Movics
Real Videos
Object Manipulation
4,986 (37.60%)
1,416 (38.78%)
3,788 (39.42%)
Person Interaction
4,288 (32.34%)
1,198 (32.81%)
2,950 (30.70%)
Person Movement
3,987 (30.07%)
1,037 (28.40%)
2,872 (29.89%)
22

---

Published as a conference paper at ICLR 2026
Table 8: Distribution of the number of persons per video in MicroG-4M. Each cell shows the number of videos followed by its proportion (%)
Persons per Video
All Videos
Movies
Real Videos
Single Person (1)
3,983 (83.69%)
816 (61.26%)
3,167 (92.41%)
Two Persons (2)
623 (13.09%)
395 (29.65%)
228 (6.65%)
Three or More (23)
153 (3.22%)
121 (9.09%)
28 (0.94%)
G.3 PER-CLASS AP RESULTS
Building on the aggregate results in Table 1 and Table 2. analyze per-class AP on the MicroG-4M test set. Figures 8-14 report the per-class curves with the 50 AVA-derived MicroG-4M actions ordered in Table 9.
Paired comparison. For Slow and SlowFast, provide matched plots under both training regimes, MicroG-MM-McrGG-MM and -Me--MGGO (Figures 8, 9, 10, I1). MicroG-4M fine-tuning lifts per-class AP on gravity-dependent person-movemente and transient-contace object manipulations, including lie/sleep, run/jog, sit, close (e.g.. a door, a box). enter, hit (an object), lift/pick up, pur down, work on computer, and hug (a person). The AVA AVA-MMccoGG-4M counterparts drop on the same groups, aligning with the quantitative plateau and ranking inversion discussed earlier and with the qualitative signatures that microgravity removes gravity- posture cues and shortens haje-object contact.
Across remaining backbones. For C2D, I3D, MViT family, and X3D, MicroG-4M fine-tuning generally improves the posture-sensitive actions and short-contact manipulations above, yet several classes remain hard across architectures: pull (an object), push (an object), close (c.g.. a door, a box), take photo, text on/look at cellphone, fight/hit (a person), and take (an object) from (a person). These actions share low visual salience or momentary cues such as small handheld objects, brief hand-hu hand transfers, global camera/scene drift in spacecraft cabins, which limits per-class AP even after in-domain training (see Figures 12, 13, 14, and 15).
Backhone-specific observations. MV1Tv1, MViTv2, and X3D exhibit comparatively limited gains after MicroG-4M fine-tuning. Two factors are consistent with this trend. First, temporal windowing and sampling may not balance long drift with short contact, which requires both extended context and reliable short-term discrimination. Second, inductive biases differ: backbones with stronger local encoding and non-local augmentation align better with microgravity-speciie contact cues than global nnteention-dominee designs with weaker local bias or very lightweight CNNs without non-local pathways. These patterns visible in the per-class plots for X3D and MVIT (Figures 14 and 15), and they are consistent with the aggregate results Tables and 2, which together explain the test mAP plateau and indicate that in-domain fine-tuning recovers part of the gap while several action categories remain challenging in orbit.
postures and
G.4 CROSS-DATASET VQA DENSITY AND DESIGN RATIONALE
To contextualize our choice of six QA pairs per 3-second clip (2 QA/s), we compare MicroG-4M with representative video-VQA corpora terms of clip granularity, QA density, and question types. This unified view allows controlled discussion of annotation budget and reasoning load across datasets with heterogeneous clip lengths.
MicroG-4M targets short clips high QA density (2.00 QA/s), comparable to MSVD-QA and exceeding other widely used corpora. A fixed per-elip budget supports semantic diversity while avoiding redundancy typical of long-form settings. Our QA taxonomy balances foreground actions, spatial context, entiry/attribute grounding, and temporal/causal reasoning. and explicitly includes an unanswerable option reduce hallucination.
Because question generation protocols and scoring schemes vary across datasets, cross-corpis density should be interpreted comparability aid rather than difficulty metric. The choice of six QA
23

---

Published as a conference paper at ICLR 2026
Table Complete summary of action class distribution across all videos, movies, and real videos (sorted by action ID).
Count Movies
Action Name
All Videos
Real Videos
bend/bow (at waist)
26
14
3
crouch/kneel
20
2
18
5
fall down
10
I
9
get up
25
4
21
jump/leap lie/slecp
20
17
4
13
martial art run/jog 10
9
18
0
18
12
0
12
sit
252
239
stand
3218
698
14
walk
369
75
294
17
carry/hold object
3126
549
2577
20
climb (e.g.. mountain)
close (door/box)
24
cut
9
0
26
dress/undress clothing
31
9
22
27
drink
43
5
38
operate spaceship
20
16
eat
45
30
enter
68
5
63
34
hit object
33
3
30
36
lift/pic up
188
10
178
open (window/door)
13
play musical instrument
3
0
3
43
point to object
323
2
321
45
pull object
32
19
13
46
push object
24
8
put down
138
read 48
15
14
56
take photo
2
0
2
57
text/look at cellphone
7
0
throw
59
touch object
353
136
217
60
turn screwdriver
17
9
61
watch TVunspecified
346
316
30
62
work computer write
110
67
43
3
64
fight/hit person
27
26
65
give/serve object
46
20
26
66
grab person
41
31
10
67
hand clap
3
3
hand shake
2
69
hand wave
140
44
96
70
hug person
16
13
3
72
kiss person
1
listen person
135
76
push person
3
2
78
take object from person
15
10
79
talk to self/person
3131
504
2627
80
watch person
713
625
88
24

---

Published as a conference paper at ICLR 2026
Slow 4x16
APO3
6.-
Cooy tortee
Class Name
Slow 8x8
1.0
0.8
Arg.- 50
0.6
0.4
02
50
Seroia Food
rool
aoogerae pag
conidd the podees
Class Name
Figure 8: Per-class AP (%) on the MicroG-4M test set for Slow Feichtenhofer et al. (2019). Training regime:
MicroG-4M fine-tuning. AP@0.5 over the 50 MicroG-4M action classes; class order matches Table 9. Legend uses L x S to denote clip length x sampling stride; settings match Table 1.
Table 10: Cross-datased VQA density and scope. Statistics are compiled from the original dataset reports. "QA/sec" denotes (Avg. QA/Clip)(Cliip length in seconds). This comparison contextualizes choice of six QA per 3-second clip in MicroG-4M.
Dataset
Clips
QA pairs
Avg. QA/Clip
Clip Lea.
QAAscc
QA Types
MSVD-QA Xu et al. (2017)
1,970
50,505
25.64
10
2.56
Wh-typs
MSRVTT-QA Xu (2017)
10,000
243,680
24.37
15
1.62
Wh-type
TGIF-QA Jang (2017)
56,720
103.919
1.83
Task-hased
TVQA Lei et al. (2018)
21,793
152,545
7.001
76
0.09
Wi-type + Temporal
ActivityNa-QAA
(2019)
Motion, Sputial, Tomporal
MovieQA Tapaswi et al. (2016)
6,771
6,462
0.95
200
0.044
Stary compreheusion
VideoQA Yang (2003)
18,100
174,775
Wh-yype YesiNo
MieroG-4MM (Oues)
1,238
7,428
6,00
2,00
Wh-type Fveggrend/Rakgrrmmn Fine-Coarse- motion, Ideatity. Temporal, Causal
pairs per 3-second segment is thus motivated by controllability: it preserves clip-level coverage and evaluation stability while enabling apples-to-apples studies of domain shift in microgravity, independent of confounds from variable clip durations or QA volumes.
G.5 EVALUATION WITH GEMINI 2.5 PRO ON THE HAR TASK
We additionally evaluate a strong multimodal large language model. Gemini 2.5 Pro Comanici et al. (2025), on the MicroG-4M human action recognition (HAR) task. We restrict the label space the
25

---

Published as a conference paper at ICLR 2026
Slow 4x16
0.-
AP:0500
0.
of
Jone ansh -iin poooo
Chass you
metda fat cand
moll
do cre
Class Name
Slow 8x8
1.0
0.8
AP825 50
0.9
0.1
00
Selou
Sprii aos ve soop
4e und
you
Class Name
Slow 32x2
1.3
APO55 50
0.
6.1
Oolons
Scus ao "p sopp up
Clen dan
cip can taid
1 with
soreee of not (niii
du ion
Class Name
Figure 9: Per-class AP (%) on the MicroG-4M test set for Slow Feichtenhofer et al. (2019), Training regime: AVA fine-tuning with zero-shot evaluation on MicroG-4M. AP@0.5 over the 50 AVA-derived action classes. Legend uses L x S to denote clip length x sampling stride; settings match Table 1.
50 AVA -deiiedd action classes used in our HAR experiments and formulate HAR as a multi-label prediction problem over these 50 classes. For each 3-second clip and tracked person, Gemini is provided with a per-person video in which the target individual is highlighted by a red bounding box, together with prompt listing all 50 action IDs and names. The model is instructed return list.
26

---

Published as a conference paper at ICLR 2026
SlowFast 4x16
AP
Class Name
SlowFast 8x8
1.0
0.8
Arga 50
0.6-
02
50
souu with
Sproi Food
Corouet coip rrng
sotte de Soaaee
rool
con the gldee
Class Name
Figure 10: Per-class AP (%) on the MicroG-4M test set for SlowFast Feichtenhofer et al. (2019). Training regime: MicroG-4M fine-tuning. AP@0.5 over the 50 MicroG-4M action classes; class order matches Table 9. Legend uses L x S to denote clip length x sampling stride; settings match Table
Table 11: HAR performance of Gemini 2.5 Pro on the MicroG-4M test set, evaluated on the 50-way multi-label action space with macro-averaging over classes.
Test Result
Model
mAP (%)
Fl-score (%)
Recall (%)
AUROC (%)
Gemini 2.5 Pro Comanici et al. (2025)
14.07
25.44
46.41
71.43
of action IDs that are clearly performed by the highlighted person. We encode both ground-truth annotations and Gemini predictions as 50-dimensional {0, 13 indicator vectors over the action label space and compute macro-averaged metrics over classes. Since Gemini does not expose calibrated per-class probabilities in this setting, the binary indicators are used both as decisions (for Fl-score and recall) and surrogate scores (for mAP and AUROC), and the ranking metrics should therefore be interpreted as conservative lower bounds. The resulting global performance is summarized in Table 11.
27

---

Published as a conference paper at ICLR 2026
SlowFast 8x8
3.0
04-
AP
0.6
Canta
Levu
Locger 199
2000
Socl in coslee aa ressiaee
Poom
lng long
Oniio
Class Name
SlowFast 32x2
1.0
0.8
0.4
6.1
do in Arctoon
Wey un
Tomas p0e loop
paar then
poly poud
Coh aorkl to
th saclyye 14
fune
Class Name
SlowFast 16x8
500
63
Fo (roude
pold on oo
yeo cooups pond
sop
Class Name
Figure 11: Per-class AP (%) on the MicroG-4M test set for SlowFast Feichtenhofer et al. (2019). Training regime: AVA fine-tuning with zero-shot evaluation on MicroG-4M. AP@0.5 over the 50 AVA-derived action classes. Legend uses L x S to denote clip length x sampling stride; settings match Table 1.
28

---

Published as a conference paper at ICLR 2026
C2D 8x8
AP
6.-
Coster PN
Class Name
C2D NLN 8x8
1.0
0.8
Arga 50
0.6
0.4
07
attte de Soree
50
Do the le Ricoip
Conoutt usic 99
creae uni Chi 1901
rooll
the contld
Class Name
Figure 12: Per-class AP (%) on the MicroG-4M test set for C2D Feichtenhofer et al. (2019). NLN denotes the non-local augmentation of the backbone. Training regime: MicroG-4M fine-tuning. AP@0.5 over the 50 MicroG-4M action classes; class order matches Table 9. Legend uses L x S to denote clip length sampling stride; settings match Table 1.
G.5.1 IMPLEMENTATION DETAILS
The evaluation procedure for the Human Action Recognition (HAR) task consists of three distinct stages, detailed below.
Per-person Clip Generation. The process begins by leveraging the provided spatio-temporal annotations, which specify the bounding box coordinates, frame index, and unique identifier for each person across the video dataset. For every unique video-person person pair, corresponding short video clip constructed. Frames are decoded from the raw video files, and the target actor's bounding box in each frame is highlighted and annotated with the person identifier. These localized per-person clips are generated to ensure that the model consistently focuses its analysis on a single actor within the video.
Gemini 2.5 Pro Inference. The HAR label space encompasses 50 action classes. Each generated clip is uploaded to Gemini 2.5 Pro, and the model is prompted with predefined text containing the list of all 50 action IDs and their descriptions. The instruction is to return a JSON list of the action IDs clearly performed by the highlighted subject. The model's 's textual output is subsequently parsed into a set of predicted action identifiers, with invalid or out-of--ocabull labels being rigorously filtered.
29

---

Published as a conference paper at ICLR 2026
I3D 8x8
1.5
0.8
AP
Class Name
I3D NLN 8x8
1.3
0.8
Arga 50
0.6-
0.4
02-
50
Orivv
woll Foed
crocar uni Chil 1901
Coroest usic
moal
the coniid
Class Name
Figure 13: Per-class AP (%) on the MicroG-4M test set for I3D Carreira & Zisserman (2017). NLN denotes the non-local augmentation of the backbone. Training regime: MicroG-4M fine-tuning. AP@0.5 over the 50 MicroG-4M action classes; class order matches Table 9. Legend uses L x S to denote clip length sampling stride; settings match Table 1.
Metric Computation from Binary Indicators. Ground-truth HAR annotations are aggregated from the action label file, establishing set of true labels for each video-person instance. Both the ground-truth and the predicted label sets are then converted into 50-dimensional binary indicator vectors. Since the inference configuration was constrained to return only action IDs (a list output), the Gemini model does not provide calibrated per-class prediction probabilities. Consequently, these {0, 1} indicators are directly employed as the decision variables for computing Fl-score and Recalls, then as surrogate scores for calculating mAP and AUROC. Macro- metrics are obtained by averaging the respective scores over the 50 action classes. The global metries reported in Table are aggregated over the entire MicroG-4M test set.
Observations. Compared with dedicated video architectures trained on MicroG-4M (Table 1), Gemini 2.5 Pro achieves noticeably lower mAp and AUROC, while exhibiting relatively high recall, which indicates a tendency to over-predict actions. Together with the cross-domain transfer results in Tables 2 and 3, this highlights substantial distribution gap between terrestrial benchmarks and microgravity videos. Both conventional video encoders and a state-of ofate-of-thh-ar multimodal language model trained predominantly on terrestrial data struggle to bridge this gap the HAR task, suggesting that segrogrrvity-speccfii dynamics and context cannot be handled by generic action priors or textual knowledge alone.
30

---

Published as a conference paper at ICLR 2026
MVITv1 16x4
AP
06-
Costic
PN
Class Name
MVITv2 16x4
1.0
0.8
Ar 10
0.6
0.4
02-
50
Orivvs
corbant
Coocet moal
connact und (hii (901
dilion cortal the uoii
Class Name
Figure 14: Per-class AP (%) on the MicroG-4M test set for MViTv1 Fan et al. (2021) and MViTv2 Li al. (2022). Training regime: MicroG-4M fine-tuning. AP@0.5 over the 50 MicroG-4M action classes; class order matches Table 9. Legend uses L x S to denote clip length x sampling stride; settings match Table 1.
H SUPPLEMENTARY FOR VIDEO CAPTIONING AND VISUAL QUESTION ANSWERING (VQA) TASK
H.1 ADDITIONAL ANNOTATION EXAMPLES
Figure 16 illustrates two further representative examples of caption and VideoQA annotations in MccroG-4M. For each 3-second clip, annotators provide a dense caption that jointly encodes astronaut identity, cabin location, relevant equipment, and fine-grained hund-body-objec interactions, followed by six visually grounded QA pairs that cover entity, attribute, action, spatial context, and temporal or causal reasoning, with explicit Not mentioned option when appropriate.
H.2 COMPUTATIONAL OVERHEAD
Table 12 reports warm-start latency and GPU memory usage across models. The observed efficiency differences consistent with architectural factors such as video encoder capacity, tokenizer design, and the number of input frames, and can serve as a practical reference when choosing models under different computational budgets.
31

---

Published as a conference paper at ICLR 2026
X3D 13x6
1.5
0.8-
AP
68
Custic PN
Loc Lidion Jest SostAA
Class Name
X3D 16x5
1.0
0.8
Ar (00
0.6
0.4
02-
50
Coly andc
coniar uni Chil cif1
moal
the cortal
Class Name
Figure 15: Per-class AP (%) on the MicroG-4M test set for X3D Feichtenhofer (2020). Training regime:
MicroG-MM fine-tuning. AP@0.5 over the 50 MicroG-4M action classes; class order matches Table 9. Legend uses L x S to denote clip length x sampling stride; settings match Table 1.
Table 12: Computational overhead on MicroG-4M under warm-start conditions. We report median latency (p50) per clip and per sampled frame, together with peak GPU memory. #f denotes the number of input frames used during model inference.
Model
Precision
Caption
VQJ
Peak GPU Mom (Gill)
p50 (siclip)
pS0 (uus/ / sampled (raamedddd-rae)
pEu (s/clip)
p50 (ms sampled -romm))
Vadoo- LaVV (2004) Qwen2 5-VL, Yang et (2004)
fpt6 br16
283.00
52,20
J4.98
0.48
15.87
35.81
1.23
Tasser2 Recap 7B You at al. (2025)
br16
16
1,20
93.65
0.52
48.62
16,50
I PROMPT TEMPLATES FOR VIDEO CAPTION AND VQA ANNOTATION
This section provides the prompt templates used during caption refinement and VideoQA annotation. All prompts are applied after human annotators have written the initial captions and consulted the domain resources described in Section 3. In all cases, Multimodal Large Language Models (MLLMs) are instructed not to introduce new facts beyond what is visually or contextually supported.
1.1 CAPTION REFINEMENT PROMPT
We use the following template to refine human-written captions, conditioning on four uniformly sampled keyframes while preserving all factual content.
32

---

Published as a conference paper at ICLR 2026
Caption
Caption
the vidco astronaut Williams fooking d: altiock floniing
shows astronaut trachorue minmogggaay Indide
red patch
how's codlar wnch
khald
freely in the micorra centt mm romment shw reptains and cemoonttrone cerrain
the ISS Jagansse Experiment Module (kibo) lim uses purp tosthi ree endt
festures apareeus witiin
coner netteet m.c.lll..
aibee sester EoLaD with strnsing Tubee caneluly squeer ng waner droppen
spacessuli wirk the background foatures various pioces commoment and ai window through whichhie interior the space station
attecc
objects itts ignnly showiig space udi costtaioon challe sges The backgrou nd di spplgss stotin couloment incuding TEHOIS got sondows verroopaee conmrtlied rectuiee for twee leaks seeigh seeightleemees.
neailly amoggeed
demorrttion the preceston
qJi Winr un the astronsut Hoscing nece the sutoocc?
VQA
NASA onaut Suni Williomss
Canadian art astronaut of Chrit Hadhlle.
A2: white with black coiarr lhail shorte
Wirut moth dows Huulfoll mater to tosthbrus unl
Q3t How is Suni williams holr positines in' vidco?
brush using sher water couch with ditnking ging tu000
A3t hall flaating freay the microgter unviiormmet Q4c the aparanaci sntmaai
Winst challenges
thevicee??
dritting
chlllenges
of performing huiie in miprogoo
the nelmer has beige
Q4i W/ cat can soen in the tociggo did of the demonstittonn
QS+ Withae is viaibll the background of vizee?
Acc Stee on equip ment induding TEMIPI mas w windoux sc racks, and rroco red arrope cantees.
Ads Varioua ecuipmeent dow ahowing interior
soveral
viruut catto
Haefwikt
williams
Purple
the bow antached the spocesu
When of day the demoratration taking place?
Adc She gives light pat box
AR Not mensionas
Figure 16: Examples of video caption and VQA annotations in MicroG-4M. Each panel shows short clip, its corresponding caption, and six associated QA pairs, illustrating the level of detail and reasoning coverage used in our annotations.
[Instruction]
You are assisting with caption refinement for short astronaut videos.
You are given:
(1) four keyframes sampled from a 3-second video clip, and
(2) a human-written caption describing 3-second video clip
Your task:
Use the visual evidence from the frames as the primary reference. Improve grammar and fluency.
Increase clarity and readability.
Simplify overly complex sentences into clear, concise style.
Opticnally vary wording slightly to avoid repetition.
DO NOT add new facts or speculate beyond the caption and context.
DO NOT change any factual details.
Return a refined caption with similar length to the input, preserving all factual
[Input frames]
Frame_l: {IMAGEL) 1:
Frame_2: (IMAGE 2
Frame_3: {IMAGE_3
Frame_4: (IMAGE
[Input caption]
(CAPTION_TEXT)
[Context] (EXTT
[Output]
Ref fne caption:
33

---

Published as a conference paper at ICLR 2026
1.2 VQA CANDIDATE GENERATION PROMPT
Given a refined caption and four keyframes from each 3-second clip, we generate diverse ques- tion-answer candidates that cover multiple reasoning dimensions. The following template is used:
[Instruction]
You are generating visual question-ant answer answer (VQA) pairs for 3 3-second astronaut video clip in microgravity.
You are given:
(1) four keyframes representing the video, and
(2) a refined caption that accurately summarizes the clip.
Treat the caption and keyframe images as the only reliable information
about the video.
Your tasks:
1. Propose 8-10 diverse, natural, and meaningful Q&A pairs that are answerable from the visual content in the frames, supported by the caption.
Questions must be visually grounded in what can be seen.
2. Cover mix of:
Standard Wh-forms (who, what,, where, how, when, why).
Foreground vs. background elements.
Coarse vs. Mine-grained actions.
Identity, location, and equipment. Temporal or causal relations.
At least one question that cannot be answered from the frames and caption; its answer must be exactly "Not mentioned".
3. Use varied question types (Who / What Where / When Why / How) and avoid repeating the same content.
4. Do NOT rely on sound, speech content, or subtitles.
5. Do NOT rely internal intentions
emotions that not visible.
6. DO NOT rely on external world knowledge beyond what is evident in the frames and caption.
[Input franes]
Frame 1: { [IMAGE_ 1;
Frame { (IMAGE 2
Frame_3: { [MMGE__ 3)
Frame_4: [IMAGE_
[Refined caption]
[CAPTION_TEXT]
[Output]
Return ONLY the selected Q&A pairs in the following format, with extra commentary:
Q: [question]?
A: [answer].
1.3 VQA FILTERING AND TOP-6 SELECTION PROMPT
Candidate QA pairs are then filtered and ranked, and six diverse, visually grounded questions are selected.
runttooction)
34

---

Published as a conference paper at ICLR 2026
You are given refined caption and 1ist of candidate question-answer pairs for a 3-second astronaut video clip.
Your tasks:
1. FILTER OUT any QA pairs that:
Rely on sound, speech content, or subtitles.
Require private intentions or emotions not visible in the clip.
Depend on external world knowledge not suggested by the caption.
Are duplicate or near-duplicate questions.
2. RANK the remaining QA pairs by:
Logical consistency between question and answer.
Linguistic fluency and clarity. Semantic relevance to the caption.
Informational value (how much useful visual information is tested).
3. SELECT final QA pairs that:
Cover diverse aspects (identity, action, body motion, spatial context, background, temporal/causal). Are visually grounded in the clip.
Optionally include ONE unanswerable question, whose answer must be exactly "Not menticned".
[Input]
Refined caption: [CAPTION_TEXX)
Candidate QA list: (JSON LIST OF OA)
[Output]
Return a JSON list of the selected QA pairs in their final form, each with:
Q: [question]?
A: [answer].
ADDITIONAL DATASET VISUALIZATIONS
Figure 17 presents collage of sampled frames provide quick visual impression of MieroG- 4M, illustrating the dataset's core characteristics. The montage spans different crews, stations, and activities.
MicroG-4M's coverage is designed to approximate the breadth of everyday microgravity operations by including footage from both major contemporary orbital platforms and relevant training/simulationn environments. Specifically, the dataset incorporates clips from orbital platforms, including various distinctive pressurized modules of the International Space Station (ISS) (e.g., Node modules, Cupola, Kibo/JEM, Columbus, Destiny/US Lab, BEAM) and the Chinese Space Station (Tiangong). Beyond in-orbit footage, we include frames from simulated and conceptual microgravity contexts, such as parabolic flights and specialized film sequences, which broaden the scope of training and conceptual scenarios, particularly those used for extra-vehicular activity (EVA) preparation. This collection, sampled across different expedition periods and mission timelines, captures significant temporal and operational spread. Furthermore, the gallery highlights crew demographics and apparel consistent with documented daily life and work. Finally, the frames exhibit considerable variability, encompassing diverse illumination, camera viewpoints, aspect ratios, resolutions, group sizes, and motion types, covering both intra-vehicular tasks and EVA-related contexts.
35

---

Published as a conference paper at ICLR 2026
Figure 17: Representative frames from MicroG-4M.
36


Reviews:
[{"review_id": "UJsD2feISv", "reviewer": "Reviewer_baMz", "summary": "This work introduces MicroG-4M, the first large-scale dataset specifically curated for human action recognition and vision-language understanding in microgravity environments. MicroG-4M includes 4,759 clips covering 50 actions, 1,238 context-rich captions, and over 7,000 question–answer pairs on astronaut activities and scene understanding. MicroG-4M aims to support three core tasks: fine-grained multi-label action recognition, temporal video captioning, and visual question answering, thereby enabling a comprehensive evaluation of both spatial localization and semantic reasoning in microgravity contexts.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-written and easy to understand.\n\n2.This work introduces MicroG-4M dataset, which is a valuable contribution to fill the gap of video understanding benchmarks under microgravity scenarios.", "weaknesses": "1.One concern for this work is the technical contribution. From my point of view, the major contribution of this work comes from its data collection and organization, while the methodological contributions are missing, i.e., there is no specifically designed baselines for video understanding under microgravity scenarios nor novel insights from this work. This decreases the overall contributions of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4FJbLOmVQf", "reviewer": "Reviewer_fpxM", "summary": "The paper introduces MicroG-4M, the first benchmark dataset designed for spatio-temporal and semantic understanding of human activities in microgravity environments. It addresses a critical gap in current vision research, as most existing datasets for video captioning and action recognition are recorded on Earth under normal gravity, whereas microgravity significantly alters human motion, interactions, and visual semantics. Constructed from real-world space mission footage and cinematic simulations, MicroG-4M contains 4,759 3 second video clips at 30fps covering 50 actions, 1,238 captions, and over 7,000 question–answer pairs centered on astronaut activities and scene understanding. The dataset supports three core tasks: fine-grained multi-label action recognition, temporal video captioning, and visual question answering.", "questions": "1. For video captioning and VQA, the authors mention using large language models (LLMs) but do not describe the prompts or prompting strategy employed. Providing information about the prompts, including their format, instructions, or examples, would improve reproducibility and allow readers to better understand how LLMs contributed to annotation quality.\n\n2. It is unclear how video input is processed to generate captions. Do the authors first extract keyframes, or do they read video frames sequentially? If keyframes are used, the authors should specify the algorithm or criteria for keyframe selection. Additionally, the caption generation process using Visual-Language Models (VLMs) or Multimodal Large Language Model (MLLM) is not described in detail, clarifying whether captions are generated per frame, per keyframe, or for the entire clip would improve reproducibility.\n\n3. Employing VLMs may introduce computational overhead, especially for high-resolution frames or sequential multi-frame processing. For practical deployment, it would be helpful if the authors reported the hardware configuration (GPU type, CPU, RAM), memory usage, and average execution time per frame or video clip. This information would provide readers with a clearer understanding of the method’s efficiency and scalability.\n\n4. The authors state that each three-second clip contains six QA pairs; however, it is unclear how these questions were generated or selected. It would be helpful to clarify whether the questions were derived directly from the video content or generated based on the captions. The paper should also describe the question selection process, including examples of question types and how balance was maintained across reasoning categories such as temporal reasoning. The author could explain such as Six QA pairs covering different aspects: (1) Identity (“Who is the astronaut?”), (2) Action detail (“What does she do with her hands?”), (3) Body motion (“How do her head and shoulders move?”), (4) Spatial context (“What items can be seen behind?”), (5) Static background recognition (“What remains stationary?”), (6) Unanswerable / implicit reasoning. However, the authors do not indicate whether such systematic consideration guided their question design. A clearer explanation of the QA generation and selection methodology would greatly enhance reproducibility and transparency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  The contribution of MicroG-4M is highly interesting and original, as it introduces the first large-scale benchmark for understanding human actions, captions, and question answering in microgravity environments. The dataset offers significant potential for advancing research in vision-language modeling, domain adaptation, and embodied AI under extreme physical conditions.\n\n2. The strength of this paper lies in the detailed presentation of dataset statistics and discussion with limitation. The authors provide a clear breakdown of the distribution across broad action types, the number of persons per clip, and fine-grained action frequencies, highlighting important patterns such as the dominance of single-person clips and the long-tail distribution of actions. Additionally, the per-class AP results convincingly demonstrate the value of MicroG-4M for microgravity-specific action recognition, while the high-density, VQA annotations further support rich semantic understanding.", "weaknesses": "1. In the collection methodology section, the author’s writing style is precise, formal, and research-oriented, making it suited for submission in conference. However, it leans toward being dense and information-heavy, which could benefit from slight simplification or the inclusion of visual aids (such as tables or flow diagrams) to enhance clarity and readability.\n\n2. The author does not explicitly define the categories “Object Manipulation,” “Person Interaction,” and “Person Movement.” Instead, it appears that the reader is expected to infer their meaning from common sense or from the constituent fine-grained actions. It would improve clarity if the author briefly described each category with examples. For instance, \"Object Manipulation could be defined as actions where a person interacts with objects in the environment, including picking up, carrying, holding, pushing, pulling, operating equipment, or using tools. Examples include carry/hold object, push object, operate spaceship, or using a computer. In microgravity, these actions are particularly important because the dynamics of motion and object handling differ from those on Earth.\"\n\n3. While the authors state that all captions and QA annotations were created manually by annotators with “domain guidance,” they do not specify what type of guidance was provided or give concrete examples. It would be helpful to clarify whether this guidance included official space agency documents, astronaut manuals, mission reports, or expert review, and to briefly describe what information from these sources was used (e.g., spacecraft layout, standard operating procedures, or typical astronaut activities). Providing such details would improve transparency and help readers better assess the quality and reliability of the annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C8F8gbGZEN", "reviewer": "Reviewer_Dmaz", "summary": "This paper proposes a new benchmark on video understanding. \nThe benchmark address the novel scenario of astronautic environment in space.\nThe benchmark contain 5k clips of short videos (3 seconds) with rich annotations. \nThis paper then benchmark state-of-the-art models using their dataset, results suggest the evaluated models do not work as good as they are in normal earth-based videos.", "questions": "This paper is clearly written, thus I don’t have questions regarding clarity. \n\nI expect more qualitative results like Figure 1, 3 and 4. \n\nThe videos.zip in the supplementary does not work for me.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper addresses a novel scenario for video understanding. \n\nThe paper is well motivated. \n\nThe paper clearly documented the data collection pipeline.\n\nThe evaluation setting of baseline methods are also clearly written.", "weaknesses": "The main issue with this paper is that its distinction from the Earth-Video benchmark is not clearly presented. \nThe reported performance indicates that the benchmark is difficult, but it is unclear whether any unique aspects of the astronautic videos contribute to this difficulty. \nCurrently, the presentation of experiment section 5 feels like “just another video benchmark”, as it does not provide much surprising findings. \n\n\nRegarding the baselines of human action recognition task, I’m curious whether the authors have tried VLMs like Gemini 2.5 Pro? I feel advanced VLMs like Gemini 2.5 Pro may be able to answer this. It would be nice to have them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)