## 3 Experimental Setup  
The dataset was collected in cooperation with the Department of Obstetrics and Gynecology at Semmelweis Medical University. The data collection system [11] consisted of a conventional RGB camera (Basler acA2040-55 uc) equipped with a near-infrared (NIR) night illuminator, enabling recording during both day and night without disturbing the infants. Videos were recorded at 20 frames per second with a resolution of 500 x 500 pixels and stored using a lossless compression format to preserve all relevant visual details for post-processing.  
Physiological signals were acquired in parallel using Philips IntelliVue MP20/MP50 patient monitors. These included photoplethysmographic (PPG) pulse signals, skin resistance, and blood oxygen saturation. All signals were synchronized with the video recordings using NTP-based time alignment. Custom-developed software (written in Python 3.11.13) was used for data acquisition, storage, and annotation.  
In total, 402 h of video and physiological data were collected from neonates (primarily preterm at gestational ages 32-38 weeks) in neonatal intensive care units. More details from the population of the recorded babies and their properties can be found in Table 1. After excluding caregiving periods and empty incubator intervals, 108 h of usable recordings were retained. Behavioral states were annotated manually by three trained reviewers with expertise in neonatal care, following the NIDCAP system [24; 25; 26]. Each reviewer annotated a distinct, non-overlapping subset of the recordings. All reviewers received standardized training to ensure consistency in applying the annotation criteria. The six behavioral states were the following: (1) deep sleep, (2) light sleep, (3) drowsy, (4) quiet awake, (5) active awake, and (6) highly aroused/crying. For the purposes of this study, we merged the three awake-related classes (4-6), including crying, into a single \"awake\" category. This decision was motivated by the primary clinical objective: to reliably distinguish between sleep and wakefulness. Our medical collaborators emphasized that the most clinically meaningful information lies in understanding how much time the infant spends in each major state--awake, drowsy, light sleep, or deep sleep--rather than in differentiating between subtle subtypes of wakefulness. In addition, we aimed to incorporate physiological data, particularly pulse rate and derived metrics. However, during episodes of crying or active wakefulness, the accuracy of PPG measurements is often compromised due to sensor displacement or motion artifacts. For this reason, it was methodologically sound to focus on more stable, quieter states--such as drowsiness and sleep phases--where physiological signals are more reliable. Moreover, this study was conducted as a pilot, with a focus on feasibility and clinical alignment. In this context, reducing label granularity within the awake class helped mitigate annotation ambiguity in short observation windows and facilitated more robust model development. Future work will aim to refine and expand the label taxonomy, particularly within the awake state, as data volume and annotation reliability increase.