# Continuous Camera-Based Premature-Infant Monitoring Algorithms for NICU
## 3 Methodology
### Top-Level Classification Block
#### 3.1.1 Feature Extraction and Object Detection  
The Top-Level Classification Block has the following output categories:  
1. Baby is present and respiration-like signal is detected. 2. Baby is present showing intensive motion, interpreted as random and frequent self-motion. 3. Caring or other intervention happens. 4. No motion or minimal motion can be found in the incubator, but the baby is detected. 5. The baby is not detected in the scene, empty incubator. 6. Multiple subcategories collecting unacceptable camera image quality and possible errors: low light conditions, blurry view, camera image is saturated, consecutive frames do not differ from each other. The classification uses a set of features extracted from the resized input stream. The input RGB frame size is \([500,500,3]\), the frames are resized by linear interpolation and converted to grayscale format and collected into a stack of \([w,h,N]\) size, where \(w=128\), \(h=128\) are the width and height, and \(N=120\) is the number of frames, which corresponds to 6 s at 20 frame-per-seconds. We originally tested about 30 features, and found 14 with the largest information content (\(\textbf{F}=[F_{1},..,F_{13}]\)). The selected features belong to four groups:  
1. _Intra-frame statistics._ Mean value of the image brightness and mean value of the standard deviation of each frame in the stack: \[F_{1} =\frac{1}{N}\sum_{i=1}^{N}\overline{X_{i}^{b}},\] (1) \[F_{2} =\frac{1}{N}\sum_{i=1}^{N}\sigma(X_{i}),\] (2) where \(X_{i}\) is the \(i\)th grayscale frame of \([w,h]\) size, \(X_{i}^{b}\) is the HSV brightness of the \(i\)th RGB frame, \(N\) is the number of frames in the stack, the vinculum denotes the mean value of all the pixels (i.e., spatial average) and \(\sigma(\cdot)\) is the (spatial) standard deviation operator. 2. _Inter-frame statistics._ Difference is calculated for each frame pair and their absolute value is averaged: \[F_{3}=\frac{1}{N-1}\sum_{i=2}^{N}\overline{|X_{i}-X_{i-1}|}\] (3) In addition, dense optical flow is calculated for each consecutive frame using the Horn-Schunck method [10]. We tested multiple methods and selected this one, because it accurately finds larger displacements (like moving limbs), and can still follow smaller movements (like chest or back). The magnitude of the optical flow is averaged for each frame pair in the frame stack, providing the next feature value \(F_{4}\)Statistics of individual pixel historyIn this group simple statistics are calculated for each pixel separately as 1D vectors along the time dimension. The first such feature is the maximum of the sample (temporal) standard deviation (\(s(\cdot)\)) of the pixel vectors:  
\[F_{5}=\max_{x,y}\{s(X)_{x,y}\} \tag{4}\]  
where \(1\leq x\leq w,1\leq y\leq h\) are the coordinates of the pixels in a frame. The next two features (\(F_{6},F_{7}\)) are the minimum and maximum value of the one-sample _Kolmogorov-Smirnov test_[34] of the pixel series. This test quantifies how the individual pixel value probability distribution differs from the Gaussian null-distribution through the time window of the stack. To express the different rate periodic motions in the scene, two _frequency domain_ ratios are calculated as well. For each pixel vector \(i\), an \(N\) length DFT is calculated resulting in \(S_{i}(f)\) the power spectrum. \(F_{8}\) characterizes the mean value of the low frequency energy component (Equation (5)) from 5 to 40 Hz discounting the near DC components. \(F_{9}\) is the mean value of the signal-to-noise ratio of the breath frequency band of 40-120 BPM over the high frequency noise in the 120-200 BPM span (Equation (6)). The choice of the frequency bands originates in the physiologically relevant range (40-120 BPM) and out-of-band frequencies. According to the measurements, the upper motion energy band is flat and mostly composed of the data acquisition and instrumental noise, while the lower band is related to the movements of the infants.