# Acute Respiratory Distress Identification via Multi-Modality Using Deep Learning
## 3 Proposed Model
### Data Pre-Processing Module  
For our experimental study, we use data collected from Sainte-Justine Hospital in Montreal, Canada. The data are captured using Microsoft Azure sensors, which simultaneously record RGB and depth information. The RGB data are captured using a 12-megapixel sensor, while the depth data are captured using a 1-megapixel sensor. The depth videos are  
Figure 1: Illustration of the proposed network architecture for detecting acute respiratory distress, featuring the integration of RGB and depth temporal visual data through identical 3D convolutional neural networks.  
recorded at resolution of 512 \(\times\) 512 in NFOV binned mode. In this mode, the sensor has an operational range of 0.50 to 5.46 m. The physical pixel size is approximately 0.0087 mm at 1 m. However, the physical pixel size varies depending on the distance to the object. RGB and depth videos were not spatially aligned and as they were recorded at different resolutions. For instance, the RGB videos have a resolution of \(1080\times 1920\) and a depth have \(512\times 512\).  
To address this issue, we first align the RGB and depth videos using the Open3D library [39], which leverages the Azure Kinect Sensor SDK for alignment. Specifically, we align the depth videos to match the RGB videos' resolution. In addition, the collected data contain unnecessary background information that can negatively impact the performance of video analysis algorithms. This extraneous information can lead to overfitting, especially when working with limited data and high memory usage. To mitigate this issue and help our model focus solely on the relevant areas of the patients, we cropped both the RGB and depth videos to isolate these specific regions, as shown in Figure 2. This step is inspired by previous studies [21; 33; 40; 41], which demonstrated that deep learning models trained on relevant regions of interest outperform those trained on full-frame data. Therefore, we have adopted a similar approach, extracting the thoracic-abdominal regions, where retraction signs typically appear.  
Further, we spatially normalize the depth videos to a range of 0 to 1 for consistent scaling. We first remove outlier pixel values greater than 4000 (since the distance between the camera and the patient is not greater than that) by replacing them with zeros. Then, we compute the average distance of the thoracic-abdominal region by taking the mean of the non-zero pixels and selecting pixel values within a range of \(\pm\)400 from this mean. Finally, the selected values are divided by 800. The pseudocode for the depth video normalization process is presented in Algorithm 1.  
```
1 Input: Depth video (\(D\))
2 Output: Normalized depth video (\(D_{norm}\))
3 foreach Frame in depth video \(D\)do
4 Replace pixel values greater than 4000 with 0 (\(D_{1}\));
5 Compute the mean \(M\) of non-zero pixels of thoracic-abdominal region (\(D_{1}\)) ;
6 Select pixels of (\(D_{1}\)) in the range \([M-400,M+400]\) and set 0 ;
7 Scale selected pixel values by dividing by 800;
8
9 end fore
10 Return normalized depth video \(D_{norm}\);
```  
**Algorithm 1**Pseudo code for depth video normalization process.