# Human Height Estimation by Color Deep Learning and Depth 3D Conversion
## 3 Proposed Method
### Human Body Region Extraction
#### 3.1.1 Human Body Region Extraction Using Color Frames  
Mask R-CNN [26] consists of three parts: a feature pyramid network (FPN) [37], a residual network (ResNet) [38; 39] and a RPN. A FPN detects the categories and the boundary boxes of objects in color video. ResNet extracts an additional feature map from each boundary box. Figure 3 shows the processes of extracting the human body and the human head using mask R-CNN.  
Figure 1: Flow of proposed method.  
Figure 2: Flow of human body region extraction. (**a**) From color information; (**b**) from depth information.  
FPN condenses the scale of the input frame through bottom-up layers and expands the scale through top-down layers. Various scale objects can be detected through FPN. ResNet introduces a skip connection algorithm that the output value of each layer feeds into the next layer and directly into the layers about more than 2 hops away. The skip connection algorithm reduces the amount of values to be learned for weights of layers, so the learning efficiency of ResNet is improved. The feature map is extracted from the frame through FPN and ResNet. RPN extracts the boundary boxes and the masks which are object area in rectangle and in pixels, respectively. Comparing with faster R-CNN which applies RoIPool, mask R-CNN extends RPN to extract not only the boundary box, but also the masks. RoIPool rounds off the coordinates of the boundary box to integer. In contrast, RoIAlign allows the floating coordinates. Therefore, the detection of the object areas of mask R-CNN is more precisely than of faster R-CNN. Figure 4 shows an example for calculating the coordinates of the regions of interest (RoIs) for detecting the boundary box and the masks by RoIPool and RoIAlign. Non-max-suppression (NMS) removes overlapping areas between the boundary boxes. The size of the overlapping area is calculated for each boundary box. Two boundary boxes are merged if the size of the overlapping area is more than 70%.  
Table 1 shows the performances of mask R-CNN when various types of backbones are applied to mask R-CNN. When X-101-FPN is applied as the backbone of mask R-CNN, the average precision of the boundary boxes (Box AP), which is a metric for detecting the boundary box, is the highest. However, the times for a train and a detection are slowest. In consideration of a tradeoff between the accuracy and the time for the detection, the proposed method applies ResNet-50 FPN which consists  
Figure 4: Example of region of interest (RoI) coordinate calculation by RoIPool and RoIAlign. (**a**) RoIPool; (**b**) RoIAlign.  
Figure 3: Processes of extracting human body and human head regions using mask R-CNN.  
The human body and the human head are detected by mask R-CNN. Mask R-CNN is trained using 3000 images of COCO dataset [40] with information about the human body and the human head. In the training mask R-CNN, a learn rate and epochs are set to 0.001 and 1000, respectively. A threshold for detection of the human body and the human head is set to 0.7. If a detection accuracy for RoI is more than the threshold, then corresponding RoI is detected as the human body or the human head. The process of extracting the human body and human head regions through mask R-CNN is as follows.  
1. Resizing a color image to a certain size
2. Extracting a feature map through FPN and ResNet50
3. Extracting RoI boundary boxes from feature map by RoIAlign
4. Boundary box regression and classification for boundary boxes through RPN
5. Generating boundary box candidates by projecting the boundary box regression results onto the color frame
6. Detecting a boundary box for each object by non-max-suppression
7. Adjusting the boundary box area through RoIAlign
8. Finding pixels in boundary boxes to obtain a mask for each boundary box