# Human Height Estimation by Color Deep Learning and Depth 3D Conversion
## 3 Proposed Method
### Human Height Estimation  
Human height is estimated by measuring a length in the 3D real world between the head-top and foot-bottom points. In order to measure the length on the real world, 2D image coordinates of two head-top and the foot-bottom are converted into 3D real-world coordinates by applying a pinhole camera model [41] as follows:  
\[\begin{split} X&=\frac{(x-W/2)}{f}d(x,y)\\ Y&=\frac{(y-H/2)}{f}d(x,y)\\ Z&=d(x,y),\end{split} \tag{2}\]  
where \(X\), \(Y\), \(Z\) are the real-world coordinates, \(f\) is a focal length of the depth camera, which means the parameter of the depth camera, and \(W\) and \(H\) are the horizontal and vertical resolutions of the depth image, respectively. In (2), the origin of the image coordinate system is the top-left of the image, but the origin of 3D camera coordinate system is the camera center. In order to compensate for the difference in the position of the origin between two coordinate systems, the coordinates of the image center are subtracted from the image coordinates. the real-world coordinates of the head-top (\(X_{h}\), \(Y_{h}\), \(Z_{h}\)) and of the foot-bottom (\(X_{f}\), \(Y_{f}\), \(Z_{f}\)) are calculated by substituting the real-world coordinates and the depth values of the head-top and the foot-bottom for (2), respectively, as follows:  
\[\begin{split} X_{h}&=\frac{(x_{h}-W/2)}{f}d_{h}\\ Y_{h}&=\frac{(y_{h}-H/2)}{f}d_{h}\\ Z_{h}&=d_{h},\end{split} \tag{3}\]  
\[\begin{split} X_{f}&=\frac{(x_{f}-W/2)}{f}d_{f}\\ Y_{f}&=\frac{(y_{f}-H/2)}{f}d_{f}\\ Z_{f}&=d_{f},\end{split} \tag{4}\]  
Figure 5: Extracting the head-top and foot-bottom points.  
Figure 6: Extracting the foot-bottom point in case of apart human legs.  
Mask R-CNN occasionally detects slightly wider human body region than the actual region. In particular, the region detection error in the lower part of human body may causes that a point on the ground is extracted as the foot-bottom. Assuming the ground is flat, the difference in a depth gradient is little between two vertically adjacent pixels that is in the ground area. The depth gradient of certain pixel \((x,y)\) is defined as follows:  
\[g(x,y)=d(x+1,y)-d(x,y). \tag{10}\]  
If certain point is on the ground, then the difference in the depth gradients is the same between the point and one pixel down. In order to determine whether the extraction of the foot-bottom is correct, two depth gradients are compared as follows:  
\[D = g\Big{(}x_{f}-1,y_{f}\Big{)}-g\Big{(}x_{f},y_{f}\Big{)}\] \[= \Big{(}d\Big{(}x_{f},y_{f}\Big{)}-d\Big{(}x_{f}-1,y_{f}\Big{)} \Big{)}-\Big{(}d\Big{(}x_{f}+1,y_{f}\Big{)}-d\Big{(}x_{f},y_{f}\Big{)}\Big{)}\] \[= 2d\Big{(}x_{f},y_{f}\Big{)}-d\Big{(}x_{f}-1,y_{f}\Big{)}-d\Big{(} x_{f}+1,y_{f}\Big{)}.\]  
If \(D\) is 0, then the point is removed from the human body region. The comparison of the depth gradients by (11) is applied to the bottommost pixels of the human body region in order to correct the foot-bottom. If all of the bottommost pixels are removed, then this process is repeated for the points of the human body region where is one pixel up. The foot-bottom is extracted as a center pixel among the bottommost pixels which are not removed. Figure 8 shows correcting the position of the foot-bottom point.  
Figure 8: Foot bottom point correction.  
Figure 7: Flow of correcting \(d_{h}\) by calculating similarity to neighboring pixels.  
The noises of depth video may cause the temporary error of the human-height estimation. The temporary error of the human height is corrected by the average of the estimated human heights among a sequence of depth frames as follows:  
\[\begin{array}{ll}\overline{H}(n+1)=&\frac{1}{n+1}\bigg{[}\sum_{i}^{n+1}H(i) \\ =&\frac{1}{n}\bigg{[}\sum_{i}^{n}H(i)\bigg{]}\times\frac{n}{n+1}+\frac{1}{n+1}H (n+1)\\ =&\frac{n}{n+1}H(n)+\frac{1}{n+1}H(n+1),\end{array} \tag{12}\]  
where \(n\) is the order of the captured depth frames and \(H(n)\) and \(\overline{H}(n)\) are the estimated and corrected human heights in the \(n\)th frame order, respectively.