# A Room Monitoring System Using Deep Learning and Perspective Correction Techniques
## The Model of Room Monitoring System
### Proposed Person Detection Scheme
#### 3.3.3 Merging of the Bounding Boxes  
We merge the bounding boxes in the initial results to obtain the final result, which is implemented in two phases. In the first phase, we merge each tile image in the initial results to obtain so-called intermediate results and focus on the bounding boxes in the overlapping areas of neighboring tile  
Figure 4: Illustration of tile images. Figure 5: Mask region-based convolutional neural network (R-CNN) framework for instance segmentation [21]. images. In the second phase, we merge the intermediate results with the bounding boxes of the full-frame image to generate the final results. Two important concepts are used in the merging process: intersection over union (IoU) and the proposed belong to areas (BTA). IoU is a widely used metric that indicates the similarity of two arbitrary shapes, and it also represents the accuracy an object detector can achieve based on a certain dataset [45; 46]. We propose a new BTA that also aims to evaluate the similarity of two random shapes. The specific formulas of IoU and BTA for comparing the similarity of two arbitrary shapes, a and B, are given by Equations (4) and (5), respectively. \[IoU=\frac{|A\cap B|}{|A\cup B|} \tag{4}\]  
\[BTA=max\left(\frac{|A\cap B|}{A},\frac{|A\cap B|}{B}\right) \tag{5}\]  
Figure 6 shows an instance of an object residing on the border. The IoU and BTA values must be considered to merge the neighboring bounding boxes. In Figure 5(a), IoU and BTA have average and acceptable values of 0.407 and 0.647, respectively. Therefore, these neighboring bounding boxes are considered to merge. In Figure 5(b), IoU has a low value of 0.161; however, we still consider merging the neighboring bounding boxes in our proposed algorithm owing to the high BTA value of 0.934. As mentioned above, in phase 1, the bounding boxes of each tile image must be merged to obtain a new set of bounding boxes, called intermediate results. For this purpose, we propose the new algorithm described in Algorithm 1. Figure 6: Illustration of an object residing on the border. ```
1. Input:\(list\_overlap=\{box_{ij}(y_{1},x_{1},y_{2},x_{2}),i=[0,n-1],j=[0,m-1]\}\)\(list\_box=\{box_{i}(y_{1},x_{1},y_{2},x_{2}),i=[0,k-1]\}\)\(list\_overlap\subset list\_box\)\(n\): number of tiles \(m\): number of boxes in overlapping areas of tile[i] \(k\): number of boxes of all tiles
2. Output:\(list\_new\_box=\{box_{i}(y_{1},x_{1},y_{2},x_{2}),i=[0,t-1]\}\)\(t\): number of new boxes of all tiles
3. set \(threshold\_iou\)\(list\_new\_box=list\_box\)
4. for\(i\) in range(0, \(n\)-1):
5. newBox=[]
6. for\(box\) in \(list\_overlap[i]\):
7. iou=0
9. for\(nextBox\) in \(list\_overlap[i+1]\):
10. if\(iou\) < \(calculate\_iou(box,nextBox)\):
11. \(iou=calculate\_iou(box,nextBox)\)\(maxBox=nextBox\)
12. if\(iou\) >= \(threshold\_iou\):
14. \(newBox=union(box,maxBox)\)\(list\_new\_box.append(newBox)\)\(list\_new\_box.remove(box)\)\(list\_new\_box.remove(maxBox)\)\(list\_new\_box.remove(maxBox)\)\(list\_new\_box.remove(box)\)\(\#box[1]=x_{1}\)
15. if\(box[1]>=(i+1)\times(W_{tile}-d_{intersect})\):
16. \(list\_new\_box.remove(box)\)
17. return\(list\_new\_box\)
```  
**Algorithm 1** Merging the bounding boxes of each tile image in the initial results  
1. Lines (1, 2) contain the input and output information of the algorithm. The input includes \(list\_box\) and \(list\_overlap\). \(list\_box\) contains the bounding boxes of all tile images, and \(list\_overlap\) contains the bounding boxes in the overlap areas of all tile images. \(list\_new\_box\) is the result returned by the algorithm. It includes the new bounding boxes, called intermediate results. 2. The parameters are set using Lines (3, 4). \(threshold\_iou\) is a predefined threshold of IoU that is used to consider whether two neighboring bounding boxes are merged. 3. Lines (5-9) consider the neighboring bounding boxes in overlapping area of each tile image. 4. Lines (10-12) determine the coupled neighboring bounding boxes with the highest IoU value in overlapping areas.