## 3 The Sound Source Monitoring Combing Stereo Vision and Beamforming
### Coordinate System Transformation  
The camera coordinate system takes the stereo camera as the origin, and the coordinate system for beamforming usually takes the center of the array as the origin. Spatial matching is needed to coincide the two systems. Coordinate system transformation is realized through rotation and translation between two coordinate systems.  
Before the coordinate system transformation, each pixel in the depth image output by the stereo camera needs to be transformed into the camera coordinate system. According to the imaging principle of the camera, the spatial transformation relationship between the pixel coordinates and the camera coordinate system \((o,\,u,\,v,\,w)\) can be established, as shown in Equation (7):  
\[\left[\begin{array}{c}x_{1}\\ y_{1}\end{array}\right]=2l\left[\begin{array}{cc}\tan\frac{\alpha_{0}}{2}/U&0 \\ 0&tan\frac{\beta_{0}}{2}/V\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]-l\left[\begin{array}{c}tan\frac{\alpha_{0}}{2}\\tan\frac{\beta_{0}}{2}\end{array}\right] \tag{7}\]  
where \((x_{1},y_{1},l)\) is the spatial coordinate of the pixel in the camera coordinate system, and \(l\) is the depth of the pixel. U \(\times\) V is the resolution of the output depth image. \(\alpha_{0}\) and \(\beta_{0}\) are the horizontal view angle and the vertical view angle of the camera, respectively.  
The camera coordinate system \((o,\,u,\,v,\,w)\) and the array coordinate system \((Oz,Xz,Yz,Zz)\) are different spatial coordinate systems. The use of beamforming to reconstruct the sound field needs to be carried out in the array coordinate system, and their projection relationship is shown in Figure 3.  
Assuming that point \((x_{1},y_{1},l)\) in the camera coordinate system, the corresponding coordinates in the array coordinate system is \(\left(X_{bf},Y_{bf},Z_{bf}\right)\). The relationship between the two coordinate systems:  
\[\left[\begin{array}{c}X_{bf}\\ Y_{bf}\\ Z_{bf}\end{array}\right]=M_{\rm R}\left[\begin{array}{c}x_{1}\\ y_{1}\\ l\end{array}\right]-\left[\begin{array}{c}X_{zu}\\ Y_{zv}\\ Z_{zw}\end{array}\right] \tag{8}\]  
\[\begin{array}{c}M_{\rm R}=\mathbf{R}_{u}(\alpha)\mathbf{R}_{v}(\beta) \mathbf{R}_{w}(\theta)\\ =\left[\begin{array}{ccc}1&0&0\\ 0&\cos(\alpha)&\sin(\alpha)\\ 0&-\sin(\alpha)&\sin(\alpha)\end{array}\right]\left[\begin{array}{ccc}\cos (\beta)&0&-\sin(\beta)\\ 0&1&0\\ \sin(\beta)&0&\cos(\beta)\end{array}\right]\left[\begin{array}{ccc}\cos( \theta)&\sin(\theta)&0\\ -\sin(\theta)&\cos(\theta)&0\\ 0&0&1\end{array}\right]\end{array} \tag{9}\]  
where \(\mathbf{M}_{\rm R}\) is the three-dimensional rotation matrix [24], which makes the camera coordinate system rotate around the \(w\) axis by \(\theta\), rotate around the \(v\) axis by \(\beta\), and rotate around the \(u\) axis by \(\alpha\). Then the two space coordinate systems are parallel to each other. The vector \([X_{zu},Y_{zv},Z_{zu}]^{T}\) is the translation between the origins of the two coordinate systems.  
After coordinate system transformation, the target area in the depth map obtained by stereo vision is converted to the corresponding 3D point cloud. The point cloud is transformed into the microphone array coordinate system. The reconstruction points of the three-dimensional sound source position are replaced with the 3D point cloud, and the beamforming calculation is performed on the surface of the target to achieve the sound field reconstruction on the 3D surface.  
Figure 3: Relationship between array and stereo vision coordinate system.