# CFAM: Estimating 3D Hand Poses from a Single RGB Image with Attention
## 3 Method
### Channel Fusion Attention Mechanism (CFAM)
#### 3.3.1 The Frontend: A Fusion Model of the _Handmask_ and the Score Maps  
We first propose a fusion model of the _handmask_ and the score maps to consider the implicit spatial information in the RGB image in our CFAM.  
There are four parallel processing streams (called \(C_{ij},i,j=1,2\) in Figure 3) in the frontend of the network with almost identical architectures, including six convolutions with rectified linear unit (ReLU) nonlinearities. However, their parameters are not shared. We set the _handmask_\(I_{mask}\) as the input of the first two streams \(C_{1j}\) and the score maps \(p_{j}\) as inputs of the latter two streams \(C_{2j}\). After being fed into C, the output of the first stream \(C_{11}\) and the third stream \(C_{21}\) is concatenated to estimate the camera coordinate, while the second stream \(C_{12}\) and the fourth stream \(C_{22}\) are concatenated to estimate the camera rotation angle. The procedure is illustrated by the following formulas:  
\[\mathrm{d}F_{1j}=I_{mask}*C_{1j}, \tag{2}\]  
\[F_{2j}=p_{j}*C_{2j}, \tag{3}\]  
\[F_{k}=F_{1j}\oplus F_{2j},k=1,2, \tag{4}\]  
\begin{table}
\begin{tabular}{c c c c} \hline
**ID** & **Name** & **Kernel** & **Dimensionality** \\ \hline  & Input & & \(32\times 32\times\mathrm{Channel}\) \\
1 & Conv. + ReLU & \(3\times 3\) & \(32\times 32\times 32\) \\
2 & Conv. + ReLU & \(3\times 3\) & \(16\times 16\times 32\) \\
3 & Conv. + ReLU & \(3\times 3\) & \(16\times 16\times 64\) \\
4 & Conv. + ReLU & \(3\times 3\) & \(8\times 8\times 64\) \\
5 & Conv. + ReLU & \(3\times 3\) & \(8\times 8\times 128\) \\
6 & Conv. + ReLU & \(3\times 3\) & \(4\times 4\times 128\) \\ \hline \end{tabular}
\end{table}
Table 1: The frontend architecture, i.e., networks C\({}_{11}\), C\({}_{12}\), C\({}_{21}\), and C\({}_{22}\), is shown below. The networks have the same structure but differ in terms of the number of input channels. The number of channels is 3 when the input is an RGB image, while it is 21 when input is a 2D score map. In addition, the networks have different weights. Conv: convolution, ReLU: rectified linear units.  
\begin{table}
\begin{tabular}{c c c} \hline
**ID** & **Name** & **Kernel** & **Dimensionality** \\ \hline
1 & F\({}_{1i}\) & & \(4\times 4\times 128\) \\
2 & F\({}_{2i}\) & & \(4\times 4\times 128\) \\
3 & Concat(1,2) & & \(4\times 4\times 256\) \\
4 & Average Pooling & \(4\times 4\) & \(1\times 1\times 256\) \\
5 & FC & & \(1\times 1\times 32\) \\
6 & FC & & \(1\times 1\times 256\) \\
7 & Sigmoid & & \(1\times 1\times 256\) \\
8 & Multiply(3,7) & & \(4\times 4\times 256\) \\
9 & FC + ReLU & & 512 \\
10 & FC + ReLU & & 512 \\
11 & FC & & Output \\ \hline \end{tabular}
\end{table}
Table 2: The middle and backend network architectures in Figure 3 consist of two independent streams: the upper stream and the lower stream. The two network architectures are similar. One stream calculates the rotation angle of the camera, while the other calculates the coordinate. Therefore, the dimensions of the network output are different. When calculating the rotation angle, the Output in the ‘Dimensionality’ is 3; when calculating the coordinate, the Output in the ‘Dimensionality’ is 64. The following table is a stream network structure. Concat: concatenation.  
where \(F_{ij}\) is the output of the convolution process, \(*\) represents the operation on the feature maps that \(C_{ij}\) acts on, and \(\oplus\) is the concatenation of \(F_{1j}\) and \(F_{2j}\). To make full use of \(I_{mask}\), the implicit spatial information and the texture information are utilized in 3D hand pose estimation, which remedies the problem of insufficient context. More spatial and context information is obtained by the network.