## 2 Materials and Methods
### The Proposed System
#### 2.2.3 Fusion of the Global and Local Networks  
Different fusion methods may produce different recognition rates. In this paper, we compared five fusion methods, i.e., concatenation, product, sum, subtract and max fusion method. For clarity, we define a fusion function f, two feature maps \(x^{a}_{i}\) and \(x^{b}_{i}\), and a fused feature maps y, where, \(x^{a}\in\mathrm{R}^{H\times W\times D}\), \(x^{b}\in\mathrm{R}^{H\times W\times D}\), \(y\in\mathrm{R}^{H^{\prime}\times W^{\prime}\times D^{\prime}}\), where \(W\), \(H\), and \(D\) are the width, height, and number of channels of feature maps. The five fusion methods are described as follows:  
Concatenation fusion. \(y=f_{\mathrm{cat}}(x^{a},x^{b})\) stacks the two features at the same location \(i,j\) across the feature channels \(d\):  
\[Y_{ij,d}=x^{a}_{ij,d},y_{ij,D+d}=x^{b}_{ij,d} \tag{1}\]  
where \(y\in\mathrm{R}^{H\times W\times 2D}\).  
Product fusion. \(y=f_{\mathrm{product}}(x^{a},x^{b})\) computes the product of the two feature maps at the same location \(i,j\) and feature channels d:  
\[y_{ij,d}=x^{a}_{ij,d}\times x^{b}_{ij,d} \tag{2}\]  
where \(1<i<H,1<j<W,1<d<D,y\in\mathrm{R}^{H\times W\times D}\).  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Upper Stream**} & \multicolumn{3}{c}{**Lower Stream**} \\ \hline
**Layers** & **Patch Size** & **Output Size** & **Output** & **Patch Size** & **Output Size** & **Output** \\ \hline Data & - & - & 36 \(\times\) 64 \(\times\) 32 & - & - & 36 \(\times\) 64 \(\times\) 32 \\ Conv1 & 3 \(\times\) 3 \(\times\) 3 & 64 & 36 \(\times\) 64 \(\times\) 32 & 3 \(\times\) 3 \(\times\) 3 & 64 & 36 \(\times\) 64 \(\times\) 32 \\ Pool1 & 2 \(\times\) 2 \(\times\) 2 & 64 & 18 \(\times\) 32 \(\times\) 16 & 2 \(\times\) 2 \(\times\) 2 & 64 & 18 \(\times\) 32 \(\times\) 16 \\ Conv2 & 3 \(\times\) 3 \(\times\) 3 & 128 & 18 \(\times\) 32 \(\times\) 16 & 3 \(\times\) 3 \(\times\) 3 & 128 & 18 \(\times\) 32 \(\times\) 16 \\ Pool2 & 2 \(\times\) 2 \(\times\) 2 & 128 & 9 \(\times\) 16 \(\times\) 8 & 2 \(\times\) 2 \(\times\) 2 & 128 & 9 \(\times\) 16 \(\times\) 8 \\  & & & & Fusion 1 & & \\  & & & & 2048 neurons & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Configurations of local network.  
Sum fusion. \(y=f_{sum}(x^{a},x^{b})\) computes the sum of the two feature maps at the same location \(i,j\) and feature channels d:  
\[y_{ij,d}=x^{a}_{ij,d}+x^{b}_{ij,d} \tag{3}\]  
where \(1<i<H,1<j<W,1<d<D\), \(y\in R^{H\times W\times D}\).  
Subtract fusion. \(y=f_{subtract}(x^{a},x^{b})\) computes the subtract of the two feature maps at the same location i, j and feature channels d:  
\[y_{ij,d}=x^{a}_{ij,d}\div x^{b}_{ij,d} \tag{4}\]  
where \(1<i<H,1<j<W,1<d<D\), \(y\in R^{H\times W\times D}\).  
Max fusion. \(y=f_{max}(x^{a},x^{b})\) computes the max of the two feature maps at the same location i, j and feature channels d:  
\[y_{ij,d}=max\left\{x^{a}_{ij,d},x^{b}_{ij,d}\right\} \tag{5}\]  
where \(1<i<H,1<j<W,1<d<D\), \(y\in R^{H\times W\times D}\).  
Through experiments, we found that concatenation fusion method can achieve better recognition rate than other methods. The concatenation fusion method is used in this research. In this section, we adopt the same fusion method with Fusion 1 to fuse global network and local network as Fusion 2 followed by fully connected layer with 2048 neurons. We adopt a softmax layer with dimension of six to recognize facial expression.  
For the whole network, global and local networks are complementary. Either global network or local network cannot represent facial expression information solely, as their fusion significantly improves on both (28.29% over local and 6.37% over global network).