## 3 Proposed Method  
### Modeling  
Formally, the smile/non-smile prediction of the input face \(X\) is defined as \(S(X)\). The expected smile/non-smile prediction of the input \(X\) is defined as follows:  
\[E[S(X)]=\sum_{s\in S}s\cdot P(S(X)=s), \tag{1}\]  
where \(P(S(X)=s)\) is the probability that the smile/non-smile prediction of the input \(X\) is \(s\), where \(s\in S\). We define \(S=\{\)'non-smile','smile'\(\}\). We assume that the predicted smile/non-smile is dependent on the gender of the input \(X\). Compared to the traditional multi-task learning architecture as shown in Figure 1, the \(FC_{S}\) layer has been transformed into \(FC_{S|C_{G}}\) layer in the multi-task learning architecture we proposed (shown in Figure 1). \(FC_{S}\) denotes the fully connected layer that contains \(K(K\in\mathbb{N})\) smile/non-smile representation units. \(FC_{S|C_{G}}\) denotes the transformed gender dependent fully connected layer that contains \(K\) gender dependent smile/non-smile representation units, where \(C_{G}\) is the gender context. Figure 1: Comparison of traditional multi-task learning and our proposed multi-task learning architecture. (**a**) traditional multi-task learning; (**b**) our proposed multi-task learning. We feed the transformed \(FC_{S|C_{G}}\) layer into the softmax layer to predict the final smile/non-smile. The probability \(P(S(X)=s)\) in Equation (1) can be modeled as follows:  
\[P(S(X)=s)=softmax(FC_{S|C_{G}}(X)). \tag{2}\]  
The gender context \(C_{G}\) contains \(K\) gender context units \(C_{Gi}\), where \(C_{Gi}\) is the \(i\)-th (\(i=1,\,2,\,\dots\,\,\,K\)) gender context unit that is automatically chosen from the gender representation units in the \(FC_{G}\) layer. \(FC_{G}\) denotes the fully connected layer that contains \(K\) gender representation units. The dependency score function \(score(x_{Sj},C_{Gi})\) that takes a conjunction of the \(j\)-th(\(j=1,\,2,\,\dots\,\,\,K\)) input smile/non-smile representation unit \(x_{Sj}\) from the \(FC_{S}\) layer and the \(i\)-th gender context unit \(C_{Gi}\) from the \(FC_{G}\) layer to score the dependency between \(x_{Sj}\) and \(C_{Gi}\). The dependency score function can be formulated as follows:  
\[score(x_{Sj},C_{Gi})=tanh(W_{S}x_{Sj}+W_{G}C_{Gi}). \tag{3}\]  
The probability \(P(d=j\mid\textbf{x}_{S},C_{Gi})\) reveals the relative importance of \(x_{Sj}\) based on \(C_{Gi}\), where \(d\) indicates which input smile/non-smile representation unit in \(x_{S}\) is important based on \(C_{Gi}\), where \(\textbf{x}_{S}\) contains \(K\) input smile/non-smile representation units. The probability \(P(d=j\mid\textbf{x}_{S},C_{Gi})\) can be calculated using the dependency score function as follows:  
\[P(d=j\mid\textbf{x}_{S},C_{Gi})=\frac{exp(score(x_{Sj},C_{Gi}))}{\sum\limits \limits_{j=1}^{K}exp(score(x_{Sj},C_{Gi}))}. \tag{4}\]  
The importance probability distribution \(P(d\mid\textbf{x}_{S},C_{Gi})\) is defined as follows:  
\[P(d\mid\textbf{x}_{S},C_{Gi})=[P(d=j\mid\textbf{x}_{S},C_{Gi})]_{j=1}^{K}. \tag{5}\]  
The gender dependent smile/non-smile representation units in the transformed \(FC_{S|C_{G}}\) layer can be defined as follows:  
\[\hat{S_{i}}=E_{\textbf{x}_{S}\sim P(d|\textbf{x}_{S},C_{Gi})}(\textbf{x}_{S} )=\sum\limits_{j=1}^{K}P(d=j\mid\textbf{x}_{S},C_{Gi})x_{Sj}, \tag{6}\]  
where \(\hat{S_{i}}\) is the \(i\)-th (\(i=1,\,2,\,\dots\,\,K\)) gender dependent smile/non-smile representation unit that is the weighted average of the input smile/non-smile representation units. \(\hat{S_{i}}\) can be formulated as the expectation of \(x_{S}\) according to the importance probability distribution \(P(d\mid\textbf{x}_{S},C_{Gi})\). The transformed \(FC_{S|C_{G}}\) layer is generated by concatenating K gender dependent smile/non-smile representation units. The gender prediction of the input face \(X\) is defined as \(G(X)\).