## 3 Structure of the Proposed Biomodal Multitask Network
### Feature Extraction  
One of the objectives of our proposal is fusing the face and speech data at the feature level in order to implement the bimodal part of our biometric system. The state of the art work proposes to create a module within the network architecture that optimally combines the extracted features of all biometric traits [19; 44; 48]. This fusion module delivers a unique feature vector that contains the most important information of the identity of an individual.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Layers** & **Filters/Neurons/Units** & **Size** & **Stride** & **Activation Function** \\ \hline \multicolumn{4}{l}{Speech processing layers} \\ \hline Convolution 2D & 32 & 7 \(\times\) 5 & 2 \(\times\) 2 & ReLU \\ Batch Normalization & - & - & - & - \\ Convolution 2D & 64 & 7 \(\times\) 3 & 2 \(\times\) 1 & ReLU \\ Batch Normalization & - & - & - & - \\ GRU (\(\times\)3) & 512 (\(\times\)3) & - & - & Tanh/Sigmoid (\(\times\)3) \\ GRU (\(\times\)7) & 512 (\(\times\)7) & - & - & Tanh/Sigmoid (\(\times\)7) \\ \hline \multicolumn{4}{l}{Face processing layers} \\ \hline Convolution 2D & 32 & 3 \(\times\) 3 & 1 \(\times\) 1 & ReLU \\ Max Pooling 2D & - & 2 \(\times\) 2 & 1 \(\times\) 1 & - \\ Batch Normalization & - & - & - & - \\ Convolution 2D & 64 & 5 \(\times\) 5 & 1 \(\times\) 1 & ReLU \\ Max Pooling 2D & - & 2 \(\times\) 2 & 1 \(\times\) 1 & - \\ Batch Normalization & - & - & - & - \\ Fully connected & 512 & - & - & ReLU \\ \hline \multicolumn{4}{l}{Fusion and} \\ \multicolumn{4}{l}{output layers} \\ \hline Weighted Sum & - & - & - & - \\ Fully connected & 45 & - & - & Softmax \\ Fully connected & 27 & - & - & Softmax \\ \hline \hline \end{tabular}
\end{table}
Table 2: Multitask network architecture.  
Figure 1 shows a block diagram of the proposed multitask network. The diagram indicates two streams of layers that process face and speech data independently. The speech stream is divided into two paths, represented by the GRU (Gated Recurrent Unit) layers. The first path processes the feature maps that will be combined with the face features in the weighted sum module. The second path extracts the temporal information used to perform the speech recognition task.  
The input of the image stream consists of face grayscale images with dimensions of 128 \(\times\) 128 pixels. We employed 2 convolutional blocks comprised of 32 and 64 filters, respectively, with the ReLU activation function, a 2D max pooling operation to reduce the feature map's dimensions and to extract important information based on a pixel's location, and batch normalization as a feature regularizer. The extracted feature maps are fed to a fully connected layer with ReLU activation and batch normalization; this layer outputs a vector of length 512.  
The input of the speech stream consists of MFCC features, which are widely used in speaker and speech recognition works [32]. These features are represented as two dimensional signals with the cepstral coefficients containing the frequency information in Mel scale located in one dimension, and the time domain in the other. No pooling layers were included to avoid a loss of temporal information.  
The extracted speech feature maps are processed by two different paths of GRU layers. The first path with 3 layers extracts temporal dependencies from the feature maps and allows the network to handle maps with different temporal sizes. The last GRU layer outputs a unique feature vector of size 512 and discards the previous vectors in the sequence. The second path of 7 GRU layers extracts temporal information with more detail than the first path. The last layer outputs the complete sequence of processed vectors, which are fed to the speech recognition output.