# A Multi-Modal Person Recognition System for Social Robots
## 4 Experimental Results
### Experiment 1  
For each subject in the probe set, two facial images, _fb_ image and its duplicate I image, were selected from the FERET database. In addition, two out of five frames from each of skeleton information and _3D_ mesh body information were selected randomly from Walking 1 group in the RGB-D database. The rest of the samples in the FERET and RGB-D databases were used to construct the training set. Some subjects in the FERET database had only two facial images. In this case, one was used for training and the other for evaluation. Five feature vectors were constructed, as described in Section 3. Three of the feature vectors represent facial information, including the facial geometry feature vector, cross ratio feature vector, and appearance-based feature vector. The rest of the feature vectors, namely the skeleton feature vector and the surface-based feature vector, represent the body information of the attended subject. IOs generated based on these features were converted into spike times and normalized to range from _zero to 150_ ms, prior to being fed to LIF neurons in SNN, as shown in Figure 8. The SNN was constructed and simulated using the neural Circuit (CSIM) simulator [60]. The parameters of LIF neurons were set as follows: the weight synapses of neuron 1 and neuron 2 were equal and set at \(2000\times 10^{-9}\). The weight synapses of neuron 3 were set as follows: the weight synapse of dendritic branch one was set to \(2500\times 10^{-9}\) and weight synapse of dendritic branch two was set to \(2000\times 10^{-9}\), \(V_{thresh}=0.15\), \(V_{reset}=-0.067\), \(V_{reseting}=0\), \(C_{m}=5\times 10^{-8}\), \(V_{init}=0.08\), \(R_{m}=1\times 10^{6}\), \(T_{refact}=0.0025\), \(I_{noise}=50\times 10^{-9}\), \(I_{sys}(\mathrm{t})\) represents the input current supplied by the synapses, i.e., the outputs from the conversion process of IOs into input spike times. These input spike times were set in the range from _zero to 150_ ms. This selection is compatible with the natural human perception of time. The SNN were simulated for 150 ms. As described in Section 2, the first neuron that fires a spike represents the best candidate of the attended subject \(x\) from the gallery set. The overall SNN was constructed from 79 circuit blocks, as shown in Figure 8. Therefore, the total number of LIF neurons was 237. The recognition rates were calculated at two stages in the hierarchical structure of the SNN, namely stage 1 and stage 2. Stage 1 consists of the list of neurons, labeled as neuron 1 and neuron 2; stage 2 was represented by the list of neurons labeled as neuron 3. The recognition rate that was calculated from the list of neurons labeled as neuron 1 was based on body information; the recognition rates that were calculated from the list of neurons labeled as neuron 2 expressed a recognition rate based on facial information or voice information. Neuron 2 may use face geometry, face appearance, voice-based feature, or all of them in order to fire a spike. The same applies to neuron 1, which may use geodesic distances, skeleton distances, or both, in order to drive its potential to the threshold and consequently evoke a spike. The overall recognition rates were calculated based on neuron 3, which may use facial information, body information, voice information, or a combination of them. Cumulative match curves (CMCs) show the probability that the correct match of classification is found in the \(N\), the most likely candidates, where \(N\) (the rank) is plotted on the \(x\)-axis. CMCs provide the performance measure for biometric recognition systems and have been shown to be equivalent to the ROC of the system [61]. The recognition result was averaged over ten runs; the cumulative match curves (CMCs) were plotted for these recognition results and are shown in Figure 11a-c.