# Multimodal Few-Shot Learning for Gait Recognition
## 2 Method
### Few-Shot Learning  
We define the _unknown known_ and _unknown unknown_ datasets. In the _unknown known_ dataset, the samples (unit steps) are not used for training the encoding function (i.e., the CNN, RNN, or ensemble networks); instead, only a few samples are utilized for training the decision boundaries of individuals using OSVM. In the _unknown unknown_ dataset, the samples are used only for testing.  
For a positive integer \(3\leq n\leq 10\), let \(\{\mathbf{s}_{i,a}|1\leq i\leq n\}\) be the set of randomly selected unit steps of subject \(id=a\) in the _unknown known_ dataset and \(\{\mathbf{v}_{i,a}=f(\mathbf{s}_{i,a})|1\leq i\leq n\}\) be the set of corresponding embedding vectors that are generated by the trained network model which can be one of CNN, RNN, or ensemble network. For each subject in the _unknown known_ dataset, at first, the system computes the centroid of \(n\) embedding vectors. The centroid of the subject \(id=a\) is defined by \(\mathbf{M}_{a}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{v}_{i,a}\). In addition, the system learns decision functions in the latent space using the  
Figure 3: Illustration of the multimodal triplet loss.  
OSVM algorithm [22] for all subjects. The algorithm obtains \(\{\mathbf{v}_{i,a}|1\leq i\leq n\}\) as an input and solves the following optimization problem:  
\[\left\{\begin{array}{ll}&\min_{a}\frac{1}{2}\sum_{i}^{n}\sum_{i}^{n}\alpha_{i }\alpha_{i^{\prime}}\mathcal{K}(\mathbf{v}_{i,a},\mathbf{v}_{i^{\prime},a})\\ &\text{subject to: }0\leq\alpha_{i}\leq\frac{1}{vn},\sum_{i=1}^{n}\alpha_{i}=1, \end{array}\right. \tag{3}\]  
where \(\mathcal{K}(\mathbf{v},\mathbf{v}^{\prime})=e^{-\gamma||\mathbf{v}-\mathbf{v }^{\prime}||_{2}^{2}}\) is a radial bias kernel function, \(\alpha_{i}\) are the Lagrange multipliers, and \(\gamma\) and \(\
u\) are among the hyper-parameters of the system.  
Let \(\mathbf{s}_{*,\mu}\) be a unit step of an unknown subject \(u\) in either the _unknown known_ or _unknown unknown_ dataset. The symbol \(*\) denotes that the unit step can be any unit step of the subject \(u\). For each subject \(a\) in the _unknown known_ dataset, the decision function of \(\mathbf{v}_{*,u}\) for subject \(id=a\) is defined by \(h_{a}(\mathbf{v}_{*,u})=\sum_{i}^{n}\alpha_{i}\mathcal{K}(\mathbf{v}_{i,a}, \mathbf{v}_{*,u})-\rho_{a}\), where \(\rho_{a}=\sum_{i}^{n}\alpha_{i}\mathcal{K}(\mathbf{v}_{i,a},\mathbf{v}_{i,a})\) for any \(h\) that satisfies the condition \(0<\alpha_{h}<\frac{1}{vn}\) and \(1\leq h\leq n\). An unknown subject could be one who was included in either the _unknown known_ dataset or _unknown unknown_ dataset. Therefore, the system should be able to recognize a unit step if it belongs to a subject in the _unknown known_ dataset. On the other hand, the system should be able to reject a unit step if it belongs to a subject in the _unknown unknown_ dataset. The system determines the prediction of \(u\) as follows:  
1. Compute \(\mathbf{v}_{*,u}=f(\mathbf{s}_{*,u})\)
2. Find provisional subject \(p=\arg\min_{a}||\mathbf{M}_{a}-\mathbf{v}_{*,u}||_{2}\)
3. If \(h_{p}(\mathbf{v}_{*,u})\geq\tau\), then \"\(u\) is recognized as \(p\)\"
4. Otherwise, \"\(u\) is not recognized\"  
where \(\tau\) is one of the hyper-parameters of the system. A conceptual diagram of the test phase is illustrated in Figure 4.