# EFCMF: A Multimodal Robustness Enhancement Framework for Fine-Grained Recognition
## 2 Method
### Feature Fusion Module  
Finally, all the features of the output are concatenated (represented as \(cat\)) and reshaped to obtain the features \(I_{l}\) for fusion:  
\[\underset{\sum_{i}^{m}k_{i}\times 1\times 1}{Feat_{i}}=\textit{cat}\Bigg{(} \Bigg{[}\underset{k_{1}\times 1\times 1}{Feat_{1}}\ldots,\underset{k_{m} \times 1\times 1}{Feat_{m}}\Bigg{]}\Bigg{)} \tag{7}\]  
\[I_{l}=\underset{\sum_{i}^{m}k_{i}}{Feat}=\textit{reshape}\Bigg{(}\underset{ \sum_{i}^{m}k_{i}\times 1\times 1}{Feat_{m}}\Bigg{)} \tag{8}\]  
In the feature fusion module, the study adopts a feature-level fusion strategy to obtain new features by concatenating the features of the visual modality and language modality  
\[\left[ \begin{array}{c}I_{v}\\ I_{l}\end{array} \right]=cat(I_{v},I_{l}) \tag{9}\]  
The error is backpropagated through the classifier to jointly training the two feature extraction modules of the model. The weight of the fully connected layer of the module is \(W\), which can be regarded as the splicing of two weights using modal features for classification: \(W=cat(W_{v},W_{l})= \begin{bmatrix}W_{v}&\mid&W_{l}\end{bmatrix} \), and the matrix product with the feature to obtain the \(Out\):  
\[Out=W\times I=\left[ \begin{array}{c}W_{v}\end{array} \right]W_{l}\ \ \big{]}\times\Bigg{[} \frac{I_{v}}{I_{l}}\Bigg{]} \tag{10}\]  
During training, in order to simulate modal missing to enhance the ability of the model to cope with modal missing and adversarial examples attacks, the input modal features are randomly deactivated. The random inactivation of features is a random event conforming to the Bernoulli distribution, and its Bernoulli random variable is \(r_{m}\sim\text{Bernoulli}(p)\). When a modality is missing, the visual modality has a \(q\) probability of remaining intact, and the Bernoulli random variable for this event is \(r_{v}\sim\text{Bernoulli}(q)\), and the language modality is \(r_{l}=1-r_{m}r_{v}\).Then,the forward propagation process of the module is as follows:  
\[I_{v}=r_{v}*I_{v}\,I_{l}=r_{l}*I_{l} \tag{11}\]  
\[I=cat(I_{v},I_{l})=\left[ \begin{array}{c}I_{v}\\ I_{l}\end{array} \right] \tag{12}\]The output of the model appears as follows:  
\[ \begin{split} Out&=W\times I=\left[\begin{array}{c}W_{ \triangledown}\left|\begin{array}{c}W_{l}\end{array}\right.\right]\times \left[\begin{array}{c}I_{\triangledown}^{\prime}\\ I_{\triangledown}^{\prime}\end{array}\right]\\ &=(1-r_{m}(1-r_{\triangledown}))*W_{\triangledown}\times I_{\triangledown}^ {\prime}+r_{\triangledown}*W_{\dagger}\times I_{\dagger}^{\prime}\\ &=(1-r_{m}(1-r_{\triangledown}))*Out_{\triangledown}+r_{\dagger}*Out_{ \dagger}\\ &=(1-r_{m}(1-r_{\triangledown}))*Out_{\triangledown}+(1-r_{\triangledown}r_ {\triangledown})*Out_{\dagger}\\ &=(1-r_{m})*Out+r_{m}(r_{\triangledown}*Out_{\triangledown}+(1-r_{\triangledown})* Out_{\dagger})\end{split}  \tag{13}\]  
The average expectation of the output is:  
\[E(Out)=(1-p)*Out+p*(q*Out_{\triangledown}+(1-q)*Out_{\dagger}) \tag{14}\]  
At this time, the model can be considered as an ensemble model of three models with strong correlation. \(Out\) is a vector representing the confidence (denoted as \(Conf_{j}\)) of the classification result of the model for a total of \(k\) categories: \(Out=[Conf_{1},\ldots,Conf_{i}]\) Finally, the output result is normalized by softmax to obtain the predicted value \(Pred\), and the \(Loss\) is calculated as follows:  
\[ \begin{split} Pred&=\text{softmax}(Out)=\frac{1}{ \sum_{\mathbf{I}}^{k}e^{Conf_{i}}}\Big{[}e^{Conf_{1}},\ldots,e^{Conf_{i}}\Big{]} \\ Loss&=Label*\log(Pred)+(1-Label)*\log(1-Pred)\end{split}