# Image Caption Generation via Unified Retrieval and Generation-Based Method
## 3 Model Architecture
### Preliminary  
The _Transformer_ model has been widely applied in sequence-to-sequence tasks and gains significant improvements. Our model mainly acquires the attention mechanism proposed by [7]. Thus, we first introduce the preliminary knowledge of the scaled-dot product attention which is defined as:  
\[Att(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{1}\]  
where \(Q\in R^{m\times d_{model}}\) is the query matrix, \(m\) is the number of objects, \(d_{m}odel\) is the latent dimension. \(K\in R^{n\times d_{model}}\) denotes the key matrix, and \(V\in R^{n\times d_{model}}\) represents the value matrix. \(K\) and \(V\) are key-value matrix pairs. In briefly, the query matrix \(K=[k_{1},...,k_{n}]\), \(k_{n}\in R^{d_{model}}\), the value matrix \(V=[v_{1},...,v_{n}]\), \(v_{n}\in R^{d_{model}}\), and \(n\) is the number of the key-value pairs. \(QK^{T}\) stands for the product operation to compute the similarity score between \(Q\) and \(K\), which may cause the result become larger or smaller and further influence the precision of the variable. Thus, \(\sqrt{d_{k}}\) indicates the scaled dot to scale \(QK^{T}\). The attention of queries \(Q=[q_{1},...,q_{m}]\), \(q_{m}\in R^{d_{model}}\) is computed in parallel. After a series computation, \(Att(Q,K,V)\in R^{m\times d_{model}}\) is the attended output features.  
The multi-head attention contains \(h\) parallel heads, and each head corresponds to a scaled-dot-product attention function. These \(h\) scaled-dot product attention are independent. The multi-head attention concatenated them and projected to form the final attentive representation:  
\[head_{i}=Att(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}) \tag{2}\]  
\[O=MultiHead(Q,K,V)=Concat(head_{1},...,head_{h})W^{o} \tag{3}\]  
Figure 1: Our model combined the retrieval-based method and the generation-based method. The de-noising module here is used to filter the retrieved similar sequences. The decoder here used both the filtered textual features and the attended visual features to generate the final descriptions.  
where \(Concat(\cdot)\) is the concatenation function, \(head_{i}\) represents the \(i\)-th scaled-dot-product attention. \(W^{o}\in R^{h*d_{p}\times d_{model}}\) is the output of different heads, \(d_{v}\) is the output dimension of each head. \(W_{i}^{Q}\in R^{d_{model}\times d_{k}}\), \(W_{i}^{K}\in R^{d_{model}\times d_{k}}\), \(W_{i}^{V}\in R^{d_{model}\times d_{v}}\) are the linear _Transformer_ parameter matrices.  
Followed the multi-head attention is a position-wise feed-forward network(FFN), FFN takes the output of the multi-head attention as input which is defined as:  
\[FFN(x)=Dropout(max(0,xL_{f}+b_{f}))L_{ff}+b_{ff} \tag{4}\]  
where \(Dropout(\cdot)\) is the dropout [32]operation to prevent over-fitting, \(max(\cdot)\) function is the ReLU activation, \(L_{f}\) and \(L_{ff}\) represent two linear transform operations, and \(b_{f}\) and \(b_{ff}\) are the corresponding bias terms. It is worth noting that both the multi-head attention and the FFN all followed a series operation of residual connection, dropout and layer normalization [33].