# NuCap: A Numerically Aware Captioning Framework for Improved Numerical Reasoning
## 3 Methodology
### Caption Decoder  
The caption decoder generates a caption from \(F_{RAOE}\) extracted from RAOE and \(F_{SAGE}\) extracted from SAGE. To effectively integrate the two complementary visual features, the decoder uses a parallel attention mechanism to generate the output caption. Each input word is first embedded into a continuous vector space and combined with sinusoidal positional encodings, yielding the initial embedding \(X^{(0)}\). Each transformer decoder layer consists of three sub-modules: masked self-attention, parallel cross-attention, and FFN. At the \(l\)-th layer, masked self-attention captures intra-sequence dependencies among the previously generated tokens:  
\[H^{(l)}=\text{Masked Self-Attention}(X^{(l-1)}) \tag{11}\]  
The output \(H^{(l)}\) then serves as a query to two separate multi-head attention modules, attending independently to the features from SAGE and RAOE:  
\[ \begin{split} V^{(l)}_{\text{SAGE}}&=\text{MHA} \big{(}H^{(l)},F_{\text{SAGE}}\big{)}\\ V^{(l)}_{\text{RAOE}}&=\text{MHA}\big{(}H^{(l)},F _{\text{RAOE}}\big{)}\end{split}  \tag{12}\]  
At each stage, adaptive attention weighting is introduced to dynamically adjust the contributions of the two visual features. Instead of directly fusing the attended features, the decoder learns separate importance scores for each feature type:  
\[ \begin{split}\alpha^{(l)}_{\text{SAGE}}&=\sigma \big{(}W_{\text{SAGE}}[H^{(l)},V^{(l)}_{\text{SAGE}}]\big{)}\\ \alpha^{(l)}_{\text{RAOE}}&=\sigma\big{(}W_{\text{ RAOE}}[H^{(l)},V^{(l)}_{\text{RAOE}}]\big{)}\end{split}  \tag{13}\]  
where \(W_{\text{SAGE}}\) and \(W_{\text{RAOE}}\) are learnable weight matrices and \(\sigma(\cdot)\) denotes the sigmoid activation. The final fused representation is computed by applying adaptive attention weights to the two features, followed by normalization and an FFN:  
\[V^{(l)}_{\text{fused}}=\text{FFN}\Bigg{(}\frac{\alpha^{(l)}_{\text{SAGE}}\cdot V ^{(l)}_{\text{SAGE}}+\alpha^{(l)}_{\text{RAOE}}\cdot V^{(l)}_{\text{RAOE}}}{ \sqrt{2}}\Bigg{)} \tag{14}\]  
This process is repeated for \(L_{c}\) layers. Finally, the output from the last layer is projected into the vocabulary space and normalized with a log-softmax to generate the probability distribution over the next word:  
\[P(y_{t}\mid y_{<t},I)=\text{log -softmax}\big{(}WV^{(L_{c})}_{\text{fused}}\big{)} \tag{15}\]  
where \(W\) is a learnable projection matrix, and \(P(y_{t})\) is the probability of generating token \(y_{t}\) at time \(t\). **Training Objective. ** To optimize the model for numerically aware image captioning, we adopt a two-stage training strategy. In the first stage, the model is trained on the COCO dataset using a combination of Number-Focused Cross-Entropy (NFCE) loss and CIDEr-D optimization. NFCE loss encourages the model to pay greater attention to numerical tokens, while CIDEr-D optimization improves the overall fluency of the generated captions. Unlike standard cross-entropy used in conventional captioning models, our NFCE loss dynamically assigns higher weights to numerical tokens during training. The model predicts the next token in the sequence while minimizing the negative log-likelihood, defined as follows:  
\[Loss_{\text{NFCE}}=-\frac{\sum_{t=1}^{T}w_{t}\log P(y_{t}\mid y_{<t},I)}{\sum_{t =1}^{T}w_{t}} \tag{16}\]  
Here, \(w_{t}\) is set to 1 for non-numerical tokens, and to a hyperparameter \(w_{n}\) (\(w_{n}>1\)) for numerical tokens. \(y_{t}\) is the ground-truth token at timestep \(t\), and \(P(y_{t}\mid y_{<t},I)\) is the predicted probability of the token \(y_{t}\) given the previous tokens and the image representation \(I\). This formulation enhances numeric sensitivity while maintaining training stability through normalization. To further improve fluency, we apply CIDEr-D optimization using Self-Critical Sequence Training (SCST) [30]. Instead of token-level supervision, this approach evaluates the entire generated caption against reference captions and assigns a reward based on the CIDEr-D score.