FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback

## Abstract

Abstract Captions are crucial for understanding scientific visualizations and documents. Existing captioning methods for scientific figures rely on figure-caption pairs extracted from documents for training, many of which fall short with respect to metrics like helpfulness, explainability, and visual-descriptiveness [ 15 ] leading to generated captions being misaligned with reader preferences. To enable the generation of high-quality figure captions, we introduce FigCaps-HF a new framework for figure-caption generation that can incorporate domain expert feedback in generating captions optimized for reader preferences. Our framework comprises of 1) an automatic method for evaluating quality of figure-caption pairs, 2) a novel reinforcement learning with human feedback (RLHF) method to optimize a generative figure-to-caption model for reader preferences. We demonstrate the effectiveness of our simple learning framework by improving performance over standard fine-tuning across different types of models. In particular, when using BLIP as the base model, our RLHF framework achieves a mean gain of 35.7%, 16.9%, and 9% in ROUGE, BLEU, and Meteor, respectively. Finally, we release a large-scale benchmark dataset with human feedback on figure-caption pairs to enable further evaluation and development of RLHF techniques for this problem.

## 1 Introduction

For scientific articles, figures like graphs, charts and plots are key to effectively conveying the work’s motivation, methodology, and results to readers. To better understand a given figure and, by extension, the research work itself, it is then crucial that the corresponding captions are informative, i.e., a given caption can represent and complement the figure, situating it in the context of the article. While the importance of figure captions is universally acknowledged, writing a good caption is not trivial. More often than not, many scholarly works contain generic figure captions and lack descriptiveness, thus rendering the figure unhelpful. This has motivated extensive research into developing methods that can automatically generate captions for figures to assist researchers in writing better captions and improve the accessibility of scientific figures for visually impaired readers.

Recent works in figure captioning formulate the problem as a vision-to-language task and have primarily focused on developing methods to encode the figure image and metadata and decode captions effectively. For model training, these methods use figure-caption pairs extracted from existing scientific articles []. While this method of data collection is appealing due to its easy access, this also leads to the problem of poor model learning and generalization when the captions are not well written. As discussed in [], more than 50% of the captions in arXiv cs.CL papers were classified as not helpful to the domain expert readers. Thus, figure captioning methods trained on such data are not calibrated to reader preferences and thus generate captions that are uninformative.

Motivated by the above, we introduce FigCaps-HF, a new benchmark and learning framework for improving figure-to-caption generation by aligning model optimization to reader preferences. Figure [1](#S3.F1) describes our proposed framework.
Our proposed framework is designed around two key questions: (1) How can we incorporate feedback from domain experts in a computationally efficient manner without compromising on performance and usability? (2) How can we develop a scalable framework for feedback generation that minimizes human labeling efforts?

To address (1) we utilize offline Upside-Down RL (
RL
or UDRL) to align the model’s generated captions with expert feedback. Unlike previous applications of RLHF [] which uses on-policy algorithms [] for reward maximization, our approach of using offline reward-conditioned behavioral cloning for model optimization is computationally efficient. Once our reward model is trained and we predict the reward scores for each sample, we do not need the reward model during figure-to-caption model training. Furthermore, offline UDRL-like methods are known to perform equally well as their other counterparts [] while being efficient and simple.

For generating feedback for figure-caption pairs in a scalable manner, we introduce a general caption-scoring mechanism, guided by domain expert feedback, which allows us to evaluate quality of figure-caption pairs with respect to readers preference.
Specifically, we utilize a small human-annotated dataset of image-caption pairs, each rated on a variety of factors including helpfulness, OCR content, takeaway etc. to train an auxiliary model to score for a given caption on the basis of the quality measure.
This step is integral because it allows us to infer caption scores for our larger training-set.
Additionally, we publically release our benchmark dataset with feedback for future research on developing figure-to-caption models.

Our experimental results indicate an increase in performance by using our Upside-Down RL-guided approach. Firstly, our empirical results indicate that our trained reward model is very well calibrated, and the annotation statistics of our ground-truth annotations match those from our inferred annotations. Secondly, we evaluate the performance of our approach on a variety of image-to-text models and observe that models with RLHF achieve the best performance; specifically, our best-performing model has a 35.7% increase in BLEU, 16.9% increase in ROUGE-L, and 9% increase in METEOR score using RLHF. Our ablation studies show the beneficial effects of further investigation into parts of our setup, including the type and nature of feedback used.

Summary of Main Contributions.
The key contributions of this work are as follows:

- •
We introduce a novel RLHF framework for figure-to-caption generation that leverages a *small amount* of actual human feedback to learn an oracle model to infer human feedback on a larger scale for any unknown figure-caption pair encountered in the wild.
- •
We propose a technique that learns an oracle model from a small amount of human feedback, which can then be used for predicting the human feedback scores for any new unseen figure-caption pair.
- •
Extensive experiments demonstrate the effectiveness of our framework for figure-to-caption generation via human feedback.
- •
To facilitate further research on this important new problem, we release a comprehensive benchmark dataset for figure-to-caption generative models with human feedback. This new benchmark data will enable other researchers to develop even better RLHF models for figure-to-caption generation.

## 2 Background

Figure Caption Generation. Most prior work in scientific figure-captioning can be broadly divided into the following three categories based on their different input modalities: the figure-image alone, the underlying data chart of the figure, and relevant texts from the original article.
In the vision-based approach, prior works have primarily utilized a vision-encoder to encode figure-features followed by a text-decoder to generate captions. [, , ] focus on explicitly extracting different features of the figure before combining their information for downstream tasks. [, , ] create and leverage FigCAP, a synthetic figure-caption dataset and adapt an LSTM model to produce captions. More recently, [] collected a dataset, namely SciCAP, from published articles and used a CNN+LSTM pipeline to generate captions. There are few prior works which examine the abilities of utilizing SOTA image-captioning pipelines, which primarily utilize large pre-trained Transformer [] modules, for figure-captioning.
A closely related task is Figure Question Answering, which formulates the more general problem of figure understanding as a visual-question answering task; there has been a variety of works in this space towards modeling  [, , , , , , ] as well as creating curated datasets including DVQA [], FigureQA[], PlotQA [], Leaf-QA [], and ChartQA [].
In the data-driven approach, research focuses on using only the tabular data, as well as some metadata, to generate a caption. Table-to-Text[] focuses on generating captions for rows in arbitrary tables. Chart-to-Text [] creates a new large-scale dataset focusing on figure-captioning and adopt an encoder-decoder transformer model to process the underlying data table and generate summaries.

In the text-driven approach, [] focuses on utilizing only the relevant text in an article to generate a figure-caption, for example, using text explicitly referencing the figure.

Learning with Human Feedback Aligning model predictions with human preference has been shown to improve task performance in various areas, including natural language processing tasks like language model pretraining [], machine translation [, ], text summarization [], unlearning undesirable behaviors from language models [], computer vision tasks like text-to-image generation [, ] and reinforcement learning tasks like training RL agents [, , ]. In contrast to prior works, our work also aims at improving figure caption generation by optimizing model learning to align with domain expert feedback. However, unlike previous work that leverages on-policy RL [] algorithm to maximize the reward-weighted likelihood, our framework utilizes reward-conditioned behavioral cloning [], an offline variant of upside-down RL method [] to optimize model learning for reader preference. This provides a simpler and more controllable framework for human preference alignment. Furthermore, our feedback scheme allows for incorporating multiple feedback at different granularity as reward signal during the model optimization step, thus improving model learning.
We propose a general human-feedback model along with a new benchmark with feedback to enable further research in developing and evaluating methods that optimize for reader preference.

## 3 Framework

Figure: Figure 1: RLHF Framework for Figure-Caption Generative Models
Refer to caption: /html/2307.10867/assets/x1.png

In this section, we explain our proposed framework for learning with expert feedback(Figure [1](#S3.F1)). We first describe a standard figure captioning pipeline (Sec. [3.1](#S3.SS1)). Next, we provide details of designing and training a generalizable human-feedback prediction model (Sec. [3.2](#S3.SS2)). Finally, we discuss our feedback-aligned model training strategy instantiated as a simple RLHF framework (Sec. [3.3](#S3.SS3)).

### 3.1 Preliminaries

In a figure-captioning problem, we are initially provided with a dataset $D_{w}$ consisting of figure-caption pairs $\{I_{w},T_{w}\}$.
Given the dataset $D_{w}$, we can then define a model $f_{\theta}$, that takes in information corresponding to the figure and outputs a sequence of text as output. Typically, the input consists of only the figure image. However, other sources of information like figure-specific text from the corresponding document, OCR, figure metadata can also be utilized as input samples.

Assuming the general case of figure image as input, model $f_{\theta}$ is constructed using a vision encoder module to get image-based encoding and a language encoder-decoder module to encode and generate corresponding text. The weights $\theta$ can either be randomly initialized, or initialized by large-scale pretrained model weights. Furthermore, the model weights corresponding to the vision encoder and text encoder-decoder models can either be initialized with separate weights or jointly trained model weights. After initialization, model $f_{\theta}$ can then be trained for the task of caption generation.

Generally, for training such a model, Language Modeling (LM) loss is used as a standard training objective.
Let $\{I_{i},T_{i}\}\in D$ be the input to the model $f_{\theta}$, where $I_{i}\in\mathbb{R}^{n}$ is the input figure, and $T_{i}$ is the corresponding text sequence. Additionally, $T_{i}$ is represented as sequence of $K_{j}$ tokens from a fixed vocabulary $\mathcal{V}$: $T_{i}=(T_{i,1},...T_{i,K_{j}})$, where $K_{j}=|T_{i}|$.
Then the training objective is defined as:

$$ $\mathcal{L}_{\text{LM}}=\frac{1}{K_{j}+1}\sum\limits_{j=0}^{K_{j}+1}H(T_{i,j}|I_{i},(T_{i,0},...,T_{i,j-1})),$ (1) $$

where H denotes the cross-entropy loss and $(T_{i,0},...,T_{i,j-1})$ represents all the tokens in the caption prior to $T_{i,j}$.

### 3.2 Human Feedback Prediction Model

To improve figure-to-caption generation, we propose to incorporate domain expert feedback into our optimization step. To generate feedback for figure-caption pairs, we thus propose to learn a feedback prediction model to score individual datasample based on different metrics representing reader preferences. Our objective is to learn a model that can predict human feedback scores for unseen captions accurately given small set of training samples.

To this end, we first label a small control set $D_{h}$ consisting of $M$ figure caption pairs $\{I_{w},T_{w}\}$ with domain experts ratings. Here we assume that $M\ll N$, i.e. the size of the control set is significantly less than the original noisy dataset (For example, if $N~{}=100,000$, then $M~{}=100$). We can now train a model on $D_{h}$ to predict the human expert ratings for the original dataset $D_{w}$.
Specifically, given human feedback dataset $D_{h}$ containing figure-caption pairs $\{I_{h},T_{h}\}\in D_{h}$ and $k$ human expert evaluation metrics for each datasample $y\in{y_{0},y_{1},...y_{k}}$, we want to train $k$ models $R(x_{i},\theta)_{k}$ to predict the $k$ scores respectively. Here the output of a model $R(x_{i},\theta)_{k}(T_{h})$ is a scalar quantity denoting a specific metric score for the given input caption. Thus we formulate the scoring problem as a regression task.
Specifically, we can define our human-feedback prediction model as follows:

$$ $R(x_{i},\theta)_{k}(T_{h})=g(l(\theta_{l},x_{i}),\theta_{g}),$ (2) $$

where, $R(x_{i},\theta):\mathbb{R}^{N}\rightarrow\mathbb{R}$, $l(x_{i},\theta_{l}):\mathbb{R}^{N}\rightarrow\mathbb{R}^{D}$ and $g(u_{i},\theta_{g}):\mathbb{R}^{D}\rightarrow\mathbb{R}$. In the above, $l(.,\theta_{l})$ is an embedding function that takes in input data $x_{i}\in\mathbb{R}^{N}$ and generates corresponding representation $u_{i}\in\mathbb{R}^{D}$, and $g(.,\theta_{l})$ is a regression function to generate the scores respectively. We only train the regression function while keeping the weights of the embedding function fixed. For training the regression function, we use mean-squared error loss, written as: $\mathcal{L}_{\text{R}}=\sum_{i=1}^{D_{h}}(\hat{y_{i}}-y_{i})^{2},$ where $\hat{y_{i}}$ is the predicted score while $y_{i}$ is the ground-truth evaluation score.
After training the human-feedback prediction models, we compute scores for all the samples in the training dataset $D_{w}$ to construct our new set, which will be used for training the figure-caption model.

### 3.3 Reinforcement Learning with Human Feedback

Given the human-feedback prediction model described above, we can now use it as a reward model to train an image-to-text model that generates higher-quality captions. We achieve this goal, by formulating the problem as a reinforcement learning task. Specifically, for the given training dataset $D_{w}$ containing figure caption pairs $\{I_{w},T_{w}\}$, we can consider figures $I_{w}$ as the state of the environment, caption $T_{w}$ as the actions and the corresponding predicted metric scores $R(T_{w})$ for these captions as the rewards/outcomes. Then our objective is to learn a policy (which in this case would be the image-to-text model $f(\theta)$ that we want to train) that maps from states($I_{w}$) to actions($T_{w}$) such that we maximize the reward for each action. In this way, we can generate output captions that better align with human judgment of a good figure-caption.

While there are many different approaches in the reinforcement learning literature [] to achieve the above objective, we specifically focus on offline upside-down reinforcement learning (UDRL). We select offline UDRL because it computationally efficient and robustly performant without being algorithmically complex[]. In UDRL, the motivation is to learn a policy ($\pi_{\theta}$) that maps the states ($S_{t}$) to actions ($a_{t}$) conditioned on specific rewards($r_{t}$). Thus the learning problem can be formulated as a supervised learning problem, wherein we first sample the triplets of ${S_{t},a_{t},r_{t}}$ from the environment to construct our dataset, which is then used to train $\pi_{\theta}$ using standard supervised learning objective. Specifically, we can write the optimization problem as:

$$ $\max_{\theta}\sum_{t\in D}\mathbb{E}[\log\pi_{\theta}(a_{t}|S_{t},r_{t})],$ (3) $$

We follow the above UDRL framework for learning an image-text model $f(\theta)$. For our setting, we consider our image-to-text model $f(\theta)$ as our policy $\pi_{\theta}$. For each caption $T_{i}\in T_{w}$, we compute a reward score and quantize it to generate a control token $c_{i}$. Specifically, we binarize the reward score to generate two control tokens: 
 and 
. In general, the level of quantization is a hyperparameter that can be selected according to task or other factors. For each caption $T_{i}\in T_{w}$, we compute the control token by thresholding the output of $R$, i.e. if $R(I_{i},T_{i})\geq t$ then $c_{i}=$ 
, else $c_{i}=$ 
. Here $t$ is a hyperparameter. Given the additional human feedback, we fine-tune $f_{\theta}$ with the following new objective function:

$$ $\mathcal{L}_{\text{HF}}=\frac{1}{K_{j}+1}\sum\limits_{j=0}^{K_{j}+1}H(T_{i,j}|I_{i},(c_{i},T_{i,0},...,T_{i,j-1})),$ (4) $$

Where $c_{i}$ refers to the control token computed using the reward function $R$ for a given caption $T_{i}$.

**Table 1: Summary of our benchmark dataset for figure-caption generative models with RLHF.**
|  | # Fig-Caption Pairs | Human Feedback | Median | Mean | Std | Q1 | Q3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Actual | 400 | Helpfulness | 3 | 3.01 | 1.19 | 2 | 3 |
| Human Feedback | Takeaway | 2 | 2.16 | 1.22 | 1 | 2 |  |
| Visual | 2 | 2.11 | 1.08 | 1 | 2 |  |  |
| OCR | 4 | 3.83 | 0.80 | 4 | 4 |  |  |
| Predicted | 106,834 | Helpfulness | 2.89 | 2.89 | 1.07 | 2.17 | 3.61 |
| Human Feedback | Takeaway | 1.95 | 2.06 | 1.03 | 1.33 | 2.66 |  |
| Visual | 1.91 | 2.02 | 1.01 | 1.31 | 2.63 |  |  |
| OCR | 3.88 | 3.84 | 0.83 | 3.32 | 4.41 |  |  |

## 4 FigCaps-HF: Figure-Captioning with Human Feedback Benchmark

As noted before, captions from online scientific articles can be of ’low quality’ with respect to domain expert quality metrics []. This can, in turn, lead to poor figure-captioning models as these are trained to simply maximize the likelihood of the raw training data. Thus, our goal with the new benchmark is to provide additional training signals to improve figure-caption model without incurring the cost of re-creating a new dataset.

To this end, we propose our new benchmark for figure-captioning with feedback. Our benchmark consists of 133,543 figure-caption pairs [] with feedback scores. Our dataset contains feedback based on different measures to evaluate quality of the author written captions for the corresponding figure. For each figure-caption pair, we evaluate the data sample based on four quality measures: (1) Helpfulness, (2) Takeaway, (3) Visual-descriptiveness (visual) and (4) Image-text (OCR) []. Each quality metric is selected to measure the ability of the readers to comprehend and draw inferences based on the provided figure and the corresponding caption.

We compute the feedback scores for each data sample in a scalable manner by first annotating a small subset with domain-expert feedback and then predicting score for the entire dataset using the human-feedback model described in Sec. [3.2](#S3.SS2). Specifically, we select 438 randomly sampled figure-caption pairs, each annotated by domain experts []. Each pair has been evaluated on 5-point Likert scale for each of the above mentioned quality metric. Using this labeled subset, we train a human-feedback prediction model to generate scores for the remainder of the dataset. Unlike the subset, we keep the scores for the entire dataset as a continuous value. This allows the users of the benchmark to accordingly decide their own scheme for labeling each figure-caption pair based on different thresholding criteria, thus providing flexibility for fine-grained feedback.

Table [1](#S3.T1) presents an overview of the statistics related to the actual and predicted human feedback for the captioning of scientific figures. We see that the predicted human feedback values in our study show a diverse range, as indicated by the small standard deviation of $1\pm 0.2$ and a consistent mean value across all ratings. Additionally, the alignment of the median predicted scores with the actual human feedback values indicates that the model’s performance is not skewed towards any particular rating but provides an accurate assessment across the range of ratings. This suggests that the human-feedback prediction model used to infer the scores is generalizable and can accurately assess the quality of captions across various ratings. Furthermore, the proposed model provides reliable scores for captions that fall outside the typical range of scores.

For further implementation details, please refer to the section "Additional Dataset Details" in the appendix.

**Table 2: Comparison with state-of-the-art methods. For all the metrics, higher values are better ($\uparrow$).**
|  | Model | #Params | ROUGE-L | BLEU | METEOR |  |
| --- | --- | --- | --- | --- | --- | --- |
| ocr-only | Pegasus [] | 0.27B | 2.6 | 4.78e-2 | 4.2 |  |
| Figure-Only | TrOCR [] | 0.23B | 2.5 | <0.1 | 1.8 |  |
| BEiT+GPT2 | 0.24B | 14.2 | 0.5 | 1.24 |  |  |
| ViT[] + RoBERTA[] | 0.23B | 14.0 | 1.2 | 12.1 |  |  |
| ViT[] + GPT2 | 0.24B | 14.2 | 1.8 | 12.6 |  |  |
| Figure-Caption | PromptCap [] | 0.47B | 13.0 | 0.9 | 8.2 |  |
| Flamingo [] | 1.14B | 8.7 | 0.1 | 4.6 |  |  |
| GIT [] | 0.17B | 11.9 | 0.2 | 9.1 |  |  |
| BLIP [] | 0.25B | 13.0 | 1.4 | 13.2 |  |  |
| CLIPCap [] | 0.15B | 10.3 | 1.2 | 13.1 |  |  |
| rlhf | Ours-BLIP-RLHF | 0.25B | 15.2 | 1.9 | 14.5 |  |
| Ours-ViT+GPT2-RLHF | 0.24B | 13.8 | 2.0 | 12.6 |  |  |

## 5 Experiments

### 5.1 Setup

For our human-feedback prediction model, we use MCSE [] as embedding function and a 2-layer MLP as regression function. For comparative evaluation, we select the following models as our baselines based on input: (1) OCR-only: Pegasus[] , (2) Figure-only: TrOCR [], BeiT$+$GPT2 , ViT$+$GPT2 [], ViT$+$RoBERTA [, ] and (3) Figure-Caption: PromptCap [], Flamingo [], GIT [], BLIP [] and CLIPCap []. We use ROUGE-L [], METEOR [] and BLEU [] metrics to compare each model’s performance. For more details regarding individual baselines, metrics and dataset, please refer to the Appendix.

### 5.2 Results

We show our experimental results in Table  [2](#S4.T2). Specifically, we want to evaluate the performance of our RLHF framework for figure-caption generation. To this end, we compare our framework with standard fine-tuning method and benchmark the performance on the Test set of our proposed benchmark. We show fine-tuning results for all the above mentioned baselines. We use BLIP and ViT$+$GPT2 to evaluate our RLHF framework.
From Table  [2](#S4.T2), models trained using our proposed RLHF formulation performs better than simple fine-tunning.
Specifically, for BLIP, RLHF provides has a 35.7% increase in BLEU, 16.9% increase in ROUGE-L, and 9% increase in METEOR score.
For ViT+GPT2, RLHF provides a 11.1% increase in BLEU.

Aggregating the metrics, BLIP performs best, which is likely due to its aligned image encoder and text decoder which are pre-trained jointly. In contrast, ViT+GPT2’s modules are not aligned/trained jointly and the text decoder learns to attend to the vision encoder only during fine-tuning. Hence, for our approach, the type of pre-training can have an impact on the amount of model improvement.

Overall, since the performance increase is generalized among models with different pre-training strategies and overall model-structure, the results show the benefits of using this simple UDRL framework for fine-tuning. Utilizing only a small amount of human annotated data, different scoring mechanisms and prompts can be further developed to take advantage of this limited supervision and further increase performance.

Figure: Figure 2: Results of our Human Feedback Prediction Model. Here we show the three figure-caption pairs with the highest (left; green) and smallest (right; red) “helpfulness” human feedback score from our trained HF model. Notably, the figure-caption pairs rated highly by our human-feedback predictive model are those that are obviously better as they mention specific takeaways, as well as OCR from the figure, and even visual aspects are often mentioned. In contrast, the figure-caption pairs with lowest scores by our predictive model are those that are extremely vague, without actual takeaways, OCR mentions, and without mentioning any visual aspects from the figure.
Refer to caption: /html/2307.10867/assets/x2.png

### 5.3 Qualitative Results

To validate our framework’s ability to generate better reader-aligned captions than standard approaches, we conduct an extensive qualitative study. We evaluate the results of the human feedback prediction model and the figure-captioning models trained with RLHF. We provide our analysis below:

Human Feedback Prediction Model:
To evaluate the generalizability our model, we first computed the score predictions on all the figure-caption pairs.
Then we ordered the figure-caption pairs by the predicted scores and selected the top-3 figure-caption pairs with the largest score along with the bottom-3 figure-caption pairs with the smallest score.
Results are provided in Figure [2](#S5.F2).
We observe that the figure-caption pairs with the largest scores are highly helpful to the reader as they mention specific takeaways from the figure (*e.g.*, “as students make more applications, the number of students who get into their top-choice school decreases, while the number of overall acceptances increases.”), as well as mentioning specific visual aspects that are important to the understanding of it (*e.g.*, “… Vertical lines show the true p (blue) and $\beta$ (orange)”).
In contrast, the bottom-3 figure-caption pairs scored the lowest (shown in red on the right in Figure [2](#S5.F2)) are vague, without any takeaways, nor reference to visual elements in the figure.

Figure: Figure 3: Generated captions from our RLHF framework using BLIP as the base model compared to BLIP without RLHF.
Refer to caption: /html/2307.10867/assets/x3.png

**Table 3: Results with multi-labeled human feedback.**
| Model | #Params | ROUGE-L | BLEU | METEOR |
| --- | --- | --- | --- | --- |
| Binary Feedback | 0.25B | 15.2 | 1.9 | 14.5 |
| Multi-label Feedback | 0.25B | 15.3 | 2.2 | 15.1 |
| Binary + Multi-label Feedback | 0.25B | 15.6 | 1.9 | 14.8 |

**Table 4: Results with different human feedback metrics (BLIP-RLHF).**
|  | #Params | ROUGE-L | BLEU | METEOR |
| --- | --- | --- | --- | --- |
| Helpfulness | 0.25B | 15.20 | 1.86 | 14.50 |
| Takeaway | 0.25B | 16.76 | 2.30 | 15.98 |
| Visual | 0.25B | 16.78 | 2.30 | 15.95 |
| OCR | 0.25B | 16.54 | 2.23 | 15.65 |

**Table 5: Results with different embedding models for the human-feedback model.**
|  | #Params | ROUGE-L | BLEU | METEOR |
| --- | --- | --- | --- | --- |
| BERT | 0.25B | 15.65 | 1.93 | 14.73 |
| SciBERT | 0.25B | 15.77 | 2.01 | 15.09 |
| BLIP | 0.25B | 15.73 | 1.98 | 14.94 |

Figure-Caption Generative Model:
To evaluate the quality of captions, we compare the output of BLIP-RLHF and BLIP (Fine-tuned) models.
We show some of the interesting results in Figure [3](#S5.F3).
In general we see that, qualitatively BLIP-RLHF produces better captions compared to fine-tuned BLIP. In most cases, captions produced by BLIP (Fine-tuned) are either
explaining the given figure incorrectly (Figure [3](#S5.F3), leftmost sub-figure), not relevant (Figure [3](#S5.F3), middle sub-figure) or are completely uninformative (Figure [3](#S5.F3), rightmost sub-figure). On the other hand, captions produced by BLIP-RLHF method are more faithful to the figure, captures semantic relation between texts to summarize the phenomenon and utilizes visual attributes in explaining the figure. We provide more examples and analysis in the Appendix.

### 5.4 Ablation Study

We perform the following ablation experiments to better understand different components of our framework. We provide the details of our findings below.

Effect of Different Human Feedback Labels:
To understand how the level of quantization of our reward signals (Binary vs Multi-level) affect the model learning, we conduct the comparative study by modifying the feedback while training the BLIP-RLHF model.
First, we trained the model for 10 epochs using multi-labeled human feedback (Row 2) , specifically, we used 5 levels of human feedback (very bad , bad, neutral, good, very good) calculated at the 20^th, 40^th, 60^th, 80^th percentile respectively to ensure an equal number of samples.
We also experimented with varying label coarsity during the course of training (Row 3); specifically, we trained the model with 5 epochs of binary-label feedback followed by 5 epochs of multi-label feedback.
We show our results in Table [3](#S5.T3). Both aforementioned approaches with finer feedback outperform simple binary feedback and demonstrate, through our RL framework, the model’s ability and receptiveness to leverage more finer human feedback effectively. The experiment also indirectly validates the quality of our human prediction model, which is capable of providing useful labels at different levels of coarsity that can be leveraged for increased performance on a downstream task like figure-captioning. The study also shows the further potential gains that can be made by further investigating different feedback mechanisms.

Effect of different human feedback metrics
We also study the effect of using different metrics as feedback for training the figure-caption models.
In particular, we compare results of training the BLIP-RLHF model with the Helpfulness, Takeaway, Visual-descriptiveness (visual) and Image-text (OCR) feedback scores provided in our benchmark. We provide the results in Table  [5](#S5.T5). We see that training BLIP-RLHF with Takeaway, visual and COR feedback performs better than Helpfulness. This is understandable as helpfulness rating is subjective while Visual and Takeaway are objective evaluation metrics respectively. This shows that the type of feedback is important and that further gains can be made by modeling different aspects of the annotated human dataset.

Effect of different figure-caption representations
To understand the effect of using different figure-caption representations, we use BERT, SciBERT and BLIP to encode our figure-captions pairs and use their final-layer representations of the [CLS] token to train our human feedback prediction model. The different representations outperform our default MCSE implementation, indicating that our human feedback prediction model, and downstream figure-captioning performance, are sensitive to the quality of representations used. Additionally, further performance gains can be made by using different representations, for example, by encoding different modalities (text only vs joint encoding of text and vision).

Effect of human feedback position:
To understand the sensitivity of the model to the position of human feedback, we compare the performance of appending and pre-pending the human feedback labels in Table  [6](#S5.T6). Since our models generate text, during test time, without any human feedback label prompt, they can only rely on feedback during training. Additionally, due to the auto-regressive generation of our models, they only observe the label before generation, and for append, only observe the label after generation. Intuitively, pre-pending should work best since the generation is conditioned on the label. The results support this and show that ViT+GPT2 and BLIP perform better when trained with pre-pended human feedback.

**Table 6: Comparing RLHF prepend to append.**
|  | Model | #Params | ROUGE-L | BLEU | METEOR |
| --- | --- | --- | --- | --- | --- |
| rlhf-append | Ours-BLIP-RLHF | 0.25B | 13.6 | 1.8 | 13.2 |
| Ours-VIT-GPT2-RLHF | 0.24B | 13.8 | 1.6 | 11.9 |  |
| rlhf-prepend | Ours-BLIP-RLHF | 0.25B | 15.2 | 1.9 | 14.5 |
| Ours-ViT+GPT2-RLHF | 0.24B | 13.8 | 2.0 | 12.6 |  |

## 6 Discussion, Limitations & Conclusion

In this work, we contribute a new methodology to improve caption generation for scientific figures. We show that incorporating domain expert feedback in learning a model for figure-ti-caption generation improves both model performance and caption quality. Our proposed framework is scalable (requires limited manual human effort in labeling) and flexible (allows for incorporating multiple reward signals at different granularity). We also propose a new benchmark of figure-caption pairs with caption quality scores to further the research efforts in reader-aligned figure-captioning tasks. We hope that this new dataset will allow the researchers to benchmark their own methods for incorporating human feedback in figure-to-caption generation tasks and various other image-to-text generation tasks.

While we empirically show that our framework can generate better captions, it currently lack the ability to incorporate multiple complementary feedback. Furthermore, currently we need to quantize the reward score to be able to utilize it as a valid feedback when training the model. This limits the applicability of our framework in scenarios where a numerical score does not correspond to a categorical label like ’good’ or ’bad’.

As a future goal, we aim to improve our framework by focusing on the above issues.
We also aim to further explore the properties and further use cases of the human feedback prediction model. For example, we would like to further benchmark the generalizability of the human-feedback prediction model to various data and task distribution shifts. This can provide further insights into developing methods that are robust and adaptable.

## 7 Ethics Statement

Our work on improving figure caption generation is important in building accessible assistive tools for scientific community and visually impaired people. However, like many works in the area of generative AI, our work/general ideas also carry the risk of misuse i.e. our proposed method can be advertised by a third party as a deployable product, when in fact, we believe that our proposed method is a research endeavor and still has room for improvement. Another potential negative impact of our work could be complacent consideration of generating human feedback without due consideration to human subjects involved. This is our key motivation to make our dataset with feedback labels public, to allow interested researchers to develop and benchmark their own methods that require feedback.

Finally, we comment on the dataset privacy considerations for the proposed benchmark. Our proposed dataset and other datasets considered in this work are licensed for academic/non-commercial research (Creative Commons Attribution-Non Commercial-Share Alike 4.0 International License). Our proposed dataset does not contain any personal information.

## Appendix A Description of metrics used for Feedback assessment

We followed [] to evaluate a given figure-caption pair from the perspective of a reader. Specifically, we used the following measures:

- •
Helpfulness: This is a subjective measure to evaluate whether a given caption is able to inform the reader about the information conveyed in the corresponding figure.
- •
Takeaway: This measure is used to assess a given caption based on whether it is able to convey a conclusive information about the given figure image.
- •
Visual-descriptiveness (visual): We define visual descriptiveness of a given caption as a measure of how much the given caption is grounded with respect to the figure. For example, a caption that describes the visual elements of the figure like color and shape should be more informative to the readers.
- •
Image-text (OCR): We formulate OCR as a metric to evaluate if the given caption included textual elements of the figure like title, legends and labels when describing the figure.

## Appendix B Experimental Setup

### B.1 Datasets

For all our models, we use the same splits in our benchmark dataset; this portion contains 106,834 training pairs, 13,354 validation pairs, and 13,355 test pairs. The primary difference between our baseline and RLHF models are the augmented figure-captions that are used for training the latter (figure-images remain the same), testing figure-caption pairs remain the same for both.

Human-Feedback Augmented Caption
For our RLHF-trained models, we generate human-feedback augmented figure-captions to align the model to human preferences.
In this process, for each caption, we first use MCSE [] to generate text-embeddings for the captions in the human annotated dataset ( 400 pairs). An auxiliary scoring-model (MLP Regressor) is then trained to predict the reader-preference scores using these embeddings, and later used to predict human feedback scores for the entire dataset; we pick the median of these scores as a pivot and label all captions with higher scores as "good", and lower scores as "bad". After pre-pending our captions with these annotations, we effectively train our models in a UDRL framework. Code to implement and generate new human-feedback augmented captions are provided in the GitHub repository.

### B.2 Evaluation Metrics

We evaluate the generated captions using a variety of common metrics.
ROUGE-L [] is a recall-oriented metric which uses the Longest Common Subsequence between the reference and the model generated caption, we report the F1 score.
BLEU[] is a precision-oriented metric which uses n-gram overlap, and an additional penalty for sentence brevity. Here, we are using BLEU@4 (i.e $n=4$ for n-gram overlap)
METEOR[] measures generalized unigram-overlap and computes a combination of the precision and recall.
For a summary of the evaluation metrics leveraged by traditional image captioning works, see [].

### B.3 Baselines

For comparative evaluation of our proposed framework, we selected methods based on the information used to generate a caption. Specifically, we categorize the baselines models into following categories:

- •
Figure-only: We refer a method as ’Figure-only’ if the given method computes an output text based on uni-modal embedding of the input image. Model architecture under this category generally comprises of some combination of a vision encoder and a text decoder module.
- •
OCR-only: Similar to above, if a method generates an output text using only text as input to the text decoder model, we classify the same as ’Text-only’ methods. Specific to our case, we can extract some textual descriptions of a given figure by applying an off-the-shelf OCR method. Hence from here on , w would explicitly refer to methods falling under the above mentioned criteria as ’OCR-only’ models. Methods under this category utilizes a text encoder and text decoder modules as part of their model architecture.
- •
Figure-Caption: Finally for methods which compute multi-modal embedding from text and image uni-modal embeddings to be utilized for generating output text using a text decoder, we categorize them as ’Figure-Caption’ methods. All the methods under this category generally include a vision encoder, text encoder and text decoder modules as part of their model architecture.

We evaluate a variety of strong image-captioning models and a text-summarization model as our baselines. We provide details of individual models below:

Unimodal Vision-Encoder Language-Decoder Models. These models consist of a pre-trained Vision-Encoder (e.g. BEiT [], ViT []) and a pre-trained Text-Decoder/Language model (e.g. GPT-2 [], RoBERTA []). The two submodules are not pre-trained jointly, and only aligned during fine-tuning via randomly initialized cross-attention layers in the decoder. These models simply take in the figure-image and generate the corresponding caption.

Pegasus [] is a Transformer-based pre-trained model for text-summarization. We use PEGASUS to generate figure-captions by summarizing the OCR extracted from the image itself.

TrOCR [] is a Transformer-based OCR model designed to extract text from a given image. It uses BEiT/DEiT as a vision encoder and RoBERTA as a text decoder, similar to the aforementioned image-to-text models, with the addition of an OCR-focused pre-training. We fine-tuned the model to generate a caption from a given figure-image.

GIT [] is a Generative Image-to-Text model. It uses a pre-trained Vision-Transformer encoder and a randomly initialized Language Transformer decoder (e.g. BERT[]), similar to the aforementioned image-to-text models, and further jointly pre-trains them using the Language Modeling task. We evaluated the performance of both fine-tuned and pre-trained versions of GIT.

BLIP [] is a Multi-Modal Vision-Language decoder model. It has a similar architecture to the Vision-Encoder Decoder image-to-text models, but utilizes interchangeable attention layers in the text-decoder to behave as either an unimodal encoder, an image-grounded text encoder or an image-grounded text decoder. The model is pre-trained using the LM, ITM and ITC losses jointly.

PromptCap[] is a prompt-based image-captioning model. In addition to taking an image, the model can also incorporate a user-defined prompt to guide the generated caption. PromptCap utilizes a pre-trained Transformer-based encoder-decoder model, namely OFA [] which is further pre-trained. PromptCap is evaluated zero-shot using its pre-trained version due to a lack of available documentation.

Flamingo [] is a Transformer-based encoder-decoder model which has a similar structure to the aforementioned image-to-text models. However, the pre-trained vision encoder and text decoder are frozen and an additional module is used to learn transformed visual representations for the frozen language model to attend to.

CLIPCap [] is a Transformer-based encoder-decoder model. It utilizes CLIP as an image encoder, and using a mapping network, maps image embeddings to a prefix which is used by a text-decoder, namely GPT2, to generate a caption. The pre-trained modules and the freshly-initialized mapping network are simply fine-tuned during the training process.

From the set of baseline models described above, we fine-tuned ViT+RoBERTA, ViT+GPT2, BEiT+GPT2, GIT, BLIP and CLIPCap on the training set of our dataset. To understand zero-shot performance for figure-captioning task, we evaluated Pegasus, TrOCR, PromptCap and Flamingo-mini models by using their pretrained weights for inference without fine-tuning them on our dataset.

For all fine-tuning experiments, we used AdamW optimizer with $\beta_{1}=0.9$ & $\beta_{2}=0.99$. We fine-tuned ViT+RoBERTA, ViT+GPT2, BEiT+GPT2 for 5 epochs with batch size 8. We used a linear rate scheduler with an initial learning rate of $2e-5$; generation was handled using a greedy strategy. For training GIT, BLIP and CLIPCap models, we used a learning rate of $1e-5$ and used nucleus sampling for text generation during inference.

## Appendix C Qualitative analysis

In this section, we provide detailed qualitative analysis of the output of BLIP-RLHF and BLIP (Fine-tuned) models (Figure [3](#S5.F3)).
In the first example shown at the top left in Figure [3](#S5.F3), we see that the generated caption with the base model BLIP has many issues.
For instance, it seems to have identified the word “edges” from the name of the model “Deep-Edge” used in the figure, despite that the figure does not actually show the number of edges in each experiment as the caption mentions.
Instead, it shows the average epoch time in seconds for each of the different experiments, which is roughly captured by the BLIP-RLHF caption.
In the second example shown in the middle of Figure [3](#S5.F3),
the BLIP model completely hallucinates the caption whereas the BLIP-RLHF caption reveals the essence of the figure while also seemingly using the semantics of this specific chart-type, *e.g.*, the phylogenetic tree shows the evolutionary relationships between different groups of fish and from the phylogenetic tree we can see how large each group is and the similarities between the groups of fish as well.
This also illustrates the ability of our approach to generalize to a variety of different chart types as we only obtained actual human feedback for line charts.
For the captions generated for the chart shown at the right in Figure [3](#S5.F3), we see that BLIP generates a completely useless caption that has no alignment with the actual chart.
In comparison, the caption generated using BLIP-RLHF mentions the estimated and actual curves present in the chart while also correctly indicating that these curves are plotted in terms of time.
Most strikingly, the generated caption refers to the curves using their color (*i.e.*, red line, blue dots), hence, the generated caption not only mentions important text from the chart, but also refers to the visual properties of the curves when mentioning them in the generated caption.

## Appendix D Datasheet

We provide a Datasheet [] documenting details about our proposed benchmark and its usage.

### D.1 Motivation

For what purpose was the dataset created? We created this dataset to provide researchers ability to develop and evaluate their respective figure-to-caption generation pipelines for reader preference aligned caption generation.

Who created the dataset (e.g., which team, research group) and on behalf of which entity(e.g., company, institution, organization)? We would provide the details of the authors upon acceptance of the paper, due to double-blind review process.

Who funded the creation of the dataset? No funding was recieved in any form in creation of this dataset.

#### D.1.1 Author Statement

The authors of this paper bear all responsibilities for the distribution, and maintenance of our proposed dataset. This document follows the Datasheet format [] whenever applicable.

### D.2 Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? Yes, the dataset is public and available for usage on the internet.

How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? The dataset and the corresponding codebase used in generating the dataset is avaialable through following links:
Benchmark: [https://doi.org/10.6084/m9.figshare.23504517](https://doi.org/10.6084/m9.figshare.23504517)
Code: [https://github.com/FigCapsHF/FigCapsHF](https://github.com/FigCapsHF/FigCapsHF)
Documentation: [https://figcapshf.github.io/](https://figcapshf.github.io/)

Have any third parties imposed IP-based or other restrictions on the data associated with the
instances? No.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.

### D.3 Maintenance

Who will be supporting/hosting/maintaining the dataset? The authors will be supporting, hosting and maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)? We would provide the details of the contact persons upon acceptance of the paper, due to double-blind review process.

Is there an erratum? No. We will accordingly make announcements if there is any.

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? Yes. Announcements regarding any updates to dataset and code would be posted here: [https://github.com/FigCapsHF/FigCapsHF](https://github.com/FigCapsHF/FigCapsHF)

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? N/A

Will older versions of the dataset continue to be supported/hosted/maintained? Yes.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes.

### D.4 Composition

What do the instances that comprise the dataset represent? Please refer to section [D.7](#A4.SS7) for detailed description of the dataset composition.

How many instances are there in total (of each type, if appropriate)? in total we have 06,834 training pairs, 13,354 validation pairs, and 13,355 test figure-caption pairs with feedback scores.

Does the dataset contain all possible instances or is it a sample (not necessarily random) of
instances from a larger set? The dataset contain all possible instances

Is there a label or target associated with each instance? Yes. Each figure image in the dataset has a corresponding caption and a set of values representing the predicted feedback score for metrics (’helpfulness’, ’ocr’, ’visual’, ’takeaway’.

Is any information missing from individual instances? No.

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? N/A

Are there recommended data splits (e.g., training, development/validation, testing)? Yes. The dataset consists of 3 splits: Train, Validation and Test. We have explicitly provided individual splits as separate data folders.

Are there any errors, sources of noise, or redundancies in the dataset? No.

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is entirely self-contained and does not require any external resource.

Does the dataset contain data that might be considered confidential? No.

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening,or might otherwise cause anxiety? No.

### D.5 Collection Process

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and
how were they compensated (e.g., how much were crowdworkers paid)? The authors were involved in the curation of the data obtained from a publicaly avaialbe source.

Over what timeframe was the data collected? Februray 2023-May 2023

### D.6 Uses

Has the dataset been used for any tasks already? Our work on human feedback aligned figure caption generation uses the proposed dataset.

Is there a repository that links to any or all papers or systems that use the dataset? N/A

What (other) tasks could the dataset be used for? Evaluating image-to-text generation models for a domain specific performance.

Is there anything about the composition of the dataset or the way it was collected and
preprocessed/cleaned/labeled that might impact future uses? No.

### D.7 Data Format

For each figure-caption pair, the figure-image is stored as a PNG, and the figure-caption (with associated metadata) is stored in a JSON format.  [4](#A4.F4) is an example from the dataset.

In each figure-caption’s metadata file, the fields are:

- •
contains-subfigure: boolean (if figure-image contains
subfigures)
- •
paper-ID: the unique paper ID in the arXiv dataset
- •
figure-ID: the extracted figure ID of paper (the index is not
the same as the label in the caption)
- •
figure-type: the figure type
- •
0-originally-extracted: original figure-caption
–
caption: caption after each normalization
–
sentence: a list of segmented sentences
–
token: a list of tokenized words
- •
1-lowercase-and-token-and-remove-figure-index: Removed figure
index and the captions in lowercase
–
Same substructure as 0-originally-extracted
- •
2-normalized:
–
2-1-basic-num: caption after replacing the number
*
Same substructure as 0-originally-extracted
–
2-2-advanced-euqation-bracket: caption after replacing the
equations and contents in the bracket
*
Same substructure as 0-originally-extracted
- •
Img-text: texts extracted from the figure, such as the texts
for labels, legends … etc.

Within the "human-feedback" field, we have the inferred human-feedback for the different metrics (helpfulness, ocr, takeaway, and visual). The tokens are decided based on the median score of the dataset on that metric.

- •
Helpfulness: Expert’s rating on how helpful a caption is to understand a scientific figure
–
Score: predicted score
–
Token: [Good]/[Bad]
–
caption-prepend: 1-lowercase-and-token-and-remove-figure-index caption with the token
- •
Takeaway: Expert’s rating on the takeaway from the scientific image
–
Same substructure as Helpfulness
- •
OCR: Expert’s rating on the OCRs expressiveness
–
Same substructure as Helpfulness
- •
Visual: Expert’s rating on the visualness of the
scientific figure
–
Same substructure as Helpfulness

Figure: Figure 4: Human Feedback Benchmark Data Example for Figure-Caption Generation with RLHF
Refer to caption: /html/2307.10867/assets/x4.png

#### D.7.1 Reading Data

For all figure-caption pairs, all of the figure-images are in their respective train/val/test subfolders under the "No-Subfig-Img" folder. The corresponding figure-captions and associated metadata are in their respective train/val/test subfolders under the "Caption-All’ folder, bearing the same filename as their image. In order to read the data, one can read the file-names of all the figure-images in a particular data-split, and retrieve the corresponding figure-caption metadata using the image file-names (instead iterating through the captions also works). Another approach is to iterate through the "file_idx.json" file under the "List-of-Files-for-Each-Experiments/First-Sentence/(train/val/test)" folder, which contains a list of all image-names we used for that data split.

#### D.7.2 Reproducibility

We have provided easy access to the benchmark dataset which was used to conduct all of our experiments, including the augmented caption that was used during RLHF fine-tuning.

We have also provided access to a github repository, which contains the code used to: train a baseline, fine-tune a model using human-feedback, and evaluate the model on the test set.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why not train a vision-language model to evaluate image-caption pairs, instead of using a regression function over fixed embedding functions? Additionally, how was the embedding function selected?
2. What is the performance of state-of-the-art multimodal language models, including proprietary models such as GPT-4, Claude 3.5, and Gemini, as well as open-source models like Llava, Qwen-2 VL, InternVL, and MiniCPM?
3. The evaluation uses metrics (ROUGE) that have been shown to poorly measure quality of LLM-generated text. Non-overlapped based metrics, like FactScore would be more appropriate.
soundness: 2
strengths: 1. This work introduces a framework that incorporates domain expert feedback for figure caption generation, an area that is underexplored.
2. The lightweight reward model is efficiently trained on a small dataset to score figure-caption pairs, yielding impressive results as demonstrated in Table 1.
3. The reinforcement learning approach effectively enhances the caption model's performance.
summary: This paper focuses on the figure-to-caption generation task, introducing a new framework that incorporates domain expert feedback for generating and evaluating captions. A lightweight reward model is trained on a small set of figure-caption pairs and used to score other pairs, with reinforcement learning employed to train the caption model for improved performance. Experimental results show that the RL approach can enhance performance.

However, several issues are noted, including the lack of discussion on related work and the absence of evaluations using current state-of-the-art multimodal language models, such as proprietary models like GPT-4, Claude 3.5, and Gemini, as well as open-source models like Qwen-2 VL, InternVL, and IDEFICS. Additionally, it is unclear how the learned reward model sustains strong performance given the limited training data, and there is insufficient detail regarding design choices and comparisons involving vision-language models for rewards.
weaknesses: 1. The paper lacks discussion of related work, including notable studies such as ArxivCap [1] and MMSci [2].
2. There is no evaluation of current state-of-the-art multimodal language models, including proprietary models like GPT-4, Claude 3.5, and Gemini, or open-source models like Llava, Qwen-2 VL, InternVL, and MiniCPM.
3. It is unclear how the learned reward model ensures robust performance given the limited training data. Additionally, there is insufficient detail on the design choices and ablation studies related to using a vision-language model for rewards.
4. The ROUGE scores might not be good metrics for evaluating caption. Some other better metrics such as FactScore [3] should be considered.

[1] Li, Lei, et al. "Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models." arXiv preprint arXiv:2403.00231 (2024).
[2] Li, Zekun, et al. "Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension." arXiv preprint arXiv:2407.04903 (2024).
[3] Min, Sewon, et al. "Factscore: Fine-grained atomic evaluation of factual precision in long form text generation." arXiv preprint arXiv:2305.14251 (2023).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What are the reference captions used for computing ROUGE, BLEU scores etc.?
2. Are 4 different models being trained to generate the 4 respective scores? If yes, is this not computationally expensive and can this not be regressed by 1 model?
3. _Generalizability of the human feedback prediction model_ : I am not convinced that is proof of generalization. The validation set consists of the same 438 human annotated samples, which the model is trained to optimize.
4. Since the "core" idea depends on human annotations, there should be a brief discussion about the demographics in the main paper.
soundness: 2
strengths: 1. The paper is well written and easy to follow.
2. I appreciate the release of the dataset and documentation; which are a good contribution.
3. The conducted experiments and ablation, within the scope of the paper, are well motivated and justify the presented approach.
summary: The paper presents FigCaps-HF, a new framework for figure to caption generation. Specifically, the paper incorporates human feedback through a RLHF setup to improve the captioning abilities of image-to-text models. The work also introduces a benchmark, as part of this framework to evaluate the quality of figure-caption pairs by training on a small subset of human-annotated data. Experiments on metrics such as BLEU and ROUGE are performed to demonstrate the effectiveness of the approach, with ablations performed validating the varying components of the method.
weaknesses: 1. The presented baselines are missing some key models -- there are many new models such as LLaVA-NeXT that should be used as baselines. Guaranteed, they are larger models but in general does the zero-shot performance of these multimodal LLMs compare against the presented method.
2. The RLHF component should also be compared against existing methods such as direct preference optimization (DPO) to fully understand the advantages over the human feedback component.
3. There are no experiments demonstrating the zero-shot performance of the proposed method to other tasks. For example, what is the zero-shot performance on baseline vs proposed method on table captioning?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In Figures 2 and 3, the authors mainly illustrate their points through a few examples, which is relatively weak since there are always some examples that meet the requirements. Could the authors provide some statistical evidence to support these claims or use other methods to prove their points from different perspectives?
soundness: 3
strengths: **Incorporation of Domain Expert Feedback**: The FigCaps-HF framework can incorporate domain expert feedback to optimize generated captions, making them more suitable for readers' needs.

**Combination of Automatic Evaluation and RLHF**: The framework includes an automatic method for evaluating the quality of figure-caption pairs and utilizes a reinforcement learning with human feedback (RLHF) approach to further optimize the generative model.

**Release of a Large-Scale Benchmark Dataset**: A large-scale benchmark dataset on figure-caption pairs is provided to support further evaluation and development of RLHF techniques for this problem.
summary: This paper introduces an RLHF benchmark and framework for generating captions from figures using limited human feedback. The approach involves learning an oracle model from a small amount of feedback to predict human scores for new, unseen figure-caption pairs. Extensive experiments demonstrate the method's effectiveness, and the authors provide a comprehensive benchmark dataset to support future research in figure-to-caption generation via RLHF.
weaknesses: **Layout and Aesthetic Issues**: The layout of the paper could be improved, and the figures are not visually appealing, which affects the overall presentation and readability.

**Weak Baseline Models**: The paper uses BLIP and ViT+GPT2 as baseline models, which are relatively weak. Given the rapid advancement of multimodal large models, experiments should be conducted on more and stronger baselines to comprehensively validate the effectiveness of the proposed approach.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: -  How did your trained reward model perform on the validation set? 

##  Layout
There is an empty line above Table 7. I would recommend the author place tables properly  (e.g., at the top of each page) and merge short tables for better visibility.
soundness: 2
strengths: - Figure-to-caption generation is an interesting question and of great potential for aiding scientific paper writing;
- The idea of incorporating expert feedback into figure-to-caption intuitive and effective as demonstrated by the experiments.
summary: This paper investigates the figure-to-caption generation problem. The key idea is to train a feedback model on human experts' feedback and apply it to large-scale unlabeled samples. This leads to a benchmark FigCaps-HF with annotated reward scores.  
The reward signal is transformed into a control token `<GOOD>` or `<BAD>` to inform the model of the quality of the caption during the learning process.
Experiments with image-to-text models such as BLIP & ViT+GPT2 show that incorporating the feedback signal is beneficial regarding the generation metrics. Further analysis examines various aspects of the proposal such as the effect of feedback from different perspectives and reward score generalizability to OOD distributions.
weaknesses: While the explorations and efforts of this paper is greatly appreciated, I think this paper can be improved quite a lot by addressing the following concerns.

- Old backbone used: Given the rapid evolution of large vision-language models (e.g., LLaVA-series,  Qwen-VL)., they have shown promising results regarding various benchmarks including figure-to-caption [1]. I would suggest the authors incorporate the results of commercial models to serve as an indicator for the sota performance and perform experiments (i.e., reward-guided tuning) with open-sourced LVLMs. BLIP (2021) & ViT+GPT2 (2019) are not sufficient as they are using (relatively) small LLMs, where I believe the rich knowledge in recent LLMs would help a lot for producing high-quality captions.

- Develop better metrics for scientific figures captioning: In current Table 2, the BLEU scores improve from 0.01x to 0.02, rendering the results less convincing.  I understand that such an open-ended generation is hard to evaluate.  N-gram overlapping-based metrics such as ROUGE-L and BLEU are insufficient to comprehensively assess the caption. Sentence-embedding-based metrics such as Senetence-BERT and LLM-aided evaluation with the golden captions would be helpful for this scenario.

- The baseline methods are insufficient: Using the reward model to annotate `<GOOD>` and `<BAD>` indicators and then adopting it into training is common practice in RLAIF papers nowadays. I would like the author to compare the method with PPO method and unlikelihood training of `<BAD>` captions to provide more insights and justify the experimental choice better.

- Benchmark quality check: The paper claims to propose a benchmark annotated by a reward model. However, to serve as a benchmark, rigorous human check is required to ensure the quality. I would like to see a consistency check of a subset and the reward model accuracy on the held-out set. 



[1] Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models, ACL 2024