SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Inter-Rater Agreement: To determine inter-rater agreement we compute Kendall's  $\tau$  [25], Kendall's  $W$  [26], and Fleiss'  $\kappa$  [14] on "virtual raters." We define a virtual rater to be a selection of one rating for each image-caption pair. Flickr8k-Expert only has 3 ratings per image-caption pair, which are sorted from lowest to highest. In order to compare our dataset with Flickr8k-Expert, we construct 3 virtual raters by randomly sampling 3 ratings for each image-caption pair and sorting those ratings such that virtual rater  $X$  has the lowest rating, rater  $Y$  has the
(a) Flickr8k-Expert
Figure 4: Dataset rating distributions.
(b) VICR
middle rating, and  $Z$  has the highest rating. We do this 100 times and take the average for each metric. In Table 3 we report these values scaled by 100 to be consistent with prior work [20]. For Kendall's  $\tau$ , we use "method  $A$ " for aggregation and variant  $\tau_{C}$  to handle ties (see [20]). The standard deviations were between 0.08 and 0.12 for  $\tau$ , 0.05 for  $W$ , and 0.20 for  $\kappa$ .
The standard benchmark in previous work for evaluating ICR estimators is Kendall's  $\tau$  coefficient [3, 32, 20]. The performance of state-of-the-art metrics in previous literature does not surpass a  $\tau$  of 51.9 on Flickr8k-Expert [20], which we perceive to result from inherent limitations of the dataset itself. We test this by pitting the ratings of one rater against the rest as a proxy for human performance in the rating task. Therefore, we use Kendall's  $\tau$  in a new way to measure inter-rater agreement. The first row of Table 3,  $\tau_{X - YZ}$ , shows the Kendall's  $\tau$  correlation of virtual rater  $X$ 's ratings with the ratings of  $Y$  and  $Z$  as references. The other
rows are defined similarly. The resulting values from this analysis shows higher correlations from the VICR dataset, demonstrating greater inter-rater agreement.
Table 3: Inter-rater agreement.
Flickr8k	VICR
TX-YZ	47.7	75.6
TY-XZ	54.8	77.8
TZ-XY	54.0	78.1
W	84.0	93.8
κ	48.8	58.1
The fourth row of Table 3, Kendall's  $W$ , shows how well the raters  $X$ ,  $Y$ , and  $Z$  all agree with one another. VICR has a Kendall's  $W$  score that is 9.8 percentage points greater than Flickr8k-Expert, showing higher amount of inter-rater reliability.
Fleiss'  $\kappa$  measures how well the virtual raters categorically agree with one another above random chance. We collapse categories for ratings 4 and 5 to all be categorical rating 4 in the last row of Table 3 because Fleiss'  $\kappa$  is sensitive to the number of categories, and there are fewer categories in Flickr8k-Expert than there are in the VICR dataset. VICR has a Fleiss'  $\kappa$  score that is 9.3 percentage points greater than Flickr8k-Expert, showing a higher amount of agreement within the virtual raters.
Another observation from Table 3 is that  $\tau_{X - YZ}$  shows a lower correlation with the other virtual raters compared to  $\tau_{Y - XZ}$  and  $\tau_{Z - XY}$ . This is a consequence of  $X$  being the rater with the lowest rating per pair and the skew towards lower ratings. The effect appears much greater in the Flickr8k-Expert dataset because it has a much more severe skew.
Evaluation With Baseline Rating Predictor: To evaluate and study our datasets, we developed a baseline reference-free image-caption rating predictor (Figure 5). For the representation layer, the predictor takes a novel approach by employing ViL
Figure 5: Architecture of our baseline rating predictor.
BERT [37, 38] co-attention embeddings, which were trained on the Conceptual Captions dataset [44]. Specifically, the input to the model (an image-caption pair) is represented by a 2048-dimensional vector created by concatenating the image and text embeddings from the final hidden layer of a pretrained ViLBERT model.
For the rating prediction, our model uses a dense neural network with two hidden layers. The first hidden layer has 1,024 neurons with a ReLU activation, and the second hidden layer has 64 neurons with a ReLU activation. The output layer consists of a single neuron, representing the predicted
rating. We use  $80\%$  dropout on both hidden layers. We use mean squared error for the loss. We trained for 4,000 epochs, with a batch size of 256, and with a learning rate of  $10^{-5}$ , decayed by  $1\%$  every 15 epochs. ModelFlickr8k and ModelVICR were trained on the Flickr8k-Expert dataset and the VICR dataset, respectively.
We summarize the results of our experiments in Table 4 where we compare reference-based and reference-free metrics against our trained models and compare our VICR dataset with Flickr8k-Expert. Each dataset was split into  $64\%$  training,  $16\%$  validation, and  $20\%$  test. All measurements are of Kendall's  $\tau$  scaled by 100 and were computed on the corresponding test sets. We used "method  $A$  " in aggregation and  $\tau_{C}$  to be consistent with prior work [20]. Our models were each trained 5 times with different random seeds and the average Kendall's  $\tau$  value is reported. The standard deviation of ModelFlickr8k was 0.13 on Flickr8k-Expert and 0.20 on VICR, and the standard deviation of ModelVicer was 0.17 on Flickr8k-Expert and 0.03 on VICR.
All of the metrics in Table 4 are significantly higher on our VICR dataset than on Flickr8k-Expert. This indicates that VICR is a more consistent and balanced dataset that can be modeled more easily than Flickr8k-Expert. Indeed, even ModelFlickr8k, which was trained on the Flickr8k-Expert dataset, gains a  $34\%$  improvement when predicting VICR ratings. ModelVicer improves by  $43\%$  when tested on the VICR dataset as compared to Flickr8k-Expert.
Table 4: Kendall's  $\tau$  correlation with ground-truth ratings on a test subset of each dataset for various metrics and predictors.
Reference-based	Flickr8k	VICR
BLEU-1	33.7	55.8
BLEU-4	31.6	51.8
METEOR	40.4	60.2
ROUGE	33.4	53.2
CIDEr	44.1	66.6
SPICE	41.2	60.4
RefCLIPScore	51.9	71.7
ViLBERTScore	50.1	66.9
Reference-free
CLIPScore	50.7	67.3
VSEPP	48.6	65.1
VBAnglement	49.9	65.8
ModelFlickr8k	53.7	71.8
ModelVICR	53.1	75.8
Our models, ModelFlickr8k and ModelVICR, show the best performance over all metrics by a decent margin. ModelFlickr8k slightly outperforms ModelVICR on the Flickr8k-Expert dataset, while ModelVICR significantly outperforms ModelFlickr8k on the VICR dataset. We believe that ModelVICR generalizes better because it was trained on a dataset with higher fidelity, more samples, and a more balanced ratings distribution. However, ModelFlickr8k did better on the Flickr8k-Expert dataset, which we suspect is because the Flickr8k-Expert dataset is based on a 4-level scale, and skews toward lower quality, making it more difficult for ModelVICR to predict.
A significant observation from comparing the Flickr8k columns of Tables 3 and 4 is that all of the metrics, including our best model, could not achieve a Kendall's  $\tau$  of more than the best human rater,  $\tau_{Y - XZ}$ , who achieved 54.8 on Flickr8k-Expert. Indeed, it seems unreasonable to expect anything much higher from any metric, given the difficulty humans have in correlating the ratings themselves. Similarly, the metric scores on the VICR dataset are also capped below the human raters' highest score of 78.1 from Table 3. This again highlights the importance of a high-quality dataset.
The metrics we measured in Table 4 fall into two categories: reference-based and reference-free. The reference-based metrics are common NLP metrics and include RefCLIPScore [20] as well as ViLBERTScore [32]. RefCLIPScore is the reference-based version of CLIPScore and ViLBERTScore extends BERTScore [56] to the visual-linguistic domain by using ViLBERT embeddings. For ViLBERTScore, we used the fine-tuned model and reported the F1 metric (ViLBERTScore $^{*}_{\mathrm{F}}$ ). The reference-based metrics were calculated using the original reference captions from MS-COCO and Flickr8k, depending on the source of the image in question.
Besides our models, the other reference-free metrics are CLIPScore [20], VSEPP [13] and "VBAign-. ment." VBAignment refers to the visual-linguistic alignment prediction task that is used in ViLBERT pretraining [37]. The outputs of this task are two logits representing "alignment" and "non-alignment." We apply softmax to the logits and use the "alignment" probability as the VBAignment value for the image-caption pair. We use the pretrained model before fine-tuning on the 12 tasks from [38].
We used publicly available code for the reference-based metrics and CLIPScore from [19]; VSEPP from [12]; ViLBERTScore from [30]; and pretrained models and VBAnglement code from ViLBERT [36]. Our model code is written in Python [49] using Keras [8] and Tensorflow [1]. The dataset and code for our model are available through our project website<sup>1</sup>.
All experiments were run in-house on an NVIDIA RTXTM A6000 GPU server with AMD EPYCTM 7302 CPU and 256 Gigabytes of system memory. Our models took approximately 20 to 45 minutes to train. Extracting ViLBERT embeddings was the slowest part (approximately 930 milliseconds per image-caption embedding) mainly due to the image feature extraction.
Our work contributes to the growing argument on the importance of creating and maintaining high-quality datasets. Well-curated datasets have been shown to be strong competitors to large-scale alternatives [16, 43]; such datasets are smaller, more efficient, and less computationally demanding, requiring fewer resources to generate and manage. To this end, we present a high-quality and well-curated image caption rating dataset, called VICR, that was generated using a human-in-the-loop game and a validated five-level scale to facilitate image caption rating.
Our VICR dataset outperforms the state-of-the-art Flickr8k-Expert dataset in several important ways. VICR exhibits greater inter-rater agreement than Flickr8k-Expert, as well as higher correlation with reference-based and reference-free metrics. Additionally, machine learning models trained on VICR surpass state-of-the-art metrics when predicting ratings on VICR and Flickr8k-Expert. All of this suggests that our dataset is more consistent and balanced. Our approach, using a human-in-the-loop game and custom rating scale, can be extended to support other research groups in developing high-quality image caption rating datasets.
There are a few limitations of our work. The images in our dataset are limited in scope with regard to objects, actions, and locations, which constrains the applicability of the dataset. Flickr8k images have 6 categories [22] and MS-COCO images have 91 [35]. The captions are all fairly simple, without great variety in length and expression, and it may be difficult to generalize a model trained on them to evaluate more expressive captions. Another limitation of our work is that the images and captions are presented in our Rating Game without context; there is no surrounding information that gives the rater clues to the intent of the image and whether the caption accurately expresses that intent. The lack of such context hinders adaptation in cases such as improving accessibility where context has been shown to aid in image descriptions [28, 45, 46]. Our well-curated dataset and data collection methodology serve as stepping stones toward future research that incorporates contextual information. For example, we envision a version of our Rating Game that presents image-caption pairs in various contexts. A third limitation of our work is that paying the raters for data collection may have had an undue influence on the ratings by incentivizing them to prioritize monetary rewards over quality ratings. We would like to study this in future work by comparing paid raters with volunteers.
Our work holds the potential for significant social impact, both positive and negative. We anticipate that this work will lead to improved image and video caption quality, promoting accessibility and inclusivity in technology. However, we acknowledge that better image and video captions can empower bad actors in generating disinformation and manipulating public sentiment, potentially creating social unrest and decreased trust in visual media.
As part of future work, we would like to broaden the scope of images and create a wider variety of captions. Additionally, following the IIR framework, we would like to conduct a user study to further validate our scale against a simpler baseline. This will help to further quantify the current scale and will help with the development of ICR scales for other targeted applications.
We believe that our carefully curated high-quality VICR dataset enables opportunities to create consistent and reliable models that could automate the caption quality rating process. This provides new possibilities, not only for more accurate, rich, and descriptive image and video annotations, but also for searching through visual content such as for image and video retrieval. VICR gives the community a new benchmark dataset for image-caption rating.
1 https://ai.youdescribe.org
Qualitative Examples
(a) A person descends a rope from a cliff into the ocean.
(b) a person is standing in the snow with a rope
(c) A bridge filled with blue and white buses.
Ratings: 1211
(d) A man and woman look back at the camera while standing in front of a red art structure.
Ratings: 1 1 1 1 1
(e) Some kids are playing in a baseball game Ratings: 1 1 2 3
Ratings: 1 2 1 1 1
(f) girl posing on tree Ratings: 2 3 2 1 2
(g) a group of people are walking down a road.
Ratings: 1 1 1 1 2 2
(h) A man without a shirt playing tennis.
(i) Two white dogs running
(j) a car is parked next to a clock
Ratings: 2 2 3 1 3
(k) A young man climbs a mountain, another follows below.
Ratings: 14223
(1) a hot dog with ketchup and mustard on a table
Ratings: 3332
(m) a desk with two computers and a keyboard
Ratings: 3 3 4 2
Ratings: 3 3 3 3 3
(n) The cat's gaze is focused on something above his head.
(o) A man in a suit is walking with a woman carrying an umbrella.
Ratings: 3444
(p) a group of birds sitting on top of a wooden log
Ratings: 4 4 4 4 3
(q) A vase of fresh, colorful flowers on a table. Ratings: 4 5 5 5
Figure 6: A sampling of image-caption pairs from the VICR dataset with ratings.
Ratings: 4 4 4 5 4
(r) A batter is swinging his bat while a catcher squats behind him.
Ratings: 5 5 4 5
Ratings: 4 4 5 4
(s) a bunch of vegetables on a counter top
Ratings: 55555
Ratings: 4 4 5 5
(t) the big ben clock tower towering over the city of London
Ratings: 55555
This work is part of the YouDescribe Project<sup>2</sup> in collaboration with Smith-Kettlewell Eye Research Institute<sup>3</sup>, and made possible by funding from Ability Central<sup>4</sup> and a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number 90 REGE0018).
Thanks to Charity Pitcher-Cooper, Brenna Gibson Tirumalashetty, Raya Farshad, Aditya Bodi, Abhishek Das, and Yash Kant for their invaluable contributions to this project.
[1] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/Software available from tensorflow.org.
[2] Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, and Matthew Stone. 2020. Cross-modal Coherence Modeling for Caption Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 6525-6535.
[3] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision. Springer, 382-398.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425-2433.
[5] Frank B Baker. 2001. The basics of item response theory. ERIC.
[6] Alexander Mario Blum, James M Mason, Jinho Kim, and P David Pearson. 2020. Modeling question-answer relations: the development of the integrative inferential reasoning comic assessment. Reading and Writing 33, 8 (2020), 1971-2000.
[7] Eva Ceticic. 2021. Towards generating and evaluating iconographic image captions of artworks. Journal of Imaging 7, 8 (2021), 123.
[8] François Chollet et al. 2015. Keras. https://keras.io.
[9] Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5804-5812.
[10] CVPR19. 2019. The first Workshop and Challenge on Conceptual Captions CVPR19 Workshop. https://wwwconceptualcaptions.com/
[11] Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation. 376-380.
[12] Fartash Faghri. 2023. VSEPP. https://github.com/fartashf/vsepp.
[13] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 (2017).
[14] Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378.
$^{2}$ https://youdescribe.org
<sup>3</sup>https://www.ski.org/
4 https://abilitycentral.org
[51] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3156-3164.
[52] Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems. 319-326.
[53] Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018. No metrics are perfect: Adversarial reward learning for visual storytelling. arXiv preprint arXiv:1804.09160 (2018).
[54] Mark Wilson. 2005. Constructing measures: An item response modeling approach. Lawrence Erlbaum Associates Publishers.
[55] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning. PMLR, 2048-2057.
[56] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019).