Alt-Text with Context: Improving Accessibility for Images on Twitter
Authors: Nikita Srivatsan
Sofia Samaniego
Omar Florez
Taylor Berg-Kirkpatrick
Venue: ICLR 2024 poster (2024)
Keywords: alt-text
social media
twitter
clip
computer vision
image captioning
accessibility
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present a model for generating alt-text descriptions for images shared on Twitter, and put forward an accompanying dataset for this task

Abstract: In this work we present an approach for generating alternative text (or alt-text) descriptions for images shared on social media, specifically Twitter. More than just a special case of image captioning, alt-text is both more literally descriptive and context-specific. Also critically, images posted to Twitter are often accompanied by user-written text that despite not necessarily describing the image may provide useful context that if properly leveraged can be informative. We address this task with a multimodal model that conditions on both textual information from the associated social media post as well as visual signal from the image, and demonstrate that the utility of these two information sources stacks. We put forward a new dataset of 371k images paired with alt-text and tweets scraped from Twitter and evaluate on it across a variety of automated metrics as well as human evaluation. We show that our approach of conditioning on both tweet text and visual information significantly outperforms prior work, by more than 2x on BLEU@4.

Published as a conference paper at ICLR 2024
ALT-TEXT WITH CONTEXT:
IMPROVING ACCESSIBILITY FOR IMAGES ON TWITTER
Nikita Srivatsan
Sofia Samaniego
Carnegie Mellon University nsri virivatas se cmu.
sofia samani remaniego... @ggm i .om com
Omar Florez
Taylor KirkpKirkpatrick University of California San Diego tberg@ucsd. edu
LatinX in AI
maa.fooorz..esearcchhee
ABSTRACT
In this work we present an approach for generating alternative text (or alt-text) de- scriptions for images shared on social media, specifically Twitter. More than just a special case of image captioning, alt-text is both more literally descriptive and context-specific. Also critically, images posted to Twitter often accompanied by user-written text that despite not necessarily describing the image may provide useful context that if properly leveraged can be informative. We address this task with a multimodal model that conditions on both textual information from the associated social media post as well as visual signal from the image, and demon- strate that the utility of these two information sources stacks. We put forward a new dataset of 371k images paired with alt-text and tweets scraped from Twitter and evaluate on it across variety of automated metrics as well as human evalu- ation. We show that our approach of conditioning on both tweet text and visual information significantly outperforms prior work, by more than 2x on BLEU@4.
INTRODUCTION
An increasingly important aspect of the social media user experience centers around the sharing and discussion of visual content. However, for users who are blind or have low vision (BLV), this type of content is often inaccessible. One solution to this problem is alt-text, an HTML attribute associated with digital images, which is intended to provide literal natural language description of an image's content, and can be rendered by screen reader or braille display. Despite being infrequently in- cluded, incredibly important from an accessibility standpoint, as it allows users to perceive the content of the image and thus have better online experience. Some social media websites, notably Twitter, have recently given users the option to write their own alt-text for images that they upload (an example is shown in Figure [ This approach however offloads the responsibility of making the site accessible onto the users, who frequently either for convenience or lack of awareness simply do not choose do (Gleason al. 2019). We find that as many 98% of images uploaded to Twitter even after the widespread feature rollout do not have alt-text, to say nothing of the quality of those that do. When screen reader encounters such an image, it will simply say "Image", leaving the user with no meaningful information about what the image is actually of. Even when images on Twitter do have accompanying user-written alt-text, the quality is inconsistent as not all users are well informed regarding best practices.
Note that while there long line of existing research on the broader task of captioning images more generally, alt-text generation for social media is special case of this task, which in turn comes with its own challenges. Well written alt-text is generally more explicitly descriptive than a high level caption and may emphasize specific details in the image based on context (Kreiss et al.) 2022a). See Figure for an example. Furthermore, the distribution of image types on Twitter differs substantially from those found in traditional captioning datasets, and may contain digital artwork, promotional graphics, or screenshots containing text. An additional challenge is that Twitter users not well trained in the practice of writing alt-text, and therefore native "gold" examples can vary

---

Published as a conference paper at ICLR 2024
Tweet text glo con-descriphee Helyy contain into from
leteck ryy meil Teday, reeetvee election Eyors. Good lonn Vakie tetr
Our Dataset
Conceptual Captions
be pue to vote, PGa Rom. Primary
sevi chin Cored dn en dln Lidbutt boo encc fer
Image xit likety contain visual exements fex reloroncud by the tweet
Alt-text yot descripcive
Image description
Molino's
caption, onjective
Coasperted warks ccardliatee
disticr
and lunctional but
20G2 Call favic meny electiion
(key to emphasize
sisaa eloments
wen mettioneed in the bwcet
All--watt: Dilicious pink margariia sits on tobie solie chrps
Caption: having cocktail personn to
Predicted alt-text 510
ClipCap (x ): pile books table..."
Ours 114901 (l)): "A pile of election fiyers table..."
Figure I: Left: An image that requires textual context write accurate alt-text for. Without condi- tioning on the tweet text, the election flyers are indistinguishable from books to a traditional caption- ing system. Right: Two similar images from dataset and Conceptual Captions with their gold labels. The alt-text for the first image is literally descriptive while the second is more colloquial.
substantially in their level of quality. To support continued research in this area, we collect and release a first-of-its-king dataset of image and alt-text pairs scraped from Twitter.
Our central motivating idea is that despite the challenges associated with this domain, social media images come with additional contextual information that if properly leveraged can significantly im- prove the quality of automatically generated alt-text. Images posted to social media often require some amount of background knowledge to adequately caption. For example the left image in Fig- ure[may easily be mistaken for a stack of books or magazines. However, we do in fact have source of text that might provide context not present in the visual information contained in the image itself, namely the text associated with the actual post which more specifically identifies the pamphlets as election flyers. We therefore see an opportunity to augment existing captioning systems with textual conditioning in order to allow them to achieve better performance this challenging task.
One model that is easily modifiable to accept multimodal inputs is ClipCap (Mokady et al. 2021 ClipCap operates using a prefix method, where image encoder in this case CLIP (Radford et al. (021) produces a vector embedding of the image which is then projected to sequence of embeddings that occupy the same dimensionality word embeddings. These (despite not actually being the embedding of any specific language sequence) can then be taken as inputs to a language model decoder- in this case GPT-2 (Radford et al. 2019) which then autoregressively completes the sequence to produce the caption.
Our approach (shown in Figure 2) supplements the text-like image embedding sequence with an em- bedding of the tweet text, in the hopes that these two information sources will provide meaningfully non-overlapping signal to the decoder, in order to produce more accurate captions. This reduces the burden on the image encoder to be able to distinguish very similar object types, and allows the decoder access to details that may very difficult glean from the visual features.
In summary our contributions are as follows: (1) We collect novel benchmark dataset for the task of alt-text prediction on social media including both images and surrounding textual context (2) We introduce a novel alt-text generation method that allows for multimodal conditioning with unimodal language model (3) We perform a quantitative analysis of this approach on our collected dataset, outperforming prior work captioning (ClipCap, Mokady et al. (2021)) and vision-and language pretraining (BLIP-2, Li et al. (2023)) by more than 2x on BLEU @4. We also conduct human evaluation which substantiates these findings.
2

---

Published as a conference paper at ICLR 2024
Prefix in Word Embedding Space
Image Encoding
Input Image (
Image Projection
Embedded
Predicted Alt- yo
Tweet
CLIP
Map Net
thi
GPT-2
p
this is definitely pie of one of buc gualinge
Tweet Text 2())
Figure 2: Overview of the alt-text model. An image is encoded via CLIP to obtain an embedding of visual features. This gets projected via a mapping network into word embedding space, where then concatenated with an embedded representation of the text from the corresponding tweet. This prefix is passed to a finetuned GPT-2 which autoregressively generates the alt-text caption.
2 RELATED WORK
Image captioning in generic sense is relatively well studied task. Since the introduction of the
COCO benchmark (Lin et al. 2014) a variety of systems have been released, and state-of-ther -o-t----t-h-rtttt quantitative performance has risen across several metrics (Vinyals al.] 2015 Fang et al. [015] Bernardi et al. 2016). Alt-text generation is relatively understudied by comparison, and there are few established datasets for it.
Kreiss et al. 2202bb collected one such dataset of images with descriptive alt-text and contextual captions from Wikipedia. This corpus was smaller than the one we put forward here at 97k images, and also represents a fundamentally different domain as Wikipedia contains different image types than social media, more frequently has gold alt-text, and has much more carefully curated text captions. They do however demonstrate that conditioning on textual context can improve the quality of generated alt-text, suggesting the value a similar approach for our setting.
Examining alt-text in the context of social media posts has been studied as well, albeit to an even more limited extent. Twitter Al (Gleason et al. 2020) is a browser extension based on pipeline system that primarily works through scraping matching alt-text from alternate sources, only gener- ating it abstractively through a separate caption generator if other methods fail.
Facebook developed and published system they refer as Automatic Alt-Text (AAT) (Wu et al. 2017]. The approach described does not generate text abstractively, and instead essentially outputs list of tags indicating various objects detected within the image. While this is arguably more informative than nothing, it's worth emphasizing that this format does not strictly speaking adhere to alt-text best practices. The system has since been refined include additional details regarding position, count, and size (Meta] 2021). Facebook's AAT has been critically examined, notably by Hanley al. (22II) who focused particularly on how describes personal identity categories. They summarize a variety of limitations BLV users experience with it including lack of detail, a reliance on tags as opposed fluid natural language captions, and an inability to reason about the contextual salience of various parts of the image.
Chiarella al. 2020) outlined steps to increase the accessibility of scientific images posted to
Twitter. Alt- Textify (Belle. et al. (002)) proposed a method specifically designed for generating
alt-text on SVG-based data visualizations and charts, and Chintalapati et al. (2022) put forward
dataset and analysis of visualizations within HCI publications. Li et al. (2020) introduced method
for generating alt-text specifically for mobile UI elements.
One prominent recent example of more general-purpose system for aligning text with images is CLIP (Radford et al. 2021). While CLIP is not exactly a captioning system, it can be used to score text/image pairs CLIP's architecture have however found success within the pipelines of broader captioning models.
way of reranking or selecting captions from candidate list. Elements of
One example is ClipCap (Mokady al. 2021). which mapping network to project CL.IP

---

Published as a conference paper at ICLR 2024
image embedding into word embedding space and then feeds the result as prefix to a language model which outputs the caption. We inherit this basic architectural framework, but extend it to multimodal prefixes which also contain textual context.
While there are other existing datasets for image captioning, they do differ from the one put forward in this paper in few key ways. One of the most similar examples is Conceptual Captions (Sharma et al. 2018). a dataset of images with short and descriptive text captions. These captions are pro- duced by processing the alt-text descriptions found within HTML simplify and improve fluency. However, the distribution of images is fundamentally different from that found on social media, and the captions are not generally descriptive enough for accessibility purposes (as result their captions average only 10 tokens, whereas ours average 40). An example showing this contrast is provided in Figure T Furthermore this dataset does not include surrounding textual context, which we demonstrate to be extremely informative for improved alt-text generation.
More recently the emerging subfield of large language models (LLMs) has expanded into exploring multimodal systems capable of conditioning images interleaved with text. Systems including Flamingo (Alayrac et al., 2022) and GPT-4 (OpenAl| 2023) have gained popularity, although their closed nature makes them difficult to study. Further, there is strong reason to prefer federated sys- tems that are small and controllable by end users rather than through paid opaque APIs that are subject to change without notice and make scientific reproducibility impossible. Open source alter- natives such as OpenFlamingo (Awadalla et al. 2023) and BLIP (Li et al.] 2022 2023) have also emerged. These models are capable of captioning images based on few if any in-context examples, despite not being specifically trained on this task. It should be emphasized that these models are however not only incomparable due to their massive and unequal training sets, but are arguably inef- ficiently overparameterized in their capacity to model a wide variety of tasks beyond captioning. The system we propose instead is trainable on reasonable compute hardware, does not require esoteric prompting strategies, and as we will demonstrate still outperforms BLIP-2 on alt-text generation. Certain large pretrained language models have also been shown to exhibit disability bias (Herold et al. 2022 which is undesirable in general and especially in this context.
MODEL
At high level, our model's primary innovation is being able condition not just visual infor- mation from the image, but also on textual content from the tweet itself. This is in contrast to prior work which captions images in isolation from the text alongside which they appear. We will now describe the multimodal conditioning setup more formally, and over the rest of this section describe our model's architecture for parameterizing the relevant distributions. Suppose that we are given dataset X = ".,("))..,,_..00)))) (NN) of images and corresponding tweet text T (0(1), ..., ((N) I as well as gold alt-text descriptions Y (1) ,.., (N) }. Our model defines a conditional distribution p(y|æ, I; 0) over the captions given the tweet and image. Having optimized this distribution with
respect to over our dataset at train time, at test time our task is predict ý arg max, p(y) 2r, l).
Note that prior work can be thought of as a special case where the tweet is not considered.
3.1 MULTIMODAL CONDITIONING
Our model's decoder must output text in order to generate the alt-text caption. A natural choice for this is an autoregressive transformer-based language model we specifically opt for GPT-2 (Radford) ct al. 2019). Conveniently, such architectures' masked self-attention mechanisms make them inher- ently well suited to conditioning on text. In order to generate alt-text conditioned on the tweet, we can simply feed the tweet to the language model as a prompt and train the model to autoregressively complete with the corresponding alt-text. The more challenging step is allowing the language model to condition on visual information from the image as well. However, we would also like to avoid fundamentally modifying the decoder's architecture and still be able to leverage unimodal pretraining. In order to solve this, what require is way project image features into text space (or more specifically word embedding space) so that we can directly incorporate them into the tweet prefix. Fortunately, this is something that prior work has addressed. ClipCap (Mokady et al. 2021 learns a mapping network on top of a pretrained CLIP image encoder that projects the visual vector
Note (0)) = x65 if images i,j are from the same tweet.
4

---

Published as a conference paper at ICLR 2024
encoding into sequence of embeddingg that while not directly corresponding specific words, still in word embedding space and therefore acceptable by a pretrained language model.
It's worth emphasizing that the crux of this method relies on "translating" image features into format which the language model can receive as input as it were text prefix. An observable artifact of this is that the mapping network often places its outputs near embeddings corresponding to words that are relevant to the contents of the image (Mokady et al. 202T). The prefix can thus be "de-embedded" back into text as a rough means of inspecting what information about the image it encodes, Crucially, this also suggests that further context not present in the image itself could be provided to the decoder in the form of additional text added the prefix. In this way, we can effectively create prefix reflecting multimodal information that nonetheless exists in unimodal space, and can input to a pretrained unimodal language model.
See Figure 2 for an overview of our model's pipeline. Drawing inspiration from ClipCap's archi- tecture (see Mokady et (202T) for details), we pass an image æ through pretrained CLIP image encoder, and project the output word embedding space using mapping network. This is param- eterized by a multi-layer perceptron that takes a 512 dimensional vector CLIP embedding and outputs & long sequence of 768 dimensional vectors (the same size GPT-2 expects for each token). These embeddings constitute a prefix p which is the correct shape to be passed to GPT-2.
Having obtained p. sequence of token embedding- esquee vectors, the procedure for combining with the tweet text to produce a prefix containing both visual and textual information is fairly straightforward We simply concatenate that projection with the embedded tweet text t (note that they are both sequences in word embedding space) to obtain the complete prefix @ t. We can condition on this, treating it as something analogous which then autoregressively outputs the alt-text caption ý. Our decoder can therefore condition both on the tweet text, which is the same modality as its output, and the image itself which not.
multimodal prompt to our language model,
To train the model, we simply teacher force and optimize the cross entropy loss between the pre- dicted logits and the gold reference. We backpropagate into the weights of both the mapping network and GPT-2, but leave CLIP frozen both for speed and performance reasons, following Mokady et al. (202T. Further implementation details are provided Appendix C
3.2
DECODING
After training. there are important choice points for how we decode captions at test time that can significantly impact performance. While ClipCap originally used greedy decoding, we instead opt for beam search with beam size of 5. Furthermore, that ours occasionally repeated words or short phrases towards the end of captions. We solve this with the trigram blocking strategy described by Pauluus et al. (2017). For fair comparison we use this same decoding method for ClipCap in our experiments.
with many language models we noticed
3.3 RERANKING
We also experiment with reranking the candidates from beam search by their similarity to the tweet text, as the highest likelihood caption might not necessarily score best under our nondifferentiable evaluation metrics. While the tweet text is not itself a descriptive caption, may reference specific things in the image that are especially relevant, and therefore high quality alt-text will likely have a significant degree of ngram overlap with it. While the model does get to condition on the tweet text through the prefix, may wish to ensure that that information does ultimately appear in the final output. In order to steer our generations in this direction, we choose a new best caption among the top 5 candidates returned by beam search based on their ROUGE-L similarity to the tweet text t. We also experimented with reranking based on BLEU and CLIP text embedding similarity but found that neither of these performed well ROUGE-L.
DATASET COLLECTION
One contribution of this paper is the collection and release of large-scale benchmark dataset for training and evaluating alt-text generation systems. We now describe how we collect a publicly available dataset of tweets containing images and corresponding user-written alt-text descriptions.
5

---

Published as a conference paper at ICLR 2024
These tweets were scraped via the public Twitter APIv2 from 21 day period in the summer of 2022. We focused on English tweets containing still images with user-written alt-text. One important consideration was the removal of duplicates. We noticed that many accounts especially bots and promotional accounts - would post the same image multiple times, and others (e.g. those tracking extreme weather events or stock prices) often post extremely similar images with only few words differing between their corresponding alt-text captions. We addressed this by deduplicating based on textual and visual matches. For more details on filtering see Appendix B
We also took steps to reduce the incidence of names in our data. While many users caption im- ages using the real names of the people in them, predicting a person's 's name from an image would implicitly require learning to perform facial recognition, something well beyond the scope of this work. Furthermore, from a user privacy standpoint, we would not want our model to be able to output the names of real people, regardless of whether they actually appear in that image. We used the existing NER system of Mishra al. (220)); Eskander et al. (2020) to identify named entities in our alt-text, and replaced all instances of the Person type with the string "person". While this sub- stitutes no additional visual description of the person, and can lead to grammatically cumbersome transformations, we leave further improvements to future work.
This yielded a dataset of 371,270 images. We split into train (330,449), val (20,561), and test (20,260) sets based on tweet ID (i.e. images from the same tweet are assigned to the same split) to prevent leakage of tweet text. The corresponding alt-text captions were on average 144 characters long, which translates to 40 tokens. The tweet text was similarly 152 characters long on average, or 46 tokens. We also hard crop both the alt-text and tweet text at 150 tokens max length. While the raw data cannot be distributed directly in order protect users' right to be forgotten, we release the dehydrated tweet IDs and media URLs 2
In order to quantitatively assess the quality of the user-written alt-text descriptions, we randomly sampled 200 tweets from the validation set, and manually inspected them to determine if they were fluent, and descriptive of the image. We found that of the 68.0% of images that still loaded and passed our preprocessing filters (described above), 94.1% % were fluently written, and 86.0% con- tained some form of literal description of the visual contents. Of those that didn't, t, the majority simply did not provide enough detail or treated the field as a second caption that provided context for what was being depicted, but not a visual description.
5 EXPERIMENTS
Having described our method, and our dataset collection procedure, we now conduct experiments to assess how well existing models perform on this new task, and by how much our multimodal system improves over those approaches. This section gives an overview of our experimental setup.
5.1 BASELINES
We examine few key baselines and ablations in our experiments in addition to our full model. The first of these is a Nearest Neighbor system, which returns the verbatim alt-text caption associated with the most visually similar image from the training set. We determine visual similarity based on the dot product of the pretrained CLIP features of the images (the same visual encoding used by our full models). The second baseline always returns the corresponding tweet text as the predicted alt-text, which we refer to as Copy Tweet Text. This tells us to what extent the tweet text is already a reasonable alt-text caption without transformation. Following our motivation for conditioning on the tweet text, we would expect that this system would occasionally return something informative albeit likely not well formed and certainly redundant from the perspective of BLV user. This can also be thought of as simpler variant of our text only system, described below.
Our first neural baseline ClipCap, described previously, using its publicly available implementa- tion. ClipCap is already pretrained on a large captioning dataset, so we evaluate it both out of the box, and finetuned on our own data. For fair comparison, we also use the same updated decoding
strategy for our ClipCap experiments. We also compare to pretrained BLIP-2 L et al. 2023) using their OPT (Zhang et al. 2022) 2.7B checkpoint.
https://githhbbe ki tašri i vat.an Pubpaa//uttteeettuuuuuubuua

---

Published as a conference paper at ICLR 2024
Model
Decoding
BLEU@4
METEOR
ROUGE-L
CIDEr
Naive Baselines
Nearest Neighbor
0.563
1.827
5.074
Copy Tweet Text
0.230
2.066
4,801
0.453
Neural Baselines
BLJP-2 (Frzze)
BS
0.111
1.449
6.744
1.381
ClipCap (Frozen)
BS (NR)
0.372
1,400
6.690
0,830
ClipCap (Finctuned)
BS (NR)
0.681
2.685
8.620
1.557
ClipCap (From Scratch)
BS (NR)
0.696
2.643
8,483
1,588
Tweet Text Conditioning
Ours (Rand Text)
BS (NR)
0.546
2.547
7,906
1.171
Ours (Text Only)
BS (NR)
0.693
2.459
7.498
1.738
Ours (Text Image)
BS (NR)
1.826
3.067
8.652
7.661
Decoding Ablation
Ours (Text Image)
Greedy
0,587
2.247
7.166
1,450
Ours (Text Image)
Greedy (NR)
0..669
2.168
7.774
1.520
Ours (Text Image)
BS
0,400
2.311
6.275
Ours (Text Image)
BS (NR)
1.826
3.067
8.652
7.661
ClipCap (From Scratch)
BS (NR) RR
0.691
2.899
8,760
1.379
Ours (Text Image)
BS (NR)+ RR
1.897
3.257
8.818
6.791
Quantitative test results across various metrics (all numbers x100). Our method and Clip- Cap are decoded using beam search with repeats. BS indicates Beam Search, NR indicates repeats, and RR indicates ROUGE-L reranking.
Table I:
Viewing ClipCap as a variant of our model that only includes image features in the prefix begs the question of whether could similarly include the tweet text, but not the projected image prefix. We therefore also consider a baseline that includes the tweet text f in the prefix, but not the projected image prefix p (referred to Ours (Text Only)). This is effectively translation model between the tweet text and alt-text; it can transform the style and filter out irrelevant details, but cannot inject new information, and thus serves as an indicator of the performance achievable based only on surrounding context without observing the image itself. We also consider an ablation of our system that substitutes the retrieved nearest neighbor for a randomly chosen tweet, which we call Ours (Rand Text). This lets us measure how robust our system is to errors in the retrieval process and how well it can generalize to cases where a similar neighbor does not exist train.
5.2 METRICS
Most image captioning work has evaluated using quantitative metrics that roughly measure the string similarity between a model's generated caption and a reference gold caption. Since we have user-written alt-text for each image in our dataset, we simply treat these as our gold references to compare to. Specifically, we measure performance on BLEUGE @4 (Papineni et al., 22022). ME- TEOR (Denkowski & Lavie 2014). ROUGE-L (Lin & Och 2004). and CIDEr (Vedantam et al. 2015). It's worth noting however that because the "gold" alt-text captions on Twitter are written by untrained users, they can noisy, inconsistent form and specificity, and occasionally do not even describe the image contents (Gleason et al.) 2019). As result, the absolute scores may seem rela- tively low compared to numbers typically reported on other datasets, although based on qualitative inspection and human assessments they do nonetheless meaningfully correlate with output quality.
Due to this variability, as well as the broader problems with contextless evaluation for this task (Kreiss et al. 2022a). we also conduct human evaluation using Amazon Mechanical Turk. For this experiment, we subsampled 954 examples from our test set (originally 1000 but some tweets had been deleted over time), and asked group of sighted human annotators compare the outputs of our multimodal system with two variants of ClipCap for those images (all systems using beam search with repeats). For details see Appendix D
RESULTS
We present results our dataset, both using automatic metrics comparing model output to user- uver-written reference alt-text, and using human evaluation. We also perform qualitative evaluation to analyze output more subjectively.
7

---

Published as a conference paper at ICLR 2024
Model
Fluency
Deseriptiveness
Model
Fluency
Descriptiveness
ClipCap (Frozen)
26.2
ClipCap (From Scratch)
34.2
37.7
Ours (Text + Image)
61.4
66.9
Ours (Text + Image)
37.4
44.4
Equal Quality 8.8
Equal Quality
28.5
17.9
Table 2: Left:: Results from human evaluation, showing annotator preference for our model vs. frozen ClipCap terms of fluency and descriptiveness. Right: Results showing annotator prefer- ence for our model vs. finetuned ClipCap in terms of fluency and descriptiveness.
6.1 AUTOMATIC METRICS
In Table | we show reconstruction performance on our test set across a variety of metrics. Copying tweet text achieves the lowest performance on all metries except for METEOR, which is perhaps expected given that tweets are generally not descriptions of the images they accompany. While users will occasionally recycle their tweets into the alt-text field, such examples are filtered out by preprocessing. Nearest neighbor achieves slightly better performance. This makes sense as there are certain styles of image, for instance selfies, that may appear frequently and have relatively similar captions, although this approach likely makes frequent mistakes when it comes to finer details.
Of the neural methods, we see that frozen ClipCap scores lowest, almost certainly due to domain mismatch in both the distribution of images and text. Finetuning our dataset does lead to some improvement, bringing ClipCap ahead of the naive baselines on most metrics. Perhaps supporting the hypothesis that Twitter dataset is fundamentally mismatched from Conceptual Captions, find that a ClipCap trained from scratch on our dataset performs about on par with the finetuned one depending on the metric. We see similar performance frozen ClipCap from BLIP-2, which despite its extensive pretraining on data not available to other systems, fails to produce alt-text with sufficient detail and accuracy for this dataset.
However we see by far the strongest results from our approach which combines the mapped image prefix with the text of the tweet as input decoder, achieving more than 2x on BLEU and 4x on CIDEr. This is also ahead of the ablation which disregards the image entirely and only uses the tweet text in the prefix, demonstrating that these two sources of information stack with one another. The competitive performance of the text-only baseline demonstrates the strong amount of necessary context present in the original post, while the fact that it outperforms merely copying the tweet text emphasizes that alt-text is nonetheless functionally different in many cases from just another caption. Randomizing the retrieved tweet does hurt our model's 's performance as well. We find qualitatively that under this setup model is sometimes still able to produce reasonable caption presumably based solely on the signal from the image features, although it will occasionally insert spurious details from the random tweet, so it's not completely robust to irrelevant input tweet captions.
6.2 DECODING ABLATION
Table [ also shows an ablation of various decoding strategies for our multimodal model. Greedy decoding does poorly, and trigram blocking helps on some metries but hurts on others. Beam search does not do better than greedy if allowed to repeat trigrams, but the specific combination of beam search without repeats does extremely well. Reranking the top candidates from beam search with ROUGE-L similarity to the tweet text offers minor improvements most metrics, but hurts performance on CIDEr. We also observe similar rankings these methods for ClipCap.
6.3 HUMAN EVALUATION
Note that while the automatic metrics are informative, the absolute scores are below what these same systems achieve on datasets such as Conceptual Captions (Sharma et al., 2018). We suspect this is largely due to the noisiness of gold references. It's worth emphasizing that Twitter users are not explicitly trained in how to write alt-text, and as result the level of quality is extremely vari- able (Gleason et al. 2019). Furthermore, some users write alt-text that functions more as a "second caption" than literal description of the image' 's contents. This means that for an observably large proportion of our data (although the exact amount difficult to quantify), the gold reference is either undesirable match idiosyncratic enough that measuring a candidate caption's similarity it is
8

---

Published as a conference paper at ICLR 2024
coler oer weee derent perembereng grgglee
engigheen serto nay.
Hen Menday fhil lndee Bored Den,
de tociedd unter calery
wen veu lolk phfrrcatinn oen wolitiingg leg.iin................ fie
We're supporting Caters Week 2022!
Modino's
6-12
Gold p roto the coos with woves craslinng bastggrund,
Goldi Color shoto of the inne for of Roscbin Bakory store. Image ciedt: Indy Star onu courter
ous morpar to
suppo ring Carora Wook 2922. The Curers Woekk logo and danes 6--2 June the bottem the trrage
Ours: A viex sea from of Revenscer. The saves ais creaning against the mode and the dky with whtte dlouss
Ours: TWo margeritee on tanle
CripCups bavery neigtbortoood
party!
perfect
Ourst National
cripCap: ane growing exponentirly as brand
christia holiday
Figure 3: Selected tweets with the user-written alt-text alongside our prediction and ClipCap's. We that by conditioning on the tweet text, our model able focus relevant details in the images, reference named places, and provide better transcription despite not being trained on OCR.
not truly reflective of how well that caption describes the image. Going through the results manually, often find instances where the model's output is actually more appropriate alt-text than what the user wrote, or where they are both functional but phrased differently enough that there is little string overlap, leading to low scores that do not necessarily reflect output quality. We thus conclude that these automated metrics, while informative, are not sufficient for assessing model performance.
We therefore also perform two rounds of human evaluation described above, with results shown in Table 2 On the left we observe that in roughly two thirds of images the human annotators judged our model's 's output to be both more fluent and descriptive than that of the frozen ClipCap baseline, and annotators generally agreed on rankings. To the right we see similar rankings albeit with a tighter margin when we compare our system against the ClipCap trained from scratch on our data. This matches the findings of our automated metrics, and supports our hypothesis that conditioning on context from the tweet yields higher quality descriptions.
6.4 QUALITATIVE INSPECTION
In order to analyze what specific types of differences lead to these improvements and also what kinds of errors the model still makes, now describe some observations from qualitative inspection. Figure [ shows some example tweets from our dataset, accompanied by the actual user-written alt- text as well our model and frozen pretrained ClipCap's 's predictions. We generally see that our system is able to focus on contextually relevant details, such as the donuts in the bakery. It's also able to leverage text the tweet to more accurately transcribe digital text in graphics. However it does at times still struggle with counting, and sometimes omits thorough visual details. ClipCap provides reasonable albeit simplistic captions for more natural images, but struggles with images that fall farther outside the Conceptual Captions distribution. It is also unable to incorporate information from the tweet itself, or use that to influence which parts of the image to focus on.
7 CONCLUSION
In this work we presented an approach for alt-text generation that conditions both on image features and the raw text of the corresponding social media post, and collected a new dataset for this task from Twitter. Despite limitations discussed the Appendix below, we believe this work is an important step forward and has potential for various downstream applications. While the most obvious use case is to simply use the model to produce alt-text for images that do not have any, we could also see this approach being used in concert with human annotators, perhaps providing starting point for captions. Additionally, since our model can form a conditional likelihood on the caption, we could use it to flag poorly written alt-text and notify users to reread the guidelines and rephrase their description. In summary, we believe this is an important task and hope that our work encourages future research that leverages the surrounding social media context in which these images appear.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Given the limitations of our approach (discussed in more detail in Appendix A we find it critical to emphasize that simply applying this method as-is to producing alt-text captions in an actual social media production setting would be premature and likely harm the user experience for BLV users. Accessibility technology must demonstrably be of more utility than other alternatives order to justifiably be adopted, and while we hope that our work is a step in that direction, we have not yet conclusively determined that that bar has been passed. The research we perform here has the potential to be co-opted by social media platforms as cheaper alternative to hiring manual alt- text writers, or having users write their own alt-text as is Twitter's 's current approach, which would arguably worsen the online experience for BLV users whose accessibility needs are already to large extent ummet in this regard. Furthermore as with any open-ended text generation system trained on social media, our model does have the capacity to produce toxic language, regurgitate sensitive user information, and produce insensitive and harmful captions for personal photos.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo, March 2023. URL https:
zenodo. //doi. 5201/eoooooooo 7733583
Aspen Belle., Vanessa Goh., Akshay Kumar., Richard Pranjatno., Pui Man Yip., Umayangani Wick- ramaratne., and Humphrey O. Obie. Alt-texify: A pipeline to generate alt-text from svg visualiza- tions. In Proceedings of the 17th International Conference on Evaluation of Novel Approaches to Software Engineering ENASE., pp. 275-281. INSTICC, SciTePress, 2022. ISBN 978-989-758. 568-5. doi: 10.3200000109944000011777
Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler- Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Jourmal of Artificial Intelligence Research, 55:409-442, 2016.
Domenico Chiarella, Justin Yarbrough, and Christopher A-L Jackso. Using alt text make science twitter more accessible for people with visual impairments. Nature Communications, 2020.
Sanjana Shivani Chintalapati, Jonathan Bragg, and Lucy Lu Wang. dataset of alt texts from hei publications: Analyses and uses towards producing more descriptive alt texts of data visualiza- tions in scientific papers. Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 2022.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop statistical machine translation, pp. 376-380, 2014.
Ramy Eskander, Peter Martigny, and Shubhanshu Mishra. Multilingual named entity recognition in tweets using wikidata. In The fourth annual WeCNLP (West Coast NLP) Summit (WeCNLP), virtual. Zenodo. Oct 2020. doi: 10.528 11ozeoooo700440000
Hao Fang, Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1473-1482, 2015.
10

---

Published as a conference paper at ICLR 2024
Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris, Kris M. Kitani, and Jeffrey P. Bigham. "it's almost like they re trying to hide it": How user- provided image descrip- tions have failed to make twitter accessible. The World Wide Web Conference, 2019.
Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M. Kitani, and Jeffrey P. Bigham. Twitter ally: A browser extension make twitter images accessible. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020.
Margot Hanley, Solon Barocas, Karen Levy, Shiri Azenkot, and Helen Nissenbaum. Computer vision and conflicting values: Describing people with automated alt text. In Proceedings of the 202/ AAAI/ACM Conference AI, Ethics, and Society, AIES '21, pp. 543-554, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384735. doi: 10.1145/3461 702.3462620. URL https:://pi..i.... http:////ooorgg//..//o wrt./o..g............ 1145/ 61 702. 3462620
Brienna Herold, James Waller, and Raja Kushalnagar. Applying the stereotype content model to assess disability bias in popular pre-trained NLP models underlying Al-based assistive technolo- gies. In Ninth Workshop Speech and Language Processing for Assistive Technologies (SLPAT- 2022), pp. 58-65, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. N0.11655/11/2022.sppps 1.8. URL URLhttps: http:p:s//aaaaaathooloooo http://laaaaoooooooooo///2//2220 slpat-1..p........
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, and Christopher Potts. Context matters for image descriptions for accessibility: Challenges for ref- erenceless evaluation metrics. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
Abu Dhabi, United Arab Emirates, December http1//.222221p//. 11022eemlp--mann399 main.309.
2022a. Association for Computational Linguistics. doi: 10.
URL https: http:s///aclaatthooogoooo org/ 2022 www.nmnlmmmaae mac.-mmin... 309
Elisa Kreiss, Fei Fang, Noah Goodman, and Christopher Potts. Concadia: Towards image-based text generation with purpose. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 4667 4684, Abu Dhabi, United Arab Emirates, December
2022b. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022.co www.022.eemmlpnneemmm mainlp-main.30..
owg. wttpg//222.emnnaaao emnlp main.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision vision-language understanding and generation. In International Conference on Machine Learning. pp.
PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 2rX1v:2301.12597, 2023.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495- 5510, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.186555 11633/11120 20.emnlp-main.445 URL / org/2020 emnlp mai ma.in. 443
Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04). pp. 605 612, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Meta, Nov 2021. URL http://ww......aaooe....o/.o..a. [ttp::/aeeecci-aaaaaaaoooooeeee. Be

---

Published as a conference paper at ICLR 2024
Shubhanshu Mishra, Sijun He, and Luca Belli. Assessing demographic bias in named entity recognition. In Proceedings of the AKBC Workshop on Bias Automatic Knowledge Graph Construction, 2020. arXiv, 2020. doi: 10.48550/ARXI..00000 20000000..... URL https:
www.ooggg/abba.aaooooeo...eeeb./ 2008 03415
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0973:: 2021.
OpenAI. Gpt-4 technical report, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv: 04304,, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-878 8763. PMLR, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyee,, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18 1238. URL wtt.:htttt/..cm //acl antho ogy.oog..goooo................o.
www.orrg/... htgg/115 8 1238
Jake Snell, Karl Ridgeway, Renjie Liao, Brett D Roads, Michael C Mozer, and Richard Zemel. Learning to generate images with perceptual similarity metrics. In 2017 IEEE International Con- ference on Image Processing (ICIP), pp. 4277-428 IEEE, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern 2015.
recognition, pp.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2015.
Shaomei Wu, Jeffrey Wieland, Omid Farivar, and Julie Schiller. Automatic alt-text: Computer- generated image deseriptions for blind users on social network service. Proceedings of the 2017 ACM Conference Computer Supported Cooperative Work and Social Computing, 2017.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 20050:2005.01068, 2022.
12

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
This work still has several limitations which are important to acknowledge. The dataset we collected and trained our model on is variable in quality as the gold references are user-written, and this necessarily introduces a gap between the training objective and actual alt-text best practices. could also be improved with more sophisticated identification and removal of bots, and using entity linking instead of just named entity recognition to provide more finegrained descriptions of people in lieu of their names. The model itself, while it frequently produces reasonable captions, still at times makes mistakes regarding specific details, is imprecise at transcribing text found in images as it does not contain an OCR system, and occasionally outputs the tweet text verbatim which despite at times serving as reasonably descriptive caption is nonetheless redundant. Since our system conditions on an accompanying social media post, is not easily adaptable to settings where images appear without any textual context, where the specific section of text meant to correspond to the image is ambiguous such as a long form web post or news article. We also rely on having pretrained image encoder which means that the visual features extracted cannot be informed by the context, they instead must feed into the language model as independent information signals. Finally, it's worth emphasizing that human evaluation is performed using sighted reviewers, and we therefore cannot make any claims about the downstream efficacy of our method without larger scale study using BLV users which was unfortunately beyond the scope of this work.
B DATASET COLLECTION
B. FILTERING
Reply tweets were included, but retweets were not. We filtered based on tweets containing still im- ages, as alt-text for GIF's tends to be very low quality and requires different modeling approaches. We also restricted our search to only include tweets which Twitter's language identification model categorized as English, thereby also ensuring that they all contained at least some amount of tweet text. Additional preprocessing included stripping all leading phrases such as "Image of" "Photo of" (as these are generally considered redundant for alt-text purposes), removing any examples with alt-text that were identical to the text of the tweet itself or simply contained placeholder text such as "Image", and removing any examples with alt-text containing URLs, user handles, and hashtags. We also only included alt-text that was least space-separated tokens long, as text shorter than this is rarely sufficiently descriptive.
B.2 DEDUPLICATION
We only retained the oldest of any images that contained identical alt-text. This largely eliminated any exact duplicates posted multiple times by spammy users. Next identified clusters of visual matches based on their downscaled pixel overlap. To do this, we resized images and center cropped to max size of 32 32 pixels, computed all pairwise pixel diff's within our data, and agglomera- tively grouped them into clusters using a < 100 differing pixels threshold to the nearest match as a cutoff for cluster membership. Within each cluster, we only retained the oldest tweet. It's also worth mentioning that while we did consider further deduplication based on SSIM (Snell et al.) 2017) similarity and ngram similarity of the alt-text itself, these approaches proved computationally infeasible.
C IMPLEMENTATION DETAILS
We train our models with a batch size of 100 and an initial learning rate of le 4 using the Adam optimization algorithm (Kingma& & Ba 2015). Our prefixes are of size A = 10. This allows training to fit on a single A6000 GPU. Our implementation is written in Py PyTorch (Paszke et al. 2019) and inherits some code from the ClipCap (Mokady et al.; (021)) repository. Experiments train roughly 6-18 hours depending on the model, the hyperparameter settings, and when early stopping happens (we perform early stopping based on model likelihood on the held-out validation set).
13

---

Published as a conference paper at ICLR 2024
D HUMAN EVALUATION SETUP
For each example, turkers were shown the image, the original tweet the image was from (including its text and any other images it contained), and the two candidate alt-text captions. They were never shown the original user-written alt-text. After providing them with detailed explanation of alt-text and the list of desirable criteria according Twitter's accessibility guide, we asked two questions: first which candidate alt-text was more fluent, and second which had a better literal description of the image. The models were also anonymized and their order was shuffled.
In order to ensure high quality labels, we performed qualification round on our reviewers. Con- sidering only turkers who had completed over 10, 000 HIT's, we showed them three example images where the two candidate alt-text captions were the gold user-written reference and manually writ- ten, intentionally clearly lower quality caption. Only reviewers who successfully picked the real user-written caption for both fluency and descriptiveness for all three examples were allowed to par- ticipate in the final larger scale comparisons between our model and frozen ClipCap, and our model and ClipCap from scratch.
14


Reviews:
[{"review_id": "qMKs5OzNQs", "reviewer": "Reviewer_xHBk", "summary": "This paper studies on an important problem: how to generate Alt-text to help improve the accessibility of social media images. More specifically, the authors collected and cleaned a large dataset of images incorporated with the alt-text labeled by users. With this large dataset, they are able to evaluate the proposed method and some baselines. Overall the proposed solution is not novel for the Machine Learning Community. But the research problem is interesting and meaningful. Also, the collected dataset is important for the community of HCI and Social Media Analysis.", "questions": "1. Will the tweet-based reranking improve the baselines like BLIP-2 and ClipCap?\n\n2. Is there some other reranking strategies that you tried? Such as comparing the Clip-based similarity in the representation space?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The collected dataset is important and useful. The data preprocess ensure its usability.\n\n2. The research problem raised in this paper is important.", "weaknesses": "1. The novelty of the proposed method is really limited excpet the tweet-text-based reranking.\n\n2. The experiment is somewhat not extensive. For example, from Table 1, it seems that the tweet-based reranking is the most important component. But the authors did not tried to incorporate the reranking with the baselines, which is not fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RcBq3Piuzn", "reviewer": "Reviewer_1eBv", "summary": "This paper proposes a method for generating \"alt-text\" for social media images. In this paper, \"alt-text\" is explained as a more detailed description and context-specific than a generic image caption.\nThe proposed method takes an image and the social media text that accompanies the image and outputs the alt-text of the image. The input image is encoded by CLIP and combined with the encoded text. The combined information is then input to GPT-2 to generate the alt-text.\nThe author has collected tweets and their corresponding images with alt-text to create a dataset for alt-text generation research. The evaluation shows the results of using this dataset to compare the proposed method with several baseline methods.", "questions": "I wondered if the \"alt-text\" would be written differently for different people, even for the same image. Is it possible for the proposed method to learn well in such a case?\n\nIn addition, although the paper uses crowdsourcing for evaluation, I think it is possible that the evaluation by the viewer of the \"akt-text\" may also differ from person to person. How did the evaluation go this time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The author points out that in social media, images are often posted in addition to textual information, but there is little information describing the images, and in such situations, information about the images is not conveyed by text-to-speech software for the visually impaired, for example. I agree with this point and understand its importance as a study.\n\nAs for the proposed method, its basic structure consists of encoding images using CLIP and generating text using GPT-2. This structure itself is not unique, as it is a concept that has been used in existing research. I believe that the originality lies in the extended part of the basic configuration, where not only the image but also the text to which the image is attached is input.\n\nThe fact that an original data set was constructed for this study is commendable. It is also commendable that in the evaluation using this data set, comparisons with various methods were made to show the characteristics of this task.", "weaknesses": "As mentioned in the \"Strengths\" section, the focus on \"alt-text\" is highly evaluated, but there is room for improvement in that \"alt-text\" is not clearly defined in the paper.\n\nIn the evaluation dataset, the \"alt-text\" entered by twitter users is used as the correct answer, but it is written as \"alt-text captions on Twitter are written by untrained users, they can be noisy, inconsistent in form and specificity, and occasionally do not even describe the image contents\" in the paper, and it seems that the authors are discussing the generation method without knowing what \"alt-text\" is.\n\nAlthough the difficulty is understandable, I think that the discussion should start with clarifying what \"alt-text\" is, and then discussing the generation method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EhjzECJw4c", "reviewer": "Reviewer_2NmJ", "summary": "The paper presents a method to generate accessibility captions for images shared on Twitter. The proposed approach combines CLIP embeddings for the images, as well as additional context that is included in the tweet’s text to create an embedding that is then fed to GPT-2 to generate the accessibility description. The paper’s evaluation demonstrates that the proposed approach can outperform naive and neural-based approaches like ClipCap and BLIP-2.", "questions": "1. What is the rationale for using user-generated captions as gold standards and do you have an idea how this affects the presented results?\n2. How is the BLEU@4 score calculated and did you use a modified version of the metric? \n3. How is the user-study conducted and what are the background/expertise of the recruited participants? Also, have you obtained an IRB approval before conducting the user study? How did you ensure that participants were not exposed to harmful information?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "First of all, I would like to applaud the authors for working on this important and timely problem. I believe that this research is very important and can have the potential to improve the lives and online experience of many people with visual impairments. Overall, I believe that this research focuses on an important problem, and there is potential for a big impact. Second, the paper collects a large-scale dataset of images and user-generated accessibility captions from Twitter; this dataset is far bigger than previous research efforts focusing on similar research problems. Third, I believe that the paper’s approach is a simple, creative, and effective method to combine CLIP embeddings, tweet text, and LLMs to generate accessibility captions for images. The paper’s approach is easy to understand and combines important features for generating contextual and useful accessibility descriptions for images shared on Twitter. Finally, I like that the paper evaluates the performance of the proposed approach both with quantitative metrics as well as via a user study that aims to assess how users perceive and compare accessibility descriptions generated from the proposed method and baseline approaches.", "weaknesses": "I have several concerns with the paper, mainly related to the lack of gold standards for accessibility captions, the lack of important and adequate methodological details, the paper’s evaluation, the paper’s approach to releasing data, and the paper’s ethical considerations.\n\nFirst, there is a disconnection between the paper’s motivation and how the paper evaluates the performance of the proposed method. I agree with the paper’s motivation that the user-generated accessibility captions are of questionable quality, given that most users are unaware of best practices for generating accessibility captions. On the other hand, however, the paper collects user-generated accessibility captions and treats them as gold standards (i.e., ground truth captions). This is problematic as in the evaluation, the paper compares the generated captions from their approach and compares them with captions that are of questionable quality. Therefore, it is not clear what is the actual performance of the proposed methods. A way to potentially mitigate this issue is to apply the proposed approach to other datasets released by previous research that include gold-standard captions (i.e., captions that adhere to the best practices for generating accessibility descriptions for images).\n\nSecond, I am puzzled about how the BLEU@4 score is calculated in the evaluation. To the best of my knowledge, the BLEU score ranges from 0 to 1 and aims to assess the precision of the n-grams included in the generated text compared to the ground truth. In the paper’s evaluation, the paper mentions that the proposed approach has a BLEU@4 score of around 1.8. I suggest to the authors to clarify how they calculated the BLEU scores (e.g., if they used a modified version) and better describe how we can interpret these BLEU@4 values.\n\nThird, the paper lacks important and adequate details on the paper’s methodology. Particularly, the paper refers to several appendices so that the reader can get more information, however, there are no appendices in the manuscript. This hurts the readability of the paper and does not allow us to assess the quality and robustness of the presented results in the paper. I suggest including the appendices so that we can understand how the paper conducted various steps of the research. In particular, I would have liked to read more on how the paper conducted the user study, how they recruited users, what is their background and expertise with regards to the best practices for generating accessibility descriptions, etc. All these details are paramount for understanding the quality of the presented research.\n\nFourth, I have some concerns about the paper’s approach to releasing the dataset. Given the recent changes to Twitter’s API, it became extremely hard to rehydrate tweets based on their IDs. So by simply releasing the Twitter IDs and the media URLs, interested researchers will not be able to reproduce the paper’s results and further use this dataset for further studying this problem. I suggest to the authors to consider releasing more attributes from the dataset (specifically the tweet’s text) so that interested researchers can reproduce the paper’s results without relying on the closed and expensive new Twitter APIs.\n\nFifth, the paper does not properly explain how the qualitative assessment is done (in Section 6.4), which does not allow the reader to understand if it’s done in a systematic way or how representative/generalizable the insights are. I suggest to the authors to include more details on how the samples for the quantitative analysis are selected and, more importantly, how the qualitative assessment is undertaken (e.g., are the people experts in the domain of accessibility description generation, are they aware of the best practices, etc.)\n\nFinally, the paper does not discuss the ethical considerations when conducting this research. This is important as the paper conducts a user study and shows participants’ images shared on Twitter. For instance, did the paper ensure that there are no harmful images in the dataset and that no participants were exposed to harmful information?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)