Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models (2024)

Abstract: Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they “fool” four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs’ multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade.

heading: 1 Introduction
text: Image-based advertisement is not only a crucial component of marketing campaigns, but also an interesting example of sophisticated multimodal communication. Ads often feature unusual visual elements (e.g., objects that are non-photorealistic, outside of their usual context, atypical, etc.) or examples of figurative language (e.g., metaphors, allegories, play on words, etc.) designed to make a long-lasting impression on the viewer. Figure 1 provides an example of an ad with a non-photorealistic object (a whale made of wires) that, as the text suggests, is used to convey a complex metaphorical message about the product (i.e., a wireless device).
These elaborate uses of images and text make automatic ad understanding a challenging task requiring multiple non-trivial abilities, e.g., object detection, scene-text extraction, figurative language understanding, and complex image-text integration. Ad understanding was first proposed as a deep-
learning task by Hussain et al. (2017), who introduced the Pitt Ads dataset, consisting of imagebased ads along with explanations capturing their underlying message (e.g., I should purchase this stereo system because wireless is less messy). This dataset was then used in a retrieval-based challenge requiring to identify a plausible explanation for an ad within a set of possible candidates.1
Early work on this task has employed ensemble predictors (Hussain et al., 2017; Ye and Kovashka, 2018) and graph neural networks (Dey et al., 2021) that were designed and trained ad hoc. More recently, the development of large vision-andlanguage models (VLMs) pretrained with imagetext matching (ITM) objectives has opened the possibility of performing the task in zero-shot, i.e. by using an off-the-shelf model instead of training one from scratch. Following this approach, Jia et al. (2023) tested multiple VLMs (ALBEF, Li et al. 2021; CLIP, Radford et al. 2021; and LiT, Zhai et al. 2022) on the task by computing imagetext alignment scores between ads and their possible explanations. They observed an excellent zeroshot performance for all models, documenting an accuracy of 95.2% for CLIP.
While the results reported by Jia et al. (2023) seem to suggest that the tested models developed the reasoning abilities necessary to succeed at ad understanding, we note that this conclusion is in contrast with a great deal of existing work. Extensive research investigating whether VLMs develop reasoning skills as a result of their contrastive ITM pretraining has exposed several weaknesses of these models. They have been shown to be limited in their abilities to identify noun mismatches in image captions (Shekhar et al., 2017), reason compositionally (Thrush et al., 2022), capture spatial relations (Liu et al., 2023), understand verbs (in-
1https://eval.ai/web/challenges/ challenge-page/86/overview
870
stead of just nouns) (Hendricks and Nematzadeh, 2021), and handle various linguistic phenomena (Parcalabescu et al., 2022) and basic constructions (Chen et al., 2023).
Importantly, this line of work focused on a set of traditional visuo-linguistic tasks but not specifically on ad understanding. Here, we ask whether the performance previously documented on the Pitt Ads dataset reflects genuine understanding abilities or is driven by simpler heuristics. We conduct a thorough analysis of the evaluation setup originally proposed to test ad understanding and reveal that it has key flaws, which allow models to exploit grounding heuristics. We introduce a new test set, TRADE (TRuly ADversarial ad understanding Evaluation), which controls for the identified issues. Our experiments show that several contrastive models tested zero-shot, including CLIP, perform at chance level on TRADE, while humans excel at the task. More generally, our findings highlight the need to better operationalise ad understanding in order to obtain reliable assessments of VLMs’ multimodal reasoning abilities.

heading: 2 A Closer Look at the Evaluation Setup
text: The Pitt Ads dataset2 by Hussain et al. (2017) consists of 64832 ads, each annotated with 3 explanations in English written by 3 different expert annotators. These explanations (in the form I should ⟨action⟩ because ⟨reason⟩) aim at capturing the persuasive message behind the ads. While explanations may be subjective, the intuition behind the image-to-text retrieval task proposed along with
2https://people.cs.pitt.edu/~kovashka/ads/
the dataset is that a model which can understand ads should be able to match them with a plausible explanation. Specifically, each ad is paired with 15 messages, 3 positives corresponding to the annotations for that ad and 12 negatives randomly sampled from annotations for different ads. Figure 1 provides an overview of the task setup.
Previous work has hinted at possible limitations of the evaluation setup. Kalra et al. (2020) observed a significant overlap between the text present in the ad and the matching explanations and noticed this was “a major discriminating factor” that their finetuned BERT model could exploit. Similarly, Jia et al. (2023) pointed out that the candidate set lacks “hard negatives” and proposed to increase the set size, but could not provide a solution ensuring the negatives were actually hard.
We conduct a quantitative analysis on the original evaluation setup to uncover potential shortcuts that VLMs may be exploiting to solve the task. We hypothesise that the models may take advantage of two factors that do not necessarily reflect ad understanding: simple relationships between (1) the candidate explanations and the text present in the ad (i.e., the degree of textual grounding of the explanations) and (2) the entities mentioned in the explanations and those depicted in the image (their degree of visual grounding). To test our hypotheses, we define several visual- and textualgrounding scores and check whether they correlate with the CLIP-based alignment score used by Jia et al. (2023) to retrieve the ad explanations.3
3More details on the scores can be found in Appendix A.
Textual-grounding scores are computed between candidate explanations and the text extracted from the ad with Optical Character Recognition (OCR). We calculate (1) text overlap as the proportion of content-word lemmas from the explanation that are also present in the OCR-extracted text, and (2) text similarity as the cosine similarity between a sentence-level embedding of the explanation and that of the OCR-extracted text, derived with MPNet (Song et al., 2020).
Visual-grounding scores include (1) object mention as the proportion of nouns in the candidate explanation that are present in a set of objects we automatically extracted from the image by a ResNet50 model (He et al., 2016), and (2) caption similarity as the cosine similarity between the sentencelevel embedding of the candidate explanation and the embedding of the ad caption we obtained with BLIP-2 (Li et al., 2023). Our motivation for examining both detected objects and generated captions is driven by the observation that they capture complementary information. More specifically, while detected objects are not mediated by language models, they may often incorporate non-salient objects that people would unlikely mention when describing a picture or contain lexical choices that differ from the human ones. On the other hand, generated captions refer to objects in a more human-like way but, at the same time, may contain hallucinations due to linguistic priors.
We compute the grounding scores and CLIP’s alignment score for the test split of the Pitt Ads dataset, consisting of 12805 samples. As hypothesised, we observe a positive correlation between all our grounding scores and CLIP’s alignment score. All the Spearman’s correlation coefficients are significant (p ≪ 0.001) and range from 0.14 and 0.61 (see Appendix A for details). In addition, as shown in Table 1 (left), we find that in the original setup the matching explanations are significantly more grounded than the non-matching explanations for each ad. While the elements (OCR text, objects, captions) detected by other models are not necessarily the same as those identified by CLIP, these results suggest that reasonably similar information is indirectly extracted by CLIP and exploited to solve the ad-understanding task. This finding also agrees with results from previous work showing that CLIP develops OCR capabilities and can successfully classify objects (Radford et al., 2021).
Overall, these results indicate that the original
evaluation setup is flawed and that the outstanding zero-shot performance obtained by VLMs on the retrieval task may be due to simple image-text alignment.

heading: 3 TRADE: A New Adversarial Test Set
text: To test the extent to which VLMs capture elaborate visuo-linguistic relationships present in imagebased ads beyond image-text alignment, we develop TRADE (TRuly ADversarial ad understanding Evaluation), a new diagnostic test set with adversarial negative explanations. TRADE consists of 300 randomly selected ads from the Pitt Ads dataset, each associated with 3 options (1 positive and 2 negatives). Concretely, for each of these ads, we randomly select one valid explanation from the available annotations and create two adversarial negative explanations—see Figures 1 and 2 for examples (more examples in Appendix C). The adversarial explanations were created by 4 expert annotators who were instructed to do their best to come up with non-plausible explanations that nevertheless mention objects and fragments of text present in the image. Annotators were also asked to approximately match the length of the positive explanation when writing these adversarial sentences. Appendix B contains more details about the creation of the adversarial negatives, including the guidelines provided to the annotators.
We validate TRADE in two ways. First, we compute the textual- and visual-grounding scores introduced in Section 2. This shows that in TRADE the gap between positive and negative explanations is radically reduced compared to the original setup, as can be seen in Table 1 (right). Second, we confirm that humans are not affected by the high level of grounding of both positive and negative examples and are able to identify the plausible explanation in
the TRADE samples with an accuracy of 94%.4
To allow for a direct comparison with an evaluation setup with random negatives, akin to the original task setup, we also create TRADE-control: a version of TRADE where the two negative explanations per ad are randomly sampled from the explanations for other ads. TRADE-control includes 10 versions created with different random samplings.
TRADE and TRADE-control are publicly available at https://github.com/dmg-illc/trade under a Creative Commons Attribution 4.0 International (CC-BY) license.

heading: 4 Experiments
text: We use TRADE to test four contrastive pretrained VLMs zero-shot. Three of these models (CLIP, Radford et al. 2021; ALBEF, Li et al. 2021; and LiT, Zhai et al. 2022) have been shown to achieve high zero-shot performance on the original task setup (Jia et al., 2023). Here we challenge them with TRADE and consider an additional model (ALIGN, Jia et al. 2021).

heading: 4.1 Models and Setup
text: Except for ALBEF, all the models we test encode visual and textual inputs separately and are pretrained with an image-text matching objective. ALBEF has an additional multimodal module, but here we only use its unimodal encoders, which are also pretrained contrastively. A more detailed overview of these VLMs is reported in Appendix E.
All four models allow for the computation of an image-text alignment score, here defined as the dot product between the normalized image embedding and the text embedding of each candidate explanation. As in previous work (Jia et al., 2023), we evaluate the models by computing alignment scores for every ad-explanation pair and consider the explanation yielding the highest alignment score as the model’s retrieved option. We report average accuracy, as (mean) rank is not very informative with only 3 candidates.

heading: 4.2 Results
text: Table 2 shows the performance of the models on TRADE and TRADE-control. All models achieve an accuracy higher than 80% in the control condition, with CLIP reaching 98%. However, the performance of all models in the adversarial setting—
4Each of the 300 samples was annotated by two annotators external to the project; more details available in Appendix D.
where humans achieve 94% accuracy, cf. Section 3—nears chance level, i.e., 33%. Figure 2 provides an example of model- and human-chosen ad explanations on a TRADE instance. These results provide compelling evidence that the evaluated VLMs rely on visual and textual grounding when retrieving ad explanations. As a result, they can achieve excellent accuracy in an evaluation setting where negatives are poorly grounded, but are easily “fooled” by grounded adversarial distractors that are extremely easy for humans to discard.
To get more insight into the models’ performance, we examine their predictions and observe that, while all models perform equally poorly on TRADE, there are 23 samples (8% of the dataset) for which the four models succeed at identifying the target explanation. An analysis of the explanations correctly retrieved by all models reveals that most of them exhibit grounding scores that are higher than the average scores for matching explanations. Figure 3 visualises this finding.

heading: 5 Conclusions
text: Our work exposes key limitations of the evaluation setup that was previously used to benchmark VLM’s ad understanding abilities. We introduce a new adversarial test set (TRADE) that controls for the identified issues and show that, while humans excel, contrastive VLMs perform at chance level on TRADE. This result has the following implications.
First, it shows that, when processing imagebased ads, contrastive VLMs are strongly biased towards textually and visually grounded explanations, regardless of their plausibility. This is in agreement with previous work (Hendricks and Nematzadeh, 2021; Parcalabescu et al., 2022; Thrush et al., 2022; Liu et al., 2023; Chen et al., 2023) and points to the need to use caution when interpreting models’ zero-shot accuracy on “naturalistic” (i.e., non-adversarial) setups as proof that they develop sophisticated reasoning abilities via pretraining.

heading: Ad TRADE explanations Chosen by
text: Second, our work highlights issues with the current retrieval-based operationalization of ad understanding as a task to evaluate VLM’s multimodal reasoning abilities. We emphasise that TRADE’s aim is to control for a confound—the grounding gap between positives and negatives—that we identified as crucial when testing a specific type of VLMs, i.e., those pretrained with an ITM objective. However, defining which abilities are necessary to conclude that a model developed a good “understanding” of image-based ads and designing a task that truly evaluates them remain open issues for future research.
Limitations
The current study and previous work have operationalised ad understanding as an ad-explanation retrieval task. In particular, we have focused on testing contrastive pretrained VLMs zero-shot on this task. Consequently, the question of whether
VLMs trained or finetuned on the Pitt Ads dataset would be more robust against our adversarial explanations remains open and could be investigated in the future. Nevertheless, we emphasize that the retrieval-based setup has limitations (e.g., the impossibility of providing task-specific instructions to the models) and may not be the most appropriate to evaluate VLM’s ad understanding skills and their multimodal reasoning abilities more generally. An interesting direction for future research could be to formulate the task differently, e.g., as a generative task. This would solve some issues of the retrievalbased setup, but also posit novel challenges, such as identifying the most effective prompt and defining meaningful protocols to evaluate the generated explanations.
On a methodological note, we highlight that visual and textual alignment are complex constructs that encompass different aspects and can be analysed at different levels of granularity. Therefore, we do not intend our grounding scores as precise and comprehensive metrics, but simply as indicators that can reflect general trends.
Ethical Considerations
TRADE does not introduce new ad-images, but simply links to the existing Pitt Ads dataset along with the set of adversarial explanations we have created. However, it is worth emphasizing that the ads present in Pitt Ads were originally collected by querying Google Images. This posits two ethical concerns.
First, offensive/harmful content or stereotypes may be present in the images, as already pointed out by Jia et al. (2023). To minimise this potential problem when developing TRADE, we made sure the annotators who created our adversarial explanations had the possibility of flagging ads that they deemed inappropriate (they did so a couple
of times). However, we cannot fully guarantee that the ad images used in TRADE are completely free from harmful content. As for the adversarial distractors created for TRADE, we have not systematically examined all of them manually to make sure they do not contain harmful content, but we believe this is very unlikely given the guidelines and the fact that they were created in a very controlled setting partially by us and partially by close colleagues.
The second concern is about the license of the images. The Pitt Ads dataset was released without a license and the curators do not clarify whether the images are copyrighted or not.
Finally, we note that our study does not take into account the personal and cultural factors which may play a substantial role in people’s perception of ads or in the values they associate with certain products. Although TRADE includes only one matching explanation for each ad, we emphasize that we do not intend this as a “ground truth”. We hope that future research on automatic ad understanding will adopt evaluation protocols that reflect a diverse set of possible interpretations.

heading: Acknowledgements
text: We warmly thank the current members and alumni of the Dialogue Modelling Group (DMG) from the University of Amsterdam for the support they provided at different stages of this project. Special thanks are due to Sandro Pezzelle, who suggested the name ‘TRADE’ for our introduced dataset. Our heartfelt gratitude also goes to the colleagues who assisted us with the creation of TRADE negatives and to the colleagues, friends and partners who kindly volunteered to participate in our experiment to assess human performance on TRADE. The present work was funded by the European Research Council under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 819455).

heading: A Grounding Scores
text: Textual Grounding The textual grounding scores were computed between candidate explanations and the ad OCR-extracted text output by Google Vision API5 and made publicly available6 by the authors of Savchenko et al. (2020). OCR text was present for 12304 ad-images of the test set and 294 images from TRADE. The text overlap score was computed as the proportion of words from the candidate explanation that were also present in the OCR-extracted text. Before computing the overlap, we lemmatized the text and removed stop-words. These preprocessing steps were performed with the NLTK7 package.
The text similarity score was defined as the cosine similarity between the embedding of the explanation and that of the OCR-extracted text. The embeddings were obtained using the Sentence Transformers8 framework. Specifically, we used an MPNet (Song et al., 2020) pretrained model, which was indicated as the best-performing one.
Visual Grounding To compute the visual grounding scores, we considered two sources of visual information: the objects identified by an object detector, and the ad-image captions. Our object detector was a ResNet50 model (He et al., 2016) pretrained on MS COCO (Lin et al., 2014). We used the implementation from the Detectron29 framework by Facebook. The model detected an average of 3.74 ± 3.85 objects from the Pitt Ads dataset test split and 3.51 ± 3.38 from TRADE. At least one object was detected on 11351 images from the Pitt Ads dataset test split and on all the ads
5https://cloud.google.com/vision/docs/ocr 6https://figshare.com/articles/dataset/OCR_
results/6682709 7https://www.nltk.org/ 8https://github.com/UKPLab/ sentence-transformers?tab=readme-ov-file 9https://github.com/facebookresearch/ detectron2
from TRADE (300). Ad captions were obtained using BLIP2 (Li et al., 2023) with OPT 2.7B as language decoder. BLIP2 was used in its Hugging Face implementation.10
The object mention score was computed as the lemmatized nouns in the AR statement that were part of the set of detected objects. The caption similarity score, on the other hand, was defined similarly to the text similarity score, with the caption in place of the OCR-extracted text.
Correlation with CLIP’s alignment scores We computed Spearman correlations between all the grounding scores and the CLIP-alignment scores for both the Pitt Ads test set and TRADE.
All results are summarized in Tables 3 and 4.

heading: B Creating TRADE
text: The adversarial negatives were designed by two of the authors and two internal collaborators who volunteered for the task and are all proficient in English. Due to the complexity of this annotation task, we deemed it not suitable for crowdsourcing. The instructions given to the annotators were the following:
1. The sentence should be inconsistent with the image, meaning that it should not be a valid answer to the question “What should you do, according to this ad?”. Keep in mind that the answer should be patently wrong, i.e. it should require very little thinking to figure out it does not match the message of the ad.
2. The sentence should be in the form “I should [action] because [reason]”
3. The verb you use after “should” should be the same as the one from the right sentence. For example, if the right sentence starts with “I should buy”, your wrong annotation cannot start with “I should fly”
4. The sentence should be as grounded as possible, meaning that you should avoid mentioning objects/words that are not present in the ad as much as you can. Please keep this in mind, it is very important!
5. If possible, privilege salient visual elements over nonsalient ones. More concretely, try to mention large writings instead of small ones, and big foreground objects instead of small background ones.
6. When describing visual objects, try to be efficient instead of verbose. For example, if an ad depicts a famous man (say, Mr. X) driving a car of a specific brand (say, Brand Y), you should write something like “I should buy Mr. X because he drives a cool Brand Y car” instead of “I should buy a man with short hair and sunglasses because he drives a red four-wheeled vehicle”
7. Please avoid extra-long sentences. Your wrong answers should be approximately the same length as the correct ones. You don’t need to be as strict as to count the exact
10https://huggingface.co/docs/transformers/ model_doc/blip-2
number of words but try to avoid large mismatches (e.g. correct answer being not even one-line long and wrong answer being two lines)
8. Only include the name of brands/celebrities if they are also mentioned in the provided annotation
9. The sentence (e.g., “I should buy this perfume because roses are red and violets are blue”) but it should not be ungrammatical (do not write something like “I should hello world because rainbow”)
Rule 8 was introduced as there is evidence (Goh et al., 2021) that CLIP is sensitive to proper nouns. Therefore, we wanted to avoid our negatives being preferred by the model simply because they contained more detailed information.
Our annotation interface allowed annotators to flag ads in case of:
1. Presence of inappropriate/offensive/harmful content.
2. Low readability of the text. 3. Low image resolution. 4. Being unable to understand the ad (e.g., be-
cause the text was not in English). 5. Being unable to create a distractor meeting all
the requirements.

heading: C Dataset Examples
text: Some additional examples of the adversarial explanations we collected are shown in Figure 4 along with their TRADE-control counterparts.

heading: D Human Accuracy on TRADE
text: To quantify the human accuracy on TRADE, we used the crowdsourcing platform Appen to present participants with the ad along with the question “What should you do according to this ad, and why?” and 3 options, i.e., a matching explanation and two adversarial grounded negatives. After some unsatisfactory pilot experiments where crowdworkers were not able to pass very simple test questions, we established that the task was not suitable for crowdsourcing. Therefore, we recruited 17 participants who volunteered for the task of judging the 300 samples in TRADE. They were not involved in the creation of the adversarial explanations and were informed that their anonymised data would be included in a study about automatic ad understanding. We ensured all annotators were proficient in English. Each question was answered by 2 different participants. They annotated an average of 35 ads each (std = 14, max = 50, min = 10). The mean accuracy calculated over the 600 collected
judgements was 94%. The cases where both participants selected the target explanation were 270 (90%).

heading: E Tested Models
text: Here we provide an overview of the models used in our experiments.
CLIP (Radford et al., 2021) is a contrastive model where image and text are separately encoded by two transformer-based models and then projected to the same vector space. CLIP is trained with a contrastive loss that minimizes the cosine distance between matching pairs of image and text embeddings. We used it in the Hugging Face implementation.11
ALIGN (Jia et al., 2021) is also a contrastive vision-and-language model trained with the same loss function used for CLIP. It mainly differs from the latter in its encoders (EfficientNet for images and BERT for text) and in that also leverages noisy data during the training process. We used the Hugging Face model implementation.12
LiT (Zhai et al., 2022) is a contrastive model where the image encoder is “locked” (i.e. frozen) during pre-training, whereas the language encoder is initialized with random weights and trained from scratch with a contrastive loss. We used the Vision Transformer implementation13 by Google Research.
ALBEF (Li et al., 2021) is a vision-and-language model consisting of two separate transformer-based encoders from image and text and a multimodal encoder. The uni-modal modules are pre-trained contrastively and their outputs are then fused in the multimodal module, which is pre-trained with masked-language-modeling and image-textmatching objectives. We used the LAVIS implementation by Salesforce.14
11https://huggingface.co/docs/transformers/ model_doc/clip
12https://huggingface.co/docs/transformers/ model_doc/align
13https://github.com/google-research/vision_ transformer
14https://github.com/salesforce/LAVIS

heading: Future Work
text: However, defining which abilities are necessary to conclude that a model developed a good “understanding” of image-based ads and designing a task that truly evaluates them remain open issues for future research. Limitations
The current study and previous work have operationalised ad understanding as an ad-explanation retrieval task. In particular, we have focused on testing contrastive pretrained VLMs zero-shot on this task. Consequently, the question of whether
VLMs trained or finetuned on the Pitt Ads dataset would be more robust against our adversarial explanations remains open and could be investigated in the future. Nevertheless, we emphasize that the retrieval-based setup has limitations (e. g. , the impossibility of providing task-specific instructions to the models) and may not be the most appropriate to evaluate VLM’s ad understanding skills and their multimodal reasoning abilities more generally. An interesting direction for future research could be to formulate the task differently, e. g. , as a generative task. This would solve some issues of the retrievalbased setup, but also posit novel challenges, such as identifying the most effective prompt and defining meaningful protocols to evaluate the generated explanations. On a methodological note, we highlight that visual and textual alignment are complex constructs that encompass different aspects and can be analysed at different levels of granularity. Therefore, we do not intend our grounding scores as precise and comprehensive metrics, but simply as indicators that can reflect general trends. Ethical Considerations
TRADE does not introduce new ad-images, but simply links to the existing Pitt Ads dataset along with the set of adversarial explanations we have created. However, it is worth emphasizing that the ads present in Pitt Ads were originally collected by querying Google Images. This posits two ethical concerns. First, offensive/harmful content or stereotypes may be present in the images, as already pointed out by Jia et al. (2023). To minimise this potential problem when developing TRADE, we made sure the annotators who created our adversarial explanations had the possibility of flagging ads that they deemed inappropriate (they did so a couple
of times). However, we cannot fully guarantee that the ad images used in TRADE are completely free from harmful content. As for the adversarial distractors created for TRADE, we have not systematically examined all of them manually to make sure they do not contain harmful content, but we believe this is very unlikely given the guidelines and the fact that they were created in a very controlled setting partially by us and partially by close colleagues. The second concern is about the license of the images. The Pitt Ads dataset was released without a license and the curators do not clarify whether the images are copyrighted or not. Finally, we note that our study does not take into account the personal and cultural factors which may play a substantial role in people’s perception of ads or in the values they associate with certain products. Although TRADE includes only one matching explanation for each ad, we emphasize that we do not intend this as a “ground truth”. We hope that future research on automatic ad understanding will adopt evaluation protocols that reflect a diverse set of possible interpretations