Identification of Multimodal Stance Towards Frames of Communication
Authors: Maxwell Weinzierl
Sanda Harabagiu
Venue: EMNLP 2023 Main (2023)
Keywords: stance detection
covid-19
social media
twitter
multimodal
images
multimedia
TL;DR: Constructed first multimedia text/image stance detection towards communication framings dataset and demonstrated high stance detection performance exploiting relations between text and image stance.

Abstract: Frames of communication are often evoked in multimedia documents. When an author decides to add an image to a text, one or both of the modalities may evoke a communication frame. Moreover, when evoking the frame, the author also conveys her/his stance towards the frame. Until now, determining if the author is in favor of, against or has no stance towards the frame was performed automatically only when processing texts. This is due to the absence of stance annotations on multimedia documents. In this paper we introduce MMVax-Stance, a dataset of 11,300 multimedia documents retrieved from social media, which have stance annotations towards 113 different frames of communication. This dataset allowed us to experiment with several models of multimedia stance detection, which revealed important interactions between texts and images in the inference of stance towards communication frames. When inferring the text/image relations, a set of 46,606 synthetic examples of multimodal documents with known stance was generated. This greatly impacted the quality of identifying multimedia stance, yielding an improvement of 20% in F1-score.

Identification of Multimodal Stance Towards Frames of Communication
Maxwell A. Weinzierl and Sanda M. Harabagiu Human Language Technology Research Institute, The University of Texas at Dallas (maxwell. weinzierl, sanda)@utdallas..
Abstract
COMMUNICATION:
Scientists have working Coronavirus for decades, The COViD-19 vaccine has been tested, tracked safe.
Frames of communication are often evoked in multimedia documents. When author de- cides to add an image to text, one or both of the modalities may evoke a communication frame. Moreover, when evoking the frame, the author also conveys her/his stance towards the frame. Until now, determining the author is in favor of, against or has no stance towards the frame was performed automatically only when processing texts. This is due to the absence of stance annotations multimedia documents. this paper we introduce a dataset of 11,300 multimedia documents re- trieved from social media, which have stance annotations towards 113 different frames of communication. This dataset allowed us to periment with several models of multimedia stance detection, which revealed important in- teractions between texts and images in the infer- ence of stance towards communication frames. When inferring the text/image relations, of 46,606 synthetic examples of multimodal documents with known stance was generated. This greatly impacted the quality of identifying multimedia stance, yielding an improvement of 20% in Fi-score.
Nho is going have the covid vaccine
JUST HAD COV
VACCINE
VAX-STANCE,
FEELING GREAT!
Figure 1: Frame evoked in multimedia document.
computational methods (Card et al., 2016; Hart- mann et al., 2019; Mendelsohn et al., 2021) have focused on the problem of discovery of communica- tion frames, more precisely, identifying when a frame is evoked in a text. However, frames are also evoked in images, not only in texts, as advocated in Entman (2003), as text/images combinations con- vey memorable, emotionally-charge messages.
1 Introduction
MULTIMODAL STANCE
Frames of communication select particular aspects of an issue and make them salient in communi- cating a message (Entman, 1993). For example, when discussing the confidence in the COVID- 19 vaccine, selecting the aspects concerning the development of the vaccine may convey the mes- sage that the vaccine is safe because (a) scientists have worked for decades on coronavirus vaccines, and (b) the vaccine has been tested and tracked, as shown in the frame illustrated in Figure Frames of communication are ubiquitous in social media discourse and can impact how people understand issues and, more importantly, how they form their opinions (Chong and Druckman, 2007). Previous
Evokes
Text / Image
Author
Frame of Communication
Evokes
/ Image
Author
MULTIMODAL STANCE
Figure 2: Communication frame evoked multimedia.
In documents where frame of communication is evoked, the authors of the documents also ex- press their stance towards the frame. This allows to consider the maltimodal stance towards frames as an extension of the original notion of frame evo- cation introduced in Entman (2003), as shown Figure 2.

---

Figure 1 illustrates a frame evoked in a tweet. Understanding why the frame is evoked requires accounting for the interaction between the tweet text and the image. While the question from the text of Figure does not necessarily articulate the problem of vaccine safety, the image shows the re- sult of vaccination, and the text superposed on the image is sarcastic. Together, the text and the image evoke the frame because they address the aspects highlighted by the frame, referring confidence in the COVID-19 vaceine. Moreover, the tweet au- thor also conveys her/his stance towards the frame, namely that they disagree with it.
formation. This allowed us to consider several data augmentation methods, which proved im- prove the results of multimedia stance detection. The MMVAX-STANCE dataset, code, and other resources are available on GitHub'.
The remainder of the paper is organized as fol- lows. Section 2 presents the communication frames that were used in the multimodal annotation of stance. Section describes the annotations, while models used for multimodal stance detection are presented in Section Section 3 outlines the exper- imental results, which are also discussed. Section 6 describes the related work, while Section 7 summa- rizes the conclusions.
The goal of stance detection to determine whether the author of a document is in favor of, against, or has no stance toward a specific target (Mohammad et al., 2016; Hardalov et al., 2022). Instead of considering stance towards target, we consider stance towards a frame of communica- tion, which articulates precisely the relevant as- pects highlighted by the frame. In this case, the stance is judged against explicit aspects of the frame, not against some unspecified properties of a target. Moreover, since until now stance identi- fication methods relied only on texts, we propose to also consider images and their interaction with texts in the determination of the stance of doc- ument towards frame of communication. To be able develop computational methods capable of inferring the stance of the author of multime- dia document towards a frame of communication, we need to rely on annotated examples. To our knowledge, there are no such annotations currently available.
2 Frames of Vaccine Hesitancy
As reported in Glandt et al. (2021), the recent COVID-19 pandemic has given rise to a large va- riety of opinions regarding the public health man- dates designed to contain the spread of the virus, While sancce-nnotated dataset including tweets targeting these measures was introduced in Glandt et al. (2021), no opinions determined by the emer- gence of the COVID-19 vaccines were captured. These vaccines were hailed with enthusiasm by many, but substantial opposition and hesitancy to- wards vaccines also emerged on social media. This motivated our interest in capturing the stance to- wards multiple frames of communication that high- light aspects of vaccination.
Attitudes towards vaceination, e.g. vaccine readi- ness or hesitancy, are modeled by a set of seven components or factors that increase or decrease an individual's likelihood of getting vaccinated (Geiger et al., 2022). These components are listed in Table along with their definitions and examples of frames of communications that address aspects of each component. We note that the vaccination components listed in Table I can be interpreted as dimensions of the frames, similar to the frame di- mensions introduced in the Media Frames Corpus (Card et al., 2016).
In this paper we introduce a new dataset of Twit- ter postings which (1) contain a combination of text and image(s) that (2) evoke a frame of communi- cation, annotating (3) the stance towards the frame of communication expressed by the tweet author. For this purpose, we have considered 113 differ- ent frames of communication related to COVID-19 vaccination, described in Section 2, which were evoked across 11 1,,300 such tweets. The resulting XMMVAX-STNNEE dataset enabled us to experi- ment with several stance detection methods to as- certain bascline performance for this specific task. employ
In our work, we have considered the same 113 frames of communication that were reported in Weinzierl and Harabagiu (2022c), resulting from complex answers to questions that addressed vari- ous aspects of each of the components used in the 7C model (Geiger et al., 2022). The majority of the 113 frames that we considered address more than one of the components listed in Table 1. total of
To further improve the performance, self-training that models the interactions between text and images. We also employ stance detec- tion operating only on text which was trained on a separate, large dataset anmotated with stance in-
httpp:: thub. com http/S/permamma//WWWaa--s--- Stance

---

Component Confidence
Definition
Examples of Frames Communication
Trust in the security and effectiveness of vaccinations, the health authorities, and the health officials who recommend and develop vaccines.
Plizer COVID-I9 vaccine may cause anaphylaxis
in people with polyethylene glycol (PEG) allergy.
G The Government has provided plenty of safety
information about the COVID-15 19 vaceines.
Complacency
Complacency and laziness to get vaccinated due to low penceived risk of infections.
Preterence for getting COVID- 19 and lighting it off than vaccinating.
Constraints
Structural or psychological hurdles that
D It takes courage both to vaccinate against
make vaccination difficult costly.
COVID-19 and to refuse the vaccine.
Calculation
Degree to which personal costs and benefits of vaccination are weighted.
0 COVID-19 vaccines protect against the emerging
variants,
Collective
Willingness to protect others and to
G Vaccination key protecting yourself and others
Rexponsibility
eliminate infectious diseases.
against COVID-19.
Compliance
Support for societal monitoring and sanctioning
People choosing not to get the COVID-19 vaccine
people who are not vaccinated.
should not lose venue access/travel to some countries.
Conspiracy
Conspiracy thinking and belief
COVID-I9 vaccines make you 5G compatible.
fake news related to vaccination.
The COVID vaccine renders pregnancies risky.
Table 1: Components of Vaccination Hesitancy and Examples of Frames of Communication addressing them in MMVAX-STANCHS
57 frames address Confidence in vaccines (53%); 43 frames address Calculation (43%); 37 of the frames address Conspiracy (33%); 28 of the frames address Compliance (23%); 14 frames address Col- lective Responsibility (12%); 11 frames address Complacency (10%) and only 7 frames address the Constraints (6%). Moreover, the large number of frames of communication that we have consid- ered sets apart our multimodal sunce-annotatie effort when compared with other existing stance detection annotated datasets using a much smaller number of targets.
final collection of 1,099,645 multimodal tweets. Data selection. To identify from the collection of 1. 1,099,455 tweets those that are relevant to any of the 113 frames of communications described in Section 2, we relied on multiple retrieval systems which employ dense indices utilizing CLIP (Rad- ford et al., 2021) with FAISS (Johnson al., 2021). CLIP a powerful image-text encoder, which was trained on image-caption pairs to produce embed- dings that are close in distance both the caption and image are aligned semantically. Index lz was constructed using CLIP's image encoder, which produced embeddings of size 768 for each image in the collection of 1, 1,099,645 multimodal tweets. In- dex IT was constructed using CLIP's text encoder, which produced embeddings of size 768 for the text of each tweet. Index L was constructed using both of CLIP's encoders, where the centroid embedding of size 768 was produced from each tweet's 's text embedding and image embedding. Each frame of communication was also embedded using each of the above three approaches, and these frame em- beddings were used to query all three dense indices for relevant tweets by finding the closest embed- dings by distance. The three indexes informed our selection of the data in the following way: The top 100 most relevant tweets were selected across all three indices for each frame of communication, producing total of 1,,300 tweets to be annotated. Table 2 presents the top 15 most common tokens present in these tweets, along with the top 15 most common hashtags, mentioned users, and linked URL domains. Tokenization was performed with spaCy (Honnibal et al., 2020), with 31 + 18 tokens discovered per tweet.
3 The MM Dataset
Data collection. After approval from the Univer- sity of Texas at Dallas Institutional Review Board was obtained, (IRB-21-515 stipulated that our re- search met the criteria for exemption #8(ii)) of Chapter 45 of Federal Regulations Part 46. (b)), we used the Twitter historical API with the fol- lowing query: [(covid OR coronavirus) vaceine lang:en], retrieving 33,566,030 original tweets from December 18th, 2019, to January 1st, 2022 (approx. 2 years). 1,920,923 of these tweets (6%) contained not only text, but also images, thus they are multimodal documents. 75% of multimodal tweets had one image, while 14% had two images, and 1 1 % had three or more images. A large fraction of these multimodal tweets were duplicates, which required filtering. Perceptual Hashing (pHash) (Za- uner, 2010) is a commonly used technique for im- age duplication detection on a massive collection of images. We utilized pHash to remove duplicate images, and their corresponding tweets, resulting in

---

Tokens
vaceine vaccinate vaccinated, COVID/ COVID-19, get getting people, immunity, ty, virus coronavirus, children. still. Pfizer. effective. need. know. MRNA, read, new #covid19 #covid / #covid_19, #vaccine #vaccines / #covidvaccine #vaccination, #ccoonavirus, tpfizer, #pandemic, #mrna, thealth, fastrazeneca, fivaccineswork, Pdeltavariant
Hashtags
Mentions
@who, @cdcgov, 8us_fda, @patus, @pfizer, @realdonaldtrump, @rwmalonend, @nytimes, @bharatbiotech, @cdcdirector, @cdedirrctoor@@oeroggn @joerogan, θcnn, @fda, @ocugen, @randpaul
Domains
cdc.gov. medrxiv.org nebraskamed.. con, goa. gov.uk, health.gov..e saalthhgovuu, youtube. com, nytimes. timesofindia. indiiniiess nhs.uk, uk, nature.com, int, livemint.com. theguardian.com tga. aga.gov.au
Table 2: Top 15 most common tokens, hashtags, mentioned users, and linked URL domains in the tweets from MMVAX-STANCE.
Accept
Reject
No Stance
Not Relevant
Total
Tweets
Train
2,371
1,236
1,857
2,448
7,912
4,882
Dev Test
196
117
333
397
1,043
646
578
332
642
793
2,345
1,390
Total
3,145
1,685
2,832
3,638
11,300
6,918
Table Distribution of stance values and unique tweets for each split in MMVAX-STANCE.
Data annotation. First, researchers from The University of Texas at Dallas judged that 7,662 of the retrieved multimodal tweets evoke the corre- sponding frames of communication. The Cohen's 's Kappa score that we obtained for inter-annotator agreement was 0.82. Each tweet was labeled with stance values by three researchers from The Uni- versity of Texas Dallas. The questionnaire pre- sented to each annotator, detailed in Appendix A, allowed us to annotate each tweet that evoked a frame of communication with one of the stance values: Accept, Reject, or No Stance. The results were: 3,145 tweets which Accept their frame, 1,685 which Reject their frame, and 2,832 with No Stance. The Cohen's Kappa score for stance inter-annotate agreement was 0.69.
STANCE and report the results in Section 5.
Multimodal Models of Stance
4.1 Baseline Models
Several models were used to establish baseline re- sults on MMVAX-STANE pre-trained language models such as Bidirec- tional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) are commonly uti- lized for text-based stance identification (Hossain et al., 2020; Weinzierl al., 2021; Weinzierl and Harabagiu, 2022b; Barbieri et al., 2020). Each tweet's text is provided along with the text of the communication frame, and a softmax layer is added on top of the contextualized "[CLS]" em- bedding to perform stance identification. Improve- ments to text-based stance identification have been found through combining BERT with lexical, emo- tional, and semantic Graph Attention Networks (Veličković al., 2018; Weinzierl et al., 2021) and Moral Foundation (MF) Hopfield pooling (Ram- sauer et al., 2021; Weinzierl and Harabagiu, 2022c), referred to as the LES-GAT-MF system. Text-based stance identification can also benefit from exploit- ing implicit attitude consistency relationships be- tween agreeing and disagreeing tweets (Weinzierl and Harabagiu, 2022a). This approach is referred to as the LACRscore system. Each of the text-based stance identification methods was image-informed by providing textual description of images. We considered two methods: Optical Character Recog- nition (OCR) and image captioning (Caps). OCR
Transformer-base
Benchmark subsets. Tweets were further sep- arated into training, development, and test collec- tions to foster experimental reproducibility, out- lined in Table 3. Care was taken to ensure each col- lection had entirely unique tweets, such that there was no overlap between training, development, and testing. The training collection, which consists of 7,912 multimodal stance judgments, was utilized to train our automatic multimodal stance identifi- cation systems, described in Section 4. The devel- opment collection, which consists of 1,043 multi- modal stance judgments, used to select system hyperparameters. The test collection, which con- sists of 2,345 multimodal stance judgments, was used to evaluate the several stance identification ap- proaches, enabling us to experiment with MMVAX-

---

was performed utilizing the Tesseract OCR engine (Smith, 2007), while image captioning was per- formed by BLIP-2 (Li et al., 2023).
part and the image of the document. Multiple types of relations between texts and images appearing in the context of the same document have been covered in Bateman (2014). For the inference of the stance of the author of a multimedia document towards a commumication frame we have consid- ered whether the text and the image pull in the same direction of the stance, thus they are conver- gent, or divergent. According to the distinction proposed in Kloepfer (1976), convergent relations between texts and images can be further separated as parallel, when the text and the image convey the same stance towards the frame, or additive, when the meaning of one of the modalities adds to the meaning of the other in determining the stance.
CLIP was also considered, which was trained produce aligned image/text embeddings through contrastive learning with text-based encoder and a separate image-based encoder. CL.IP-Text uti- lized the text encoder of CLIP, and therefore did not have access to any image information for each tweet. Both the tweet's text and the communica- tion frame's s text would be encoded through CLIPPs text encoder, with both embeddings concatenated together. Inmge similarly utilized the image encoder of CLIP, and therefore did not have access to any text information for each tweet. CLIP-Joint utilized both the text and the image encoder to pro- duce a CLIP text embedding and a CLIP image embedding for each tweet, which were concate- nated together. A final softmax layer was added to the concatenated embeddings to perform stance identification for each CLIP- method.
We hypothesized that relations between the text and the image of each multimodal tweet can be inferred if we compare the annotated multimodal stance with the stance inferred only from the text. As currently, there are many text-based stance de- tection models that operate quite well, we were able to produce silver annotations of the text- based stance for each example from MMVAX- STANCE. For this purpose, we have used the text- based stance detection reported in Weinzierl and Harabagiu (2022b). This allowed us to establish seven different prototypical examples which com- bine the values of the annotated multimedia stance with the values of the text-based stance annotations. The prototypical examples are illustrated in Fig- ure 3. We found that 1,578 of the examples from the training set of MMMAX-STANCN represent pro- totype 1; 249 examples represent prototype 2; 684 examples represent prototype 3; 249 examples rep- resent prototype 4; 1,052 examples represent pro- totype 5; 303 examples represent prototype 6; and 544 examples represent prototype 7.
Multimodal transformer methods were consid- ered for stance identification, as a fusion of context across both text and image modalities could be nec- essary to fully understand the stance of a tweet. VILT (Kim et al., 2021) is Vision-and--aagguaee Transformer that models interactions between the language text and the visual components in an image using a cross-attention transformer encoder architecture. FLAVA (Singh et al., 2022) is a Foun- dational Language and Vision Alignment Model, which was pre-trained on about times more data than VILT, and also included additional contrastive and unimodal pre-training tasks. BridgeTower (BT) (Xu et al., 2023) bridges the gap between text and image encoders by introducing a cross-modal en- coder that attends to various levels of the text and image encoders, enabling the text encoder to be modeled differently than the image encoder, while the cross- modsl encoder can benefit from the spe- cialization of both encoders. Each multimodal sys- tem encodes the tweet's text, the communication frame, and the tweet's image and produces single contextualized embedding, which is provided to a softmax layer that performs stance identification.
Inference of Text/Image Relations: The pro- totypcall examples inform the possible relations between the text and the image of each multimodal tweet, which in turn help us derive the possible stance of the image.
O In Prototype and Prototype 3, as seen in Fig- ure 3, the multimedia stance and the text-only stance have the same value: Accept for Prototype and Reject for Prototype 3. This indicates that there are two possible relations between the text and the image in each of these prototypes: either the text and the image have a Parallel relation, each con- tributing separately to the value of the multimodal stance; or the text and the image share a Diverging
4.2 Accounting for Text-Image Relations
As we have seen in the example illustrated in Fig- ure 1, the stance of a multimodal document towards a frame of communication depends on the meaning derived from the interaction between the textual

---

Annotated Multimedia Stance
Annotations of Text-Based
Inferred Text/Image Relation
Possible Image Stance
Prototypical Examples
Prototype
Parallel
Accept
Image
Text A/lmage
Accept
Accept
Diverging
Image
Protatype
Reject
Image
Text B/lmage
Reject
Accept
Divergent
Reject
Image B
Prototype
Text C/lmage
Reject
Reject
Parailei
Reject
Image
No Stance
Image
Prototype
Diverging
D/Image
Accept
Reject
Accept
Image
Prototype
Diverging
Accept
Image D
Text E/image
No Stance
No Stance
Parallel
Image
Prototype
Text F/lmage
Reject
No Stance
Additive
Reject
Image
Prototype 7
Additive
Accept
Image G
Text G/Image G
Accept
Figure Inference of text/image relations enables deriving the possible image stance values.
relation, in which the image has a stance of a differ- ent value towards the frame than the text. When Parallel relation is inferred, the possible stance of Image A is only Accept and the possible stance of Image Reject. But when a Diverging relation is inferred, the stance of Image A, or Image C, may either No stance or Reject.
Hence, all images from tweets only having Accept possible stance values, due to Additive Paral- lel relations, were paired with texts from tweets having Accept text-based stance values from Par- allel relations. In the same way, all images having only Reject possible stance values, due to Additive or Parallel relations, were paired with texts from tweets that were predicted to have Reject stance values from Parallel relations.
O In Prototype 2 and Prototype 4, the stance of the multimedia document towards the frame of commu- nication is different from the stance predicted for the text, indicating that the only possible relation between text and image is Diverging. Therefore, the possible stance value for the image is the same as the multimedia stance value.
Generating Synthetic Multimedia Stance Exam- ples: Figure shows that there is prototype for which the text stance is deemed Accept and the relationship between the text and the image of the tweet could Parallel: Prototype 1. There are also two prototypes that have possible image stance value only as Accept: Image D from Proto- type 4, due to a Diverging relationship, and Image G from Prototype 7, due to an Addirive relationship. If we pair all the texts from Prototype with all the images from the examples of Prototype 7, we generate 8,371 synthetic examples with multi- modal stance Accept. Similarly, if we pair all the texts from Prototype with all the images from the examples of Prototype 4, we generate 3,732 synthetic examples with a multimodal stance Ac- cept. In this way, we have created 12,103 synthetic multimodal documents with the multimodal stance Accept, which refer as S Accept. Figure 3 also
Prototype 5, because both the multimedia stance and the text stance have the value No Stance, this enables us to infer that the text and the image share Parallel relation. Therefore, the only possi- ble value for the stance of Image E is No Stance. C In Prototype 6 and Prototype 7, because the stance value of the text is No Stance, the relation between Text F and Image F, or between Text G and Image G, Additive. This entails that Image can have only a stance value of Reject, while Image G can have only a stance value of Accept.
To create large set of synthetic examples, we con- sidered the interplay between Parallel, Additive, and Diverging relations between texts and images.

---

System
Macro
Macro
Maero
Accept
Accept
Accept
Reject
Reject
Reject
P
F1
P
BERT
37.69
37.67
37.88
45.81
44.02
47.75
29.57
31.31
28.01
BERT OCR
37.45
39.84
35.66
45.98
45.94
46.02
28.92
33.73
25.30
BERT Caps
44.07
44.90
43.51
50.55
49.10
52.08
37.60
40.70
34.94
BERT OCR Caps
40.30
39.54
41.49
50.48
46.88
54.67
30.13
32.19
28.31
LES-GAT-MF
39.02
39.67
38.45
46.17
45.75
46.61
31.86
33.60
30.29
LES-GAT-MF OCR
38.40
39.11
37.91
45.38
44.18
46.65
31.43
34.05
29.18
LES-GAT-MF Caps
42.66
42.13
43.22
48.21
46.96
49.53
37.11
37.31
36.92
LES-GAT-ME OCR Caps
42.26
42.10
42.46
48.09
35.39
36.11
34.70
LACRscore
41.74
40.39
43.22
47.21
46.35
48.10
36.27
34,42
38.34
LACRscore + OCR
41.27
39.65
43.06
46.08
44.83
47.41
36,46
34.46
38.70
LACRscore + Caps
44.40
43.34
45.52
50.09
49.27
50.93
38.71
37.41
40.11
LACRscore + OCR Caps
43.96
43.32
44.66
48.81
47.31
50.41
39.11
39.32
38.91
Texp-Text
39.70
39.33
40.25
47.06
44,87
49.47
32.34
33.78
31.02
CLIP-Image Image
CLIP-
28.67
33.57
50.13
53.93
37.14
98.44
3.41
30.00
1.81
CLIP-Joint
46.79
43.42
53.78
56.29
45.60
73.53
37.29
41.24
34.04
VILT
44.37
38.62
52.52
53.52
48.39
59.86
35.21
28.85
45.18
FLAVA
47.63
42.93
53,52
55.76
49.60
63.67
39.51
36,27
43.37
BT
52.91
46.84
60.86
61.74
53.87
72.32
44.09
39.81
49.40
BT SAU! DT
54.46
50.79
58.73
59.78
55,29
65.05
49.15
46,28
52.41
BT SSame
57.22
53.12
62.03
62.30
57.32
68.23
52.14
48.92
55.82
BT Accept
60.65
57.74
63.92
69.27
65.03
74.11
$2.03
50.44
53.72
BT Steject
62.10
59.78
64.70
65.16
61.29
69.55
59.04
58.26
59.85
BT SAu
71.32
71.51
71.16
79.45
78.64
80.28
63.19
64.38
62.05
Table Results from the stance identification experiments on the test collection from MMVAX HIMVAX-STANCE.
shows that there is one prototype for which the text stance Reject is held towards frame of communi- cation, and the relationship between the text and the image of the tweet could be Parallel: Proto- type 3. Image B used in Prototype 2 and Image F used in Prototype 6 both have only the possible stance value Reject, through a Diverging and an Additive relationship, respectively. Therefore, by pairing all the texts from Prototype 3 with all the images from Prototype 2, we obtain 1,875 syn- thetic examples with multimodal stance Reject. Similarly, if we pair all the texts from Prototypes 3 with all the images from the examples of Prototype 6 obtain 2,758 synthetic examples with multi- modal stance Reject. In this way, we have created 4,633 synthetic examples with a multimodal stance Reject, which we refer to as Steject- Finally, if we pair all texts from Prototypes 1, 3, and with all the images from the examples of Prototype 5 we produce 29,870 synthetic examples with un- changing multimodal stance, Image E will have No Stance and not modify the Accept, Reject, and No Stance stance values of the texts respectively. We refer to this collection synthetic, unchanged multimodal stance examples as S total, by inferring the text/image relations and the resulting possible stance of the tweet images, we generated 46,606 synthetic examples SaU for training multi- modal stance detection. Synthetic examples S SAU
were created using the gold annotated training data from MMVAX-STANCE, which we will refer to Dr. BridgeTowee (BT) was selected the mul- timodal architecture train with these synthetic examples, as initial experiments demonstrated it performed best across the multimodal models on stance detection. Hyperparameters for each system are provided in detail in Appendix B.
5 Results and Discussion
To evaluate the quality of stance identification on the test collection from MMVAX-STANCS we use the Precision (P), Recall (R), and F1 metrics for de- tecting the Accept and Reject values of stance. We also compute a Macro averaged Precision, Recall, and Fj score. The evaluation results for MMVAX- STANCE listed in Table 4. The bolded numbers represent the best results obtained.
Among text-based baseline systems, the LACRscore system achieved the highest Macro Fj score, demonstrating the value of incorporat- ing attitude consistency between tweets. Image- informed text-based systems demonstrated a clear pattern: OCR did not improve the F₁ scores of text- based systems, while image captioning increased the Macro F1 scores by to 4 points over the non- raption-informed systems. Image-text aligned sys- tems illustrated a clear performance benefit when incorporating image data directly along with text

---

data for stance identification, with the CLIP-Joint system achieving an improved Macro F₁ score over the best text-based systems. Multimodal mod- els further demonstrated this phenomenon, with VILT, FLAVA, and BT systems matching or exceed- ing all text-based systems. BT achieved remark- able improvements in Macro Fi score, increasing 8.5 points above the best text-based system, and increasing 6.1 points above the CLIP-Joint sys- tem. BT + SaU Dr immediately demonstrates the value of introducing synthetic multimodal ex- amples, as this system was not trained on any of the original training examples from MMVAX- STANCE, only the synthetic examples constructed in Section 4. The performance of BT along with training on each set of synthetic examples is pro- vided to demonstrate how each type of synthetic ample contributes to improved stance detection. BT SSame shows moderate improvements F1 score across both the Accept and Reject stance values by providing more examples to learn from where the image does not contain stance information. BT + Accept demonstrates a significant increase in the Accept Fj score, these synthetic examples pro- vide additional opportunities for BT to identify the Accept stance in both the text and the images in tweet. Similarly, BT Reeject demonstrates significant increase in the Reject F score, which has a large impact on the Macro F1 score, as the Reject stance tends to be more difficult to identify by stance detection systems. Finally, BT + SAll incorporates all synthetic examples, providing sig- nificantly more silver examples for BT to use to identify all stance values. BT Sau achieves the best Macro Fj score of all systems, with score of 71.32.
benefits of the size of the synthetic dataset. We train BridgeTower on various randomly shuffled proportions of the full synthetic dataset SaU, from 10%, corresponding to BT + S10%. to 90%, cor- responding to BT + Sy0% Furthermore, we com- pare these systems with BridgeTower trained on zero synthetic examples, corresponding to BT. and BridgeTower trained on all the synthetic examples, BT SAU- the same systems reported in Table 4. Results of this ablation experiment are provided in Table 5. These results are not surprising, showing that by adding even only 10% the augmented data, the Macro F -scoe increases from 52.91 to 59.73 73. This trend continues until 80% of the aug- mented data is employed, when the Macro Fı-score reaches values of over 70%.
Related Work
Several stance detection datasets using social me- dia postings are available, all of them considering only the text of the postings. Very well-known is the stance detection dataset shared publicly with Se- mEval2016 Task 6 (Mohammad et al., 2016). This dataset consists of 4.163 tweets considered politics- relevant targets (e.g., Hillary Clinton) when an- notating the stance of a tweet's text. A newer and much larger stance-annotanl dataset (approxi- mately 50,000 tweets), introduced in Conforti et al. (2020), used targets focused financial transac- tions that involve mergers and acquisitions. Stance- annotated Twitter datasets covering topics related to the recent COVID-19 pandemic are also cur- rently available. For example, Mutlu et al. (2020) published COVID-CQ, a dataset of approximately 14,000 tweets manually annotated to capture each tweet author's stance regarding the use of hydroxy- chloroquine in the treatment of COVID-19 patients. A dataset of 1,629 tweets annotated with stance was also introduced in Miao et al. (2020), targeting lockdown regulations in New York City. In Glandt et al. (2021) a stance-annotated dataset of 7,122 tweets focused on targets related to the pandemic (e.g. "Wearing a Face Mask"). The COVIDLIES dataset (Hossain et al., 2020) of 6,761 stance- annotated tweets targeted 86 common misconcep- tions about COVID-19, Additionally, the COV- AXLIES dataset (Weinzierl and Harabagiu, 2022a) is stance-annotated dataset of 7,152 tweets focus- ing on 47 frames of misinformation targeting the COVID-19 vaccines, which was expanded into the COVAXFRAMES dataset (Weinzierl and Harabagiu,
System
Synthetic Size
Macro
BT
0
52.91
BT Sux%
4,660
59,73
BT S20%
9.321
64.10
BT Sau%
13,981
65.56
BT 840%
18,642
67.90
BT So0%
23,303
68.78
BT S60%
27,963
69.30
BT + S70%
32,624
69.64
BT + S80%
37,284
70.42
BT S00%
41.945
70.42
BT + SAN
46,606
71.32
Table 5: Stance detection ablation experiments with BridgeTTower on the test collection from MMVAX- STANCE over the size of the synthetic dataset.
We also perform an ablation experiment over the

---

2022c), containing 14,180 stance-annotated tweets focusing on 113 COVID-19 vaccine hesitancy fram- ings. All these datasets ignored the presence of images in tweets when judging the stance values. Although the multimedia stance annotations pro- vided by COMAAX-STANEE consider only 11,300 tweets, the synthetic examples that we have gener- ated provide significantly more examples.
et al., 2022). Baselines were established using sev- eral text-based and multimodal stance detection systems. Substantial interactions between the text and the image(s) of a tweet were found to be key when inferring the tweet author's stance towards frame of communication. The inference of the text/image relations led to the generation of a large set of synthetic examples of multimodal tweets, annotated with stance values. Baseline methods were greatly improved when the synthetic exam- ples were employed, indicating the role of massive sets of examples and relations between text and im- ages when learning to identify the values of stance in multimodal documents.
Previous stance identification on Twitter was cast either as (1) a classification problem, learning to predict the stance value of a tweet towards a given target; or (2) an inference problem, when tweet may entail, contradict, or does not imply the target. Classification methods: Neural architectures, rely- ing on RNNs, CNNs, and/or attention mechanisms, dominate these methods (Augenstein et al., 2016; Du al., 2017; Sun et al., 2018; Siddiqua et al., 2019). BERT was later shown to be the best model overall for stance detection on the SemEval2016 Task 6 (Ghosh et al., 2019). On the COVIDLIES dataset, Weinzierl et al. (2021) demonstrated that BERT can be further improved when using Graphic Attention Networks (Veličković et al., 2018). Fur- ther improvements were reported in Weinzierl and Harabagiu (2022a) when operating on Co- VAXLIES, by considering the implementation of pragmatics propertics of stance into knowledge em- beddings.
Limitations
Stance detection methods utilized in this paper fo- cus on social media posts from Twitter. There- fore, these methods may not work as well on posts found on other platforms, especially those allowing longer content, such as Reddit. We plan to extend the methods presented in this paper to additional social media platforms in future work.
Additionally, a central assumption of our ap- proach is the idea that relationships exist between the text of post and the image(s) in the post. We discovered these relationships utilizing a combi- nation of annotations and text-based stance pre- dictions, but either of these stance values could be incorrect due to mislabeling or mistaken pre- dictions, leading to incorrectly inferred relations. These cases certainly exist, as some of our anno- tated multimodal stance values contradicted the predicted text stance values. We had 526 instances of posts with an annotated multimodal stance of No Stance with a predicted text stance of Accept, and 279 instances of the predicted text stance of Reject with an annotated multimodal stance of No Stance. These cases represent 15% of the training dataset of which aligns with expectations from the combined annotator disagreement and the accuracy of the text-based stance detector.
Inference Methods: When the COVIDLIES dataset of stance annotations was released in Hossain et al. (2020), stance identification was presented as natural language inference problem, which can benefit from existing textual inference datasets. Bidirectional LSTM encoders and Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) were trained on three common NLT datasets: SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and MedNLI (Williams et al., 2018), helping to identify the stance of tweets. However, classifica- tion methods were able to obtain superior results on stance detection. Nevertheless, none of these meth- ods consider the information conveyed by images present in tweets for stance detection, which our experiments on MMVAX- -TANCCE demonstrate.
7 Conclusion
Finally, we focus heavily the interactions be- tween parallel and additive relations between the text and the image(s) of a post. Further work could expand on finding ways to also utilize the diverging relations, and their interactions with parallel and additive relations.
In this paper we introduce MMVAX-STANCE the first multimodal dataset of stance-annotated tweets towards 1 13 frames of communication. These frames communicate various aspects of COVID- 19 vaccination, informed by the 7C model (Geiger

---

9 Ethics Statement
Dennis Chong and James N. Druckman. 2007. Framing public opinion in competitive democracies. Ameri- can Political Science Review, (01 -655.
We respected the privacy and honored the confiden- tiality of the users that have produced the tweets pertaining to RIMAAX-STANCE We received ap- proval from the Institutional Review Board at The University of Texas at Dallas for building and work- ing with this Twitter social media dataset. IRB-21- 515 stipulated that our research met the criteria for exemption #8(ii) of Chapter 45 of Federal Regula- tions Part 46..16..... (0).)..... Experiments were performed with high professional standards, avoiding evalua- tion on the test collection until final model was selected from development performance. All ex- perimental settings, configurations, and procedures were clearly laid out in this work, the supplemental material, and the linked GitHub repository. The public good was the central concern during all en- closed research, with a primary goal of benefiting both public health and natural language processing research.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won I-they: very large dataset for stance detection on Twitter. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1715- 1724, Online. Association for Computational Linguis- tics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with sarget-specific neural at- tention. In Proceedings of the Twenty-Sixah Inter- national Joint Conference on Artificial Intelligence, IJCAI-17, pages
References
Robert M. Entman. 1993. Framing:
Toward clarification of fractured paradigm. Journal of Communication, 43-4)-51-58
Isabelle Augenstein, Tim Rocktäschel, Andreas Vla- chos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885, Austin, Texas, Association for Computational Lin- guistics.
Robert M. Entman. 2003. Cascading activation: Con- testing the white house's frame after 9/11. Political Communication, 20( 20(4):415-432
Mattis Geiger, Franziska Rees, Lau Lilleholt, Ana P. Santana, Ingo Zettler, Oliver Wilhelm, Cornelia Betsch, and Robert Böhm. 2022. Measuring the 7Cs of vaccination readiness. European Journal of Psy- chological Assessment, 2814)2261-26.
Francesco Barbieri, Jose Gamacho-Colladoo Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online. Association for Computational Linguistics.
Shalmoli Ghosh, Prajwal Singhania. Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: A comparative study. In Experimental IR Meets Multilinguality. Multimodality, and Interaction, pages 77, Cham. Springer International Publishing.
John A. Bateman. 2014, Text and Image: A Critical Introduction to the Visual/Verbal Divide. Taylor and Francis EBA. Routledge, New York.
Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021. Stance detec- tion in COVID-19 tweets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1596-161 I, Online. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Momchil Hardalov, Amnv Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey stance de- tection for mis- and disinformation identification. In Findings of the Association for Computational Lin- guisties: NAACL 2022, pages 1259-1277, Seattle, United States. Association for Computational Lin- guistics.
Dallas Card, Justin Gross, Amber Boydstun, and Noah Smith. 2016. Analyzing framing through the casts of characters in the news. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 1410-1420, Austin, Texas. Association for Computational Linguistics.

---

Mareike Hartmann, Tallulah Jansen, Isabelle Augen- stein, and Anders Søgaard. 2019. Issue framing in online discussion fora. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1401-1407, Minneapolis, Minnesota. Association for Computational Linguistics.
Ece C. Mutiu, Toktam Oghaz, Jasser Jasser, Ege Tu- tunculer, Amirarsalan Rajabi, Aida Tayebi, Ozlem Ozmen, and Ivan Garibay. 2020. A stance data set polarized conversations on twitter about the efficacy of hydroxychloroquine as a treatment for covid-19. Data in Brief, 33:1 104401
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy; Industrial- strength Natural Language Processing in Python.
Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for 19 (Part 2) al EMNLP 2020, Online. Association for Computa- tional Linguistics.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Scidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brand- stetter, and Sepp Hochreiter. 2021. Hopfield net- works is all you need.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(03):535-547.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-amd-languase transformer without convolu- tion region supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594 PMLR.
Rolf Kloepfer. 1976. Komplementaritää von sprache und bild am beispiel von comic, karikatur und reklame. Sprache in Technischen Zeitalter Statgartt, 57:42-56.
Umme Aymun Siddiqua, Abu Nowshed Chy, and Masaki Aono. 2019. Tweet stance detection using attention based neural ensemble model. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long and Short Papers), pages 1868-1873, Min- neapolis, Minnesola. Association for Computational Linguistics.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping anguage-image pre- training with frozen image encoders and large lan- guage models.
Julia Mendelsohn, Ceren Budak, and David Jurgens. 2021. Modeling framing immigration discourse on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2219-2263, Online. Association for Computational Linguistics.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A foun- dational language and vision alignment model. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15617-15629.
Lin Miao, Mark Last, and Marina Litvak. 2020. Twitter data augmentation for monitoring public opinion on COVID-19 intervention measures. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computa- tional Linguistics.
R. Smith. 2007. An overview of the tesseract ocr engine. In Ninth International Conference on Document Anal- ysis and Recognition (ICDAR 2007), volume 2, pages 629-633.
Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierar- chical attention network. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 2399-2409 Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) pages 31- 41, San Diego California. Association for Computa- tional Linguistics.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.

---

2018. Graph attention networks. In International Conference on Learning Representations.
System
Learming
Batch Size
Epochs
Rate
BERT
le-4
32
10
Maxwell Weinzierl and Sanda Harabagiu. 2022a. Identi- fying the adoption rejection of misinformation tar- geting covid-19 vaccines in twitter discourse. Pro- ceedings of the ACM Web Conference 2022, WWW '22, page 3196-3205, New York, NY, USA. Associa- tion for Computing Machinery.
LES- DES-GAT-MF
Se-4
32
10
LACRscore
le-4
32
36
Cext-Text
3e-5
32
10
Image
3e-5
32
10
CLIP--oint
3e-5
32
10
VILT
3e-5
32
10
FLAVA
3e-5
16
100
Maxwell Weinzierl and Sanda Harabagiu. 2022b. Vac- cineLies: A natural language resource for learning to recognize misinformation about the COVID-19 and HPV vaccines. Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 6967-6975, Marseille, France. European Language Resources Association.
BT
3e-5
32
10
BT + SAN - DD
3e-5
64
BT + Ssame
3e-5
64
BT + SAccepe
3c-5
64
BT + SRejent
3e-5
64
BT SAR
36-5
64
Table 6: Hyperparameters for stance detection systems on MMVAX-STANCE
Maxwell Weinzierl, Suellen Hopfer, and Sanda M. Harabagiu. 2021. Misinformation adoption or re- jection in the era of covid-19. Proceedings the International AAAI Conference on Web and Social Media, 15(1):787-795.
is expressed by the text. Do follow links in the tweet, as the stance should be determined by the text and the image of the tweet.
Maxwell A. Weinzierl and Sanda M. Harabagiu. 2022c. From hesitaney framings to vaccine hesitancy pro- files: journey stance, ontological commitments and moral foundations. Proceedings of the Interna- rional AAAI Conference Web and Social Media, 16(1):1087-1097
Accept The user accepts, agrees with, and/or propagates the provided frame. Reject The user rejects, disagrees with, rebuts, debunks, demonstrates the opposite stance towards the frame. No Stance The user does not demon- strate acceptance or rejection towards the frame, but the tweet is relevant to or is about the frame.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Not Relevant The user's tweet has nothing to do with the frame, and was incorrectly predicted to be relevant by retrieval system.
Xiao Xu, Chenfei Wu, Shachar Rosenman, Va- sudev Lal. Wanxiang Che, and Nan Duan. 2023. Bridgetower: Building bridges between encoders in vision- languae representation learning. Proceed- ings of the AAAI Conference on Artificial Intelligence, 37(9): 10677-16777
Annotators were next provided a series of exam- ples of each stance value towards a frame.
Each example to be annotated was provided to the annotators in the following format:
Christoph Zauner. 2010. Implementation and bench- marking of perceptual image hash functions. Mas- ter's thesis, Upper Austria University of Applied Sei- ences, Hagenberg Campus.
Frame Text: frame
A Annotation Questionnaire
Tweet Text:
Annotators were first presented with the following set of instructions:
text
image
Choose the stance value that is expressed by the tweet towards the frame text. Consider both the text of the tweet along with the image. If the text and the image are conflict, choose the stance that
Annotators were then asked to select one of the four possible stance values: Accept, Reject, No Stance, or Not Relevant.

---

B
Training Details
Primary hyperparameters for each stance detection method are provided in Table All systems uti- lize the same learning rate schedule, where the learning rate starts at the maximum learning rate provided in Table 6, after which the learning rate is decayed 0 over the remainder of the training steps. Any additional hyperparameters follow train- ing hyperparameters selected in prior work, such as the size of the TransMS knowledge embeddings (8) or the margin used (4.0) by the LACRscore sys- tems. We al always attempted to train with the largest batch size (as power of 2) which could fit on our Nvidia Titan V GPU. Hyperparameter decisions were entirely guided by maximizing F1 score the development collections of MM VAX-STANCE, and only 5 10 experiments with different hyper- parameters were performed before the best model configurations were selected to be evaluated on the test collection. Any additional details and code can be found in the linked GitHub repository.


Reviews:
[{"review_id": "QdTkGNTDfC", "reviewer": "Reviewer_anjp", "summary": "", "questions": "A: See \"Reasons To Reject 2\".", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the annotating difficulty behind the multimodal stance dataset proposed. Addressing concerns and questions given for rejecting the paper:\n\n1.a. We searched the ACL anthology as well as AAAI/IJCAI papers and could not find any paper tackling multi-modal stance detection on social media. There may be many other multimodal datasets, but they are used to process other tasks – and not stance detection. If the reviewer has knowledge of such datasets, providing a reference would help us to correct our statement and link to the reference. Otherwise, we will have no reason to not continue claiming that to our knowledge this task and dataset is novel.\n\n1.b. The new “object”, referred by the reviewer, is defined as frame of communication because such frames are well-known in the Theory of communication. Lines 028-030 of the paper – which correspond to the first sentence of the introduction, define these frames  – and cite an influential paper that introduced them. Frames of communication are important because they promote a particular interpretation and moral evaluation of problems promote a particular problem definition, causal interpretation, moral evaluation and/or treatment evaluation.  Moreover, they are ubiquitous in natural language communications, (As we elaborate in lines 037-041) therefore they merit attention from NLP research.  Previously, papers focusing on frames of communication have been accepted to ACL/EMNLP (search on aclweb anthology proves it). We feel that by not providing such a definition and discussing some undefined “objects” would make the paper ill-defined and confusing.\n\n2.While there are 9 possible combinations of [Multimodal_Stance_VALUE, Text_Stance_VALUE] only 7 of these combinations bring any value to the data augmentation that we have performed.  The two cases which we ignored are: (Case 1) Multimodal_Stance_VALUE= “No Stance” and Text_Stance_VALUE= “Accept”; and (Case 2) Multimodal_Stance_VALUE= “No Stance” and Text_Stance_VALUE= “Reject”. Multimodal_Stance_VALUE is decided by a human judge, whereas the  Text_Stance_VALUE is decided by an automatic stance detection system. Since we always trust the human judge, which decided the stance based on both the text and the image of the tweet, we infer that in both Case 1 and Case 2, the automatic stance detection was incorrect. Therefore, we did not consider any cases when we knew that the text stance detection was incorrect. Only 15% of all multimedia tweets pertained to Case 1 or 2.   \n\nThank you very much for reviewing our work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work is well motivated.\n2. The proposed dataset has a solid theoretical background.\n3. Due to the annotating difficulty, the work of this paper is abundant.", "reasons_to_reject": "1. The task of multi-modal stance is not novel and there are some existing relevant datasets. There is no need to define a new object \"Frame of Communication\".\n2. There are totally 9 possible combinations of multi-modal annotations and text-based silver annotations. Are \"No Stance (multi-modal anno.) - Accept (silver anno.)\" and \"No Stance (multi-modal anno.) - Reject (silver anno.)\" not prototypical enough? Or do they just not exist at all?", "questions_to_authors": "A: See \"Reasons To Reject 2\".", "paper_topic_and_main_contributions": "The main contribution of this work is proposing a new multi-modal stance detection dataset, namely MMVAX-STANCE. Besides, how multi-modal information impacts the multi-modal stance detection is investigated.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FHi48Sz39T", "reviewer": "Reviewer_GuKD", "summary": "", "questions": "A. Could you explore the influence of synthetic data size (e.g., 10%, 40%, 70%) on the results? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer recognizing that a multimedia stance detection dataset could promote the development of related research. Addressing concerns and questions given for rejecting the paper:\n\n1. Lines 184-192 describe how we performed de-duplication, therefore there are no duplicates in the dataset. When selecting tweets relevant to a frame of communication, we also took into account the fact that a tweet may evoke more than one frame, therefore the tweets may be relevant to more than one frame. Figure 2 illustrates the property of frames, which cannot be ignored. However, we made sure that if a tweet evokes more than one frame, it should be used either only in the training dataset of all the frames it evokes, or it should be used only in the test dataset of those frames, or in the development dataset of all the frames. Therefore, no duplicates exist in evaluations. We would be glad to report the number of unique tweets, in contrast to tweet judgements, in each split: 4,882 in training, 646 in development, and 1,390 in testing.\n\n2. As shown in Table 3, we have considered 18 simpler methods, which were applied on our dataset. The results of these simpler methods did not obtain better scores than 52.91 F1-score. By using the data augmentation methods that we present in this paper as well as a text-based stance detection we were able to obtain an augmentation of the values of the F1-score by nearly 20 points, which we feel is significant. Although the data augmentation method may seem complicated, it makes it possible to obtain results for multimodal stance detection that are comparable with text-only stance detection results. We have also experimented with self-training and the results were poor, therefore we did not feel that they are worth reporting. In lines 330-344 we mention relations between texts and images in general citing two books that detail them, which all multimodal data exhibits– there is nothing specially designed about them.\n\n3. We agree that an ablation study over the synthetic data size would be insightful, and we would be glad to add these results to the paper:\n\n0%: 52.91 F1 (BridgeTower baseline)\n\n10%: 59.73 F1\n\n20%: 64.10 F1\n\n30%: 65.56 F1\n\n40%: 67.90 F1\n\n50%: 68.78 F1\n\n60%: 69.30 F1\n\n70%: 69.64 F1\n\n80%: 70.42 F1\n\n90%: 70.29 F1\n\n100%: 71.32 F1 (BT-ALL)\n\nThese results are not surprising, showing that by adding even only 10% of the augmented data, the F1-score increases from 52.91 to 59.73. This trend continues until 80% of the augmented data is used, when the F1-score reaches values of over 70%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Previous work only relies on text to detect the stance. The authors collect a multimedia stance detection dataset towards 113 frames of communication about various aspects of COVID-19 vaccination, which might promote the development of related research.\n2. Many unimodal and multimodal models are utilized to perform stance prediction. Experimental results on the constructed dataset reveal the effectiveness of the combination of texts and images.\n", "reasons_to_reject": "1. The authors select the tweets which are relevant to the frame of communication when collecting the data. However, this might result in a lack of diversity in the collected data. Besides, they don’t conduct the deduplication for the retrieved similar data. Actually, the same multimodal tweets might be contained in the retrieved data.\n \n2. The method based on text-image relations for generating synthetic multimedia stance data is specially designed and somewhat complex. It might be better if you could use other generic and simple methods (e.g., self-training) to synthesize data.\n", "questions_to_authors": "A. Could you explore the influence of synthetic data size (e.g., 10%, 40%, 70%) on the results? ", "paper_topic_and_main_contributions": "This paper addresses detecting multimodal stance towards frames of communication. Previous work most predicts the stance from the text. However, social media users tend to combine images and texts to convey their stances. Therefore, the authors consider 113 frames of communication and use the frames to retrieve relevant multimodal tweets for constructing the first COVID-19 multimodal stance detection dataset. Besides, authors also utilize text/image relations to generate synthetic multimedia examples. Experimental results on the dataset show the effectiveness of the designed method for synthesizing data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l8cS2nS663", "reviewer": "Reviewer_Sfdh", "summary": "", "questions": "1. Why count \"Not Relevant\" tweets (according to Appendix A) into the total number of the training/development/test collection? It makes the quantitative relations in Table 2 confusing (for example, 3,145 + 1,685 + 2,832 = 7,662 ≠ 11,300).\n\n2. Since it is necessary to \"consider both the text of the tweet along with the image\" when annotating the multimodal stances according to Appendix A, then why not annotate the text-based and image-based stances at the same time instead of using the model-predicted text-based stances when generating synthetic samples? It costs nearly no more effort to do this and if this is done, all the incorrectly predicted relations (such as contradictory prototypes 2 and 4) could be avoided.\n", "limitations": "", "rating": 1, "confidence": 4, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Addressing concerns and questions given for rejecting the paper:\n\n1. Regarding the need to clarify the concept of “communication frames” – we start the introduction with a definition of these frames (lines 28-30) and a citation of an influential paper that is discussing them. Then in lines 037-041 we state why they are important- as they help people understand issues and more importantly – form their opinions! Therefore, the direct application is inferring attitudes and understanding the causes of those attitudes. In the lines 074-080 we state clearly that it is a new task – stance detection towards frames – and we explain its advantages when compared with stance towards a target, i.e. “the stance is judged against explicit aspects of the frame, not against some unspecified properties of a target.\" The direct advantage of using stance towards frames is in the clarity of the judgement – when deciding the stance towards a target, it is not clear what exactly about that target makes a judge agree or disagree with the target.\n\n2. We present an empirical method that is based on data augmentation obtained by (1) considering the possible relations between texts and images and (2) the results of text-only stance detection.\n\nThis data augmentation is new. Even if this method is simple, it proved to be performing very well. When state-of-the-art multimodal processing techniques were used (e.g. BridgeTower) their results alone were not very impressive, whereas when the augmented data was used with BridgeTower – a significant increase in the quality of results was obtained (see Table 3).\n\n3. As intuitive as this approach seems after reading our paper, this approach is entirely novel and was not obvious to others in the field. We feel that the fact that it leads to a significant increase in performance over state-of-the-art methods (BridgeTower) justifies the need to report the methodology, as it may inspire others in the field. Moreover, performing multimodal stance detection with the same level of performance as text-only stance detection (over .70 F1-score) is not trivial.  \n\n4. Without considering the text-image relations, the data augmentation would not be possible. Therefore, we would only have the current state-of-the-art results generated by BridgeTower. Our simple, empirical method provides new state-of-the-art results. All experimental results listed in Table 3 support this claim.\n\nAnswering the questions:\n\n1. We count “Not Relevant” tweets in the total size of the collection because some tweets do not evoke any frame of communication, therefore they are not considered relevant to any frame. These evocation judgements were made by researchers with expertise in the Theory of Communication and we believe that these judgments should not be discarded, as they reflect the fact that people do not form attitudes based on the content of each tweet, but rather based on the content of some tweets, which happen to evoke frames. Regarding Table 2, we would be glad to include an additional column for the number of “Not Relevant” judgements in each dataset split such that these numbers are less confusing to readers.\n\n2. There are several reasons that justify the annotation of the entire tweet’s stance – considering both the text and the image. First, as shown in Figure 1, tweet authors do not decide to use both images and texts by accident. When making that decision, they rely on  the possible relations between text and images to convey their message. This motivates our decision to ask a human judge to infer the stance of an entire tweet towards a frame, and not only of the text or image part. Secondly, while text-based stance detection was studied before, image-only stance detection suffers from the quality limitations of image processing – highlighted in the Table 3 when using CLIP-Image. Third, asking annotators to make stance judgements for the entire tweets (considering both the text and the image) and then also asking them to make stance judgements based only on the text, and finally make judgements only based on the image will cost three times more than the cost of asking them to make stance judgements based on all content of each tweet. For this annotation to be fair, annotators would need to independently judge these stance values in isolation, otherwise the order of presenting the text and image would bias annotation efforts. Moreover, when we designed the annotation process, it was not obvious that the stance values of the text and image in isolation would lead to improved results of multi-modal stance detection. Finally, by having gold-standard annotations of the text-only stance values and of the image-only stance values, the data augmentation technique would be the same. The text-based detection method might have some slight improvements, because it is fine-tuned to the gold -data, which would also lead to some slight improvements on the multi-modal stance detection. But it will not be of the same impact that we have seen from current SOTA results (with BridgeTower) to the results of using this data -augmentation on silver data.\n\n3. Prototypes 2 and 4 are not incorrectly predicted relations, as we clearly outline in Section 4.2. Diverging relations (covered by these 2 prototypes) are a key type of relation between texts and images, which exploit to produce synthetic examples.\n\nAdditionally, we would like to ask what the reviewer felt was ethically dubious about our work, as they have marked “Ethical Concerns” as “Yes.” We clearly state that we received approval from our Institution Review Board (IRB). Please clarify your ethical concerns.\n\nFinally, we would like to question the reviewer’s issue with reproducibility: we have provided an anonymized code repository along with a commitment to release annotations. Why does the reviewer feel they would never be able to reproduce our results? Please clarify your reproducibility concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper annotates a multimodal dataset for the stance detection task towards communication frame .", "reasons_to_reject": "1. The paper does not clarify the concept, application scenarios, and related works of \"communication frames\" clearly. Is stance detection towards communication frames a new task? Is there any advantage of detecting the stance towards a specific communication frame rather than a traditional target? These questions should be answered in the paper.\n\n2. The paper only evaluates some existing baselines and does not propose any new approach for the stance detection task.\n\n3. The proposed data augmentation approach is intuitive, but not very novel.\n\n4. Is it necessary to present Text-Image Relations? Are there some experiments to support your claim?\n\n5. This paper poses a great reading challenge. Writing needs to be greatly improved.\n", "questions_to_authors": "1. Why count \"Not Relevant\" tweets (according to Appendix A) into the total number of the training/development/test collection? It makes the quantitative relations in Table 2 confusing (for example, 3,145 + 1,685 + 2,832 = 7,662 ≠ 11,300).\n\n2. Since it is necessary to \"consider both the text of the tweet along with the image\" when annotating the multimodal stances according to Appendix A, then why not annotate the text-based and image-based stances at the same time instead of using the model-predicted text-based stances when generating synthetic samples? It costs nearly no more effort to do this and if this is done, all the incorrectly predicted relations (such as contradictory prototypes 2 and 4) could be avoided.\n", "paper_topic_and_main_contributions": "The paper first introduces a new dataset of 11,300 multimodal tweets with stance annotations towards 113 different frames of communication related to COVID-19 vaccination. On this basis, the paper evaluates some baseline models on the dataset and proposes a data augmentation approach that generates 46,606 more synthetic training samples.", "excitement": 1, "reproducibility": 1, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes multimodal stance classification and provides a new dataset. The dataset collection procedure is well explained. However, it could benefit from more details about the annotators: who were they? How long does it take to label? How much were they paid? I would suggest statistics about the dataset's language content: #tokens, #words, etc. The proposed method is reasonable, and the experimentation is exhaustive. Listed issues have been addressed and defended in rebuttal comments, and there is openness to clarify different aspects of the final version of the manuscript. The dataset might be of interest to the community, and it would be available upon publication.

Pros
* Novel dataset and stance-classification-related task
* Exhaustive evaluation and promising results

Cons
* Missing minor dataset information about annotators and language content
Decision: Accept-Main