Ideology Takes Multiple Looks: A High-Quality Dataset for Multifaceted Ideology Detection
Authors: Songtao Liu
Ziling Luo
Minghua Xu
LiXiao Wei
Ziyao Wei
Han Yu
Wei Xiang
Bang Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Ideology detection
multifaceted ideology schema
dataset
political spectrum
TL;DR: A new schema, dataset and benchmarks for a new multifaceted ideology detection task.

Abstract: Ideology detection (ID) is important for gaining insights about peoples’ opinions and stances on our world and society, which can find many applications in politics, economics and social sciences. It is not uncommon that a piece of text can contain descriptions of various issues. It is also widely accepted that a person can take different ideological stances in different facets. However, existing datasets for the ID task only label a text as ideologically left- or right-leaning as a whole, regardless whether the text containing one or more different issues. Moreover, most prior work annotates texts from data resources with known ideological bias through distant supervision approaches, which may result in many false labels. With some theoretical help from social sciences, this work first designs an ideological schema containing five domains and twelve facets for a new multifaceted ideology detection (MID) task to provide a more complete and delicate description of ideology. We construct a MITweet dataset for the MID task, which contains 12,594 English Twitter posts, each annotated with a Relevance and an Ideology label for all twelve facets. We also design and test a few of strong baselines for the MID task under in-topic and cross-topic settings, which can serve as benchmarks for further research.

Ideology Takes Multiple Looks: A High-Quality Dataset for Multifaceted Ideology Detection
Songtao Liu", Ziling Minghua Xu Lixiao Wei', Ziyao Wei", Han Yu', Wei Xiang', Bang Wangl. "School of Electronic Information and Communications, Huazhong University of Science and Technology. Wuhan, China "School of Journalism and Information Communication, Huazhong University of Science and Technology, Wuhan, China
{ (liusongtao, luoziling, xuminghua, wangbang) edu. cn
Abstract
Amnolated Manually
Mulúi
Authors
Source
Size
faceted
Ideology detection (ID) important for gaining insights about peoples" opinions and stances on our world and society, which can find many applications in politics, economics and social sciences. It is not uncommon that a piece of text can contain descriptions of various issues. It is also widely accepted that person can take different ideological stances different facets. However, existing datasets for the ID task only label a text as idcologically left- or right-leaning as whole, regardless whether the text containing sues. Moreover, most prior work annotates texts from data resources with known ideologi- cal bias through distant supervision approaches, which may result in many false labels. With some theoretical help from social sciences, this work first designs an ideological schema con- taining five domains and twelve facets for multifaceted ideology detection (MID) task to provide a more complete and delicate de- scription of ideology. We construct a MITweet dataset for the MID task, which contains 12,594 English Twitter posts, each annotated with Relevance and an Ideology label for all twelve facets. We also design and test few of strong baselines for the MID task under in-topic and cross-topic settings, which marks for further research.
lyyer
Books
X
Preotine Pietro et al. (2017)
Twitter
4,833,133
x
Kulkami ad. (2018)
News articles
120,000
X
X
Kiesel al. (2019)
News articles
1,273
X
Kiesel al. (2019)
News articless
754,000
X
X
Baly et all (2020)
News articlles
34,737
X
Liu (2022)
News articles
2,331,552
X
X
Garefa-Díar al.
Twitter
37,560
(2022)
MITWBET
Twitter
12,594
different is-
Table 1: Comparison of ideology detection datasets.
al., 2018; Schwarz, 2019). Ideology detection is critical task for quantitative political and so- cial science (Wilkerson and Casas, 2017; Liu et al., 2022), which can help policy makers to analyze public opinions for making wise decisions (Xiao et al., 2020). Tracking ideology on social media is also important for monitoring online communities and detecting signs of ideological radicalization (Grover and Mark. 2019; Aldera al., 2021).
It is widely accepted that individuals or groups can take different idcological stances in different facets (Boyd and Jackson, 1967; Feldman and Huddy, 2014). Motivated by this, many efforts have been devoted to characterizing ideology from various facets (Ferguson, 1952; Rokeach, 1973; Grunow et al., 2018). From computational lin- guisties viewpoint, some texts may contain descrip- tions of different issues and reflect the author's ideology from various aspects. As shown in Fig- ure ((b), the authors convey their multiple ideolo- gies by expressing opinions on topics they care about. Therefore, it is necessary to detect the ideol- ogy of texts from multiple facets, as provide more comprehensive and nuanced picture for fur- ther sociological research.
bench-
1 Introduction
Ideologies are the collection of beliefs or philoso- phies attributed to a particular social group or class (Eagleton, 2007). They shape how we see the world and interact with each other. In this infor- mation age, social media has rapidly developed, allowing people to express their thoughts and opin- ions online, which offer strong cues about their ideologies (Preojiuc-Pietrr
al., 2017: Kulkarni
Equal contribution: S. Liu and Z. Luo. Wang 'Datasct and codes are available on sttps://githhr com/LST Con/LST1836/MB 336//ITweet
Corresponding author:
In order to facilitate progress on idcology de- tection, a number of annotated datasets have been

---

[Topic: Abortion]
Rigln
Regine
believe both bodily aulcnomy and the sanctity of life. In some situations, abortion is necessary, Being pro-life requires being chocce. Overturning Roe Wade one small vote for group of mostly men- and one giant setback for human rights.
Grade
Society
Pollcics
Merddggrstt
Recial Bereeagen
CLude
Uniemtation
Nerket
Social Devilpement Personal Right
E Economy
Chanryy
[Tople: Russo-Ukrainis War]
Equeliy
History books will have a wonderfully detailed account nearly every minute leading to Russia's invasion of U'raine. And a student will ask the teacher one day: if we knew all this for long such detail, why did we do little more than take notes for history?
Diplomacy
Cam
Strwiegs
Culture
Filled
Prrvess
Callora
Charcil- State
V'anar
Medations
tletis Miltary Fooce (Right) Social Divelopreent
Matiifaened labet Dipkeratic
(a) Our proposed multifaceted ideology schema.
(b) Examples from our MiTweet dataset.
Figure I: (a) An illustration of our proposed multifaceted ideology schema. Five fans represent the five domains and twelve petals represent the facets under domains. The inside and outside each petal denote the left-leaning and
ideologies of that facet respectively. (b) Examples from our MITweet dataset. Pentagrams are used to mark the multifaceted ideology label. Colored petals represent the facets that the text involves.
released, as shown in Table 1. However, two draw- backs of existing datasets limit the research of this task: (1) Only one facet. Due to the absence of the- oretical guidance from the multifaceted ideology. existing datasets only label a text as ideologically left- or right-lcaning as a whole, regardless whether the text containing one or more different issues. For example, the upper text in Figure 1(b) can be la- beled with left-leaning due to its radical attitude. However, this simple label conceals the domains discussed in the text (i.e., culture and society), and ignores that the author supports the protection of individual rights, which is conservative or right- leaning in sociology. (2) Noisy labels. Most of pre- vious work crawl data from resources with known ideological bias and project the bias label to all texts gathered from it, which is form of distant supervision. As discussed in Baly et al. (2020), while the distant supervision assumption generally holds, there are still many instances that defy it, which introduces noise into datasets.
of twelve facets. We also define the ideological attributes for the left- and right-lcaning of each facet to achieve a more detailed illustration of ide- ology. Based on the new schema, we construct a new high-quality dataset, MITweet, for a new multifaceted ideology detection (MID) task. The dataset contains 12,594 English Twitter posts, each manually annotated with a Relevance (and an Ide- ology) label for all twelve facets. As the examples in Figure 1(b), MITweet makes up the absence of comprehensive annotation by annotating ideo- logical labels for multiple facets involved in texts. In addition, MITweet covers 14 highly controver- sial topics in recent years (e.g., abortion, 19 and Kusso-Ukrainie war), which can facilitate re- search in cross-topic MID.
Using MITweet as the benchmark, we develop strong baselines for the new MID task based on several widely-used pre-trained language models. We conduct thorough experiments and analysis un- der both in-topic and cross-topic settings. Exper- iment results show that: (1) BERTweet (Nguyen et al., 2020) performs the best on overall metrics for both subtasks of MID. (2) The performance of weRTweet can be further improved by using indi- cators as part of input, which is detected with the weighted logoodds-ratio technique with informed Dirichlet priors (Monroe et al., 2008). (3) Cross- topic setting is quite challenging and the achieved
In an effort to address these issues, in this work, we first design a multifaceted ideology schema to provide a more complete and delicate description of ideology, by considering traditional ideology di- vision and emerging social topics. As shown in Figure (a)) the schema covers five orthogonal do- mains, including politics, economy, culture, diplo- macy and society, under which, there are a total

---

performances are far from promising.
five ideology profiles: egalitarian, essentialist egal- itarian, intensive parenting, moderately traditional, and traditional. During the COVID-19 pandemic, the far-right ideology has taken on new characteris- tics, including a disregard for scientific expertise, distrust of news media, and allegiance to Trump- Christian nationalism (Perry et al., 2020). These new social issues call for a flexible ideological eval- uation system, which can fully explain contempo- rary issues in various social fields and describe the existing variation ideology from multiple facets.
2 Multifaceted Ideology
To accurately measure the ideology of an individ- ual or group, sociologists have designed a series of scales to categorize political attitudes based on more facets. Ferguson (1952) developed ten stance scales that encompass attitudes towards birth control, the death penalty, censorship, com- munism, evolution, law, patriotism, theism, crim- inals, and war. By carefully analyzing the results of the popular test, he was able to condense the ten scales into three distinct ideological facets: reli- gious, humane, and ethnic. Soon after, Hans (1957) classified political ideology into "Radicalism" and "Conservatism". To demonstrate the inadequacy of the above research, Rokeach (1973) proposed a dual-axis model of ideology based on the principles of freedom and equality. This model similar to the Nolan Chart (Nolan, 1976). political diagram that uses two facets to divide people's 's ideological positions into four different quadrants. The hori- zontal axis represents "economic freedom", while the vertical axis represents "individual freedom". Ideologies located in different quadrants are catego- rized as modern liberalism, conservatism, libertari- anism, and nationalism. These prior work proposed different facets for dividing ideologies and laid the theoretical foundation for multifaceted ideology. but a core problem is that they usually study spe- cific domains, such politics or economics, and different facets may overlap have ambiguous boundaries.
By considering previous work and emerging social issues, we put forth a new multifaceted ideology schema (see Figure I(a), detailed in Appendix A). Firstly, our multifaceted ideology schema divides ideology into five orthogonal do- mains to ensure that each domain is independent and has clear boundarics. Secondly, the multi- faceted ideology schema now incorporates new contemporary issues. For instance, the facet "Cul- tural Value" is linked to the anti-abortion move- ments, the facet "Justice Orientation" is associated with the Black Lives Matter movement, and the facet "Military Force" is connected to peace-war issues. Finally, our multifaceted ideology schema covers twelve facets, each is defined with ideolog- ical attributes for the left- and right-leaning, pro- viding more complete and delicate description of ideology.
3 The MITweet Dataset
Based on the proposed multifaceted ideology schema as described in Section 2, we annotate tweets manually to build MfTweet. In this sec- tion, we describe how the data is collected, the design of annotation process, dataset statistics and the methods to ensure the quality of annotation.
With the development of society, the study of multifaceted ideology and political spectrum has brought attention to new issues, including anti-war movements, women's 's rights, environmental pro- tection, animal welfare and other related concerns. Mueller (2017) argued that color-blind racism is a form of covert and highly institutionalized discrim- ination, which differs from the more overt forms of racism seen during slavery and legal segregation. Therefore, colorblindness serves as the prevailing ideological support for historically unique form of structural white supremacy known as color-blind racism. Grunow et al. (2018) argued that defining gender ideology as one-dimensional construct that ranges from traditional to egalitarian is prob- lematic in contemporary society. They proposed an alternative framework that considers the multi- dimensionality of gender ideologies and identified
3.1 Data Collection
We collect tweets using Twitter streaming API based on different topics. To maximize the propor- tion of tweets containing the author's 's ideology and improve annotation efficiency, we select 14 topics that meet the following two criteria: (1) highly con- troversial or extensively discussed in recent years; (2) related to politics, economy, culture, diplomacy or society, i.e. the five domains in the ideology schema. Then we use set of query keywords as seeds to collect topic-related tweets. For each topic, we set the query period during which the event in- volved many intensive discussions on social media,

---

#Raw Tweets
#Cleaned Tweets
#Annotated
Topic
Query Keywords
Query Period
Tweets
Inflation Reduction Act (IRA)
#IRA, #inflation
Aug Sep, 2022
1114
811
427
Capitol Hill Riot (CHR)
#CapitolHilRioo
Jan Feb, 2021
2330
1440
346
CHIPS Science Act (CSA)
#CHIPSAct
Aug 2022
2159
346
Abortion (Ab)
#Roe
Jun Aug. 2022
3222
2376
1893
Russo-UKeee Ukrainian War (RUW)
#UkraineWar War, Kyiv
Feh - Oct, 2022
7390
5672
2100
Mass Shootings (MS)
#GunsShot, #crime
Feh 2018 Jun 2022
2325
1494
624
George Floyd (GF)
#GeorgeFloyd
May Aug, 2020
896
582
423
Black Lives Matter (BLM)
#Blectl.vessMater
Aug 2014 Jun 2020
3877
707
Political Parties (PP)
Democrat, Republican
Mar 2020 Oct 2022
1692
1253
502
Mexico-US Border (MUB)
#BorderCrisis
Mar May, 2021
3337
2331
912
Democracy (Dm)
democracy
Sep Nov, 2022
750
392
Covid- 19 (C19)
covid, vaceine
Feh 2020 Jul 2022
3718
3070
369
Energy Crisis (EC)
#Energy Crisis, #oil
Mar Apr, 2022
2205
1523
661
Women's Right (WR)
women right
Dee 2021 May 2022
6405
5170
2892
Total
41499
28967
12594
Table 2: 14 topics in MITweet. The left part presents examples of query keywords and the query periods when collecting data. The right part presents the numbers of raw tweets, cleaned tweets after preprocessing and final annotated tweets for each topic.
as shown in Table In total, we gather around 41.5k raw tweets for all topics.
public management, social science and communi- cations. Before starting the annotation, we conduct sufficient training for annotators and several rounds of annotation trials, which will be discussed in de- tail in Section 3.4.
To ensure the quality of MITweet, we preform several preprocessing steps for raw tweets: (1) We remove duplicates and retweets. (2) We remove @USER and URLs in tweets, as they often do not con- vey semantic information literally. (3) We remove tweets with less than 15, more than 130 words. We set this length limit because tweets that too short are usually semantically incomplete, while tweets that are too long may contain many irrele- vant content, which can introduce noise into the dataset. (4) We focus on tweets that have received a lot of attention on Twitter. Tweets that express a clear or controversial point of view or represent the perspectives of many people often have higher popularity and research value.
For each tweet, each annotator needs amno- tate Relevance (and an Ideology) label for each of twelve facets our idcology schema described in Section This process was done in two steps. Firstly, annotators are asked label "Related" or "nrellated" for each facet. For example, if an an- notator believes that tweet expresses the author's 's ideology regarding "Social Development", then "Related" should be labeled on this facet, and vice verse. Secondly, for each facet labeled with "Re- lated" in the previous step. annotators are further required to label one ideological category: "Left", "Center" "Right". Note that the "Center" class means that tweets are biased towards a centrist ide- ology in one facet, not lacking ideological bias (e.g., purely descriptive news reports), which is labeled with "Unrelated".
For each tweet, we calculate a spread score based on the number of likes, comments and retweets, a user score based the number of tweets, followers and followees of the author. Then final heat score is obtained by weighting the spread score and user score. We remove tweets with heat scores below a certain threshold. After preprocessing, we have 29.0k cleaned tweets total and randomly sample 17.0k of them for annotation.
Each tweet is annotated by three random anno- tators, and the final results are obtained through major voting. If major voting is not possible, that is, three annotators label facet with completely different ideological categories, then the tweet will be discarded due its possible high ambiguity. When only two annotators label "Related" on a facet, they must reach an agreement on ideological
3.2 Data Annotation
We invite 56 annotators to participate in our anno- tation, including graduate students from schools of

---

Relevance
Ideology
Domain
Facet
Related Ratio (%)
#Related
#Left
#Center
#Right
Political Regime (PoR)
112
0.9
39
14
59
Politics
State Structure (SS)
291
2.3
67
88
136
Economic Orientation (EO)
759
6.0
294
297
168
Economy
Economic Equality (EE)
672
5.3
520
119
33
Ethical Pursuit (EP)
2935
23.3
1976
465
494
Culture
Church- State Relations (CSR)
68
0.5
17
18
Cultural Value (CV)
1.2
48
Diplomatic Strategy (DS)
1572
12.5
711
421
Diplomacy
Military Force (MF)
1837
14.6
132
575
1 1130
Sncial Development (SD)
1737
13.8
1236
287
Society
Justice Orientation (IO)
3452
27.4
3058
281
Personal Right (PeR)
3516
27.9
171
241
3104
Table Relevance and ideology distribution, Related ratio refers the proportion tweets related to facet in the entire MITweet.
labels; Otherwise, the tweet will also be discarded. Finally, 74.1 percent of labeled tweets pass the inter-annotator agreement check and compose the MIlweet dataset.
not appropriate label texts just as either left- or right-leaning and that characterizing ideology from multiple facets can provide a more complete and nuanced view for further analysis.
3.3 Dataset Statisties and Analysis
3.4 Quality Control
Overall Statisties MITwwet contains total of 12,594 tweets, 11,649 of which related least one ideological facet in the schema. Each tweet is labeled with a Relevance label, and an Ideology label the relevance label "Related", along each facet. Mcanwhile, MITweet involves 14 topics from a variety of fields, and their statistics are provided in Table 2 and Appendix C.
As described in Section 1, most previous related work annotate datasets using distant supervision approaches, which can lead to many noisy labels in datasets. In contrast, our MITweet is annotated manually and is therefore inherently more reliable and diverse.
In order to further ensure the data quality, we carried out a strict annotation workflow. First of all, we explain the annotation schema in detail for an- notators and provide some examples. Then several rounds of annotation trials are performed, After each trial, we collect questions from annotators and discuss frequent inconsistencies. based on which we retrain annotators and then new round trial will start. The above process iterated for sev- eral times until the pass rate of agreement check reaches 0.70 and the average inter-annotator agree- ment reaches 0.75 (Krippendorff's alpha). During formal annotation, we annotate 14 topics one by one to avoid interference between topics and duce the difficulty in dealing with continuously changing topics. Before starting the annotation of each topic, we introduce the background of the topic and provide some representative instances for annotators.
Relevance Distribution As shown in Table 3, PeR, JO and EP are the most frequently related facets; MF, SD and DS also comparatively com- mon, while CSR and PoR are the rarest facets, with a related ratio of less than I %. In general, most facets have a relatively low related ratio, result- ing in an imbalance data distribution and posing challenges for facet relevance detection task. Nev- ertheless, we do not take downsampling since this is reflection of the real-world data distribution.
Ideology Distribution The right part of Table 3 presents the ideology statistics. It is obvious that different facets have very different data distribu- tions. For example, most tweets are left-leaning towards SD and JO, while it is the opposite for MF and PeR. This once again indicates that it is

---

Facet
PoR
SS
BO
EE
EP
CSR
CV
DS
MF
SD
JO
PeR
Avg.
K's K's alpha
92.8
88.1
86.1
77.5
80.3
89.3
93.0
90.2
71.6
84.0
68.6
75.7
83.1
Table 4: The Krippendorff's alpha of each ideology facet.
We compute the Krippendorff's alpha along each facet as the measure of agreement between annota- tors, as shown in Table 4. We can observe that Krip- pendorff's s alpha of most facets are above 80, with an average of 83.1, which is satisfactory score, indicating high consistency among annotators and that the dataset is reliable for further research.
MITweet reflects people's idcological leanings to- wards various facets in the real world and therefore has high credibility.
Experiments
4.1 Dataset Split
We split the MiTweet dataset two different ways for different scenarios. (1) Random split. We randomly divide the training, development, and testing sets following a 70/15/15 split. This is for in-topic MID since all three sets share the same topics. (2) Topic split. Firstly, several topies are selected as the test topics. Then, tweets of the remaining topics are randomly divided into training set and development set according to the ratio 4:1. This can serve as the setting of cross-topic MID, where the model is first trained and validated on source topics, and then tested on target topics.
3.5 Credibility Analysis
With the well-designed schema and strict annota- tion workflow, the data distribution of MITweet is generally consistent with the sociological charac- teristics of each topic involved. Take the topic of BLM as an example.
BLM (Black Lives Matter) is social and civil rights movement. It advocates for the rights and cquality of Black communities in various aspects of society, including law enforcement, criminal jus- tice, education, employment and healthcare. From the perspective of multifaceted ideology, BLM is mainly related to Social Development, Justice Ori- entation, Personal Right (Society domain) and Eth- ical Pursuit (Culture domain). In addition, the na- ture of BLM, which is the pursuit of black rights, racial equality and social reform, suggests that it stands for Revolutionism (Left) in Social Develop- ment, Result Justice (Left) in Justice Orientation, Individual Right (Right) in Personal Right and Eth- ical Liberalism (Left) Ethical Pursuit.
4.2 Tasks and Models
We split the multifaceted ideology detection pro- cedure into two sub-tasks in a pipeline manner: (1) Relevance Recognition aims to recognize the facets that a text is related to. We model this task as a multi-lahel classification problem. (2) Ideol- ogy Detection predicts which idcology a text holds regarding the related facets. This task can be mod- eled as a multi-class classification problem.
We use Transformer (Vaswani et al., 2017) en- coder based pre-trained language models (PLMs) as backbone models. Three PLMs are com- pared in experiments: BERT (Devlin et al., 2019), RoBERTa (Liu al., 2019), and BERTweet (Nguyen et al., 2020) pretrained on 850M En- glish Tweets following the training procedure of RoBERTa. The base version of these models from HUGGINGFACE are used. We fine-tune the PLMs to predict the relevance or ideology by appending a linear classification layer on top of the hidden rep- resentation of the [CLS] token. For the relevance recognition lask, tweet text is the input. For the ideology detection task, we concatenate the tweet with the facet name (e.g., "Political Regime") and separate them using the [SEP] token, and then feed the sequence into PLMs.
The topic distribution of MITweet is shown in Appendix C, we can see that the label distribution of BLM topic is consistent with the analysis men- tioned above. The facets with the largest number of related tweets are Justice Orientation, Social Development, Personal Right and Ethical Pursuit, accounting for 88.4%, 36.5%, 23.9% and 5.4% of all BLM tweets, respectively. The ideology dis- tribution also aligns with the sociological charac- teristics of BLM. Most tweets related to Ethical Pursuit (81 (81.6%). Social Development (80.6%) and Justice Orientation (89.0%) are left-leaning in their respective facets. And 87.6% of tweets related to Personal Right are right-leaning in this facet.
For other topics, we can observe similar consis- tency with sociological analysis. This indicates that

---

Facet
Indicators
For the relevance recognition task, first, we can observe that BERTweet performs the best on most facets and achieves competitive performance on EP and SD, indicating the effectiveness of this model, which is pretrained on tweets. Second, facets with low results, such as CV, SS, CSR and PoR, are also the facets with the most unbalanced data distribu- tions as provided in Section 3.3. This demonstrates that dealing with the low proportion of positive in- stances this task is huge challenge and needs more research efforts. One exception is the SD facet, which has a similar rated ratio as DS and MF, but much worse performance. By analyzing the prediction results, we find that many SD-related tweets use metaphors and hide the intent behind the literal sense(e.g.. the lower example in Figure (()),, which is difficult for models to understand.
PoR
Socialism, Communist, capitalism Democrats, Republicans, partisan #inflation, #IRA, market insurance, relief, tax Christian, religious, church women, #BlackLivesMatter, rights
sS
EO
EE
CSR
JO
Table 5: Examples of indicators.
Inspired by Kawintiranon and Singh (2021), which used the weighted log-odds-ratio technique with informed Dirichlet priors (Monroe et al., 2008) to compute stance words, we propose using the sim- ilar method to obtain the indicators for each facet. Specially, we extract tweets from training set to form corpus C2 and a corpus Cy. where C, con- tains tweets related to a facet, while C2 contains the rest, i.e., unrelated tweets. Then, by using weighted log-odds-ratio, we compute the usage difference of each word among two corpora and find the top-k significant words in C₁ as indicators of the facet. Therefore, indicators are the words in training set that best represent facet. Examples of indicators are provided Table 5. Obviously, compared with the abstractness of facet names, indicators have more concrete meanings and different indicators describe different perspectives of a facet. We set k = 18 and concatenate tweet with indicators as an alternative input for the ideology detection task.
For the ideology detection task, we can observe that: (1) The achieved performances low across all facets, with no facet exceeding 60%, which is far from practically usable, indicating the inherent challenge of this task. (2) Different from the rele- vance recognition task, in this task, each model has its own advantageous facets. For example, BERTweet leads other models by large margin in CSR, especially after using indicators, while BERT and RoBERTa exhibit the best performances in five facets respectively.
(3) The performances of eight facets are improved when using indicators as part of input. This is likely due to the fact that indica- tors come from the words that appear in the dataset and may contain more topic-related in-contextual meanings compared with bare facet names. That is, indicators can establish a deeper interaction with texts and help models mine the idcological bias contained in texts more accurately.
4.3 Evaluation Metrics
We adopt the Accuracy (Acc) and Fl metrics for each facet. In order to evaluate the overall performance of models, macro methods to integrate metries from each facet. Macro-Acc/F1 treats each facet equally and the average of Acc/Fl scores across all facets. Micro- Acc/F1l is calculated globally by putting together predictions from all facets. Note that due to the highly unbalanced data distribution in the relevance recognition task, we only use Fl-related metrics in this task for more credible evaluations.
use both micro and
Overall results for relevance recognition and ide- ology detection are presented in Table 7 and Ta- ble 8 respectively. BERTweet achieves the best overall performance on both sub-tasks, except for Macro-FF the ideology detection tasks, which is also competitive with the best result. This is not unexpected, because BERTweet is pre-trained on tweets and is therefore more suitable for mod- eling the tweet texts in this task, whereas BERT and RoBERTa have domain adaptation problems to some extent. Interestingly, after using indica- tors, we can see significant improvement on BERTweet, but not for BERT and RoBERTa. We think this is likely because indicators contain some words with Twitter style (e.g., #inflation, #Black-
4.4 Results and Analysis
4.4.1
In-topic Setting
Under in-topic setting, we train, validate and test models on the same 14 topics in MiTweet. Fl scores of each facet are shown in Table We also provide Ace results of the ideology detection task for each facet in Appendix D.

---

PoR
SS
FO
EE
EP
CSR
CV
DS
MF
SD
PcR
Relevance Recognition
BERT
a0.05,.1
30.914.:1
46.44
61.59 as
82,644..0
34.07ass
8.46g.s
61 060..a ca
S6.730.r
42,100,44
74.59L.s
69.386.s
RoBERTa
H4.00us
47,
68 a8.
61.041a6
81,56125
L95
18.39
62,08
86.10
44.542.77
T0.13
69. 199770p.57
BERTweet
10.92..0
32.711.3
71.05
63.29
82 2in as
35.0
19.5224s
62.73
85.99.a4
11,02, mm
75.55
70.71n.4
Idcology Detection
Facez Name
26,00g
16
53 23.28
48.51
49.901
33.0514.1
44.97
$7.599.5a
46.377,.2
42.32,,4a
37-18
36.31a.co
RоBERтa
28.38
29.61aur
51.84227
46 16.963.03
83.662.0
31.60g.0
47. 1.
b6.001.st
47.41s.00
42.033.u
37.M
40.20ga7
BERTweet
27.40
28.97am)
47.98
51.30g
26.42.
46. 46.1513.0
57.23: T3
44.03;
42.58
36.5
34.06g
Indicatory
BERT
31.79
29.9 29.93,
54.42.97
46.58%
51.44261
19.96ir2
41.74aor
54.62,,
47. 47.112.06
44.088
38,83
40.044,2
RoBERTa
03.45
52, 18
43.74
52.413m
28.17
51.77
56.03:
46.88..1
45.00...
36,31: mo
39. 34..0
BERTweet
24.40
27.50%
52.26
41.25
52.43
49.93
43.37
57.00at
48.39
43.02
36.55
35.04a
Table 6: Fl scores (%) of the two sub-tasks for each facet under in-topic setting. Facet Name and Indicators denote the two types of input described in Section 4.2. Bold and underlined values are the best and second- results in the sub-task, respectively. We report the average of along with their standard deviation in the subscript.
Macro-F1
Micro-F1
Relevance Recognition
Idcology Detection
Target Topics
BERT
55.64o.s
69.366.39
Micro-F1
Micro-Ace
Micro-F1
RoBERTa
56.92,
69.87,
CHR&GF
59.60p.30
70.20
52.41p8t
BERTweet
$7.48..53
70.32 48
BLM&Dm
54.60 20.20
80 640.2
58.89pur
Table Overall results (%) of relevance recognition.
Table 9: Zero-shot results (%). We report averages along with their standard deviations over 5 random test sets from target topics.
Macro-F1
Micro-FI
Macro-Ace
Micro-Ace
Name
BERT
41.721.81
65.39%.67
62.50,.10
74.53
RoBERTa
42.78
67.80:
63. 75
75.59h
A model is first trained and validated source top- ics, and then tested on target topics. (2) Few-shot. In addition texts of source topics, a model can be further trained on few samples from each tar- get topic, with the parameters of PLM frozen or not. We ensure that the test sets of few-shot and zero-shot are the same, for fair comparison.
BERTweet
42.271.08
68,42,
64.511.74
75.10,.a
Indicators
BERT
41.961.1
66.48 29
62.35
74.26) sn
RoBERTa
41.78 18..8
67.63o
64. 7
75.98,
BERTweet
42.68
69.28..66
65.88 4545
76.38..42
Table 8: Overall results (%) of ideology detection.
Most of the topies are domain domain-specific, and it is likely that there is little or no relevant text on some facets. For example, topic BLM has no rel- evant texts in EO and CSR, and only one two relevant texts in PoR, SS, DS and MF, as shown in Appendix C. Therefore, we only use micro-Ace and micro-F1 metrics to focus on facets that are more relevant to a topic.
LivesMatter), which common during BERTweet pre-training, but may confuse BERT and RoBERTa. Moreover, the Macro- metrics are always signifi- cantly lower than the Micro- metrics, indicating that the models are good at making overall predic- tions across all facets, but not ideal at distinguish- ing between different facets.
Zero-shot results are provided in Table 9. We can observe that the Micro-Ace of ideology de- tection is relatively high and even reaches 80.64% when BLM&Dm are used target topics, while the Micro-F1 of both sub-tasks rather low. This indicates that the model tends to predict majority categories and does not have a good understand- ing of the relevance and idcology features of texts. The main reason is that under cross-topic setting,
4.4.2 Cross-topic Setting
New topics come up every day, but acquiring large amounts of annotated texts for new topics is time- consuming and labor-intensive. is hence neces- sary to evaluate the models' generalization abil- ity to transfer knowledge from annotated topies. We next conduct two sets of experimentations with BERTweet under cross topic setting: (1) Zero-shot.

---

Micro-F1 of Relevarce Recegnition
Micro-Acc of ideology Delection
Mccr-F11 of ideology Delection
74
74
58
72-
73
70
58
55
86
71
54
82
Rinelinee
52
Fhadma
Frezze
Freaze
Freezo
68
10
68
51
20
30
No.
Figure 2: Few-shot results (%). Target topics are CHR&GE "Fine-tune" and "Freeze" respectively denote that fine-tune and freeze the parameters of PLMS during model training. We report averages over 5 random training and test sets from target topies.
models must be able to discover the association between source and target topics to gain better per- formance. However, without any knowledge about the target topics, models can only rely connec- tions on text level, the relevance and ideology.
Limitations
MITweet only covers English tweets, which limits the linguistic features covered by MITweet and the scope of applications built on In some facets (e.g., CSR and PoR), the number of relevant texts is relatively small and may not train the model well. Therefore, we will annotate more texts with the schema by combining emerging new topics and extend MITweet to other languages in future work.
is hard to correctly identify
In Figure 2, we present the results for few-shot experiments. We can see clear upward trend in performance as the training data increases, and almost all results under the fine-tune setting are better than those under the freeze setting. For the relevance recognition task, the model gains consid- erable improvements with only 5 training examples. When the number of training examples increases 40, there is large gap of about 10 compared with the results in zero-shot. However, for the ide- ology detection task, the performance improves slightly even 40-shot, suggesting that, is still hard for the model to transfer ideological features from source topics with few training examples and maybe there is a need to incorporate knowledge about target topies into the model in future work.
Only several base PLM models chosen in our experiments, and more ideology detection methods are worth of further exploring.
Under cross-topic setting, we just train models with texts from source topics. As the experi- mental analysis shows, it is worth considering injecting prior knowledge of target topics improve the generalization ability of models. We leave this work for the future.
Ethics Statement
The proposed multifaceted idcology schema is accordance with the general ethics in social science research to understand people, society and politics. The MITweet dataset is only a peek into the huge amount of Twitter data, and the released dataset has removed all identity related information.
5 Conclusion
this paper, we have presented new multi- faceted ideology schema covering five domains and twelve facets to provide a more complete and delicate evaluation system for ideology. Based on the schema, we have constructed a high-quality dataset, MfTweet, as benchmark for a new MID task. Experiments show that the MID task is quite challenging, especially under cross-topic setting, which requires the knowledge transfer capability models. We believe that this work has the poten- tial to positively impact both the rescarch and the applications of ideology detection.
Acknowledgement
This work is supported in part by Major Project of National Social Science Foundation of China: "AI and Precise International Communication" (Grant No. 22&2D317) and National Natural Science Foundation of China (Grant No. 62172167). We thank all the annotators for their hard work.

---

References
Mohit lyyer, Peter Enns, Jordan Boyd-Grabel and Philip Resnik. 2014. Political ideology detection us- ing recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1113-1 1123,112, Baltimore, Maryland. Association for Computational Linguistics.
Saja Aldera, Ahmad Emam, Muhammad Al-Qurishi, Majed Alrubaian, and Abdulrahman Alothaim. 2021. Online extremism detection in textual content: A systematic literature review. IEEE Access, 9:42384- 42396.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, Online. Association for Computa- tional Linguistics.
Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4725-4735, Online. As- sociation for Computational I Linguistics.
J. Edwin Boyd and Douglas N. Jackson. 1967. The perceived structure of social attitudes and personality A multidimensional scaling approach. Multivariate Behavioral Research, 2(3):281.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, Payam Adineh, David Comey, Benno Stein, and Martin Potthast. 2019. SemEval- 2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop Semantic Evaluation, pages 829-839, Minneapolis, Minnesota, USA, Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3518- 3527, Brussels, Belgium. Association for Computa- tional Linguistics.
Terry Eagleton. 2007. Ideology: An Introduction. Verso.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint
Stanley Feldman and Leonie Huddy. 2014. Not so sim- ple: The multidimensional nature and diverse origins of political ideology. BEHAVIORAL AND BRAIN SCIENCES, 37(3)312--13..
Yujian Liu, Xinliang Frederick Zhang. David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022. POLI- TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. In Find- ings of the Association for Computational Linguis- tics: NAACI,2022 pages 1354-1374, Seattle, United States. Association for Computational Linguistics.
L. W. Ferguson. 1952. Review of the book personality measurement. Journal of Consulting Psychology, 16(6):475.
José Antonio Gareía-Díaz Ricardo Colomo-Palacion and Rafael Calencia-Garcia 2022. Psychographic traits identification based on political idcology: An author analysis study on spanish politicians' tweets posted in 2020. Future Generation Computer Sys- tems, 130:59-74.
Burt Monroe. Michael P. Colaresi, and Kevin M. Quinn. 2008. Fightin' words: Lexical feature se- lection and evaluation for identifying the content of political conflict. Political Analysis, 16(4)3372-433.
Ted Grover and Gloria Mark. 2019. Detecting poten- tial warning behaviors of ideological radicalization in an alt-right subreddit. Proceedings of the Interna- tional AAA/ Conference Web and Social Media, 13(01):193 204.
Jennifer C. Mueller. 2017. Producing colorblindness:
Everyday mechanisms of white ignorance. SOCIAL PROBLEMS, 64(2):219-238
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, On- line. Association for Computational Linguistics.
Daniela Grunow, Katia Begall, and Sandra Buchler. 2018. Gender ideologies in europe: A multidimen- sional framework. JOURNAL OF MARRIAGE AND FAMILY, 10.4)22..0.
Eysenck Hans. 1957. Sense and nonsense in psychology. Sense and nonsense in psychology.
David Nolan. 1976. Principles of pharmaceutical mar- keting. Journal of Marketing, 40(2):123.

---

Samuel Perry, Andrew L. Whitehead, and Joshua Grubbs. 2020, Culture wars and covid-19 conduct: Christian nationalism, religiosity, and americans' be- havior during the coronavirus pandemic. JOUR- NAL FOR THE SCIENTIFIC STUDY OF RELIGION, 59(3):405-416
State Structure (SS) The organizational form of the state, i.e., the distribution of power among agencies.
Centralism (Left): the power should be con- centrated in the central authority.
Daniel Preotiuc-Pietre Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction of Twitter users. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729-740, Vancouver, Canada. Association for Computational Linguistics.
Federalism (Right): the power should be dis- tributed between central authority and the constituents.
A.2 Domain 2: Economy
Economic Orientation (EO) Any of the ways in which humankind has arranged for its material provisioning.
Milton Rokeach, 1973. The nature of human values. American Journal of Sociology, 89(2).
Juliana Schwarz. 2019. Detecting political ideology in youtube comments using machine learning. In Seminar Social Media and Business Analytics.
Command Economy (Left): the government should take responsibility for making most the important economic decisions.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Market Economy (Right): economic decisions should be guided by the interactions of indi- viduals, organization, companies.
Economic Equality (EE) The orientation that ought be adopted in achieving equality through economic policies.
John Wilkerson and Andreu Casas. 2017. Large-scale
computerized text analysis in political science: Op- portunities and challenges. Annual Review of Politi- cal Science, 20( 20(1):52-5544
Outcome Equality (Left):: all groups should receive the same treatment or distribution.
Zhiping Xiao, Weiping Song. Haoyan Xu, Zhicheng Ren, and Yizhou Sun. 2020. Timme: Twitter ideettgy-detectioon via multi-task multi-relational bedding. Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD '20, page 2258-2268 New York, NY, USA. Association for Computing Machinery.
Opportunity Equality (Right): all groups should have equal access to resources.
A.3 Domain 3: Culture
Ethical Pursuit (EP) The guiding principles and standards that govern the collective existence and conduct of individuals.
A Multifaceted Ideology Schema
Ethical Liberalism (Left): the mainstream cul- ture should support sexual liberation, same- sex marriage, abortion, and other related is- sues.
The multifaceted ideology schema contains five domains that reflect different aspects of society, Under the five domains, there are twelve facets with idcological attributes of left- and right-leaning, shown in Table 10. Detailed definitions as follows.
Ethical Conservatism (Right): the mainstream culture should restrict individuals' behavior based on moral norms and religious doctrines.
A.1 Domain 1: Politics
Church-State Relations (CSR) The relations between religious ideology and the political con- sciousness of nation.
Political Regime (PoR) The formal and informal structure and nature of political power in a country.
Socialism (Left):: the ownership or control of the property should be public-owned.
Secularism (Left); the religious power and state power should be separated.
Capitalism (Right): the ownership of the means of production should pwivd.
Caesaropapism (Right): the religious power and state power should be unified.

---

Domain
Facet
Left
Right
Political Regime (PoR)
Socialism
Capitalism
Politics
State Structure (SS)
Centralism
Federalism
Economic Orientation (EO)
Command Economy
Market Economy
Economy
Economic Equality (EE)
Outcome Equality
Opportunity Equality
Ethical Pursuit (EP)
Ethical Liberalism
Ethical Conservatism
Culture
Church-State Relations (CSR)
Secularism
Caesaropapism
Cultural Value (CV)
Collectivism
Individualism
Diplomatic Strategy (DS) Military Force (MF)
Globalism
Isolationism
Diplomacy
Militarism
Pacifism
Social Development (SD)
Revolutionism
Reformism
Society
Justice Orientation (JO)
Result Justice
Procedural Justice
Personal Right (PeR)
Social Responsibility
Individual Right
Table 10: The proposed multifaceted ideology schema.
Cultural Value (CV) The cognitive framework shared by the members of society.
Revolutionism (Left): it is necessary to take direct and noticeable action to achieve social goals.
Collectivism (Left):: an individual should be seen as subordinate to social collective.
Reformism (Right): the social changes should take place in gradual way.
Individualism (Right): all values should be human-centred and the individual should be of supreme importance.
Justice Orientation (JO) The orientation that ought to be adopted in achieving justice through social policies.
A.4 Domain 4: Diplomacy
Diplomatic Strategy (DS) Fundamental princi- ples and guidelines for nation's diplomatic en- deavors.
Result Justice (Left): people should be fairly distributed and treated in various social activi- ties.
foreign policy should be planned with an international perspective.
Globalism (Left):
Procedural Justice (Right): the authorities should make fair decisions.
Isolationism (Right): political and economic entanglements with other countries should be avoided.
Personal Right (PeR) The standards by which individual rights are measured in the formulation of social policies.
Military Force (MF) The disposition of a na- tion or political faction towards the utilization of military force.
Social Responsibility (Left): there should be a greater emphasis on fulfilling individual re- sponsibilities.
Militarism (Left): is necessary to use strong armed forces to gain political or economic advantages.
Individual Right (Right): there should be a greater emphasis on protecting individual rights.
Pacifism (Right): all types of violence be- tween countries are incorrect.
B Calculation Process of Tweet Heat Score
A.5 Domain 5: Society Social Development (SD) The collective stance societal members towards the flux of eras.
For each tweet, the spread score is calculated by weighting the number of likes L, comments and retweets R. We also consider the influence

---

of the tweet author and calculate the user score by weighting the number of tweets T, followers Fr and followees Fe of the author. Then the final heat score obtained by weighting the spread score and user score.
spread L a1 C. 02 + R * az, user T 31 Fr B2 + Fe 33,
heat spread μ4 + user * μ22
During data collection, we set 01 0.3, 02 0.6, ag = 0.1, 31 = 0.3, 32 = 0.6, Bg = 0.1 and An 0.6, F2 0.4. We remove tweets with heat scores below threshold which is adjusted manually according to the topic heat on Twitter.
C Topic Distribution of MITweet
See Table 11.
Additional Experimental Results
See Table 12.

---

Topic
IRA
CHR
CSA
Ab
RUW
MS
GF
BLM
PP
MUB
Dm
C19
EC
WR
#Tweets
427
346
346
1893
2100
624
423
707
502
912
392
369
661
2892
#Reated Tweets
407
330
319
1890
1995
242
411
699
447
837
379
330
531
2832
#Left
2
1
6
0
9
PoR
#Center
2
0
3
0
0
2
0
0
#Right
1
3
8
0
0
22
7
6
2
2
#Left
3
2
0
0
0
49
3
4
0
0
SS
#Center
7
56
#Right
6
2
0
21
6
0
2
63
4
20
5
0
6
#Left
112
0
30
0
5
0
0
0
27
3
2
7
108
0
EO
#Center
115
0
22
107
3
#Right
63
11
0
4
0
0
0
1
80
0
#Left
78
8
30
10
5
1
7
96
34
8
134
50
56
EE
#Center
41
4
2
1
0
0
2
29
0
17
10
5
#Right
11
0
2
1
0
1
#1.eft
0
1268
2
13
0
31
5
637
EP
#Center
0
2
0
297
3
50
2
4
2
4
1
0
0
100
#Right
107
3
#1.cft
CSR
#Center
0
0
7
2
0
0
0
0
1
0
6
#Right
3
0
7
1
0
0
0
1
2
#Left
3
4
4
2
1
4
36
3
29
2
CV
#Center
0
0
0
2
#Right
5
0
15
6
10
2
0
5
#L.cft
7
3
109
0
383
4
1
2
101
14
14
68
4
DS
#Centee
112
12
#Right
53
0
55
0
0
0
271
1
12
43
0
#l.eft
3
0
111
0
0
1
0
11
0
MF
#Center
5
0
#Right
0
6
2
1069
4
5
3
3
27
4
#Left
41
104
24
125
13
18
122
208
51
114
120
65
63
168
SD
#Center
27
67
5
42
4
12
32
8
20
10
12
34
#Right
101
0
24
0
10
6
0
22
#Left
147
17
352
556
24
149
58
35
1639
JO
#Center
5
1
13
2
7
17
50
6
23
9
2
0
145
#Right
1
5
#Left
15
4
26
6
10
13
14
22
29
PeR
#Center
0
0
61
4
3
2
2
5
16
6
0
133
#Right
5
23
5
532
21
29
69
148
24
101
175
39
14
1919
Table I1: Topic distribution of MITweet. "Related Tweets" means tweets related to at least one facet in our schema.
PaR
EO
EP
CSR
CV
MF
SD
JO
PeR
Facet Name
BERT
42.50,,20
35.29
58.912.1
76.91
70.21
38.18
59.174.30
60.34,a6
65.41gas
67.37
88.46, as
87.07
RoBERTa
38.7513.55
m
18.15
74.85
70.722.40
18,98
70.00%
59.23,
70.22, m
69.14,13
88.39
89.05,03
BFRTweer
42.50
40.50
61.45:
77. 77.1
70.499.54
47.27
63 63.333
58. 80:80
67 67.26),
70.21
87.730
87.34,
Indicators
BERT
48.75
38.82
59.82 89.82
73.81
69 00.065.mm
25.45 05..5..m
64.17,27
57. 52..2
67. 67.34
68.57
86 86.31as
88.6 88.02
RoBERTa
43.75 75
42.36..0
09.82278
77. 17.32
72.53
40.00
64.17:78
59 06.006..
09.33,,m
72 72.88
87 87.502a2
87 94.30
BERTweet
13.75
35.30b
00.91gas
77.53,
58.19
63. 33.333..2
39. 39.322,4
74.71
88.85mus
87 92..9
Table 12: Acc scores (%) of the ideology detection sub-task for each facet under in-topic setting.


Reviews:
[{"review_id": "s1ThCmO0Oh", "reviewer": "Reviewer_6B2A", "summary": "", "questions": "Q1. How is the spread score and heat score calculated (for cleaning the tweets)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review and recognition of our work. We address the comments as follows.\n\n> **Q1: How is the spread score and heat score calculated (for cleaning the tweets)?** \n\n**A1:** Due to considerations of paper length and structure, we only provide a brief description in Sec. 3.1. Here are the details.\n\nFor each tweet, the spread score is calculated by weighting the number of likes, comments and retweets. We also consider the influence of the tweet author and calculate the user score by weighting the number of tweets, followees and followers of the author. Then the final heat score is obtained by weighting the spread score and user score.\n\n\\begin{align}\n\\text{spread score} &= \\mathrm{\\\\# likes * \\alpha_1 + \\\\#comments * \\alpha_2  + \\\\#retweets * \\alpha_3,}  \\\\\\\\\n\\text{user score} &= \\mathrm{\\\\#tweets *\\beta_1 + \\\\#followees *\\beta_2  + \\\\#followers *\\beta_3,}  \\\\\\\\\n\\text{heat score} &= \\mathrm{spread score * \\mu_1  + user score * \\mu_2.}\n\\end{align}\n\nDuring data collection, we set $\\alpha_1$=0.3, $\\alpha_2$=0.6, $\\alpha_3$=0.1, $\\beta_1$=0.3, $\\beta_2$=0.1, $\\beta_3$=0.6 and $\\mu_1$=0.6, $\\mu_2$=0.4. We remove tweets with heat scores below a threshold which is adjusted manually according to the topic heat on Twitter.\n\nWe will add the above to the appendix in next paper revision.\n\n> **Q2: (Presentation Improvements) While we know there are 12 facets, it is mainly abbreviated (in Table 3) and other mentions. It's hard to pinpoint which facet is which, until the reader reaches the appendix.**\n\n**A2:** Thank you for pointing out this issue. We will modify Table 3 and add the full name of each facet in next paper revision. Although the full names have been shown in Figure 1(a), we acknowledge that the figure is mainly used to visually show the overall framework of the schema, and it is a little difficult to find the details of each facet.\n\n> **Q3: (Weakness) While the dataset is manually annotated, there is no external measure to determine whether the ideology detected matches. A case study on a specific topic could help.**\n\n**A3:** Thank you for your constructive suggestion.\n\nWith well-designed schema and strict annotation workflow, the relevance and ideology distribution of MITweet is generally consistent with the sociological characteristics of each topic. Take the topic of BLM as an example.\n\nBLM (Black Lives Matter) is a social and civil rights movement. It advocates for the rights and equality of Black communities in various aspects of society, including law enforcement, criminal justice, education, employment, and healthcare. The movement emerged in 2013 in response to the acquittal of Trayvon Martin's killer and gained significant momentum following the high-profile deaths of other unarmed Black individuals, such as Michael Brown and Eric Garner. Therefore, BLM is mainly related to Social Development, Justice Orientation, Personal Right (Society domain) and Ethical Pursuit (Culture domain) in the schema. In addition, the nature of BLM, which is the pursuit of black rights, racial equality and social reform, suggests that it stands for Revolutionism (Left) in Social Development, Result Justice (Left) in Justice Orientation, Individual Right (Right) in Personal Right and Ethical Liberalism (Left) in Ethical Pursuit.\n\nThe topic distribution of MITweet is shown in Table 11 in Appendix, we can see that the label distribution of BLM topic is consistent with the analysis mentioned above. The facets with the largest number of related tweets are Justice Orientation, Social Development, Personal Right and Ethical Pursuit, accounting for 88.4%, 36.5%, 23.9% and 5.4% of all BLM tweets, respectively. The ideology distribution also aligns with the sociological characteristics of BLM. Most tweets related to Ethical Pursuit (81.6%), Social Development (80.6%) and Justice Orientation (89.0%) are left-leaning in their respective facets. And 87.6% of tweets related to Personal Right are right-leaning in this facet.\n\nWe will add the above in Section 3.3 in next paper revision.\n\n> **Q4: (Reproducibility) The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available.**\n\n**A4:** Thank you for pointing out this issue. We tune the hyperparameters based on the development set. We will add the parameter setting in Section 4 in next paper revision. The ways of data split are described in Section 4.1 and we will provide our division of data in the supplement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A well-annotated and designed dataset with an average of 83.1 for Krippendorff's alpha. \n2. A valid experiment setup and reporting of both the average and standard deviation of the F1 score.\n3. Show that their approach is generalizable as their model with zero & few-shot setting on new topics still perform well.", "reasons_to_reject": "Weakness: While the dataset is manually annotated, there is no external measure to determine whether the ideology detected matches. A case study on a specific topic could help.", "questions_to_authors": "Q1. How is the spread score and heat score calculated (for cleaning the tweets)?", "paper_topic_and_main_contributions": "The paper constructs a multifaceted ideology detection (MID) task and dataset based on 12K English Twitter posts. Ideology is split into five main domains: Culture, Diplomacy, Economy, Politics, Society. Within each domain, it is split further into petals, which is then split into left, center & right. Their MITweet composes of 14 set of different topics, with each having at least 300 tweets annotated.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. While we know there are 12 facets, it is mainly abbreviated (in Table 3) and other mentions. It's hard to pinpoint which facet is which, until the reader reaches the appendix."}, "extra_text": {}}, {"review_id": "T47Nbywtvp", "reviewer": "Reviewer_4GHc", "summary": "", "questions": "How do you avoid/ reduce the bias when  measuring the ideology?\nAre all the 56 annotators  in the article from graduate school?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review. We address the comments as follows.\n\n> **Q1: Ideologies are often complex and can have multiple dimensions. The approach however did not mentioned on solution or steps taken when the ambiguity and overlap between different ideologies occurs.**\n\n**A1:** We agree that ideologies are often complex and can have multiple dimensions. With this in mind, we take thorough considerations during the design of the schema to minimize the ambiguity and overlap between different ideologies. As described in Section 2, we conduct research in social sciences and refer to a series of scales that categorize political attitudes based on one or more facets (e.g., the dual-axis model of ideology). We also consider the characteristics of contemporary social issues (e.g., the gender ideology in women’s rights) to describe the existing variation in ideology. Based on this, we propose the schema. The five domains in the schema reflect **different aspects** of society and the 12 facets are **clearly defined** and have **clear boundaries**, as shown in Appendix (the definitions of 12 facets will be added in next paper revision). Therefore, by careful design of the schema, we minimize the ambiguity and overlap between different ideologies.\n\nIn addition, during data annotation, we try to minimize annotators' comprehension bias by carrying out a strict workflow. Before formal annotation, we provide thorough training for annotators and conduct multiple rounds of annotation trials. After each trial, we collect questions from annotators and discuss frequent inconsistencies, based on which, we retrain annotators and revise the schema if necessary. Each tweet is annotated by 3 random annotators and a strict agreement check is performed. The final average inter-annotator agreement score (Krippendorff’s alpha), which is 83.1 (Table 4), also demonstrates that the ambiguity and overlap between different ideologies are reduced effectively. The above are discussed in detail in Section 3.2 (Data Annotation) and 3.4 (Quality Control).\n\n>**Q2: How do you avoid/ reduce the bias when measuring the ideology?**\n\n**A2:** Please refer to Answer 1.\n\n>**Q3: Are all the 56 annotators in the article from graduate school?**\n\n**A3:** Yes. The 56 annotators are all from graduate school, including schools of public management, social science and communication. And we paid them handsome wages.\n\n>**Q4: (Reproducibility) The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available.**\n\n**A4:** Thank you for pointing out this issue. We tune the hyperparameters based on the development set. We will add the parameter setting in Section 4 in next paper revision. The ways of data split are described in Section 4.1 and we will provide our division of data in the supplement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The baselines used  for the MID task under in-topic and cross-topic settings.", "reasons_to_reject": "Ideologies are often complex and can have multiple dimensions. The approach however did not mentioned on solution or steps taken when the ambiguity and overlap between different ideologies occurs.", "questions_to_authors": "How do you avoid/ reduce the bias when  measuring the ideology?\nAre all the 56 annotators  in the article from graduate school?", "paper_topic_and_main_contributions": "The author proposed  a  multifaceted ideology schema  that covers five domains and twelve facets to provide a complete evaluation system for ideology. Based on the schema, they constructed a high-quality dataset known as  MITweet, as a benchmark for a new multifaceted ideology detection (MID) task.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wWswro9l7P", "reviewer": "Reviewer_bU4Y", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review. We address the comments as follows.\n\n>**Q1: Lack of limitations of the new schema. The multi-faceted schema significantly overlaps with topical identification e.g. Politics, Economics, etc this can make the downstream value of this schema less relevant.**\n\n**A1:** This is an important concern. However, we believe that although the 5 domains and 12 facets in our schema may be regarded as topics in topical identification to some extent, our schema still has unique scientific research value and application prospects in the field of social science and computational linguistics.\n\nFirst, the 5 domains and 12 facets in the new schema are not randomly chosen, but specially designed for multifaceted ideology detection with theoretical research in social sciences, as described in Section 2. We refer to a series of scales that categorize political attitudes based on one or more facets (e.g., the dual-axis model of ideology). We also consider the characteristics of contemporary social issues (e.g., the gender ideology in women’s rights) to describe the existing variation in ideology. In the new schema, the 5 domains reflect different aspects of society and the 12 facets are all issues of great concern in contemporary society in their respective domains. (We will add the explicit definitions of 12 facets in next paper revision.) As a result, our new schema avoids the ambiguity and incompleteness in previous works, and is more suitable for contemporary issues.\n\nSecond, for each of the 12 facets, we define the ideological attributes for left- and right-leaning to provide a more precise understanding for measuring multifaceted ideology. In contrast, such definition of ideological attributes does not exist in topical identification.\n\nThird, with this new multifaceted ideology schema, we are able to measure ideology in a more complete and delicate way. As described in Section 1, some texts may contain descriptions for different issues and reflect the author’s ideology from various aspects. Therefore, it is necessary to detect the ideology of texts from multiple facets, so as to provide a more comprehensive and nuanced picture for further sociological research. To our best knowledge, we are the first to propose a multifaceted ideology schema and detect ideology from multiple facets in the field of computational linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "New annotation schema which moves beyond binary labels and provides more fine-grained information\nBaseline results\nNew benchmark \n", "reasons_to_reject": "Lack of limitations of the new schema\nThe multi-faceted schema significantly overlaps with topical identification e.g. Politics, Econoics, etc this can make the downstream value of this schema less relevant. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a new idealogy identification schema for annotating ideologies in social media text moving beyond the traditional left and right labels. They create a new dataset called MITweet based on tweets on multiple topics annotated with this new idealogy schema. The authors also ran experiments using different text encoders to present a reasonable baseline for the dataset", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9oAXhc0w38", "reviewer": "Reviewer_upds", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We greatly appreciate your thoughtful review and recognition of our work!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The main strength lies in its novel approach to ideology detection through a multifaceted lens. The authors effectively present the concept of exploring ideologies from multiple perspectives, which is crucial for gaining a more comprehensive understanding of this complex and multifaceted phenomenon. \n(2) The idea of creating a high-quality dataset to capture these nuances is commendable and fills a gap in the existing research landscape.", "reasons_to_reject": "no", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper construct a MITweet dataset for the multifaceted ideology detection(MID) task, which contains 12,594 English Twitter posts each annotated with a Relevance and an Ideology label for all twelve facets. It also design and test a few of baselines for the MID task under in-topic and cross-topic settings, which can serve as benchmarks for further research. This work is sound and interesing, which is beneficial for the reserarcher on fine-grained ideology detection.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study introduces a multifaceted ideology detection dataset comprising 12K English tweets. In addition to the dataset, the research presents benchmark results using various pre-trained language models.

All reviewers acknowledged the value of the dataset, though they had a few minor concerns. The authors addressed these concerns in their rebuttal. It would be beneficial to incorporate these responses into the paper to provide greater clarity for readers.
Decision: Accept-Main