The Truth, The Whole Truth, and Nothing but the Truth:  A New Benchmark Dataset for Hebrew Text Credibility Assessment
Authors: Ben Hagag
Reut Tsarfaty
Venue: EMNLP 2023 Findings (2023)
Keywords: NLP and social media
NLP application
Fake news detection
Fact Checking
Credibility Assessment

Abstract: In the age of information overload, it is more important than ever to discern fact from fiction. From the internet to traditional media, we are constantly confronted with a deluge of information, much of which comes from politicians and other public figures who wield significant influence. In this paper, we introduce HeTrue: a new, publicly available dataset for evaluating the credibility of statements made by Israeli public figures and politicians. This dataset consists of 1021 statements, manually annotated by Israeli professional journalists, for their credibility status. Using this corpus, we set out to assess whether the credibility of statements can be predicted based on the text alone. To establish a baseline, we compare text-only methods with others using additional data like metadata, context, and evidence. Furthermore, we develop several credibility assessment models, including a feature-based model that utilizes linguistic features, and state-of-the-art transformer-based models with contextualized embeddings from a pre-trained encoder. Empirical results demonstrate improved performance when models integrate statement and context, outperforming those relying on the statement text alone. Our best model, which also integrates evidence, achieves a 48.3 F1 Score, suggesting that HeTrue is a challenging benchmark, calling for further work on this task.

The Truth, The Whole Truth, and Nothing but the Truth:
A New Benchmark Dataset for Hebrew Text Credibility Assessment
Ben Hagag
Reut Tsarfaty
Bar-Ilan University
Bar-Ilan University
ben. hon.hhagg@biu..c.o
reut teat.
Abstract
to the spread of fake news either, with fake news being connected to stock market fluctuations and large trades. For example, fake news claiming that Barack Obama, the 44th President of the United States, was injured in an explosion, wiped out $130 billion in stock value (Zhou and Zafarani, 2018). Also, recent studies have shown that false content reaches "farther" and wider audience of readers than real content (Vosoughi et al., 2018).
In the age of information overload, it is more important than ever to discern fact from fic- tion. From the internet to traditional media, we are constantly confronted with deluge of information, much which comes from politi- cians and other public figures who wield sig- nificant influence. In this paper, we introduce HeTrue: a new, publicly available dataset for evaluating the credibility of statements made by Israeli public figures and politicians. This dataset consists 1021 statements, manually annotated by Isracli professional journalists, for their credibility status. Using this corpus, we
Even if correction of the false information reaches the misinformed audience, simply provid- ing the correct information is ineffective, as con- tinued reliance misinformation is likely when the misinformation conforms to person's 's pre- existing belief system, yet the correction does not (Lewandowsky et al., 2005). Retracting misinfor- mation that runs counter to a person's 's worldview can ironically even strengthen the to-be-corrected information, a phenomenon known as the world- view backfire effect (Hart and Nisbet, 2012).
whether the credibility of statements can be predicted based on the text alone. To establish a baseline, we compare text-only methods with others using additional data like metadata, context, and evidence. Fur- thermore, we develop several credibility assess- ment models, including feature-based model that utilizes linguistic features, and state-of-the- art transformer- based models with contextual- ized embeddings from a pre-trained encoder. Empirical results demonstrate improved perfor- mance when models integrate statement and context, outperforming those relying on the statement text alone. Our best model, which also integrates evidence, achieves 48.3 F1 Score, suggesting that HeTrue is challeng- ing benchmark, calling for further work on this task.
Addressing the spread of fake news necessitates a comprehensive approach to assess the credibility of information. Credibility assessment, distinct yet sometimes used interchangeably with fake news detection, involves evaluating the veracity of state- ments claims made by individuals, particularly those in influential positions. While fake news ex- plicitly entails deception, credibility assessment encompasses broader spectrum, scrutinizing the authenticity and reliability of information regard- less of the intent behind it.
1 Introduction
Our society is struggling with an unprecedented amount of falsehoods, hyperboles, and half-truths (Hassan et al., 2017). False information, com- monly termed 'fake news', is now viewed as one of the greatest threats to democracy, journalism, and freedom of expression. Distributing false content has become a significant concern in recent years when claimed have helped change public opinion in the US elections of 2016 (Zhou and Zafarani, 2018). Our economies are not immune
Manual credibility assessment is sufficient in effectively combating the rapid spread of false information, leading to a pressing need for alter- native approaches. In particular, there a need for automatic credibility assessment tools, which leverages statistical and machine learning methods to analyze and determine the credibility of state- ments or news they come in, thereby avoiding the human-labe bottleneck.
To effectively combat fake news and assess the

---

credibility of information, there is a pressing need for comprehensive datasets that facilitate the de- velopment and evaluation of automatic detection tools. Existing datasets primarily focus on English language content (Guo et al., 2022), leaving a sub- stantial gap in the study of other languages, includ- ing Hebrew, which known for its unique linguistic characteristics (Tsarfaty al., 2019). Additionally, many available datasets lack crucial supplementary information such as context, semantic features, and metadata associated with the statements in question. Including these features is essential for building ro- bust NLP models, enhancing their understanding and accuracy in credibility assessment tasks.
tion, and prioritization. These challenges result in a change in the normal elements of the speaker's is lan- guage. Prior works have found that these linguistic signals can also specifically assist in distinguishing between truth and falsehood in politicians' state- ments (Wang, 2017).
We hypothesize that linguistic signals indicative of statement's credibility exist in Hebrew and that these signals correlate with the credibility as- sessments made by professional journalists, who collaborated with us to create our gold-standard benchmark. To provide broader view, we have also explored alternative methods, which will be discussed in the following sections.
The need for such fake-news detectors is partic- ularly pressing in areas of which socio-political sit- uations may be sensitive yet may have world wide effects such the Middle East. To mitigate this, introduce the HeTrue dataset, the first-ever Hebrew fake-news detection benchmark.
As Hassan et al. (2015) first introduced, achiev- ing the 'Holy Grail' end-to-end credibility as- sessment systems, also referred to as fact-checking, requires a fully automatic platform. This platform should consist of three main components: a Claim- Spotter to detect claims that need validation in real time, Claim- Checker to evaluate the veracity of these claims, and a Fact-Check Reporter justify the evaluations with convineing evidence. This sys- tem would make determinations by analyzing the claim's text, source, claimant profile, and retriev- ing evidence from databases of previously checked claims. Our work advances towards this "Holy Grail," focusing on a fully automated credibility as- sessment system for Hebrew texts. The unique fea- tures of our dataset! and experiments could also benefit credibility assessment in other languages. We call on the computing and journalism communi- ties to join on this pursuit, given the importance and timeliness of this task.
The HeTrue dataset is unique Hebrew resource composed of 1021 statements made by politicians. Each statement is meticulously labeled for truthful- ness and supplemented with context information. The dataset is further augmented with metadata and semantic features, which have been validated and annotated by nonpartisan fact-checkers, com- piled with the International Fact-Checking Network (IFCN). While the HeTrue dataset addresses sig- nificant gap in Hebrew language resources for cred- ibility assessment, its comprehensive set of features also raises the bar for similar datasets in other lan- guages, including English.
Utilizing this novel benchmark, in this paper we focus mainly on linguistic- approaches for automatic credibility assessment that is, one that relies solely on the claim, or the claim's 's con- text as input for detection. Linguistic approaches are frequently utilized to tackle this task (Wang, 2017, Roy et al., 2018), as they can operate with- out additional resources or data. This makes these approaches more scalable and straightforward to implement in real-time applications with minimal effort and budget.
The contribution of our paper is, hence, man- ifold. First, we introduce HeTrue, the first pub- licly available Hebrew dataset for fake news detec- tion. This dataset comprises 1021 statements from politicians and public figures, manually labeled by professional journalists, and serves as a valuable resource for researchers investigating fake news detection in Hebrew. To the best of our knowledge, this is the first claim assessment dataset that ac- companies each claim with its context, alongside semantic and metadata features.
These approaches hinge on the assumption that individuals tend to express themselves differently, whether verbally or in writing, when conveying false information compared to true information. Fraser (1991) asserts that these differences stem from a sense of stress, manifesting in a decrease in cognitive integration capacity, precision, organiza-
Second, we conduct an extensive experimental analysis, exploring a variety of modeling types and dataset setups. To establish baseline performance for the task, we employ several methods including hand-crafted feature- based model, recurrent neu-
'Sec comparison with other datasets in Appendix A.7

---

ral network (RNN) with static embeddings, and a model extending - Hebrew monolin- gual pre-trained encoder (Seker et al., 2021). Our experiments with these models demonstrate the presence of linguistic cues in Hebrew text that can distinguish between credible and non-credible state- ments. we provide an analysis of the key features from the hand-crafted feature model, emphasizing their significance.
metrics (such syllable and word counts), vocab- ulary and grammatical complexity, and an analysis of the usage of words denoting uncertainty, speci- ficity, and expressiveness. On the other hand, Han- cock et al. (2007) focus on linguistic traits more fre- quently associated with deceptive behavior. Their examination of 242 transcripts revealed that individ- uals inclined to deception tend to produce lengthier texts, utilize a greater number of sensory expres- sions (such as references to sight or touch), and exhibit preference for using pronouns that shift focus away from themselves and towards others, in comparison to when they are being truthful. Reis et al. (2019) incorporates various psycholinguis- tic features derived from LIWC (Pennebaker et al., 2001), to detect persuasive and biased language. In our work, we systematically adapt and extend these methodologies assess statement credibility in Hebrew.
Thirdly, we demonstrate the importance of inte- grating context and evidence in credibility assess- ment. Our novel model, which combines statement context and an evidence-retrieval mechanism, re- sults in significant performance enhancements. We further compare our results with setadata-based models, revealing that our linguistic approach out- performs these models, and demonstrating that the addition of metadata features to a linguistic (hand- crafted feature model) does not have an additive effect on the model's ' performance 2.
The second approach is Evidence retrieval, which aims to find sources supporting or refuting the claim. One strategy involves the construction of knowledge graphs, facilitating the identifica- tion of relationships between textual segments and graph instances Pan et al. (2018). Alternatively, pre- established knowledge bases can be employed garner pertinent evidence, subsequently leveraging models that capitalize on this retrieved information for text classification Thorne and Vlachos (2018).
2 Related work
Credibility assessment is an interdisciplinary field that involves evaluating the believability, trustwor- thiness, reliability, accuracy, fairness, and objectiv- ity of claims or statements (Viviani and Pasi, 2017). It is closely related to fact checking and deception detection, and focuses on assessing the quality of information and the level of trust that can be placed with respect to claims. The goal of credibility as- sessment is to differentiate between credible and non-credible claims, without necessarily determin- ing the root cause of false information, such as whether it is due incorrect factual claims or the intent of the speaker (Giachanou al., 2019).
The third approach is metadata-based, and it is often used alongside other methods. It involves analyzing text-related information like publication date, claimant's demographies, and source type (Long et al., 2017). Additionally, with the rise of fake news on social networks (de Souza et al., 2020), behavioral analysis becomes important to identify unreliable patterns through user or mes- sage interactions (Ruchansky et al., 2017).
There are three categories of approaches for cred- ibility assessment and fake news detection: linguis- tic (text-based) evidence-retrieval and metadata- based, which typically incorporate machine learn- ing techniques for training effective and accurate classifiers (de Souza et al., 2020).
In terms of the credibility levels to be predicted, the work by Rashkin et al. (2017) was first in- troduce the task of credibility assessment in scale of 1-6, in their work with the Liar dataset (Wang, 2017). They used LSTM with GLOVE word em- bedding and showed that stylistic cues can help determine the truthfulness of text. Roy et al. (2018) applied CNN and BiLSTM models to the same dataset and task, aiming to extract patterns from short statements and understand the unique behav- iors of source speakers using different dataset at- tributes, underscoring the importance of including speakers' profile information for fake news classifi-
Linguistic approaches attempt to identify clues in the text in order to verify veracity (Conroy et al., 2015, Zhou et al., 2004). The work of Afroz et al. (2012) posits that changes in certain linguistic char- acteristics can signal an attempt to conceal writ- ing style, aiding in the detection of deceptive texts. This research present the effectivness of specialized set of lying-detection features including quantity
"The HeTrue dataset and models are publicly availahle at: ht https: http//tttttt/ttt tuu.. com cOnl On plab// /eeab//eerzee Hee True.

---

cation.
ies employed Google's evidence retrieval to bolster claim veracity modeling. Yet, their scope appears limited, especially terms of integrating context and a rich set of metadata//semanic features. No Hebrew-specific dataset for fact verification or fake news detection existed until our contribution, fo- cusing on real-life statements.
Giachanou et al. (2019) incorporated emotional signals for credibility assessment, which presented improved performance, relying on the fact that those signals can play a vital role in detecting false information (Ekman, 1992 Vosoughi et al., 2018).
Recent progress in this field encapsulates a vari- ety of methodologies. Victor (2020), for instance, present semi-supervised deep learning (SSDL) pipeline employing attention RNN-based model. Additionally, models like X-Fact (Gupta and Sriku- mar, 2021) emphasize the integration of evidence with attention architecture. A distinctive approach is taken by FakeFllow (Ghanem et al., 2021), I), which focus on news articles and study the importance of information flow to detect fake news.
Towards Hebrew Fake-News Detection. The availability of labeled benchmark datasets is crucial for building statistical approaches for automatic fake news detection new languages. English has been extensively studied with a large amount of annotated data, while other, less-rescarched lan- guages, have limited or no annotated data available (Guo et al., 2022).
As of yet, publicly available Hebrew dataset on fake news detection or claim credibility assess- ment is publicly available. Furthermore, a limited number of works exist in the field of automatic cred- ibility and falsehood detection in Hebrew. The stud- ies of Dilmon (2004, 2007, 2013) examine the lin- guistic differences between truthful and deceptive discourse in Hebrew and aim to develop a primary test for the cognitive and emotional processes in- volved in deception MaaCohen-Kernee et al., 2015). Although these studies do not analyze natural inputs and instead focus on laboratory-created in- puts, they offer valuable insights into the cognitive and emotional processes involved in deception in the Hebrew language. In their corpus, comprising 48 pairs of stories told by 48 subjects, they found several distinguishing features:
Context plays pivotal role in understanding and verifying claims. However, its inclusion in datasets is rare, and also varies significantly (Guo et al., 2022). Some datasets, such those created by Mitra and Gilbert (2015) and Ma al. (2016), in- corporate context derived from related threads. In these instances, a claim contextualized with a set of pertinent posts, often originating from the same thread, to compensate for the limited context within an individual post. On the other hand, Ghanem et al. (2021) adopt distinctive strategy, considering the segments of an entire article analyze informa- tion flow. Our work uniquely provides context for sentence-level claims, drawing directly from pre- ceding text or speech. This approach captures the immediate discourse surrounding the claim, high- lighting the critical role of textual interactions in credibility assessment. We believe this direct inte- gration of granular context at the sentence level is novel contribution to the field.
Morphological criteria: False stories exhibit increased use of 3rd person verbs and de- creased use of 1st person verbs, while true sto- ries show intensified use of past tense verbs.
While English remains the predominant lan- guage for fake news detection datasets, other lan- guages are often underrepresented. As observed, most efforts to date, such as those by Vlachos and Riedel (2014); Wang (2017), have extracted real- world claims from dedicated English-based web- sites like Politifact. However, Gupta and Sriku- mar (2021) are notable exceptions, having curated claims from 25 languages. Their multilingual base- lines, including models like Claim Only, Attention- based Evidence Aggregator, and Augmenting Meta- data Model, have established a promising founda- tion for multilingual fact-checking. Another ini- tiative is by Baly et al. (2018), who compiled a dataset of 219 Arabic statements. Both these stud-
Syntactic criteria: False stories tend have increased use dependent clauses and de- creased use of independent and conjunction clauses.
Semantic aspects: False stories involve in- tensificd use of synonym words, relative pro- nouns, negation, and generalized words.
Leveraging insights from these Hcbrew-focused studies, alongside the English-based works, en- hance credebility assessment in Hebrew. Our work extends the existing knowledge base, constructing comprehensive linguistic feature-based models and evaluating their effectiveness on our task.

---

While our contribution heralds a significant leap in Hebrew-centrie datasets for fact verification, is not confined to that. further includes a wide range of experiments and the introduction of empirical evaluation of novel models tailored explicitly for this task: Context-Bass Based Model (CBM), Attention- based Evidence Aggregator (Attn-AE) and Context and Evidence Model (CCEM).
third journalist made the final decision, ensuring highly reliable and consistent dataset. Although this rigorous process resulted in a smaller dataset size than the initially available set of claims, it guarantees a high-quality and reliable dataset.
In addition to the sentence-level statements and their journalist's verified label, we introduce in this dataset the following features: 'claim context' and jjournalisteditedd claim'. 4 Crucially, the context capsulates the sentences around the claim," adding new dimension claim assessment. our work, we examine the improvement gained by incoporat- ing context with the statement itself.
3 HeTrue: New Benchmark Dataset for Hebrew Credibility Assessment
In this work, we present HcTrue, unique and first- of-its-kind Hebrew dataset for credibility assess- ment, meticulously compiled through a collabora- tion with professional journalists from "The Whis- tle", an Israeli fact-checking organization. The dataset comprises 1021 statements from Israeli politicians and public figures, each accompanied by its credibility score.
Additionally, we analyzed semantic features in- cluding Field, Subject, Title, and tags. This two- fold process first involved a "The Whistle" journal- ist, followed by editor who reviewed the sug- gested values and refined them as needed. The full feature list including additional metadata features can be found in Appendix A.6. The label distri- bution of the dataset is presented Table 3 in the Appendix.
"The Whistle" operated an independent NGO from 2017 2018 before integrating into Globes newspaper in January 2019. It is noteworthy for being the only Israeli institute complying with the International Fact-Checking Network (IFCN), up- holding transparency, impartiality, and fairness in fact-checking. Our partnership with "The Whistle" ensured the integrity of data and alignment with scientific research standards, involving necessary adjustments during collection and annotation.
Assessment Scenarios and Data Splits To gain better understanding of the underlying task, we created four instances of our dataset for further experimentation.
FULL-SPECTRUM: The entire dataset, five
credibility scores.
TF-ONLY: Statements only with a "True" or "False" label. Other labels are removed.
The statements. spanning one to sentences, were manually collected by journalists from "The Whistle" in the area between February 2017 June 2023.
TF-BINS: Following previous studies (Popat et al., 2018; Giachanou et al., 2019), labels are grouped into binary classes, with true, mostly true and half true forming one class (i.e., true), and the rest as false.
Each statement was assessed on a nuanced 5- point truthfulness scale, ranging from "True" to 'False', with intermediary labels including MMoslly- True', "Partly-True' and Mostly-Falss'' Consis- tently with recent studies (Gupta and Srikumar, 2021), we also introduced two additional labels: "Unverifiable" for claims lacking sufficient evi- dence and 'Other' for cases that do not fit into any of the aforementioned categories. This approach is widely adopted by most fact-checkers (Guo et al., 2022).
PS-WRITTEN: Statements published in writ- ten media, such as Facebook posts newslet- ters. Excluding transeribed statements.
For the experiments described below, we excluded the "Unverifiable" and "Other" claims, leaving five credibility scores for our empirical investigation.
HeTrue Credibility Assessment
4.1 Experimental settings
To ensure label accuracy, "The Whistle", in partnership with the authors, implemented a rigor- ous three-stage inter-annotator agreement process. Each statement was initially examined by one pro- fessional journalist, followed by an independent review by a second. If disagreements occurred, a
In this paper, we examine the claim credibility as- sessment task by establishing a rigorous experimen- tal framework fcaturing a variety of computational models. Our approach employs both traditional
'Journalists often revise the original claim brevity and clarity prior to publication
Averrgge statement length (tokens) is 25.
Averagge context length is 51 tokens.

---

linguistic methodologies and advanced deep learn- ing algorithms, each utilizing different aspects of the dataset. This diversified strategy provides us with a broader understanding of the task and its complexities, and sets a comprehensive benchmark for future research. The implemented models are presented in Section 4.3.
The results, described in Table 1, reveal sev- eral noteworthy observations. Firstly, it is clear that the models significantly outperform the meta- data models across all four experimental setups. A marked improvement is observed between the full- spectrum and written setups, which is not found in the metadata models. The latter achieved 5% more compared to the former setup. These results align with two key conclusions. First, that there are lin- guistic cues within the text that can attest to its credibility, which is evident across all four setups and corresponds to the model performance. Second, that these cues are stronger in written statements as opposed to transcribed ones. This finding likely indicates that information is lost or that noise is in- troduced during the transcription process, such punctuation marks or writing bias, which impairs the model's learning.
4.2 Evaluation
In line with previous work (Guo et al., 2022), we employ the macro Fl score as the evaluation metric. To ensure the statistical robustness of our findings, we employ bootstrap technique (Dror al., 2018) that involves generating 10,000 resampled varia- tions of the test set. By calculating the standard deviation of the performance metric across these iterations, we evaluate the stability of our results.
Training Setup We employed nested cross vali- dation to evaluate the performance of our models, This approach allows us to tune the hyperparame- ters of the model in an inner loop while evaluating its performance on unseen data in an outer loop. This helps in avoiding overfitting and providing more robust estimate of model's performance on unseen data. We used k=5 for both inner and outer loop. The hyperparameters are optimized by training the model using different values on the sub- folds, and then evaluating the performance of cach set of hyperparameters on the sub-fold reserved for testing. The final performance of the model was determined by computing the mean Fl score and standard deviation of the outer loop held-out test set 10,000 times using bootstrap resampling. This eval- uation was conducted after optimizing the model through the inner loop. Hence, hyperparameters are optimized independently of the testing data. For hy- perparameter optimization we used Optuna (Akiba et al., 2019), a framework for sequential model- based optimization (SMBO) (Hutter et al., 201 I) with TPE as the sampling algorithm (Bergstra et al., 2011), to find the best combination of hyperparam- eter values for given machine learning model. For further details on preprocessing and hyperparame- ter tuning, please refer to Appendix A.
Furthermore, when examining the performance of the model that takes context into account against the model that only considers the statement, it is ob- served that the former yields superior results (see Table 2). This suggests that additional informa- tion present in the context is erucial for prediction. This conclusion aligns with the findings of similar task in news articles (Ghanem et al., 2021). )). See Appendix A.3 for full feature importance analysis.
Metadata-based Models We examine the per- formance of a model based solely metadata and semantic features, and analyze the combination between these features and the strong linguistic baseline. The Feature list and the preprocessing steps are described in Appendix A.4.
We outline our empirical results in Table 1. We observe that in most dataset setups, the metadata models perform better than their counterparts. In the TF Bins setup. the majority baseline outper- forms. We hypothesize that the binning technique might have an adverse impact on the data and fea- ture distribution, affecting the results. Both the metadata feature-based model and the combined model obtain significant improvements and outper- form the majority baseline in other setups, indi- cating that substantial information correlated with claim credibility exists in those features.
4.3 Models
Additionally, the combined model achieved bet- ter results than the metadata-only model. Analysis of feature importance using Shapley values (SHAP- LEY, 1953), shows that the top features in the all- features metadata model are: media source, party, Knesset, gender (objective features), and field (Se-
Strong Linguistic Baseline Drawing from our hypothesis that linguistic cues vary with the credi- bility of a claim, we engineered linguistic features, based on previous works, from both the statement and its full context, elaborated in Appendix A.

---

mantic features). The inclusion of semantic fea- tures improves performance significantly.
contextualizee--asse model with a transformer ar- chitecture. We utilized the state-of-the-at Aleph- BERT (Seker et al., 2021) which was pre-trained on 17.6G of Hebrew text and we fine-tuned it on our task.
Finally, we examine the effect of integrating metadata with hand-crafted linguistic features. We found that results were comparable. This was some- what surprising and it suggests a connection be- tween the linguistic and metadata features. Further research is needed to better understand this rela- tionship. Table summarizes all results including the comparison Majority baseline.
Model 3: Attention-based Evidence Aggregator (Attn-EA) Building upon the attention- bassed ev- idence aggregation (Attn-EA) model proposed by Gupta and Srikumar (2021) significant work identified by Guo et al. (2022) we made refine- ments to the evidence collection pipeline. Our vised attention-based evidence aggregation model, aggregates evidence from the top five Google search snippets associated with each claim. To re- duce potential bias, we curated an 'Excluded Web- sites' list, which filters out the original source of the claim or any websites that disclose the claim's 's ve- racity, such as "The Whistle". Moreover, snippets containing more than 85% of the original claim are omitted. We employed AlephBERT (Augenstein al., 2019) separately encode the claim and each piece of evidence, extracting the output of the CLS token: c, [c1, 62, en)- The dot-product attention mechanism (Luong et al., 2015) then applied to compute attention weights [an, Q22 .... On. and corresponding linear combination: e = E, The combined representation is subsequently con- catenated with and fed to the classification layer.
4.4 Deep Learning Models
Manual extraction of linguistic features might be a demanding task. In addition, such methodol- ogy might not take into account hidden patterss text, and such patterns may be unknown to the researchers or hard to manually extract.
Hence, recent works on credibility assessment and fake news detection exploited various deep neural architectures. In this work, we explore deep learning methods and incorporate recent ad- vancements in the field including state-of-the-as transformers-basee contextualized word embed- dings in Hebrew (Seker et al., 2021). Additionally, we suggest a novel architecture that combines both evidence and context to enhance the performance on our task.
We conducted experiments in two setups: Statement-Foous and Context-Aware. For each setup we have conducted several experiments as will be described below. We ran our experiments with the Huggingface transformers library Wolf et al. (2020) available under an Apache-2.0 license. For all models, we used the publicly available base checkpoints on Huggingface.
4,4.2 Context-Aware Models
The second category of our models incorporates the full context into the analysis. In this setup, we developed and evaluated three models.
Model 1: RNN with Fest and Full Context This model uses the same RNN-based architecture as the statement-focus setup but incorporates the full context into the analysis.
4.4.1 Statement-Focus Models
The first category of our models focuses only on the statements. In this setup, we implemented and compared three models.
Context-Based Model (CBM) We propose model architecture that leverages the power of the AlephBERT model for text under- standing and incorporates an attention mechanism to capture the joint interaction between the state- ment and its context. Given statement S and its context C, we tokenize and encode them using Ale- phBERT, yielding embeddings Og and Oc- We then employ an attention mechanism, with Oş the query and Oc the key and value matrices. The attention mechanism computes the attention weights and generates the merged representation Osc- This merged representation captures the con-
Model 2:
Model 1: RNN with Static Word Embeddings The first model is based on RNNs using static word embeddings. We experimented with Word2Vec (Mikolov et al., 2013) and fastText (Bojanowski al., 2017) embeddings, and selected fastText due to its superior performance on our task.
Model 2: AlephBERT with a Transformer Ar- chitecture. To capture more complex patterns and dependencies the data, we employed a
"httos://hugginggfe co/models

---

Full Spectrum
Only
TF Bins
FS Written
Majority
11.0 + 1.2
43.0 # 3.0
47.0 3.3
10.0 1.1
Metadata only
21.1 1.8
51.6 + 3.6
46.9 3.3
21.9 + 1.9
Meranita+Semmntic
22.5 + 1.9
54.7 # 3.8
46.6 3.3
22.6 + 2.2.
Linguistic
25.0 + 1.8
60.0 + 3.4
51.0 2.8
30.0 +
Linguisti++Mettaattt-Seeee
25.4 1.9 60.7 + 3.5
50.3 2.7
31.0 + 2.2
Table 1: Comparison of the Fl scores with standard deviations for several models including: Metadata, Meta- datanSemantic, Linguistic (Hand-Crafted Features), and also included for comparison. All models were tested on the HeTTree dataset.
Majority baseline is
Model
Full Spectrum
TF Only
TF Bins
FS Written
Linguistic + M/S
25.4 + 1.9
60.7 3.5
50.3 2.7
31.0 1
Statement- Focus
RNN fastText
29.6 + 2.3
65.2 3.4
56.7 2.7
34.2 2.3
AlephBert-based
44.0 3.0
72.0 4.1
63.0 3.5
50.0 4: 3.0
Attn-EA
47.8 3.2
73.5 + 4.2
64.9 3.6
52.7 + 3.2
Linguistic + M/S
27.4 2.2
61.3 3.3
49.6 2.5
31.3 4: 2.2
Context-Aware
RNN fastText
31.2 -
66.1 3.5
58.3 = 2.8
44.1 2.4
CBM
46.0 + 3.1
71.0 + 4.0
64.0 + 3.5
53.0 + 3.1
CCEM
48.3 3.2
72.8 4.3
66.7 3.7
54.5 + 3.2
Table 2: The table presents Fl scores for four linguistic models: the hand-crafted feature model, RNN+fas(Text AlephBERT, and seleence-based in two scenarios "Statement only' and "Full context'. Standard deviations are also reported as subscripts
scores.
textual interactions between the statement and its context. It is then passed through fully connected layer with weights W and bias b to produce the output Z. Finally, the class probabilities P are obtained by applying the softmax function Z.
fused representation OSCE that encapsulates the interactions between the statement, its context, and the external evidence. Finally, the class probabil- ities P are extracted from OSCE using softmax function.
4.4.3 Results
Combined Context and Evidence Model (CCEM) Recent studies showed the importance of evidences and context to claim- assessments tasks. However, models doing so not exist. The Combined Context and Evidence Model (CCEM) architecture leverage the princi- ples of both the Context-Based Model (CBM) and the Attention-based Evidence Aggregator. Initially, similar to the CBM, forms merged context- statement representation Osc using the statement and its context C. Concurrently, applies the evidence aggregation approach of the Attention-based Evidence Aggregator to obtain a combined evidence representation E, derived from the top Google search snippets associated with the claim. The representations Osc and E' are then processed through transformer-based integration layer. This layer uses Osc and E inputs, con- ducting a multi-head self-attention operation. The attention mechanism considers Osc as the query and as the key and value matrices, resulting in
Model 3:
Table highlights several significant patterns and key findings from experiments.
One primary observation from our experiments is the consistent outperformance of deep learning models, compared to simpler hand-crafted feature- based models. The latter approaches do highlight some useful features for this task and manage surpass number of baseline models, indicating that there are explicit features that can be used for Hebrew credibility assessment task. However. the more complex architectures, including models with static word embeddings and transformer-based models with contextualized embedding, prove to be more effective and deliver better performance overall. A surprising result is the improvement in performance exhibited by the "Evidence-based model in the 'Statement only" setup. This model surpasses the performance of the advanced Ale- phBERT model, even with the CBM architecture, highlighting the significant impact of integrating
model

---

external evidence aggregation on model effective- ness.
methodologies introduced in this study. Finally, the HeTrue dataset, with its unique combination of fea- tures, holds potential for broader range of NLP applications such as stance classification, argument mining, topic modeling, and rumor detection. This broad utility further underscores the value of our contribution and opens new pathways towards more sophisticated fake news detection methodologies.
In the Context-Aware setup, the 'Combined Con- text and Evidence Model' (CCEM) marginally out- performs other models, highlighting the effective- ness of utilizing both full context and external ev- idence. The RNN + fastText models also demon- strated substantial improvement when transitioning from "Statement only' 'Full context'. This en- hancement emphasizes the impact of context in tex- tual understanding, pointing out the limitations of models that focus solely on individual statements. Interestingly, slouueent-focused AlephBERT outperforms the Context-Aware RNN model. This could be attributed to AlephBERT's advanced pre- training, which enhances its capabilities in natural language understanding, generalization, and pos- sibly even in encoding writing styles. Throughout the work, we observe that written statements con- sistently lead in terms of performance. This trend strengthens the hypothesis that there are linguistic and other cues not faithfully captured or altered during text transcription.
Limitations
The dataset is curated by Israeli professional jour- nalists who carefully analyze claims made by pub- lic figures and politicians Hebrew media. How- ever, as stated in the paper, some of the claims are written by the public figures themselves, and others are transcribed by "The Whistle" journal- ists. This could lead to potential nuances in the way the claims are transcribed, which could influ- the results of the analysis. One might assume that the transcription is biased towards the outcome of the claim's 's truthfulness. To address this issue, we created the written-only setup that evaluates statements that are originally from written media type, such as Facebook, Twitter or opinion articles. Additionally, the dataset used may reflect some col- lection bias, as the journalists collected the claims that they view as most interesting to the public. This may result in the collection of statements that use provocative language, are false, or have other controversial characteristics. This may limit the generalizability of the findings for naturally occur- ring texts. Future work could focus on discerning the effects of transcription nuances and overcoming potential biases.
5 Conclusions
Our study introduces HeTrue, a unique and rich dataset designed for automatic credibility assess- ment Hebrew. This dataset stands out not only as the first of its kind in Hebrew, but also as the first to incorporate comprehensive set of features in- cluding the statement context, the statement itself, the journalist's version. and additional semantic and metadata features annotated by professional journalists in a rigours process.
The findings from our experiments underscore the effectiveness of linguistic approaches in the challenging task of credibility assessment. Our results consistently demonstrate enhanced perfor- mance when context is integrated into the mod- els across various setups. Furthermore, show that incorporating evidence-retrieval mechanisms alongside context further contributes to the models' performance.
Ethics Statement
The automatic detection of fake news is com- plex task that raises important ethical considera- tions. One of the key concerns is the risk of using claimants profiling, which could lead to the stigma- tization of certain individuals or groups based on their data, such as their gender or political party. Furthermore, there is a risk of bias towards certain ways of expression, thus, linguistic approaches also need to be carefully considered to avoid any form of bias. Additionally, the use of automatic detcc- tion systems raises questions about accountability and transparency. The models used in this task can be opaque and difficult to interpret, making challenging to understand how and why certain decisions are being made. This can make it difficult
Despite the advancements made, credibility as- sessment, particularly in morphologically rich lan- guages like Hebrew, remains a challenging task. While pre-trained language models like Aleph- BERT show promise in effectively understanding and generalizing language even with limited con- text, the quest for better accuracy continues. Future research will aim to refine and further improve the

---

identify and correct errors, and can also make difficult to hold those responsible accountable for any negative consequences that may arise from the use of these systems. These concerns are amplified in today's s information landscape where data can be created or manipulated by Generative Artificial In- telligence, further complicating the discernment of facts from fiction and truths from lies. Given these ethical considerations, we urge careful handling of automatic credibility assessment and encourage inclusive, responsible system development.
academic community unite in condemnation of these unspeakable atrocities committed by Hamas, who claim to be acting in the name of the Pales- tinian people. We call all to join us in advocating for the prompt and safe return of the abductees, as we stand together in the pursuit of justice and peace.
This paper finalized in the wake of these events, under great stress while we grieve and mour.. may not be as polished as we would like it to be.
8 Broader Impact
Acknowledgments
This paper is submitted in the wake of tragic terrorist attack perpetrated by Hamas, which has left our nation profoundly devastated.
We would like to thank "The Whistle" at "Globes" for their essential support creating the HeTrue dataset, serving as a foundational element of our paper and facilitating future research in this domain. This research was funded by the Israeli Ministry of Science and Technology (MOST) grant No. 3- 17992, and an Israeli Innovation Authority grant (IIA) KAMIN grant, for which we are grateful. In addition, This project has received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and inno- vation programme, grant agreement No. 677352.
On October 7, 2023, thousands of Palestinian terrorists infiltrated the Israeli border, launching a brutal assault on 22 Isracli villages. They methodi- cally moved from home home brutally torturing and murdering more than a thousand innocent lives, spanning from infants the elderly. In addition to this horrifying loss of life, hundreds of civilians were abducted and taken to Gaza. The families of these abductees have been left in agonizing uncer- tainly, as no information, not even the status of their loved ones, has been disclosed by Hamas. This is not fake new. This is first-hand evidence from family, friends, relatives and acquaintances, the ones that survived the horror.
References
Sadia Afroz, Michael Brennan, and Rachel Greenstadt. 2012. Detecting hoaxes. frauds, and deception in writing style online. In 2012 JEEE Symposium on Security and Privacy, pages 461-475 IEEE
The heinous acts committed during this attack, which include acts such as shootings, sexual as- saults, burnings, and beheadings, are beyond any justification.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD interna- tional conference on knowledge discovery data mining, pages 2623-2631.
Beyond the loss we suffered as a nation and as human beings due to this violence, many of us feel abandoned and betrayed by members of our research community who did not reach out and were even reluctant to publicly acknowledge the inhumanity and total immorality of these acts.
Isabelle Augenstein. Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Chris- tian Hansen, and Jakob Grue Simonsen. 2019. Mul- tife: A real-world multi-domain dataset for evidence- based fact checking of claims. the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 4685-4697.
This horrific situation highlights the devastating impact of misinformation, fake news and propa- ganda, and what they can inflict on society. this case, manipulated stories and misleading narratives contributed to justifying or diminishing the severity of Hamas' atrocities, underscoring the urgent need for improved methods in assessing the credibility of information, such as the methods advocated for and developed in this work.
Proceedings of
Ramy Baly, Mitra Mohtarami, and James Glass. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of NAACL-HLT, pages 21-27.
We fervently call for the immediate release of all those who have been taken hostage and urge the
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 201 Algorithms for hyper-parameter

---

optimization. Advances in neural information pro- cessing systems, 24.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos. 2022. A survey on automated fact- fact-checking. Transactions of the Association for Computational Linguistics, 10:178-206.
Piotr Bojanowski. Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the associa- rion for computational linguistics, 5:135-146.
Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact check- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 2: Short Papers), pages 675-682.
Jianlin Cheng, Zheng Wang, and Gianluca Pollastri. 2008. A neural network approach to ordinal regres- sion. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computa- tional intelligence,, pages 1279-1284. IEEE.
Yaakov HaCohen-Kerner, Rakefet Dilmon, Shimon Friedlich, and Daniel Nisim Cohen. 2015. Distin- guishing between true and false stories using various linguistic features. In Proceedings of the 29th Pa- cific Asia Conference on Language, Information and Posters, pages 186-186.
Nadia K Conroy, Victoria L Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proceedings of the association for information science and technology, 52(1):1-4.
Computation:
João Victor de Souza, Jorão Gomes Jr, Fernando Marques de Souza Filho, Alessandreia Marta de Oliveira Julio, Jairo Francisco de Souza, al. 2020. systematic mapping on automatic classification of fake news in social media. Social Network Analysis and Mining, 10(1):1-21.
Jeffrey T Hancock, Lauren E Curry, Saurabh Goorha, and Michael Woodworth. 2007. On lying and be- ing lied to: A linguistic analysis of deception computer-mediated communication. Discourse Pro- cesses, 45(:)11-
P Sol Hart and Erik C Nisbet. 2012. Boomerang effects in science communication: How motivated reason- ing and identity cues amplify opinion polarization about climate mitigation policies. Communication research, 39(6):701-723
R Dilmon. 2004. Linguistic differences between lie and truth in spoken hebrew. Doctoral diss., Bar Ilnn University, Ramat Gan, Israel.[in Hebrew].
Rakefet Dilmon. 2007. Fiction or fact? comparing true and untrue anecdotes. Balšanwt iybriyt, 59.
Naeemul Hassan, Bill Adair, James T Hamilton. Chengkai Li, Mark Tremayne, Jun Yang. and Cong Yu. 2015. The quest to automate fact-checking. In Proceedings of the 2015 Computation+ Journalism Symposium.
Rakefet Dilmon. 2013. False speech, linguistic aspects in hebrew. The Encyclopedia of Hebrew Language and Linguisties, Leiden, Brill, Boston and Tokyo, pages 542 546.
Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Jo- sue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al. 2017. Claimbuster: the first-ever end-to-end fact-checking system. Proceed- ings of the VLDB Endowment, 10( 1945-198888
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's s guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1383-1392.
Frank Hutter, Holger H Hoos, and Kevin Leyton- Broww. 2011. Sequential model-basee optimization for gen- eral algorithm configuration. In International confer- ence on learning and intelligent optimization, pages 507-523. Springer.
Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169-20)
JR Flesch. 1965. Flesch-kincaid readability formula.
Bruce Fraser. 1991. Questions of witness credibility. Harvard Law School.
Stephan Lewandowsky, Werner GK Stritzke, Klaus Oberauer, and Michael Morales. 2005. Memory for fact, fiction, and misinformation: The iraq war 2003. Psychological Science, 1603)5190-195.
Bilal Ghanem, Simone Paolo Ponzetto, Paolo Rosso, and Francisco Rangel. 2021. Fakeflow: Fake news detection by modeling the flow of affective informa- tion. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 679-689.
Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake news detection through multiipprrppective speaker profiles. In Proceedings of the eighth international joint conference on natural language processing (volume 2: Short papers), pages
Anastasia Giachanou, Paolo Rosso, and Fabio Crestani. 2019. Leveraging emotional signals for credibility detection. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pages 877-880.
Minh-Thang Luong, Hicu Pham, and Christopher D Manning. 2015. Effective approaches to attention- based neural machine translation. In Proceedings

---

of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsar- faty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew nip applica- tion with, arXiv preprint arXiv:2104.040) 04405.
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bemnar J Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks.
LLOYD SHAPLEY. 1953. A value for n-person games. contributions to the theory of games ii, kuhn, h., tucker, a.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint wrXiv:1301.3787.
James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and fu- ture directions. In Proceedings of the 27th Inter- national Conference on Computational Linguistics. pages 3346-3359 3359..
Tanushree Mitra and Eric Gilbert. 2015. Credbank: A large-scale social media corpus with associated credibility annotations. In Ninth international AAAI conference on web and social media.
Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What's wrong with hebrew nip? and how to make it right. In Proceedings the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNPP): System Demonstrations, pages 259-264.
Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019. Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and case study from modern Hebrew. Trans- actions of the Association for Computational Linguis- tics, 7:33-48.
UBOHO Victor. 2020. Robust semi-supervised learning for fake news detection. Ph.D. thesis, Prairie View A&M University Prairie View, TX, USA.
Jeff Z Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yang- mei Li, and Jinshuo Liu. 2018. Content based fake news detection using knowledge graphs. Inter- national semantic web conference, pages 663. Springer.
Marco Viviani and Gabriella Pasi. 2017. Credibility in social media: opinions. news, and health infor- mation-a a survey. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 7(5)re
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):201.
Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACI. 2014 workshop on lan- guage technologies and computational social science. pages 18-22.
Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018. Declare: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pages 22-32.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380): 1466::11166111 1151.
William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and polit- ical fact-checking, In Proceedings of the 2017 con- ference on empirical methods in natural language processing, pages 2931-2937.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2020. Transformers: State-of-the-aa natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38-45.
Julio CS Reis, André Correia, Fabrício Murai, Adriano Veloso, and Fabrício Benevenuto. 2019. Supervised learning for fake news detection. IEEE Intelligent Systems, 34(2):76-81.
Arjun Roy, Kingshuk Basak, Asif Ekbal, and Pushpak Bhattacharyya. 2018. A deep ensemble framework for fake news detection and classification. arXiv preprint arXiv:1811.04670
Lina Zhou, Judee K Burgoon, Jay F Nunamaker, and Doug Twitchell. 2004. Automating linguistics- based cues for detecting deception bastd asynchronous computer-mediated communications. Group decision and negotiation, 13(1):81-106.
Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. Csi: A hybrid deep model for fake news detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 797- 806.
Xinyi Zhou and Reza Zafarani. 2018. Fake news: A sur- vey of research, detection methods, and opportunities. arXiv preprint

---

A Appendix
Last names usage
Lexical features char based
A.1 Text preprocessing
Total number of chars
Before creating and extracting the linguistic fea- tures, prepare data for that. Cleans- ing(cleaning) the dataset correct corrupt or inac- curate records and extracting basic linguisties ele- ments that will be used later to create the features. To do so we used YAP (Yet Another More et al., 2019 developed by Reut Tsarfaty and Amir More. To preprocess the statements, the following stages were taken:
Digits percentage
Letters percentage
Bag of chars tf-idf
Syntactic features
Count of punctuation marks
Count of special chars including
@$%
Count per dependency parts depen- dencies between words (sub-obj for in- stance)
Removing irrelevant special characters
Removing double qoutes for acronyms
Count per POS
Removing Stop words (but save the original text separately)
Count per Person(first,seconndttti in a statement.
Tokenization for specific features that are token-based.
Count per verb tense (BEINONI, PAST, FUTURE, IMPERATIVE).
Stammer for specific features that are stem- bases.
Structural features
Number of inner quoutes
Lemmatizer for specific features that lemma-based.
Hashtags count
Semantic features
In addition we used YAP to extract the following information to later use for feature extraction:
Positive, negative, and objective seman- tic based on Bert based model built on tagged tweets (Twitter).
POS tagging
Dependencies parts extractions
We extracted these features on both claim and context.
Grammatical person
Verbs tense
A.3 Strong Linguistic Baseline Features Analysis
A.2 Linguistic features extraction
We developed set of hand-crafted linguistic fea- tures based on previous studies that have indicated a correlation between these features and the psycho- logical state of claimants. This psychological state can be used to infer the truthfulness of a claimant statement, as is reflected in the language of the claim. The features are divided into the following categories:
We conducted a feature importance analysis us- ing Shapley values on our best XGBoost model after optimization using the nested cross validation approach. The key finding are as follows. First, false statements tend to make greater use of relative clauses and punctuation marks, particularly com- mas. Second, true statements tend have more positive sentiment and make greater use of comple- ment clauses and inseparable prepositions. The use of definite markers is more prevalent in false state- ments. Next, true statements tend to make use words with greater complexity as measured by the Flesch-Kincaid index (Flesch, 1965). In addition, the overall statement length was not found to be significant factor in determining truthfulness.
Lexical features word based
Bow tf-idf
Complexity measure
Readability measure
Average word length
Count of H' hayedia
Count of distinct words
A.4 Metadata features
Count of words
Count of syllables
The dataset we curated from "The Whistle" con- tains in addition to the claim additional metadata features. These features can be divided to semantic
Usage of uncertainty words
First names usage

---

features and objective features. The objective fea- tures are: date, speaker, role, party, media, knesset and gender. The semantic features are: subject, title, field, and tags.
Label
# Claims
True
86
Mostly true
102
Half true
122
Mostly False
252
Metadata preprocessing One of the "metadata features" R is 'media'. The 'media' feature describes the platform in which the claim was produced. The way "The Whistle" journalists described the plat- form is in the most granular level. For example, claim produced in radio station called "GLTZ" in radio program called "Nahon Le-Haboker", was described in the 'media' feature as "Nahon Le- Haboker GLTZ". We hypothesis that adding the type of media, the media source and the producer type(spoken/written) will improve the accuracy of our predictions. This is because different types of media can have different effects people. For ex- ample, video YouTube may have a different ef- fect than news article website. Additionally, different media sources can have different biases, which may influence how an individual interprets and reacts to piece of media.
False
343
Unverifiable
88
Other
28
Table 3: Label distribution in HeTTrue dataset
as the majority class. We finally chose to upsample only the True score statements. We chose if use the original statement with or without stopwords, and chose without. For the full context as input we chose with stopwords(original) as it performed the best.
For the Deep learning models, we experiment with two variations: classification approach using softmax in the final layer and ordinal regression approach using the method that was introduced by (Cheng et al., 2008) leveraging the ordinal charac- teristic of our labels.
A.5 Hyperparameter Optimization
For the RNN model we examined both word2vec and TastText and we chose fastText. Also, we com- pare the performance of GRU LSTM and we chose GRU. The architecture chose is: Dropout 0.5 after the embedding layer. than GRU layer with dim=8. Than dropout of 0.2 and sigmoid as activa- tion function for the final layer (for ordinal model) and softmax (for classification model). The batch size chosen is 32 and number of epochs 10. We also used the same upsampling method desribed above. Here, also considered the use of stopwords, and the decision was the same as above.
As stated in the training setup section, we employed nested cross validation to evaluate the performance of our models and to choose the best hyperparam- eters. For the hand-crafted feature model the metadata models and the integrated (metadata + hand-crafted features) models. We used XGBoost the classification model and we searched among the following parameters: we searched booster in dart .bbtree .gblinear and chose gbtree. We searched lambda in le-8, 1.0 and chose 6.421 We searched alpha in le-8, 1.0 and chose 0.0008. We searched min_chi _d__ld__eeeee wit in 1-10 and chose 6. We searched subsample in 0.01 1.0 and chose 0.385. We searched cols bolsample _eeeeee in 0.01 1.0 and chose 0.4487. We searched max_ _ee__depthhh in [1-9] and chose 5. We searched eta [le-8 1.0] and chose 0.7542. We searched gamma in [1e-8 1.0] and chose 1.005 We searched grow_policn policy in "depthwise", "lossguide" and chose lossguide. Also we implemented feature selection using sklearn SelectKBest method. We search k in [100- 8000] and chose 2127. Also, we examine the use of up-sampling method overcome the unbalance of our data. We chose hetween upsampling the True score statements, the True score statements + Mostly true statements or upsampling all to count
For the AlephBERT model we searched the learning rate in le-6. 5e-5, le-5 and chose with le-4. We experiment two architectures two fully connected layers on top of pooled out- put of the pre-train AlephBERT and the other is one fully connected layer. Both with dropout layer and ReLU as activation function. We finally chose : dropout 0.5 (searched in [0-0.5] and one fully connected layer. We searched the batch size in 16, 32, 64 and chose 8. To optimize performance we used gradient accumulation for total batch size of 16. We searched weight decay in 0.1 and chose le-4. We also used the same upsampling method desribed above. Here, we also considered the use of stopwords, and the decision was the same as above.
with

---

A.6 HeTrue Dataset Features and Definitions
A.8 Dataset Example
An example from the HeTrue dataset can be found in Table 5 below.
Dataset label distribution can be found in table table 3 Feature list of HeTrre dataset described below.
Journalist-edited claim The claim as appeared on "The Whistle" website. Sometimes the claim is being transformed by the website edi- tors for various reasons.
Original claim The claim produced by the claimant and appeared in the original media.
Claim Full context The original claim and the surrounding sentences, up to two sentences
before and after the claim.
Claim date The claim's producing date.
Claimant The person full name who made the
claim.
Gender Male/Female.
Role The claimant's 's role.
Party The claimant's political party.
Field General subject area or discipline that the
claim relates to.
Subject Specific topic that the claim is about.
Title A brief, descriptive phrase that summarizes
the main idea of the claim.
Media The media in which claim is produced.
Knesset# The Knesset number in which the claimant is part of.
Tags Added by "The Whistle" journalists label and categorize the claims.
Label "Truth score" given by the "The Whistle" journalists.
A.7 Comparison of Credibility Assessment Datasets
See Table 4 for comprehensive comparison of credibility assessment datasets, focusing on sentence-level claims. For each dataset, we present the number of instances, the language, the number labels and the source of the dataset. Addition- ally, we highlight the comprehensiveness of each dataset in terms of additional features: semantic annotation, metadata features, and context. Se- mantic annotations pertain to features extracted by professional journalists. Metadata features cover attributes related to the claim and claimant. Con- text encompasses surrounding text relevant to the statement. HeTrue stands out as the sole dataset integrating all three of these additional features.

---

Dataset
Rlnpets
Semantic Annot.
Metadata
Contest
Ciasses
Sourees
Lang
CrineVett (Bachenko al., 2008)
275
NIA
NIA
NA
Crime
En
Politifact (Vlachos and Rledel, 2014)
N/A
Yes
N/A
Check
StatPPropettie (Vlachos and Riedel, 2015)
7,092
N/A
N/A
N/A
Numeric
Internet
En
Emnergent (Ferreira and Vlacbos,
Emergent
CrediAAssess (Popar et ol., 2016)
5,013
N/A
N/A
NAA
2
Fact Chock/Wili
En
PunditFact (Rashkin al., 2017)
4,361
NIA
N/A
N/A
2/6
Fact Check
En
Lar (Wang. 2017)
12,836
N/A
Yes
NAA
6
Fact Check
En
Verify (Baly (l. 2018)
422
N/A
N/A
N'A
2
Fact Check
Anfti
CheckThati 8-T2 (Ramon-Code
2018)
N/A
Transcript
Snopes (ILanellowski et 2019)
2019)
6,422
N/A
YES
N/A
3
Fact Check
En
MuluHC (Augenstcin
Yos
Yes
Check
Climate- FEVER (Diggelmann et al., 2020)
1,535
N/A
NAA
NAA
Climate
En
SeiFact (Waslen al. 2020)
1,409
NIA
NA
NA
Science
En
PUBHEALTH (Kotoeya and Tonl, 20206)
11,832
Yes
N/A
COVID-RRet (Saakyan al,, 2021)
4,086
N/A
N/A
N/A
Forum
En
FaFtct (Gupta and Srikumar,
N/A
Many
Hefroe
1021
Yes
Yes
Yes
7
Faet Check
He
Table 4: Overview of datasets for credibility assessment, focused on individual statements.
Edited claim
DYUINTO 25%6-2 pm D'WIN MTDD mNb7 1200 Emnnuum manh muntiNn b'ow2 ww"w 1721p
In recent years, the parole boards have given fewer and fewer people a third, only about 25% of people received a third.
Original claim
-3 """t brUIR mhon mnbh urd b'IMuuh MYT MarTAsh biwwo hith 7 MN 7 7TH "OR "m pn22 177 17nnant phnnn, whhw t2p broinbo 25% .D'TONT muo n8 EYUN? 7Nun 1 pn T 1971 roNT 7u what wan You should know that in recent years the parole boards have given fewer and fewer people third, only about 25% of the people received third, and part of solution, the High Court of Justice said that the living conditions of the prisoners should be increased, and therefore the number of prisoners should also, among other things, be reduced.
Full context
whiw namta brmmim MYMMEDU Mrupra an buuetr mhath tn thrt 12 1n to Nosuth TTu 877 mmbroorn mano mumann mith broing brmmoumpp 'IFT poиnrr ITT1 'Dn 51ato TOx to 19hn pina D'Nitt Ntw bnnnk tom hD5 MMN tun wrbe 17 noi 7T7 IN 1o1b K Ki EN na1o than mnmann ENU Kin TIN tmpppia 878, momph Thoo hown 87 128 who to namo momm DN moth breIN mino mnbh 1211 bmnmou m1mt mammmet buwww norh Thu 10111 "Tow TIN th 7 127 thneet phmnnn wubt tharp treaNn 25%6o p77 whhr MMO MR DSN3? 7NUT 1 7T'm 1oin .OYTON 'r monttm wanh 28 7mant
I both Ministry Justice and the media when parole board gives a third, the criteria should be good and dangerous behaviour, and there no need to invent kinds other criteria are not in law, every prisoner who comes before parole board, opinion that if his behaviour was good and if he not dangerous, then be should To give him third, and even more so the committee gives him third, don't think necessary appeal, except very extreme cases. should know that recent years the parole boards have given fewer and fewer people third, only sbout 25% of the people received third, and as part of our solution, the High Court Justice said that living conditions the prisomers should be increased, and therefore the number of prisoners should also, among other thines, be reduced.
Claim date
06.05.2018
Claimant
Ayelet Shaked
Gender
Female
Role
Minister of Justice
Party
Yamina
Field
Law and Order / tow/t p1n
Subject
Release of prisoners /b"TON "17Mt
Title
the approval of early releases for prisoners p'7row> pronp1o pmnin "1n"
Media
Daalman-Lieberann Kan-Bet/ 1B -1pna"7-po7p
Knesset##
The 20th Knessett
20-7 no127
Tags
Ayelet Shaked, prisons /mmb vna, v1,,7p ni""
Label
True
Table 5: Example from HeTrue dataset. For reference, translations also shown


Reviews:
[{"review_id": "Jhwgp10hRC", "reviewer": "Reviewer_uzQB", "summary": "", "questions": "A. Not clear how the data has been gathered from The Whistle - is it just scraped / is the dataset released together with the Whistle?\n\nB. I would have liked a more focused related work section also describing similar dataset for other non-English / underrepresented languages. Also, the context and evidence based methods are very shallowly presented in the related work. Can you tackle this points in the final version? From my side, you can remove some more generic parts about using network analysis for fact checking as they are completely unrelated to the current work. \n\nC. I would have liked to see some simpler baselines for using context and evidence, e.g. just concatenating the two with a <SEP> token or another special token. Have you tried any of these simpler experiments? This would justify that the proposed Attn-EA, CBM, CCEM models add value compared to a simpler method.\n\nD. Can you provide an example of statement, evidence and context in the Appendix? \n\nE. How do you know that the context and evidence do not contain any information about the actual credibility label for the statement? Are these manually inspected? \n\nF. How where the statements in the HeTrue dataset selected?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincere gratitude to the reviewers for their meticulous evaluation and constructive feedback. We deeply value the insights provided, which have offered us the opportunity to enhance the quality and clarity of our work. In this response, we have systematically addressed each of the concerns raised and have made corresponding revisions to our manuscript where necessary.\n\nWe've carefully reviewed each query and concern and have diligently addressed them in our response.\n\n**Reviewer's Concern A :Clarity on data gathering from The Whistle**\n\nWe regret that our original manuscript did not clearly explain how the data from \"The Whistle\" was gathered. We have revised the paper to provide a more detailed explanation.\n\nWe didn't simply scrape data from \"The Whistle.\" Instead, we formed a direct collaboration. Our partnership ensured the integrity of data and alignment with scientific research standards. \"The Whistle\" made necessary adjustments during collection and annotation. As highlighted in the \"HeTrue: a New Benchmark Dataset for Hebrew Credibility Assessment\" section, their journalists manually curated the HeTrue dataset over years. Each claim was annotated for metadata, semantic features (see full list of features in Appendix), and its full context. Claims were also refined by \"The Whistle\" for user clarity (“Edited claim”).\n\nFor the claim credibility score, we co-designed with \"The Whistle\" a framework ensuring compatibility with both scientific and journalism standards. The annotation process can be summarized as:\n\n* **Multi-level Review:** As outlined in our paper, we used a rigorous three-stage inter-annotator agreement procedure. Statements were examined by two journalists. Discrepancies prompted a third journalist's review, curbing individual bias\n\n* **Diverse Annotation Team:** Journalists from varied backgrounds ensured factual and unbiased labels\n\n* **IFCN Guidelines:** The participating journalists strictly follow the IFCN guidelines, which uphold transparency, impartiality, and fairness in fact-checking.\n\n* **Nuanced Labeling:** We introduced a detailed 5-point truthfulness scale, complemented by 'Unverifiable' and 'Other' categories. This granularity captures the complexities of real-world statements, avoiding potential bias and oversimplification.\n\n* **Comparison with Established Methods:** Our labeling process aligns with methods employed in recent fact-checking studies, highlighting its consistency with prevailing community standards.\n\n* **Transparent Scoring Rationale:** To further reduce bias and ensure transparency, every claim's rationale for scoring is public and documented.\n\nMoreover, the dataset is released in conjunction with \"The Whistle,\" ensuring that users and researchers have access to both the original source and our refined data for comprehensive insights.\n\nThough our rigorous process yielded a smaller dataset, it ensures the highest quality and reliability.\n\nLastly, all text-based features were double-checked for accurate transcription. The semantic features benefited from a two-tier verification process involving both a \"The Whistle\" journalist and an editor. We trust this elucidation sufficiently addresses your concerns.\n\n\n\n**Reviewer's Concern B:** The reviewer emphasized the need for a more focused related work section, highlighting similar datasets for non-English or underrepresented languages. Additionally, they indicated that context and evidence-based methods were not detailed adequately. A recommendation was also made to remove generic parts concerning network analysis for fact-checking, as it didn't align closely with the current research.\n\n**Our Response**\nWe truly appreciate the reviewer's suggestions regarding the related work section. We have undertaken the following improvements:\n\n* **Datasets for Underrepresented Languages:** While English remains the predominant language for fake news detection datasets, other languages are often underrepresented. As observed, most efforts to date, such as those by Vlachos and Riedel (2014) and Wang (2017), have extracted real-world claims from dedicated English-based websites like Politifact. \n\nHowever, Gupta and Srikumar (2021) are notable exceptions, having curated claims from 25 languages. Their multilingual baselines, including models like Claim Only, Attention-based Evidence Aggregator, and Augmenting Metadata Model, have established a promising foundation for multilingual fact-checking. Another commendable initiative is by Baly in “Integrating Stance Detection and Fact Checking in a Unified Corpus” (2018), who compiled a dataset of 219 Arabic statements. Both these studies employed Google's evidence retrieval to bolster claim veracity modeling. Yet, their scope appears limited, especially in terms of integrating context and a rich set of metadata/semantic features.\n\nNo Hebrew-specific dataset for fact verification or fake news detection existed until our contribution focusing on real-life statements. \n\n\n* **Context-based Methods:** Context plays a pivotal role in understanding and verifying claims. Few datasets like those by Mitra and Gilbert (2015) and Ma et al. (2016) included claim context, often sourcing from related threads. Since an individual post may contain limited context, these works represent each claim by a set of relevant posts, for example, the thread they originate from. Another unique approach is from Ghanem in \"FakeFlow: Fake News Detection by Modeling the Flow of Affective Information,\" which considers an entire article's segments for information flow. Building on the hypothesis that text interplay is vital, our work is pioneering in presenting sentence-level claims made by public figures, complemented by context.  \n\nWhile our contribution heralds a significant leap in Hebrew-centric datasets for fact verification, it's not confined to that. As our wide-range of experiments and the introduction of empirical evaluation of novel models tailored explicitly for this task: Context-Based Model (CBM), Attention-based Evidence Aggregator (Attn-AE) and Context and Evidence Model (CCEM).\n\nA comprehensive comparison of our dataset against previous works can be found in Table 4 in the appendix.\n\nWe hope that these revisions address the reviewer's concerns, and we genuinely thank them for their insightful feedback, which has been instrumental in refining our work.\n\n\n\n**Reviewer's Concern C:**\n I would have liked to see some simpler baselines for using context and evidence, e.g. just concatenating the two with a <SEP> token or another special token. Have you tried any of these simpler experiments? This would justify that the proposed Attn-EA, CBM, CCEM models add value compared to a simpler method.\n\n**Our Response**\nThank you for highlighting the importance of foundational methodologies. Due to space constraints and our emphasis on advanced models like Attn-EA, CBM, and CCEM, we limited our discussion on simpler methods. However, post-review, we recognize the value of elaborating on these methods, and we will include them in the final version:\n\nBeyond the Attn-EA, CBM, and CCEM models, we have ventured into more simpler architectures for our context-based models. Specifically, we've also explored: \n* **“Linguistic + Metadata Model”:** This model factors in linguistic intricacies and metadata related to context and evidence. The linguistic features were carefully designed based on past works. \n* **“RNN + FastText:** This approach encapsulated the context, utilizing the power RNN with FastText embeddings.\nThe results of which are presented in Table 2. \n\nRegarding simpler methods for amalgamating both evidence and context, our experiments encompassed:\n\n**TF-IDF Vectorization**: Applied TF-IDF to represent the significance of terms in context and evidence. Performance metric: F1 score of 26.2 ± 1.6\n**Direct Concatenation:** Merged context and evidence without the introduction of special tokens. Performance metric: F1 score of 34.3 ± 1.4\n**Sentence Embeddings with a <SEP> Token:** Implemented fixed-size embeddings for the combined representation. Performance metric: F1 score of 44.3 ± 2.4\n\nWe trust that these additional insights and experiments, clarify the nuances and value of our proposed models in contrast to the foundational methods. We're genuinely grateful to the reviewer for pointing out this aspect, as it has enabled us to present a more rounded perspective on our research.\n\n\n\n**Reviewer's Concern D:**\n Can you provide an example of statement, evidence and context in the Appendix?\n\n**Our Response**\nIn response to the reviewer request for a specific example of a statement, evidence, and context from the dataset, we provide the following example translated from Hebrew to English:\n\nOriginal Claim: \"You should know that in recent years the parole boards have given fewer and fewer people a third, only about 25% of the people received a third, and as part of our solution, the High Court of Justice said that the living conditions of the prisoners should be increased, and therefore the number of prisoners should also, among other things, be reduced.\"\n\nEdited Claim: \"In recent years, the parole boards have given fewer and fewer people a third, only about 25% of the people received a third.\"\n\nEvidence Snippet: \"The review of the characteristics of the cases and the decisions that were made opens for the first time a window into the work of the parole boards in Israel, which allows for an in-depth study of the results of the proceedings and the reasons that guide the...\"\n\nClaim Full Context: \"I told both the Ministry of Justice and the media that when the parole board gives a third, the criteria should be good and dangerous behavior, and there is no need to invent all kinds of other criteria that are not in the law [...] You should know that in recent years the parole boards have given fewer and fewer people a third, only about 25% of the people received a third, and as part of our solution, the High Court of Justice said that the living conditions of the prisoners should be increased, and therefore the number of prisoners should also, among other things, be reduced.\"\n\n\nWe hope this example provides clarity on the structure and depth of the dataset entries.\n\nThe example in the paper currently excludes this evidence snippet, and we acknowledge the oversight. We will ensure that the \"Evidence Snippet\" section is incorporated in the revised version of the paper for better clarity and comprehensiveness.\n\nThank you for your valuable feedback, which will aid in enhancing the quality of the paper.\n\n\n**Reviewer's Concern E:**\nHow do you know that the context and evidence do not contain any information about the actual credibility label for the statement? Are these manually inspected?\n\n\n**Our Response**\n\nIn response to your query regarding the potential inclusion of credibility labels within the context and evidence:\n\nWe have rigorously manually inspected the evidence to ensure the absence of any explicit credibility labels. Through this meticulous process, we identified and addressed instances with problematic evidence. To systematize this and ensure reproducibility without intensive manual oversight, we adopted a two-step procedure(also mentioned in the paper)\n\n1.Excluded Websites List: As articulated in the manuscript, to mitigate possible biases, we established an 'Excluded Websites' list. This list filters out sources which might inadvertently reveal the claim's veracity, including the original claim source and websites like \"The Whistle\".\n\n2.Content Overlap Restriction: For added assurance, any snippets with content overlap exceeding 85% of the original claim are systematically discarded.\n\nIt's crucial to emphasize that explicit references to the credibility level are largely confined to \"The Whistle\" or citations thereof. This makes the exclusion process straightforward. Moreover, other sources that implicitly discuss the claim's credibility — identified by their content overlap with the statement — are also eliminated.\n \nIn light of your feedback, we will be sure to emphasize these points more prominently in the final version of our manuscript. We hope this provides clarity on the steps we've taken to maintain the purity and impartiality of our dataset, ensuring its value and reliability for research.\n\n\n\n**Reviewer's Concern F:**\nHow were the statements in the HeTrue dataset selected? \n\n**Our Response**\n\nThank you for your inquiry regarding the HeTrue dataset's statement selection process. The statements in the HeTrue dataset were principally curated by professional journalists from \"The Whistle.\" Their selection was based on a myriad of criteria, encompassing the statement's potential societal impact, its alignment with current events, and the prominence of the statement's author. To ensure impartiality and objectivity in our dataset, we, in close collaboration with “The Whistle”, diligently worked to guarantee that the dataset was free from biases (explained in detail above). We emphasized a balanced representation across factors such as political efficacy, gender, party allegiance, among others, to uphold the dataset's integrity.\n\n\n\n**Reproducibility**\n\nWe sincerely appreciate your feedback regarding reproducibility concerns. The integrity and transparency of our research is of utmost importance, and we understand the value of ensuring our results can be easily replicated by the community. To address the concern you raised:\n\nIt is important to emphasise that both the dataset and the codebase will be made publicly available upon acceptance of the paper.\n\n\nFor all reported experimental results:\n\nWe have evaluated all our models across four dataset setups, detailed in Section 3. The experimental setup, encompassing both evaluation and training, is elaborated upon in Section 4.\n\n**Strong Linguistic Baseline:** Various models were examined, with Xgboost being selected based on hyperparameter optimization, as delineated in Appendix A.5. The feature extraction process involves text pre-processing steps, exhaustively described in Appendix A.1 and A.2. We extracted features from both the statement and its encompassing context.\n\n**Metadata-based Models:** Several models based on metadata were utilized. All feature extraction processes, inclusive of pre-processing, are meticulously detailed in Appendix A.4.\n\n**Deep Learning Models:** We have approached experiments in two distinct setups: Statement-Focus and Context-Aware. For each:\n\n- **Statement-Focus Models:** Three models were employed. This included the RNN with Static Word Embeddings utilizing fastText embeddings, selected for its exemplary performance post hyperparameter optimization (detailed in the appendix). Next is the AlephBERT model, a Hebrew BERT variant fortified with a Transformer architecture. It underwent pre-training on 17.6G of Hebrew text and fine-tuning for our specific task. The Attention-based Evidence Aggregator (Attn-EA) model leverages AlephBERT for encoding both claims and evidence, utilizing dot-product attention, as outlined in Section 4.4 (Full formula can be found there too).\n\n- **Context-Aware Models:** Our experiments encompassed the RNN with FastText and Full Context. The Context-Based Model (CBM) and the Combined Context and Evidence Model (CCEM) both employ AlephBERT. The latter integrates statement, context, and external evidence from Google search snippets for assessment. Comprehensive formulas are available in Section 4.4. Complete parameter settings are listed in Appendix A.5.\n\n**Source code** will be made publicly accessible upon acceptance, accompanied by precise specifications and dependencies.\n\n**Evaluation Measure/Statistics:** Metrics, including the macro F1 score, are elucidated in tables 1 and 2 of Section 4. We employ the bootstrap technique to ensure statistical robustness. Comprehensive details are presented in Section 4.2.\n\n\n\n**Regarding the dataset:**\n\n- **Dataset Statistics:** HeTrue encompasses 1021 statements with label distribution outlined in Table 3.\n- **Train/Validation/Test Splits:** We leveraged a nested cross-validation strategy with 5-folds in both inner and outer loops. The detailed approach and methodologies are documented in the appendix and Section 4.\n- **Data Exclusions and Pre-processing:** Only the labels \"Unverifiable\" and \"Other\" were omitted. All remaining pre-processing phases are exhaustively detailed in Section 3 and the appendix.\n- **Language:** The dataset is entirely in Hebrew.\n- **Dataset Link:** This will be shared upon paper acceptance.\n- **Data Collection Process:** Detailed in Section 3, including our collaboration with \"The Whistle\" and the three-stage inter-annotator agreement process.\n\nFor all models, we used the publicly available base checkpoints on Huggingface, with Pytorch 2.0.\nAll hyperparameters details for our models, described thoroughly in the appendix A.5. We have used Optuna library. \n\n\n\nWe hope this comprehensive response should adequately address reproducibility concerns.\n\n\n–\n\n*We're grateful for you bringing to our attention the typos and grammatical mistakes; they have been corrected in the final version of the manuscript.\n\n*All references cited in this response can be found in the bibliography section of our paper.\n\nWe deeply appreciate the time and dedication the reviewer has committed to evaluating our manuscript. We have endeavored to address each concern with thoroughness and rigor. It is our sincere hope that, upon reviewing our revisions and the effort we have put into addressing the feedback, the reviewer finds our submission to be significantly enhanced and more robust. Thank you once again for your invaluable insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* New dataset for text credibility assignment / fact checking in Hebrew. Dataset quality looks very good, and also provides information like metadata, context and evidence.\n* Good explanation for the experiments, nice integration of context and evidence in the final decision. ", "reasons_to_reject": "* Not clear how the data has been gathered from The Whistle - is it just scraped / is the dataset released together with the Whistle?\n* I would have liked a more focused related work section also describing similar dataset for other non-English / underrepresented languages. Also, the context and evidence based methods are very shallowly presented in the related work.\n* I often felt that some sections are longer and without added value to the paper - maybe a short paper format might have been better.\n* I would have liked to see some simpler baselines for using context and evidence, e.g. just concatenating the two with a <SEP> token or another special token .", "questions_to_authors": "A. Not clear how the data has been gathered from The Whistle - is it just scraped / is the dataset released together with the Whistle?\n\nB. I would have liked a more focused related work section also describing similar dataset for other non-English / underrepresented languages. Also, the context and evidence based methods are very shallowly presented in the related work. Can you tackle this points in the final version? From my side, you can remove some more generic parts about using network analysis for fact checking as they are completely unrelated to the current work. \n\nC. I would have liked to see some simpler baselines for using context and evidence, e.g. just concatenating the two with a <SEP> token or another special token. Have you tried any of these simpler experiments? This would justify that the proposed Attn-EA, CBM, CCEM models add value compared to a simpler method.\n\nD. Can you provide an example of statement, evidence and context in the Appendix? \n\nE. How do you know that the context and evidence do not contain any information about the actual credibility label for the statement? Are these manually inspected? \n\nF. How where the statements in the HeTrue dataset selected?", "paper_topic_and_main_contributions": "The paper propose a dataset, called HeTrue, for text credibility assignment / fact checking in Hebrew. The dataset contains information collected from the Whistle website [1], including some metadata about the source of the statement being investigated and some textual evidence about the credibility assignment decision. \n\nThe authors also use HeTrue for a series of experiments, both using feature engineering and standard ML and using AlephBERT-based solutions. The main contribution with regard to experiments is the usage of context and evidence together with the statement under consideration using some additional attention mechanisms (which are different for Attn-EA, CBM, CCEM models).\n\n[1] - https://thewhistle.globes.co.il/feed", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I would have liked a more focused related work section also describing similar dataset for other non-English / underrepresented languages. Also, the context and evidence based methods are very shallowly presented in the related work.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper might use an extra round of proofreading, e.g.\n\n- The former argue that => argues\n- This study use both => uses\n- For example Long et al. (2017) show => For example, Long ...\n- English has been focused on with much more annotated ... => rephrase"}}, {"review_id": "nWxCCFdpHw", "reviewer": "Reviewer_UQD5", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincere gratitude to the reviewers for their meticulous evaluation and constructive feedback. We deeply value the insights provided, which have offered us the opportunity to enhance the quality and clarity of our work. In this response, we have systematically addressed each of the concerns raised and have made corresponding revisions to our manuscript where necessary.\n\n\n**Response to reviewer Bias concern:** \n\n\nWe apologize for not highlighting this crucial aspect sufficiently in our initial submission. We understand the paramount importance of clarifying any potential biases, especially in datasets like ours, and are committed to elucidating this further in our final version. Bias Mitigation in Annotation:\n\n* **Multi-level Review:** As outlined in our paper, we used a rigorous three-stage inter-annotator agreement procedure. Each statement underwent initial examination by one journalist, then an independent review by a second journalist. Any discrepancies between their assessments brought in a third journalist for the final decision. This minimized potential bias from any individual journalist.\n\n* **Diverse Annotation Team:** Journalists annotating the data hail from diverse backgrounds and maintain \"The Whistle's\" professional standards. Their commitment to objectivity ensures labels are based on factual accuracy, free from personal or organizational biases.\n\n* **IFCN Guidelines:** The participating journalists strictly follow the IFCN guidelines, which uphold transparency, impartiality, and fairness in fact-checking.\n\n* **Nuanced Labeling:** We introduced a detailed 5-point truthfulness scale, complemented by 'Unverifiable' and 'Other' categories. This granularity captures the complexities of real-world statements, avoiding potential bias and oversimplification.\n\n* **Comparison with Established Methods:** Our labeling process aligns with methods employed in recent fact-checking studies, highlighting its consistency with prevailing community standards.\n\n* **Transparent Scoring Rationale:** To further reduce bias and ensure transparency, the entire rationale behind our scoring system is publicly available. Additionally, for each claim annotated, the specific rationale behind its score is clearly documented and made transparent. This level of openness allows for external scrutiny, assuring users and researchers of the thoroughness, objectivity, and integrity of our scoring methodology.\n\nThough our rigorous process yielded a smaller dataset, it ensures the highest quality and reliability.\nLastly, all text-based features were double-checked for accurate transcription. The semantic features benefited from a two-tier verification process involving both a \"The Whistle\" journalist and an editor. We trust this elucidation sufficiently addresses your concerns.\n\n\n\n** Response to reviewer Contribution concern**\n\nWe deeply value your feedback and understand the significance of clearly articulating the distinctiveness of our research. We believe our work embodies contributions in the NLP domain, and we're committed to elucidating these in detail.\n\n\n* **Introduction of \"HeTrue\" Dataset:** A principal contribution is the development of the \"HeTrue\" dataset. Distinctively, this dataset captures both claims and their contextual surroundings, establishing a potential reference point for subsequent datasets in this field. The dataset is curated in a thorough process with unique features (such as semantic features, edited claim and context).  As far as we know, this is the first-ever claim assessment dataset that accompanies each claim with its context, in any language.  More details can be found in table 4. \n\n** Extensive Experimental Analysis:** A cornerstone of our research is the comprehensive experimental framework we employed. We systematically investigated various modeling types and techniques, encompassing both traditional NLP methods and cutting-edge transformer architectures and designed unique transformer based models. We believe that such a broad and deep experimental setting provides the community with a clearer understanding of the strengths and limitations of different techniques when applied to the challenge of credibility assessment. \n\n* **Linguistic Analysis in Hebrew:** Our work delves into the linguistic nuances of Hebrew, addressing a gap in datasets oriented towards fake news detection. We identified particular linguistic markers in Hebrew that distinguish between credible and non-credible statements (Appendix A3). This contributes to our understanding of linguistic variations in credibility assessment, with possible implications for other languages. Our hand-crafted linguistic features, informed by previous studies, spotlight essential cues in credibility detection.\n\n* **Integration of Context:** We've highlighted the salience of statement context in the analysis. Our work shows that the combination of statements with their contexts can significantly enhance predictive accuracy, a finding that has implications for both further research and practical applications. As previously noted, there are no other works that focus on sentence-level claims (not from news articles) paired with context. We hope that our findings will stimulate the creation of such datasets in the future.\n\n* **Interplay of Metadata and Linguistic Features:** Our experiments integrating metadata with linguistic attributes presents intriguing findings. As detailed in Table 1, our results suggest a nuanced relationship between these two domains.\n\n\n\n* **Attention-based Evidence Aggregator (Attn-EA) Model:** Expanding on the work by \\citet{gupta2021x}, our Attn-EA model offers a refined approach to evidence curation, ensuring a meticulous integration of evidence from top Google search snippets and rigorous bias filtering.\n\n* **Advancement in Context-Aware Models:** We introduced models such as the Context-Based Model (CBM) and the Combined Context and Evidence Model (CCEM). The CCEM, in particular, integrates features from both CBM and Attn-EA, providing a more rounded approach by integrating statements, their contexts, and external evidence. Our approach uniquely combines statement context with external evidence, addressing an aspect not frequently explored in prior research. The methodologies we employ, particularly in models like CCEM, contribute a novel perspective to credibility evaluation. \n\n\nIn summary, we believe our work offers novel insights and methodologies in several areas of NLP research. We are grateful for the opportunity to elucidate our contributions more comprehensively.\nIn light of our clarifications, we sincerely hope that the distinctiveness and value of our contributions become apparent.\n\n---\n\n\n\n\n\n\n\n** Response to reviewer reproducibility concern**\n\nWe sincerely appreciate your feedback regarding reproducibility concerns. The integrity and transparency of our research is of utmost importance, and we understand the value of ensuring our results can be easily replicated by the community. To address the concern you raised:\n\nIt is important to emphasise that both the dataset and the codebase will be made publicly available upon acceptance of the paper.\n\nFor all reported experimental results:\n\nWe have evaluated all our models across four dataset setups, detailed in Section 3. The experimental setup, encompassing both evaluation and training, is elaborated upon in Section 4.\n\n**Strong Linguistic Baseline:** Various models were examined, with Xgboost being selected based on hyperparameter optimization, as delineated in Appendix A.5. The feature extraction process involves text pre-processing steps, exhaustively described in Appendix A.1 and A.2. We extracted features from both the statement and its encompassing context.\n\n**Metadata-based Models:** Several models based on metadata were utilized. All feature extraction processes, inclusive of pre-processing, are meticulously detailed in Appendix A.4.\n\n**Deep Learning Models:** We have approached experiments in two distinct setups: Statement-Focus and Context-Aware. For each:\n\n- **Statement-Focus Models:** Three models were employed. This included the RNN with Static Word Embeddings utilizing fastText embeddings, selected for its exemplary performance post hyperparameter optimization (detailed in the appendix). Next is the AlephBERT model, a Hebrew BERT variant fortified with a Transformer architecture. It underwent pre-training on 17.6G of Hebrew text and fine-tuning for our specific task. The Attention-based Evidence Aggregator (Attn-EA) model leverages AlephBERT for encoding both claims and evidence, utilizing dot-product attention, as outlined in Section 4.4 (Full formula can be found there too).\n\n- **Context-Aware Models:** Our experiments encompassed the RNN with FastText and Full Context. The Context-Based Model (CBM) and the Combined Context and Evidence Model (CCEM) both employ AlephBERT. The latter integrates statement, context, and external evidence from Google search snippets for assessment. Comprehensive formulas are available in Section 4.4. Complete parameter settings are listed in Appendix A.5.\n\n**Source code** will be made publicly accessible upon acceptance, accompanied by precise specifications and dependencies.\n\n**Evaluation Measure/Statistics:** Metrics, including the macro F1 score, are elucidated in tables 1 and 2 of Section 4. We employ the bootstrap technique to ensure statistical robustness. Comprehensive details are presented in Section 4.2.\n\n\n\n**Regarding the dataset:**\n\n- **Dataset Statistics:** HeTrue encompasses 1021 statements with label distribution outlined in Table 3.\n- **Train/Validation/Test Splits:** We leveraged a nested cross-validation strategy with 5-folds in both inner and outer loops. The detailed approach and methodologies are documented in the appendix and Section 4.\n- **Data Exclusions and Pre-processing:** Only the labels \"Unverifiable\" and \"Other\" were omitted. All remaining pre-processing phases are exhaustively detailed in Section 3 and the appendix.\n- **Language:** The dataset is entirely in Hebrew.\n- **Dataset Link:** This will be shared upon paper acceptance.\n- **Data Collection Process:** Detailed in Section 3, including our collaboration with \"The Whistle\" and the three-stage inter-annotator agreement process.\n\nFor all models, we used the publicly available base checkpoints on Huggingface, with Pytorch 2.0.\nAll hyperparameters details for our models, described thoroughly in the appendix A.5. We have used Optuna library. \n\n\n\nWe hope this comprehensive response should adequately address reproducibility concerns.\n\n–\n*All references cited in this response can be found in the bibliography section of our paper.\n\nWe deeply appreciate the time and dedication the reviewer has committed to evaluating our manuscript. We have endeavored to address each concern with thoroughness and rigor. It is our sincere hope that, upon reviewing our revisions and the effort we have put into addressing the feedback, the reviewer finds our submission to be significantly enhanced and more robust. Thank you once again for your invaluable insights", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset can be used to develop more robust models. ", "reasons_to_reject": "This paper presents a dataset containing statements from Israeli public figures and politicians. The dataset is annotated by professional journalists and different techniques were used for evaluating the dataset. The proposed model achieves a 48 F1 Score as the best score. However, how the bias was removed in the annotation process. Moreover, it is very unclear how the proposed dataset can be reproduced or adds new knowledge in terms of advancing the field. \n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a dataset containing statements from Israeli public figures and politicians. The dataset is annotated by professional journalists and different techniques were used for evaluating the dataset. The proposed model achieves a 48 F1 Score as the best score. \n\n", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AhjOwgLzE8", "reviewer": "Reviewer_BPdV", "summary": "", "questions": "How the findings in your experiments about the relative performance of different models and approaches are compared to the finding from experiments with other datasets and languages? \n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincere gratitude to the reviewers for their meticulous evaluation and constructive feedback. We deeply value the insights provided, which have offered us the opportunity to enhance the quality and clarity of our work. In this response, we have systematically addressed each of the concerns raised and have made corresponding revisions to our manuscript where necessary.\n\n\n\n**Response to the Reviewer's Query on Model Comparisons Across Datasets and Languages**:\n\nWe appreciate your question regarding the relative performance of our models and how they compare with experiments on other datasets and languages.\n\nIn our experiments, all models incorporating context consistently outperformed their counterpart models that did not use context. As far as we know, we are the only to establish experiments focusing on sentence-level claims with their context. However, different context types of different claim types examined before. Our finding is congruent with existing literature where the integration of context has demonstrably enhanced model performance. For instance:\n\n1. **Ghanem (2021)** in \"Fakeflow: Fake News Detection by Modeling the Flow of Affective Information'' showcased the value of leveraging context to bolster fake news detection. They segment news articles, and treat each segment as “claim” and the other as context. \n  \n2. **Nakashole and Mitchell (2014)**, \"Language-aware Truth Assessment of Fact Candidates\", proposed a methodology that harnesses Subject-Predicate-Object (SPO) triplets extracted from the Web, also can be considered as context,  emphasizing language objectivity analysis to determine veracity. \n\n3. **Mitra and Gilbert (2015), Ma et al. (2016), and Zhang et al. (2021)** leveraged posts related to the main post (which needed verification) as context. This approach resulted in improved model performance.\n\n\nIn our work, we closely observed how different evidence forms influenced model performance. As highlighted in the literature, evidence plays a pivotal role, especially in the realm of automated fact-checking. However, the nature and structure of this evidence can often dictate the efficacy of the model.\n\n\n1. **Popat et al. (2018)**, through \"DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning\", accentuated the utility of external evidence. Their work closely mirrors our findings, where models leveraging external sources, especially textual ones, generally showcase heightened performance accuracy.\n\n2. **Guo (2022)** in \"A Survey on Automated Fact-Checking\" conducted a comprehensive dive into various evidence types used for fact-checking. His study corroborated our observation that textual sources, spanning from news articles and academic papers to Wikipedia documents, remain among the most dominant and effective forms of evidence in the space. \n\nHowever, while textual sources are prevalent and widely accepted, they are not without challenges. As highlighted in the cited literature, using metadata as evidence might offer complementary information but lacks the depth to ground claims conclusively (Wang, 2017; Potthast et al., 2018). Similarly, though structured knowledge, such as knowledge bases, presents an organized approach, their inherent incompleteness can sometimes impede conclusive evidence generation (Bordes et al., 2013; Socher et al., 2013).\n\nIn our experiments, we found a semblance of these trends and challenges. Our models that embraced richer textual evidence and blended it with metadata / semantic features demonstrated a nuanced ability to tackle fake news detection, resonating with findings from works like Vlachos and Riedel (2015) and Gupta et al. (2020).\n\nOur findings on evidence-based models align with the broader trends and insights of the NLP community.\n\n\nRecent research underscores the significance of either context, evidence, or both, in various capacities. While our findings resonate with these works, a distinguishing feature of our approach is the simultaneous integration of both context and evidence, an amalgamation not extensively explored in the literature to date. The novel approach that we suggested outperforms past architecture, and we believe it can encourage the community to further research this avenue. \n\n*We're grateful for you bringing to our attention the typos and grammatical mistakes; they have been corrected in the final version of the manuscript.\n\n*All references cited in this response can be found in the bibliography section of our paper.\n\nWe deeply appreciate the time and dedication the reviewer has committed to evaluating our manuscript. We have endeavored to address each concern with thoroughness and rigor. It is our sincere hope that, upon reviewing our revisions and the effort we have put into addressing the feedback, the reviewer finds our submission to be significantly enhanced and more robust. Thank you once again for your invaluable insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tA well-structured and well-written paper, including the attachments.\n•\tThe area of automatic claim verification is a hot topic.\n•\tA new dataset is collected and annotated by human experts.\n•\tThe collection and annotation of the dataset is done very professionally, with care to all details. \n•\tExperiments are conducted to establish a baseline and state-of-the-art approaches are applied.\n•\tThe conducted experiments are well-designed.\n•\tThe new dataset is suitable as a benchmark.\n", "reasons_to_reject": "Limited discussion of how the findings of the experiments are compared with other works. Do the models perform in the same/similar/different way?", "questions_to_authors": "How the findings in your experiments about the relative performance of different models and approaches are compared to the finding from experiments with other datasets and languages? \n\n", "paper_topic_and_main_contributions": "The paper introduces HeTrue: a new, publicly available dataset for evaluating the credibility of statements made by Israeli public figures and politicians. The dataset consists of 1021 statements, manually annotated by Israeli professional journalists, for their credibility status. \nExperiments are conducted to establish a baseline and compare text-only methods with others using additional data like metadata, context, and evidence. Several credibility assessment models are developed including a feature-based model that utilizes linguistic features, and state-of-the-art transformer-based models with contextualized embeddings from a pre-trained encoder. Results from the experiment show that:\n•\tthe best performance is achieved when models integrate statement and context\n•\tthe HeTrue is a challenging benchmark and can be used in the future for model training and evaluation.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The first part of the title “The Truth, The Whole Truth, and Nothing but the Truth:” can be obliterated.\nAnd some of the motivation in the introduction could be shortened. This will give a bit of space to move some parts from the appendixes to the main body of the paper.\n\n"}}]

Author rebuttal:

Meta-review: First of all, thanks to the authors for working on "fact-checking" in a low-resourced language Hebrew. The problem itself is super important in the current scenario. The reviewers pointed out some shortcomings of the paper, which the authors also duly acknowledged. These include a detailed discussion on data collection, including recent relevant works on low-resourced languages, exemplifying the problem definition with relevant artifacts from the dataset, including a few simple baselines, etc., which I believe can be done easily. I would also request the authors to polish the paper in terms of language (reduce verbosity) and, if space permits, move critical discussions from the rebuttals/appendix to the main manuscript. I think the paper has most of the desired information for a resource paper, and the fact-checking dataset on Hebrew is a novel resource. I hope the authors would take the suggestions and cues from the reviewers to improve their paper.
Decision: Accept-Findings