HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark
Authors: Amir David Nissan Cohen
Hilla Merhav-Fine
Yoav Goldberg
Reut Tsarfaty
Venue: EMNLP 2023 Findings (2023)
Keywords: hebrew
machine reading comprehension
question answering
dataset
TL;DR: We created HeQ, a novel Hebrew Reading Comprehension dataset, with bespoke guidelines and metrics, showing that standard measures fail in morphologically rich languages and morpho-syntactic models may falter on semantic tasks.

Abstract: Current benchmarks for Hebrew Natural Language Processing (NLP)  focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding. To bridge this gap, we set out to deliver a Hebrew Machine Reading Comprehension (MRC) dataset, where MRC is to be realized as extractive Question Answering.
The morphologically-rich nature of Hebrew poses a  challenge to this endeavor: the indeterminacy and non-transparency of span boundaries in morphologically complex forms lead to annotation inconsistencies, disagreements, and flaws of standard evaluation metrics.
To remedy this, we devise a  novel set of guidelines, a controlled crowdsourcing protocol, and revised evaluation metrics, that are suitable for the morphologically rich nature of the language.
Our resulting benchmark, HeQ (Hebrew QA), features 30,147 diverse question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. Our empirical investigation reveals that standard evaluation metrics such as F1 Scores and Exact Match (EM) are not appropriate for Hebrew (and other MRLs), and we propose a relevant enhancement. In addition, our experiments show low correlation between models' performance on morpho-syntactic tasks and on MRC, which suggests that models that are designed for the former might underperform on semantic-heavy tasks. The development and exploration of HeQ illustrate some of the challenges MRLs pose in natural language understanding (NLU), fostering progression towards more and better NLU models for Hebrew and other MRLs.

HeQ: a Large and Diverse
Hebrew Reading Comprehension Benchmark
Amir DN Cohen'
Hilla Merhav-Fine" Yoav Goldberg 1.3
Reut Tsarfaty'
Bar-Ilan University, Ramat-Gan, Israel
Webiks, Tel Aviv, Israel
Allen Institute for AI, Seattle, WA
mairdno@@mai1.com hillaswebiks com, yoav. yoav.goldae www.llbergggmai....mm com, reat. caatt.sssraaccymaaa.... com
Abstract
QA systems find utility in large spectrum of applications, from chatbots (Gao al., 2018) and search engines (Guu et al., 2020) to the evalua- tion and assessment of pre- trained models' (Devlin et al., 2019; Joshi et al., 2020: Lan et al.. 2019) comprehension and reasoning skills. Despite these developments, the landscape of MRC is primar- ily dominated by datasets and models for English, where low- medium-resouree languages such Hebrew remain largely underrepresented.
Current benchmarks for Hebrew Natural Lan- guage Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the se- mantic dimension of language understanding. To bridge this gap, we set out to deliver a Hebrew Machine Reading Comprehension (MRC) dataset, where MRC is to be real- ized as extractive Question Answering. The morphologically-rich nature of Hebrew poses a challenge to this endeavor: the indetermi- nacy and con-transpareney of span boundaries in morphologically complex forms lead to an- notation inconsistencies, disagreements, and flaws in standard evaluation metries. To edy this, we devise a novel set of guidelines, a controlled crowdsoureing protocol, and re- vised evaluation metrics, that suitable for the morphologically rich nature of the language. Our resulting benchmark, HeQ (Hebrew QA), features 30, 147 diverse question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. Our empirical investi- gation reveals that standard evaluation metrics such as Fl Scores and Exact Match (EM) are not appropriate for Hebrew (and other MRLs), and we propose a relevant enhancement. In addition, our experiments show low correla- tion between models' performance morpho- syntactic tasks and on MRC, which suggests that models that designed for the former might underperform on semantics-heavy tasks. The development and exploration of HeQ il- lustrate some of the challenges MRLs pose in natural language understanding (NLU), foster- ing progression towards more and better NLU models for Hebrew and other MRLs.
Although recent research in NLP shows in- creased interest in the Hebrew language (Seker et al., 2022; Guctta et al., 2022), the focus remains largely on corpho--yntactic tasks. With the excep- tion of ParaShoot (Keren and Levy, 2021), the ab- sence of Natural Language Understanding (NLU) benchmarks for Hebrew leads to the design of mod- els primarily tailored for morpho-syntacee tasks. However, it remains an open question whether the crrpttssyntactic bench- marks correlates with actual natural language un- derstanding, and indeed, as we demonstrate in $5, this is not necessarily so.
performance on existing
A central challenge in the development of QA systems for Hebrew, a Morphologically Rich Language (MRL), compared with morphology- impoverished ones, is the identification of precise answer spans in morphologically rich internally complex surface forms. This exacerbated by mul- tiple processes of affixation in Hebrew (prefixes, suffixes, clitics) that may obscure the boundaries between linguistic units, making it harder to pin- point the correct span of answers. For instance, given the sentence .m2 'M7 S1DNK (literally: "yesterday I-was at-the-house"), valid responses to the question "Where was I?" can be either m22 (at-the-house), mT (house). Current annotation guidelines, crowdsourcing protocols and evaluation metrics, however, assume the annotated spans re- spect space-delimited token boundaries, and fail to accommodate such morphological patterns, conse-
1 Introduction
Machine reading comprehension (MRC), critical skill of NLP systems, is often assessed by means of testing the ability models to answer questions about a given passage, a setup known in NLP also by the name Question Answering (QA).

---

quently penalizing boundary alterations and caus- ing annotation disagreements. To counter this, we propose a revised set of guidelines and new evalua- tion metric, Token-Level Normalized Levenshtein Similarity (TLNLS), which manifests lower sensi- tivity to affixation changes.
to be as important as the size, underscoring the im- portance of dataset design and the value of our new Hebrew MRC benchmark, HeQ. Note that data, tagging platform, user manual, and the training and evaluation scripts are publically online.
The Challenge: Extractive Question Answering in Morphologically Rich Languages
We thus present HeQ, new benchmark for MRC in Hebrew, modeled around the format of the widely-used Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 201 2a, 20) SQuAD is a benchmark dataset for MRC that was collected via crowd-soureing and consists of questions and answers based on Wikipedia articles. However, SQuAD has been shown to contain biases and spurious regularities that allow models to exploit heuristies and shallow cues without fully under- standing the text (Jia and Liang, 2017). Further- more, SQuAD covers only a limited range of topics and domains (Ruder and Sil, 2021). Hence, this work, we introduce several novelties. First, we work with fixed set of human annotators and train each annotator individually based on their previous annotations to increase the quality and diversity of their annotations. We also add a data source from news domain (GeekTime). besides Wikipedia, in- crease HeQ genre diversity. Finally, we change the unanswerable questions generation methodology so they become more challenging for the model.
Extractive Question Answering (Extractive- -Ariiiiiee-AA))) is a common and widely used task where a model is tasked with locating and reproducing an answer to a question posed directly with regard to a source text. In morphologically-rice languages, this task be- comes more complex due to the linguistic features that these languages manifest. Morphologically rich languages, such as Hebrew, Turkish, or Arabic, exhibit high morphological complexity, including extensive inflection, agglutination, and derivational morphology. This characteristic makes Extractive- QA more challenging because the model needs to process and understand complex word forms and grammatical constructions to correctly pinpoint the answer spans.
In these languages, a single word can carry infor- mation typically spread across several words in lan- guages with more impoverished morphology, such as English. This means that word- and sentence- boundaries may not always align with semantic units or logical chunks of information. Conse- quently, Extractive-QA models must be able to handle this morphological richness to accurately lo- cate and extract the correct answer. Due the high inflectional nature of these languages, the correct answer may appear in different forms within the text, requiring the model to understand and match these varied forms. This aspect necessitates an un- derstanding of the language's morphological rules and richly annotated dataset to provide the model with enough examples to learn from.
In our experiments, we observe that mul- tilingual model, mBERT outperforms all other Hebrew-trained models despite being exposed to the least amount of Hebrew data during training, and having the smallest Hebrew token vocabulary. This suggests that pretrained models Hebrew can greatly benefit from pretraining on other lan- guages. Furthermore, we find that models trained on our new dataset, HeQ, perform significantly better than those trained on the earlier ParaShoot dataset, which emphasizes that the quality of the collected data is greatly influential model per- formance (in fact, a model trained on subset of HeQ with the same number of questions as in the ParaShoot dataset, outperforms a model trained on ParaShoot, also on the ParaShoot test set). Lastly, models based on the Geektime section outperform the Wikipedia ones and demonstrate better domain transferability, likely due to the former's varied text structure.
On top of that, existing evaluation metrics, pri- marily designed for morphologically simpler lan- guages, often fall short when applied to languages with higher morphological richness. A major con- tributor to this is affixations, which are a prevalent feature in MRLs. These languages often merge what would be considered separate words in lan- guages like English into single unit. This means that an answer span might not align with the tradi-
In conclusion, despite the complexity of Hebrew, multilingual pretraining proves beneficial for MRC tasks. The quality and diversity of the data appear
'tttps://gi httpg/ttt...ccco..oeeeooooectt Com//NT.P-TT. Hebrew- Quest i ion Anwwering- Dataett

---

tional notion of word boundaries.
In addition, the annotators were monitored and have been given feedback on the overlap between the questions and the text, aiming to minimize it.
Current evaluation metrics do not take this into account, leading to potential harsh penalties for mi- nor boundary deviations. Metrics like Exact Match (EM) or F1 score, which function well in morpho- logically simpler languages, may not accurately reflect a model's performance in morphologically rich languages. For instance, slight changes in word endings might result in dramatically differ- ent EM score, even if the overall answer meaning remains intact. Similarly, an Fl score may fail to capture the nuances word formation and changes that are inherent to these languages.
Quality over quantity. While HeQ is much smaller than SQuAD (30K vs 150K samples), we argue that the quality, diversity, and difficulty of the data are more important than the sheer number of examples. To demonstrate this, in $5 we com- pare different sizes of He and compare HeQ to the earlier ParaShoot dataset (Keren and Levy, 2021). I). In the following subsections, we describe how created HeQ based these principles.
3.2 Data Preparation
3 Collection Process
The HeQ dataset was created by combining two distinct data sources, namely, Hebrew Wikipedia and Geektime, Israeli technology newspaper. From the Hebrew Wikipedia, we selected the 3,831 most popular articles create the Wikipedia para- graph pool. Subsequently, we randomly extracted paragraphs from these articles, with a constraint that each paragraph should contain between 500 and 1,600 characters and no more than three para- graphs per article. We removed paragraphs that contained nen-verbal sections, such as mathemati- cal equations, and those that could be considered highly sensitive or inappropriate for our dataset, such as those containing violent, graphic, or sexual content. Some paragraphs were manually replaced with alternative paragraphs to ensure a diverse set of topics and styles. After this filtering and manual replacement process, the paragraph pool consisted of 5, 5,850 paragraphs. across total of 2,523 articles.
This section provides a comprehensive account of the data preparation, annotation, and validation pro- cedures employed in this study. Specifically, we describe the methods used to preprocess the raw data and prepare it for annotation, the rigorous an- notation process carried out to ensure high-quality data, and the validation steps taken to evaluate the reliability and accuracy of the resulting dataset.
3.1 Annotation Philosophy
HeQ created with four principles in mind:
Diversity. There is evidence that the quality of the model (in both pre-training and fine-tuning) is based on the diversity of the datasets (Zhou et al., 2023). We increase the diversification of the datasets in three ways: First, we generate the data from two sources, Wikipedia and News, which have very different structures, and topics, which re- sulted in different question types. The second the paragraphs chosen for annotation, we made sure that the initial pool of paragraphs was as diverse as possible. And thirdly, annotators were explic- itly instructed to create different and diverse sets of questions. We also monitored the diversity of annotations during the questions collection process.
To obtain the Geektime paragraph pool, used articles published from the year 2017 onwards, ex- cluding those the Career Management category as they contained job descriptions and not news. We randomly selected paragraphs from the remain- ing pool of articles, with minimum length of 550 characters and at least 300 characters in Hebrew.
Accuracy Accuracy is important for both train- ing robust models and accurate evaluation. We detailed the measurements we took to ensure very high accuracy in $3.3 and present the validation metrics in $3.4.
3.3 Annotation Process
Croked-workers recruitment. We ensured an- notation quality by recruiting native Hebrew speak- ers, who self-identified as fluent, via the Prolific crowd-sourcing platform. Candidates were pro- vided annotation guide, outlining task require- ments and guidelines for question creation and an- swer identification.
Difficulty SQuAD was criticized to encourage the model learn shallow heuristics (Elazar et al.,
2022). We aimed to overcome this by instructing the annotators to produce questions that require some inference (see "gold" questions in Table I).
The selection process involved a trial task, where candidates crafted questions and identified answers

---

from given paragraphs. Their performance was meticulously evaluated by the authors on parame- ters like correct answer span, question-answer lex- ical overlap, and attention to detail, ensuring the exclusion of frequent typographical errors, poorly articulated questions, or incomplete answers. Only candidates demonstrating comprehensive task un- derstanding and the ability to deliver high-quality annotations were recruited.
development (5%), and test (5%) sets, with the constraint that questions related to specific article were assigned to the same set. Each sample in the test and train was augmented by creating several correct answer spans for better validation.
3.4 Data Assessment
To ensure the quality of the evaluation, we man- ually validated the test and development sets and corrected wrong answers where necessary. Overall. reviewed 34% of the data, with 6.36% being rejected due to quality concerns. However, this rejection rate was largely attributed disqualified annotators and does not reflect the overall quality of the dataset as shown in the following paragraph. Also, note that the full development and test sets were validated for correction.
The Annotation Process For the annotation, we used custom UI, based on the UI used in ParaShoot. where in each step the annotator is pre- sented with paragraph and is required to write question and an answer, and mark if the question is answerable. We show the UT in Figure 1 under Appendix A. For each paragraph, the annotators were instructed to create 3-5 questions.
To validate the final dataset quality, we randomly sampled 100 questions from Wikipedia and 100 questions from Geektime for accuracy testing. We measured the quality of questions according to two parameters: the percentage of correct questions (93% for Wikipedia and 97% for Geektime) and the percentage of answers with a correct answer span (100% for Wikipedia and 99% for Geektime).
Crowdsource Monitoring. During the annota- tion process, we ensured ongoing quality control over the work of our annotators by providing per- sonalized feedback to each individual as needed. This feedback highlighted areas for improvement and areas of strength. Annotators who successfully applied the feedback and demonstrated improve- ment were invited to participate in additional anno- tation tasks, while those who did not improve were disqualified. Initially, feedback was given to each annotator after they completed their task. However, as the annotation process progressed, experienced annotators were randomly selected for feedback to allow for closer monitoring of new annotators and greater freedom for trusted annotators. For most annotators, We observed significant improvement in question quality after several rounds of feedback.
We include more analysis on the quality of the dataset in Appendix
4 MRC evaluation for MRLs
Evaluating the performance of MRC models is fundamental aspect of their development and finement. However, this evaluation process isn't 't as straightforward it might seem - especially when it comes to MRLs. The traditional evalua- tion metric of MRC is F1 over answer span words (described in Appendix B. or a variation thereof. The F1 score is sensitive to small changes in the an- swer span boundary. For example, if the gold span is "in the house" while the extracted span is "house" the Fi score will be 3.2 English somewhat allevi- ates this problem by removing the words "a", "an",, and "the" for the evaluation, which the former example will increase the F1 to 0.5.
To assess the quality of the questions, we as- signed each question a score based on four levels: rejected, verified, good, and gold. Questions that did not meet the minimum criteria for the dataset, typically due to semantic morpho-syntactic er- rors, unclear questions, questions that were clas- sified "unanswerable" but were answerable some extent, were classified as "rejected". "Verified" questions passed the minimum threshold but were relatively straightforward or used similar wording to the relevant sentence in the paragraph. "Good" questions had distinct wording, either lexically or syntactically, from the relevant sentence in the para- graph. Finally, "gold" questions required inference. Table I presents examples of each question type. The dataset was randomly split into train (90%),
Unfortunately, the boundary issues in MRLs, and in Hebrew specifically, are much more com- mon and result in biased scores. For example, if consider the same example, but in Hebrew, the phrase "in the house" is translated to the single word m2n, while the word m= corresponds "house". So while the English phrase will get an Fi score of 0.5, the Hebrew span using a token-based

---

Quality lahel
Example
Verified
Question/na3mp p'an miahn 1nt 17wh pabiin numben ny nuTenn oith aiee niiin Sentence: ,nt im paap paapp and
11 deal mu7Tnn
Answer: 11 1w22
Good
Question: "1an1o 'NT nw 7'2nn Sentence: 50 1522 apaoo nomb benn maar "manonn nuвиn moroon 1939 THNOER
pon тOmo 72N8 nw 1520 aaai Tрpan monor nunn mara mnn man T 7p712 7822 careL 1a12 ((" (21 12N")
177717 [..]]
jon
Gold
Question pam 7oo pENnn 5117 ærpan
Sentence: pruaon 7hnoa
wrann [..] m0d5 1932 non bruut matap nmban .p "D 7pM
am [i.] pap mir noon 1-0 by msen Ldtaaa aрmin 7nx by wran
Answer: 171 m31"
Table I: Example for quality levels of different samples.
or burd-based evaluation will be scored 0, leading to an underestimation of model performance. This is the result of compounding, combining two or more words into single word in Hebrew. Com- pounding very common in Hebrew and happens in many situations like possessive, Pluralization, definite article, prepositions, and other affixations. While some of these affixations also appear in En- glish (like Pluralization), they are much more com- mon in the Hebrew setting.
Speed. The metric should have a short run time.
4.2 The Token-Level Normalized Levenshtien
Similarity (TLNLS) Metric
The Metric. candidate for MRC evaluation is the Normalized Levenshtien Similarity metric (which we describe in Appendix 1)..1). This a character-level metric that has the benefit of being less affected by change in affixation but causes the evaluation to be skewed when dealing with long words and long sequences, which is undesirable. To overcome this, we present hybrid metric derived from F1 and Levenshtien similarity. Given two tok- enized spans, a predicted span = (PI,...,PM, and a gold span G = {gi..... In), where and Is are tokens, we define TLNLS as:
One way to avoid the issue is to do complete morphological decomposition of each word in the predicted span and the gold span and remove af- fixations from the evaluation. Unfortunately, the morphologically rich forms are ambiguous, and Hebrew morphological disambiguation is context- dependent, requiring a dedicated model to perform. We seek an evaluation method that is not depen- dent on the existence of disambiguation model and which will not be influenced by mistakes of different disambiguation models. Rather want method that is simple, unbiased, and easy to run on multiple MRLs. We thus present novel evaluation metric for MRC tailor-made for MRLs.
TLNLS(P.G) 1
Σ max (lscrlss(gs (1) 9.CG PIEP
max((G),
Tokenization.
TLNLS assumes tokenization of the input phrases. There are several ways to tok- enize a phrase, using white spaces, using a parser (like YAP (More et al., 2019)), and using a pre- trained model tokenizer. In this work we use white spaces for TLNLS, for several reasons: first, this is language- -nngguaggnnnde approach, that can be used for any MRL without any adjustment. Second, the design of TLNLS such that words with similar stems, but different infliction get a high score. third, any more advanced tokenization might be biased towards specific model.
4.1 Qualities for Evaluation Metric for MRL
We propose a new evaluation metric that takes into account the unique characteristics of MRLs. This metric should have the following characteristics: o Inflection invariance. The metric should give a similar score to word and its inflected and affixed variants (e.g., m2, m2n and m277).
Span invariance. The metric should give simi- lar score different spans that represent the same answer ("king David" "David").
Dates and numbers. limitation of these ap- proach concerns dates. F.g. the spans "1948" and "1921" will result in F1 score of O, but a TLNLS score 0.5. For this reason in the number of digits
o language independence. We aim to create general evaluation metric for MRLs, that does not require specific knowledge of the target language.

---

Type
Pi
Edit
TLNLS
Span
Span 2
TLNLS
P
Positive
0.576
0.389
0.727
MusicaNeto-1
MusicaNeto
0,909
0
Negative
0.019
0.233
0.093
owarobbo
0.9
pwmm
0.875
Table 2: Performance Metrics. The value in the brackets are without dates.
0.5
0
Table 3: Qualitative comparison of TLNLS and Fi. We
sampled 4 examples that had a high gap between Fi and TLNLS.
in either span is more than half, we revert to the original Fj.
Model
EM
Fi
TLNLS
4.3 Evaluating the Metric
Aleph Bert
57.91
67.66
76.07
Aleph Bert Gimel
57.12
67.37
75.3
Quantitative Evaluation We evaluate the score the metric assigns to correct spans (positive evalua- tion) and wrong spans (negative evaluation). Our development set in each case contains several cor- rect spans for each sample.
mBERT
62.7
71.42
78.2
mBERT (ParaShoot)
36.77
51.08
59.20
ChatGPT
9.38
32.19
37.91
Table 4: Performance Comparison of Different Models on the HeQ Dataset. All results except Aleph Bert and Aleph Bert Gimel are sgnificant.
The positive evaluation is done by taking all the samples in the development dataset that have more than one answer span, and evaluating the similarity between these spans based on each metric. For example, if sample contains the spans "nan"" "mas", and "mv=", we compare each span to the others (In our example, three comparisons, one for each pair) and report the mean similarity for each pair based on each of the metries. We assume that a better metric will give a high score for similar spans.
4.4 Recommended Metric
A good metric is vital for the correct evaluation of different models. Our experiments show that the Fi is too strict in the sense that it gives a low score to good spans (shown by our positive evaluation), including spans that differ only by single affix. To compensate for this, we suggest a new metric TLNLS, which gives significantly better results for good spans while still giving low scores to bad spans. For the evaluation, we use the two tradi- tional metries (Fi and exact match), and the new TLNLS metric, which we offer for general use.
In the negative evaluation, we collect negative spans and validate that the metric gives low score to these spans compared to the gold answers. To collect these spans we use the trained QA model on the development set and collect answer spans that received very low F1 scores (< 0.1) compared to the gold spans, we manually verify that these spans are incorrect. The result is 100 spans that contained a wrong prediction of the model. We compare the model (wrong) output to the gold labels using different metrics.
5 Experiments
We experiment with HeQ in several scenarios. First, we create a baseline based on several known models, then we evaluate the improvement of HeQ trained model vs ParaShoot trained model. We continue by analyzing the impact of the data size of the train data the models' performance. Finally, we assess the effect of the different do- mains in the dataset on the overall performance and the cross-domain performance on HeQ. We discuss the training setup in Appendix E
Results. As shown in Table 2, TLNLS achieves the highest score with large margin in the positive evaluation. The currently used F1 metric achieves mediocre score of 0.56. In the negative evaluation, Fi achieves the best (lowest) score, while TLNL achieves a slightly worse score of 0.093.
Improvement compared to ParaShoot. In table 5 we evaluate the test set of ParaShoot using the ParaShoot train data compared to theh HeQ train data, On all models, we see a significant improve- ment when training of HeQ compared ParaShoot.
Qualitative Evaluation We measure the samples with the greatest score difference between Fi and TLNLS. We iterate over the development set and calculate for each sample the Fi and TLNLS scores (like in the positive evaluation). We show 4 random samples in Table 3. All the samples are under- evaluated by Fi, (receive a score of 0) even though the returned span is valid.
5.1 Baseline Results
We evaluate several known pre- trained models on the HeQ test set. Unless stated otherwise, all mod-

---

Model
Geektime Between the Hebrew-monolingual models (AB and ABG), the results are almost comparable, giv- ing AB a small insignificant advantage over ABG, Based on this result we suspect that while the in- creased vocabulary size has a positive effect on morpho-syntactic tasks like the ones AB and ABG were evaluated on, it has lower significance in se- mantic tasks like MRC.
ParaShoot
HeQ
EM
Fi
TLNLS
TLNLS
AB
26
49.6
60.41
39.23
62.42
74.73
ABG
24.68
48.87
59.8
38 96
61.92
74.11
mBERT
32.08
56.25
66.83
44.29
65.58
75.38
Table Performance Comparison of Models trained on the ParaShoot and the HeQ Datasets and evaluated on the Parashoot dataset.
Trainsct
EM
TLNLS
Next, the mBERT (ParaShoot) model is under- performing compared to the HeQ trained model. which is not surprising considering the size of HeQ compared ParaShoot and has better quality.
58,38
68.00
75.80
Wikipedia
56.05
65.72
73.47
Geektime Wikipedia
62.70
71.42
78.21
Geektime + Wikipedia (15K)
59.24
67.52
75.27
Geektime Wikipedia (1.7K)
48.20
59.26
67.83
Surprisingly, the worst model in our evaluation chatGPT. While it is the largest model evaluated, achieves the lowest results, the returned spans are often mistaken, and even when the model returns the correct answer the span boundaries are wrong.
Table 6; Performance Comparison of Different Training Sets (sizes and domains) on HeQ.
els were trained on the HeQ train set. o AlephBERT (AB) (Seker et al., 2022): A He- brew pre-trained model that is currently used as a baseline for most Hebrew-related tasks.
5.2 The Effect of Data Size and Type on Model Performance
o AlephBERTGimel (ABG) (Guetta et al., 2022): A variation on AB which has larger vocabulary size which resulted in fewer splits and improved performance the AB evaluation benchmarks. mBERT (Devlin al., 2019): A multilingual variation of the known BERT model.
In this subsection, we examine the effect of the number of samples, and the type of data, on the overall performance of the trained models. The results are presented in Table 6.
First, we consider three splits of the data: Geek- time (news) only, Wikipedia only, and subset of the dataset that is roughly the same size the two other splits (Geektime + Wikipedia 15K). All models were evaluated on the HeQ test set. The Geektime dataset outperforms Wikipedia and the combined split. We hypothesize that the reason for that is that while Wikipedia has more diverse topics, its text follows relatively rigid structure.
o mBERT (ParaShoot) (Keren and Levy, 2021) BBRT trained on the parashot dataset.
o chatGPT: ChatGPT is a well-known conversa- tional model that was finetuned on a large LLM from openAI. We evaluate chatGPT using the fol- lowing prompt: "Answer the following question based the provided context. If the answer is present in the text, please provide it as a span. If the answer does not exist in the text, reply with an empty string. Qucstion: <question> Context: <context>".
The second experiment is evaluating the model performance based on different data sizes. We com- pare three data sizes, the full dataset (Geektime Wikipedia), the 15K version, and 1.7K version, which is comparable in size to ParaShoot. The big- ger the dataset is the more performance improves, but interestingly, moving from 1.7K to the full ver- sion (x20 increase) increases the final score by only 11.4% TENLS points, and moving from 15K to the full version (x2 increase) increases the fi- nal score by only 3% TENLS points. Based on this assume that to improve future models 100K+ dataset is needed. In addition, the large gap between the 1.7K version and ParaShoot (67.83% to 59.2% TLNLS) shows that most of the perfor- mance comes from high-quality and challenging questions and not from the sheer size of the dataset.
We present the results in Table 4. Similarly to ParaShoot, the best-performing model on all met- ries mBERT, which is surprising considering that BBERT is the pre- pre-trained model that was exposed to the least amount of Hebrew texts during training. We attribute this result to the fact that during the multilingual pretraining, mBERT is the model that was exposed to the largest corpora overall. Based this hypothesis, future Hebrew pre- pre-trained mod- els might benefit from jointly pre-training En- glish or other high-resoure languages.
"t the time of writing this paper we could not access the
GPT4 APL.

---

Geektime
Wikipedia
Split
EM
Fi
TLNLS
EM
Fi
TLNLS
Gecktime
61.07
70.59
77.35
55.70
65.36
74.36
Wikipedia
52.20
63.39
71.14
59.81
68.04
75.65
Geektime + Wikipedia (15K)
59.33
68.65
76.33
59.15
66.38
74.09
Gecktime Wikipedia
64.40
73.36
79.98
60.74
69.46
76.92
Table 7: Performance Comparison of Different Trainset on Geektime and Wikipedia Datasets
5.3 Domain Transfer
All of the above are English datasets. In contrast, TyDi QA (Clark al., 2020) is designed for mul- tilingual information-sseking QA, encompassing languages from various language types, offering a more global scope for MRC research.
HeQ contains two domains: Geektime (tech news) and Wikipedia. We evaluate the models in both in-domain and cross-domain settings. We also add Geektime + Wikipedia (15K) and the full dataset (Geektime + Wikipedia) basclines.
While the selection of MRC datasets in En- glish is large, the only MRC dataset in Hebrew is ParaShoot (Keren and Levy, 2021), consisting of Wikipedia articles and constructed in a similar way SQuAD. We improve over ParaShoot in three aspects: size, diversity and quality. HeQ has 30K questions compared to 3K; uses news and Wikipedia domains; and was much more rigorously validated. In particular, special attention has been given to the evaluation of flexible span boundaries.
The results are presented in Table 7. Similarly $5.2, we see that the Geektime-traine mod- els achieve better results than the Wikipedia one, having a 6.21 points increase in TLNLS on the Geektime test, while the Wikipedia train model has only 1.29 points TLNLS increase.
We again attribute this to the similarity in text structure in Wikipedia compared the less strict text structure Geektime.
While many MRC datasets leverage external sources (user queries and trivia sites), these re- sources are not available for Hebrew. Therefore, we create HeQ in a similar fashion to SQuAD 2.0. However, revised guidelines have been introduced in order to ensure diversity, question difficulty, and sorting our matters concerning span boundaries.
6 Related Work
MRC datasets (often these take the form of question-answering (QA) tasks) have been devel- oped by the NLP community for long time (Hirschman et al., 1999). While newer datasets are being developed, the structure of QA data paragraph (from Wikipedia), question about this paragraph, and an answer still the way most datasets are constructed nowadays. Perhaps the most well-known datasets are the SQuAD and SQuAD 2.0 (Rajpurkar et al., 201 2018b) which use Wikipedia as source for paragraphs, creat- ing 100,000 QA instances. SQuAD 2.0 improve over the first version by including another 50,000 unanswerable questions.
7 Conclusion
In this work, we presented the challenges of MRC for MRLs. We introduced HeQ, Hebrew MRC benchmark designed enable further research into Hebrew NLU. Experimental insights highlighted the surprising efficacy of mBERT, trained on di- verse languages, over Hebrew Hebrew-monoline monolingual models, suggesting the value of multilingual pretraining.
Other known datasets are NewsQA (Trischler et al.. 2017) that were collected from news arti- cles, TriviaQA (Joshi et al., 2017), a large-scale dataset (650K samples) that used questions from quiz web sites aligned with Wikipedia and web re- sults, and CoQA (Reddy et al., 2019), dataset fo- cused on conversational question answering, offers a unique context-based approach with dialogues and answers. Another recent QA dataset is natural questions (Kwiatkowski et al., 2019), which dis- tinctly uses natural queries from Google searches, aligned with Wikipedia paragraphs.
However, data quality was seen to be as impor- tant as size, underlining the need for high-quality, diverse Hebrew datasets. The complexity of evalu- ating models in MRLs was also recognized, leading to the proposal the TLNLS metric, better suited to handle morphological variations.
Overall, this work advances the understanding of MRC in morphologically rich languages, offering avenues for improved benchmarks, datasets, and evaluation metrics. invites further exploration to broaden knowledge and enhance MRC perfor- mance in underrepresented languages.

---

8 Limitation
who claim to be acting in the name of the Pales- tinian people. We call all to join us in advocating for the prompt and safe return of the abductees, as we stand together in the pursuit of justice and peace.
We recognize two main limitations of the dataset: This paper was finalized in the wake of these events, under great stress while we grieve and mourn. It may contain subtle errors.
Dataset Size. Although HeQ is of high qual- ity, it consists of relatively smaller number of instances compared to well-established English MRC datasets like SQuAD. The limited dataset size might impact the model's ubility to generalize and capture the full range of linguistic and contex- tual variations in Hebrew. However, efforts have been made to ensure the dataset's quality and diver- sity compensates for its smaller scale.
Acknowledgements
We thank Roei Shlezinger and Tal Geva for their invaluable contributions and assistance in shaping this paper.
Lack of Natural Questions. One of the limita- tions faced in constructing HeQ is the unavailability of a dedicated resource containing natural ques- tions in Hebrew. Natural questions, which reflect authentic and unconstrained human language use, provide a realistic and diverse set of challenges for MRC models. Without access to such a resource, the dataset may not fully capture the breadth and complexity of real-world questions, potentially lim- iting the evaluation of MRC models' performance in natural language understanding.
We thank the anonymous reviewers for their in- sightful comments and fruitful discussions.
funded by the Israeli Min- This last author istry of Science and Technology (MOST grant num- ber 3-17992), and an Isracli Innovation Authority grant (IIA KAMIN grant), for which we are grate- ful. addition, This project has received funding from the European Research Council (ERC) un- der the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT).
Despite these limitations, the introduction of a new dataset and evaluation metric provided this paper are valuable contributions to the field and the progression of Hebrew NLP research.
References
9 Ethics and Broader Impact
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski. Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seekinn question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454 470.
This paper is submitted in the wake of tragic ter- rorist attack perpetrated by Hamas, which has left our nation profoundly devastated. On October 7, 2023, thousands of Palestinian terrorists infiltrated the Israeli border, launching a brutal assault on 22 Israeli villages. They methodically moved from home to home brutally torturing and murdering more than thousand innocent lives, spanning from infants to the elderly. In addition this horrifying loss of life, hundreds of civilians were abducted and taken to Gaza. The families of these abductees have been left in agonizing uncertainty, as no infor- mation, not even the status of their loved ones, has been disclosed by Hamas.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Moshach, Yonatan Belinkov, Hinrich Schütze, and Yoav Gold- berg. 2022. Mcasuring Causal Effects of Data Statis- tics Language Model's 'Factual' Predictions.
The heinous acts committed during this attack, which include acts such as shootings, sexual as- saults, burnings, and beheadings, are beyond any justification.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neu- approaches to conversational AL. In 41st Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pages 1371-1374, Association for Computing Ma- chinery, Inc.
We fervently call for the immediate release of all those who have been taken hostage and urge the academic community to unite in condemnation of these unspeakable atrocities committed by Hamas,

---

Bylon Guetta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsar- faty. 2022. Large Pre-Trained Models with Extra- Large Vocabularies: A Contrastive Analysis of He- brew BERT Models and a New One to Outperform Them All. Technical report.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.
Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019. Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from Modern Hebrew. Trans- actions of the Association for Computational Linguis- rics, 7:33-48.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. 2020. Retrieval Augmented Language Model Pre- Trainingg In Proceedings of the 37th International Conference on Machine Learning, pages 9929-3938. PMLR.
Gonzalo Navarro. 2001. A guided tour to approximate string matching, ACM Computing Surveys, 33(1):31- 88.
Lynette Hirschman, Mare Light, Eric Breck, and John D. Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th Annual Meet- ing of the Association for Computational Linguistics, pages Colleee Park. Maryland, USA. Asso- ciation for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 Sa. Know what you don't 't know: Unanswerable questions for SQuAD. In ACL 2018 56th Annual Meeting of the Association for Computational Linguistics, Pro- ceedings of the Conference (Long Papers), volume 2, pages 784 789.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP 2017 Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 2021-2031. Association for Computational Linguis- tics (ACL).
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018b. Know what you don't I know: Unanswerable questions for SQuAD. In ACL 2018 56th Annual Meeting of the Association for Computational Linguistics, Pro- ceedings of the Conference (Long Papers), volume 2, pages 789.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving Pre-traiming by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 201 SQuad: 100,000+ questions for machine comprehension of text. In EMNLP 2016- Conference on Empirical Methods in Natural Lan- guage Processing, Proceedings, pages 2383-2392.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguisties.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Nat- ural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Omri Keren and Omer Levy. 2021. ParaShoot: A He- brew question answering dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering. pages 106- 1 12. Punta Cana, Dominican Republic. Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning, 2019. CoQA: conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-20 266.
Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. 2020. Look the first sentence: Position bias in question answering. In EMNLP 2020- 2020 Conference on Empirical Meth- ods in Natural Language Processing, Proceedings of the Conference, pages 1109-1121, Association for Computational Linguistics (ACL).
Marc-Antoine Rondeau and T. J. Hazen. 2018. System- atic error analysis of the Stanford question answering dataset. In Proceedings of the Workshop on Ma- chine Reading for Question Answering, pages 12-20. Melbourne, Australia. Association for Computational Linguistics.
Sebastian Ruder and Avirup Sil. 2021. Multi-Domain Multilingual Question Answering. In EMNLP 2021-
Tom Kwiatkowski. Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- lational Linguistics, 7:452-466
2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 17-
21.
Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022. BERPhBERT: Language model pre-training and evaluation from sub-word to sentence level. In

---

Proceedings of the 60th Anmual Meeting of the Associ- ation for Computational Linguistics ( (Volame 1: Long Papers), pages 46-56, Dublin, Ireland. Association for Computational Linguistics.
Given stream of tokens for the gold span G .. In and stream of tokens for the pre- dicted span = (PI..... ., Pn). we define the inter- section function as
Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. 2022. Look to the right: Mitigating relative position bias in extractive question answering. In Proceedings of the Fifth BlackboxNI.P Workshop on Analyzing and Interpreting Neural Networks for NLP. pages 418 425, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
intersect
Hgg 3 s.t. = E G,p. PH!
Using this we can define the known precision and recall, and Fl functions:
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Har- ris. Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: machine comprehen- sion dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
precision(G. P) I)te
recall(G, interseat(G.P))/IP)
2 (e, precision P) Fi P) recall(G, G, P) + precision G,P) P)
B.1.1 Normalized Levenshtien Similarity Levenshtien distance is defined as the minimal number of edits that are required to convert a string 81 into different string 82 (Navarro, 2001). An edit one the following functions:
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment.
Annotation UI
Removal remove character from string (e.g. "cat" + "ca").
003729 n90n n709
Addition add a new character to a string (e.g. "cat" + "cats").
nomon mis mraano nesow mnsent to nenor 1001 arandh erowngn 1n nomeo net maguo. fro ameraza 50% in maaos was cra son -na an vo p0330s yun anabour marten tlnns 1695
Substitution change a character from a string to a different character (c.g. "cat" + "cut").
crowmpn preach
prawing zrowip nonon natoron than amp nanotn wokny Inviitt phtto
wane sern nomone tuee ens hurso nomtin ia paen no mich noen eammp pareho
Formally we can define Levenshtien distance recursively
typoco mum on eru stoeat bry
npooo rar yas rloel
lev(s1, 82)
[si|=
if S2) 0
[82],
if $1 0
leu(s1,
if ss [0] s2(0]
1+
DOJ03 naRnnT
212m
nehnne
lev(sn [1 82)
min
lev(s), s2 []]
otherwise.
lev(s) :) $2 :)),
Figure I: Annotation UI.
Where the "[J" operator is the array operator in Python. To normalize the edit distance we sim- ply divide the result by the max length of the two strings. To use normalized Levenshtien similarity we use the following formula:
B Metries
B.1 Fi Score
The current metric used for many MRC datasets evaluation is the Fg score, defined as follows.
lev(s1, 82)
Is(51, 82) max (1

---

6/0
030
604
080
4.3
(a) Question answer overlap.
Figure 3: The distribution of answers inside the context on HeQ and SQuAD. The distribution is roughly the same.
Position Bias. Uneven distribution of the answer location might also lead to biases in the data (Ko et al., 2020; Shinoda al., 2022). We show the answer distribution across the context in Figure 3. Like SQuAD, the distribution of answers HeQ is mostly uniform, where there is bias towards the start of the context and a negative bias towards its end.
0.0
5. S.Cktra
(b) Question and context overlap
Figure 2: A comparison of the overlap in HeQ. on both metries, our dataset has overlap compared to SQuAD.
E Training Parameters
Parameter
Value
Table shows the peper-parameters used for train- ing. No hyper-parameter tuning was made during our experiments.
Learning rate batch size per device
3 10-5
4
GPU type
four Nvidia 2080TI
Optimizer
Adam
Epochs
(with early stopping)
Table 8: Training parameters.
C Empirical Analysis
this section, we supply more analysis driven by other studies on MRC datasets.
Data Overlaps. Biases in the data result in model that learns the biases and not the actual task (Elazar et al., 2022; Rondeau and Hazen, 2018). We aimed reduce biases as much as possible. In this analysis, we validate that there is little overlap between the question and the text and between the question and the answer (in answerable samples). We plot the overlap between the questions and the context and answer in 2. It is casy to see that HeQ has lower overlap compared to the SQuAD dataset.
D English Translation of Example from the Data
We show the translation of Table 1 in Table 9

---

Quality label
Example
Verified
Question:Wher Where will program hroadcast the coming days?>
Sentence: Rebroadcast programs associated with educational television such "Close Relatives" and "Zeho ze" will be broadcast on kan II
Answer: kan
Good
Question: Who replaced Hajj Muhammad in his position?
Sentence: In February 1931, the "Central Committee" agreed recognize Hajj Muhammad the general commander of the rebels [...] After his death, the Central Commitice appointed Ahmed Muhammad Hassan ("Abu Bakr"), former teacher the village of Burka near Nablus, to the position of general commander.
Answer: Ahmed Muhammad Hassan
Question: Who scored
goal the cup final of 1932?
Sentence: Nevertheless, the team managed to qualify in 1932 for the cup final [...] during the game in 1-0 situation Hapoel from goal by Yona Stern [...] the trophy was stolen the team's players
Answer:
Yona Stern
Table 9: Example for quality levels of different samples in English.


Reviews:
[{"review_id": "HTtum2LJ1p", "reviewer": "Reviewer_cQ9w", "summary": "", "questions": "Explain how the answers were annotated.\n\nWhat are the statistics of the final dataset?\n\nWhat kind of errors that ParaShoot makes does HeQ help mitigate?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review! We were happy to read that you also identified the need for a large-scale MRL dataset.\n\nQ: “the process of obtaining questions is described, but what are the answers?“ \n\nA: The questions and answers are written in tandem. We report this in lines 292-293\n\nQ: What are the final statistics of the dataset? Section 3.2 provides details of data obtained based on Wikipedia, but no details about the data obtained from Geektime.\n\nA: The overall dataset contains 30147 samples. out of which 15025 are from Wikipedia and 15122 are from Geektime. We did not write the size of each split, but we will be sure to add it to the camera ready once it gets accepted.\n\nQ: the authors argue for the diversity of the data gathered, but there is no quantification, and this is not difficult to provide. Even a shallow types vs. token statistic would be useful.\n\nA: We will add token uniqueness between the two domains and between intra-domain categories. If there are more diversion metrics that you see as beneficial, we’d be happy to add them too!\n\nQ: it seems like a good idea to choose negative spans from the wrongly predicted answers, but they may not necessarily be good for evaluating the string matching method. How about partly overlapping spans with the positives? Or other such spans that are more difficult negatives? The purpose of the new string match method is to deal correctly with morphological variations, including misleading ones. It would also be useful to have some error analysis for the TLNLS. What situations does it get wrong?\n\nA: The purpose of TLNLS is to be a good discriminator between two models and not between two random spans (we will clarify this point in the paper). For this reason, we believe that the best way to evaluate it is by using the actual predictions of models. While we believe this is the best way to evaluate the model, we agree that adding a partly overlapping span will strengthen the analysis of the new metric, and we will add this to the camera ready. Regarding error analysis, we believe that the negative analysis does exactly that - it evaluates how the metric scores the model when it predicts an incorrect span.  If you have further ideas for evaluation, we will be happy to include them.\n\nQ: Some discussion and error analysis on the new dataset would be useful. \n\nA: We scrutinized several aspects of the dataset (different sizes, different domains) as an analysis. We have additional error analysis that was dropped due to lack of space (In short, the model fails much more on samples that require inference, more so than those quoting the answer verbatim). We will make sure to add this analysis to the camera-ready version. \n\nQ: Contrasting this to ParaShoot would also be useful, to show the benefits of using the new dataset (e.g. examples of ParaShoot that the model trained on HeQ gets right, to illustrate what phenomena/situations the new dataset helps model better)\n\nA: We include a direct comparison between HeQ and Parashoot in the section that starts in line 590. Note that Parashoot is a very small dataset (having 1792 samples for training). This results in very simple models that use relatively shallow heuristics. For example, they are easily misled to select an answer if there is a relevant type in the text (for example, a person, when the question is “who”). The models created based on the much larger  HeQ are more nuanced in this case.  We didn't include a direct comparison because we didn’t see it as a significant contribution to our analysis - we argue that the high-quality data and the fact that new Hebrew models do not show an improvement over older ones when semantic tasks like MRC are involved is the heart of the paper. Nevertheless, we will add this to the camera ready.\n\nQ: the ParaShoot dataset is frequently referenced but not described, and not well contrasted with the dataset presented here.\n\nA: We discuss Parashoot in several places in the paper, starting with line 053. Mostly we discuss the related work section starting line 653. We will add a more elaborate description of ParaShoot, including the dataset statistics, as well as a direct comparison to it, to the camera-ready version\n\nRegarding reproducibility - our data is in the same format as SQuAD, and can be easily trained with the available SQuAD scripts. In addition, we will release all the data and trained models necessary to replicate our results. If there are additional aspects that we missed and you wish to be disclosed to allow for easy reproducibility, please let is know. We’d very much like to improve on that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset constructed, and the process used to collect it, are useful resources for NLP, and also further emphasize the different processing requirements of different languages. The issues surrounding morphologically rich languages are well presented and help justify the necessity of the dataset and the new string matching process.", "reasons_to_reject": "The main result of the work presented in the paper is the dataset, but there are insufficient information regarding it:\n\n- the process of obtaining questions is described, but what are the answers? are they developed in parallel?\n\n- what are the final statistics of the dataset? Section 3.2 provides details of data obtained based on Wikipedia, but no details about the data obtained from Geektime.\n\n- the authors argue for the diversity of the data gathered, but there is no quantification, and this is not difficult to provide. Even a shallow types vs. token statistic would be useful.\n\n- it seems like a good idea to choose negative spans from the wrongly predicted answers, but they may not necessarily be good for evaluating the string matching method. How about partly overlapping spans with the positives? Or other such spans that are more difficult negatives? The purpose of the new string match method is to deal correctly with morphological variations, including misleading ones. It would also be useful to have some error analysis for the TLNLS. What situations does it get wrong? \n\n- some discussion and error analysis on the new dataset would be useful. Contrasting this to ParaShoot would also be useful, to show the benefits of using the new dataset (e.g. examples of ParaShoot that the model trained on HeQ gets right, to illustrate what phenomena/situations the new dataset helps model better)\n\n- the ParaShoot dataset is frequently referenced but not described, and not well contrasted with the dataset presented here.", "questions_to_authors": "Explain how the answers were annotated.\n\nWhat are the statistics of the final dataset?\n\nWhat kind of errors that ParaShoot makes does HeQ help mitigate?", "paper_topic_and_main_contributions": "The paper presents a new dataset for MRC in Hebrew, built based on a rigorous data collection and filtering process, using paragraphs from Wikipedia and a newspaper with technical news. Because Hebrew is a morphologically rich language, to allow for such variations in the potential answers, a new string matching method is proposed, that does not penalize morphologically inflected words.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gy00BG3H3e", "reviewer": "Reviewer_mCKP", "summary": "", "questions": "QA: In 3.1 (Annotation Philosophy), you mention that you increased diversification in three ways. The first was clear: include different structures and different topics. The other two methods just say that you made sure the dataset was “diverse”—what metrics are you using for diversity? \nQB: How did you create the multiple answer spans to augment the train and test samples? \nQC: In line 351, what is a “correct question?”\nQD: At the beginning of 4.3, do you just mean “pairwise?”\nQE: In line 452, did you mean if the number of digits in either spam is more than half the total number of digits in each span? \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! we were glad to read you found our work well-motivated, useful, and the dataset of high quality -- this precisely what we aimed for in constructing this resource.\n\nRegarding the evaluation (reason to reject):\n\"The proposed evaluation metric (token-level normalized Levenshtein Distance) is not validated by humans. The other metrics that are reported in the paper (F1 and exact match) are shown by the authors to be ineffective. This makes it difficult to trust the assessment of model performance on HeQ.\"\n\nThank you for bringing up the critical aspect of the evaluation of the models in the case of MRLs. Firstly,  we by no means argue that EM and F1 are useless, they are the standard metric, and hence we report these results. On top of that, in MRLs some span differences are more nuances and hence we provide an additional, more sensitive, metric. All in all our evaluation shows that EM and F1 have a tendency to be strict since they cannot attend to in-token nuances, and hence should be treated as a lower bound, which is also useful for model analysis. \nSecondly, our metrics for evaluating the metrics itself, are based on good and bad spans collected from humans, and in that sense, it is effectively human-based. As we describe in line 454e we have two types of measures for evaluating the metric: positive (where we expect the metric to give high scores to correct spans) and one is negative (where we expect the metric to give low scores to incorrect spans). Both evaluations rely on manually collected gold data of good and bad spans, and we argue that this is a methodologically solid way to create a robust evaluation for the new metric.  We are open to adding additional forms of human-based evaluation for our TLNLS to the camera-ready – could you please elaborate on what kind of human judgments you would like to see on top of the human tagging of good/bad spans as we described above?\n\n\nQ: In 3.1 (Annotation Philosophy), you mention that you increased diversification in three ways. The first was clear: include different structures and different topics. The other two methods just say that you made sure the dataset was “diverse”—what metrics are you using for diversity?\n\nA: The second endeavor was to select paragraphs from different and diverse topics in wikipedia. The third endeavor was to systematically look for repeating patterns that were used by the annotators during the evaluation of the annotators. and instruct them to create different kinds of questions. We will explain this more clearly in the camera-ready version.\n\nQ: How did you create the multiple answer spans to augment the train and test samples? \n\nA: The test and validation were validated completely by the author of the paper. When we validate them we manually add all the relevant spans beside the one chosen by the annotator. We will clarify this in the camera ready version.\n\nQ:In line 351, what is a “correct question”?\n\nA: “Correct questions” are defined in the paper to be questions that didn’t fall into any one of the “incorrect question” categories. The categories of “incorrect questions” are as follows: 1. Questions that were marked “unanswerable” but were actually answerable, i.e., their answer appeared in the text; 2. “Answerable”-marked questions that their answer did not appear in the text (the marked span was wrong); 3. Questions that were judged ungrammatical. The rest of the questions are judged “correct”. \nQ: At the beginning of 4.3, do you just mean “pairwise?”\n\nA: Using “pairwise” is a lot more concise and clear. We will use that.\n\nQ: In line 452, did you mean if the number of digits in either spam is more than half the total number of digits in each span?\n\nA: No. We mean that out of all the characters in a single span, at least half are digits (we evaluate this separately for each span). For example in the string “asd23” two out of five are digits. This is used mainly to filter out dates that are expressed only as digits and punctuation which is the only scenario that we found TLNLS to be underperforming.\n\nRegarding typos - we will be sure to proofread the entire paper and correct all remaining typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This dataset is a resource for which no alternatives currently exist—the authors motivate the need for this resource well by pointing out unique aspects of the Hebrew language. \n\nThe dataset quality appears to be very high. The authors clearly detailed their data curation attempts and manually verified much of the data. \n\nThe domain transfer and data size experiments are particularly compelling. ", "reasons_to_reject": "The proposed evaluation metric (token-level normalized Levenshtein Distance) is not validated by humans. The other metrics that are reported in the paper (F1 and exact match) are shown by the authors to be ineffective. This makes it difficult to trust the assessment of model performance on HeQ. ", "questions_to_authors": "QA: In 3.1 (Annotation Philosophy), you mention that you increased diversification in three ways. The first was clear: include different structures and different topics. The other two methods just say that you made sure the dataset was “diverse”—what metrics are you using for diversity? \nQB: How did you create the multiple answer spans to augment the train and test samples? \nQC: In line 351, what is a “correct question?”\nQD: At the beginning of 4.3, do you just mean “pairwise?”\nQE: In line 452, did you mean if the number of digits in either spam is more than half the total number of digits in each span? \n", "paper_topic_and_main_contributions": "This paper introduces a machine reading comprehension dataset for the Hebrew language in the form of a QA dataset (HeQ). The authors demonstrate issues with the usual metrics for evaluating QA performance in Hebrew and propose a new metric that is a modified version of Normalized Levenshtein Similarity. Finally, the authors evaluate several models on HeQ, finding that mBERT performs the best and even outperforms ChatGPT. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are many small typos—I would give the entire paper a reread. The ones I caught:\n\nline 522 “on” not “of”\nCrowd workers spelled wrong twice in subheadings in 3.3\nWrite out MRL in abstract \nLine 525\nLines 448 and 451"}}, {"review_id": "z7T6rOv2mZ", "reviewer": "Reviewer_ujfx", "summary": "", "questions": "In lines 342-343, I am curious that if the authors will delete all annotations of those disqualified annotators after rejecting the cases they made.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We were glad to read that you found our work to be solid and a meaningful contribution to the MRC field!\nRegarding your question “In lines 342-343, I am curious if the authors will delete all annotations of those disqualified annotators after rejecting the cases they made.”- \nTo make this clear: we removed all wrong annotations from disqualified annotators, they are not part of the final dataset. We will clarify it in the camera ready.\n\nRegarding your reason to reject\nReason1: “I doubt the generalization ability of the proposed TLNLS metric considering flexible span boundaries in other morphological-rich languages. The reason for languages that have complexity in morphology is different. Prefixes or suffixes could change the meaning of words totally although they have the same stems. In Arabic, different forms of words could present same semantic meaning, and same word also could be presented in various forms. The work only applies TLNLS in Hebrew. It is still need to be investigated showing the ability of TLNLS metric in other morphological-rich languages.”\n\nAnswer: The purpose of TLNLS is to be a good discriminator between answers of models given the same question and the same text and not between two random spans (we will clarify this point in the paper). Hebrew, like Arabic, exhibits both types of morphemes: the first type includes affixes that are attached to base words to indicate grammatical information, such as inflectional morphemes, along with prefixes that function as conjunctions, prepositions or definite articles. The second type involves derivational morphemes, which are affixes that are added to a base word to create a new word with a different meaning or part of speech. The reason that we focus on the former category is that they are a lot more common as mistakes done by the model. In fact, after reading your comment we searched the data for mistakes that involved derivational morpheme, and didn’t find any. So, we argue that TLNLS can be used effectively for other MRLs, and especially Arabic, as its grammar and inflections are relatively similar to Arabic. \nFollowing your review we also applied  TLNLS on an Arabic MRC dataset ( GitHub - RanaMalhas/QRCD: Reading comprehension on the Holy Qur'an ). Based on our test, TLNLS is also very relevant for this setup, but we believe that the full analysis of TLNLS in Arabic is beyond the scope of this current paper.\n\n\nReason2: “The example in Table 1 without translation in English is not understandable.”\nAnswer: We include the translation in the appendix, on Table 9, and following your comment, we will add it to Table 1 for camera ready.\n\nReason3: In line 434, what does ls denote in the formula? It is better to give a brief explanation of it.\nAnswer: We will include a full explanation in the camera-ready version. The idea behind this equation is to split the span into words, and then we score each word based on its similarity using normalized Levenshtein similarity. Note that ls - levenshtein similarity is explained in Appendix B, we will move this notation to the relevant section in the camera ready.\n\nReason4: The caption of Table 4 is not very clear to me.\nAnswer: We will make the caption clearer, specifically we’ll explain how we calculated significance.\n\nRegarding typos and references - we will fix all of them for the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed benchmark in low-resource languages is quite valuable for the Machine Reading Comprehension (MRC) community to foster further research on it. It is quite a desired work. The motivation and idea of the work are clear and convincible. Most of claims and arguments are supported well with relevant literature. The writing is good. Overall, it is a solid work and makes meaningful contributions in the MRC field.\n\n", "reasons_to_reject": "I doubt the generalisation ability of proposed TLNLS metric considering flexible span boundaries in other morphological-rich languages. The reason of languages that have complexity in morphology is different. Prefixes or suffixes could change meaning of words totally although they have same stems.  In Arabic, different forms of words could present same semantic meaning, and same word also could be presented in various forms. The work only applies TLNLS in Hebrew. It is still need to be investigated showing the ability of TLNLS metric in other morphological-rich languages.\n\nThe example in Table 1 without translation in English is not understandable.\n\nIn line 434, what does ls denote in the formula? It is better give a brief explanation of it.\n\nThe caption of Table 4 is not very clear to me.\n", "questions_to_authors": "In lines 342-343, I am curious that if the authors will delete all annotations of those disqualified annotators after rejecting the cases they made.", "paper_topic_and_main_contributions": "The paper proposes a new Hebrew dataset, HeQ, for the Machine Reading Comprehension task. The dataset is created by following a set of guidelines and a crowdsourcing protocol to control the quality of it. Compared with the counterpart (ParaShoot), HeQ includes one more domain of news showing the diversity of it. Due to Hebrew is a morphological-rich language, the authors propose a hybrid evaluation metric, TLNLS, derived from F1 and Levenshtien similarity. Thus, the models can be evaluated fairly considering flexible span boundaries. Finally, the authors conduct experiments from different perspectives, performance of baselines on the datasets, data size and domains, to show the effectiveness and contributions of the benchmark presented in this work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Line 122: missing reference for paraShoot dataset\nLine 274-275: any website link for the Prolific crowd-sourcing platform?\n", "Typos_Grammar_Style_and_Presentation_Improvements": "I don't know what MRL stands for in line 25 of abstract when I read there.\n\nWhat does 'Edit' means in Table2?\n\nLine 525: typos in 'dname'."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a machine reading comprehension dataset for Hebrew, HeQ, consisting of approximately 30K question-answer pairs, used to demonstrate comprehension. It likewise details the collection and annotation pipelines, a revised evaluation metric (TLNLS, which is better suited for the complex morphological patterns exhibited by Hebrew), benchmarking experiments, as well as comparisons to an existing machine reading comprehension dataset ParaShoot. Reviewers praised the paper’s clarity and motivation, its literature review, and the fact that it presents a high-quality resource with few to no viable alternatives. They did raise individual concerns in their reviews, however (e.g. extensibility of the proposed evaluation metric TLNLS to other morphologically rich languages and lack of human evaluation regarding TLNLS). Reviewer cQ9w raised several additional concerns, commenting the paper would benefit from (1) added details pertaining to the dataset, e.g. human evaluation, statistics, evaluation of diversity, (2) more detailed comparison to Parashoot, and (3) detailed error analysis of the models trained on HeQ. In their rebuttals, the authors addressed all of these concerns extensively, sharing the quantitative and qualitative results of their error analysis and empirical results from their comparisons between Parashoot and HeQ.

To summarize, this paper calls for a fair number of revisions, but much of the work required for these revisions is already complete, as demonstrated in the author rebuttals. What remains is to incorporate this work in writing and ensure that all reviewers comments and concerns have been addressed.
Decision: Accept-Findings