OpenHuEval: Evaluating Large Language Model on Hungarian Specifics
Authors: Haote Yang, Xingjian Wei, Jiang Wu, Noémi Ligeti-Nagy, Jiaxing Sun, Yinfan Wang, Győző Zijian Yang, Junyuan Gao, Jingchao Wang, Bowen Jiang, Shasha Wang, Nanjun Yu, Zihao Zhang, Shixin Hong, Hongwei Liu, Wei Li, Songyang Zhang, Dahua Lin, Lijun Wu, Gábor Prószéky, Conghui He
Accepted at: acl25

OpenHuEval: Evaluating Large Language Model on Hungarian Specifics

Haote Yang1*, Xingjian Wei1*, Jiang Wu1*†, Noémi Ligeti-Nagy2*, Jiaxing Sun3*, Yinfan Wang4*, Zijian Győző Yang2*, Junyuan Gao5*, Jingchao Wang6, Bowen Jiang7, Shasha Wang1, Nanjun Yu8, Zihao Zhang8, Shixin Hong9, Hongwei Liu1, Wei Li1, Songyang Zhang1, Dahua Lin1,10, Lijun Wu1, Gábor Prószéky2, Conghui He1‡
1Shanghai Artificial Intelligence Laboratory,
2HUN-REN Hungarian Research Centre for Linguistics, 3Wuhan University,
4Shanghai University, 5University of Chinese Academy of Sciences,
6East China Normal University, 7Peking University, 8Harbin Institute of Technology,
9Tsinghua University, 10Chinese University of Hong Kong

Abstract
We introduce OpenHuEval, the first benchmark for LLMs focusing on the Hungarian language and specifics. OpenHuEval is constructed from a vast collection of Hungarian-specific materials sourced from multiple origins. In the construction, we incorporated the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs’ generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations. Ultimately, OpenHuEval encompasses eight Hungarian-specific dimensions, featuring five tasks and 3953 questions. Consequently, OpenHuEval provides the comprehensive, in-depth, and scientifically accurate assessment of LLM performance in the context of the Hungarian language and its specifics. We evaluated current mainstream LLMs, including both traditional LLMs and recently developed Large Reasoning Models. The results demonstrate the significant necessity for evaluation and model optimization tailored to the Hungarian language and specifics. We also established the framework for analyzing the thinking processes of LRM with OpenHuEval, revealing intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example. We will release OpenHuEval at https://github.com/opendatalab/OpenHuEval .

1 Introduction
Recent advancements in Large Language Models (LLMs) (Jaech et al., 2024; Team, 2024a) represent significant strides toward artificial general intelligence (AGI). However, notable performance gaps remain between English and other languages in both language-agnostic tasks (e.g., math reasoning, code generation) (Huang et al., 2025; Zhang et al., 2024) and language-specific tasks (e.g., idiom usage, cultural understanding) (Naous et al., 2023; Sun et al., 2024; Liu et al., 2024c), posing challenges to global AI deployment and equitable development. The disparities in cross-lingual performance arise mainly from two factors: First, the training data, particularly the pre-training data, is heavily skewed toward English. Second, while English evaluation benchmarks are advanced and rapidly evolving, non-English benchmarks are underdeveloped, particularly for language-specific features, limiting the identification of shortcomings in non-English languages and leading to their neglect in research.

This paper focuses on the evaluation of Hungarian language and specifics. Hungarian is spoken by around 14 million people worldwide. Research on the Hungarian language not only improves the user experience for Hungarian speakers but also offers valuable insights for similar studies in other languages and regions. Existing Hungarian evaluation datasets are largely translations of English ones, missing essential Hungary-specific elements such as language nuances, culture, history, and regional context, which are key for Hungarian users. Among the existing evaluation datasets, HuLU (Ligeti-Nagy et al., 2024) is the key benchmark for Hungarian language understanding, but its focus on multiple-choice and true/false questions limits its ability to evaluate broader LLM capabilities, such as language generation, open-domain Q&A, reasoning and instruction-following.

To address this gap, we introduces OpenHuEval, the first evaluation benchmark for LLMs focused on Hungarian language and specifics. The comparison of OpenHuEval with the existing related benchmarks is shown in Table 1. Overall, OpenHuEval has two main distinguishing features:
1) Hungarian-Specific Inspired by (Liu et al., 2024b; Sun et al., 2024), we propose eight distinct Hungarian-specific dimensions (see §2.1), cover-
<table>
  <tr>
    <th>Benchmark</th>
    <th>Real user query</th>
    <th>Self-awareness evaluation</th>
    <th>Proverb Reasoning</th>
    <th>Generative task & llm-as-judge</th>
    <th>Hungarian Lang</th>
    <th>Comprehensive Hu-specific</th>
  </tr>
  <tr>
    <td>WildBench (Lin et al., 2024)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>SimpleQA (Wei et al., 2024), ChineseSimpleQA (He et al., 2024)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>MAPS (Liu et al., 2024c)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>MARC, MMMLI U et al in (Lai et al., 2023)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>BenchMAX (Huang et al., 2025)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>MILQA (Novák et al., 2023)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>HuLU (Ligeti-Nagy et al., 2024)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>OpenHuEval (ours)</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison of related benchmarks.

![Overview of OpenHuEval. The figure shows two main sections: Data Construction Process and OpenHuEval. The Data Construction Process includes steps such as obtaining questions from multiple data sources, using Hungarian annotators, and checking questions by LLM. The OpenHuEval section shows a pie chart with categories such as Business and Finance, Language, History, Life, Culture and Customs, Education, and Geography and Place, each with a number indicating tasks. There are also sections for Multiple types of tasks and rule/similarity matching.](page_186_370_1077_393.png)

Figure 1: Overview of OpenHuEval.

ing a variety of scenarios that users may encounter when querying in Hungarian. Guided by these dimensions, we collected a vast amount of Hungarian specific material from multiple sources and used this to construct the corresponding evaluation tasks.

2) Keeping up with the latest advances in LLM evaluation Significant progress has been made in LLM evaluation, with query sources shifting from manual or rule-based constructions to real-world internet questions (Lin et al., 2024), enhancing practical relevance. Question formats evolved from multiple-choice to open-ended Q&A (Wei et al., 2024), better reflecting actual usage. Evaluation methods transitioned from rule-based approaches to LLM-as-judge and subjective assessments, improving accuracy and objectivity (Li et al., 2024a). However, these advancements primarily apply to English datasets and not Hungarian. Thus, when creating OpenHuEval, we incorporated these principles and methodologies from English evaluations.

Based on OpenHuEval, we evaluated the performance of mainstream LLMs on Hungarian language and specifics. We compared the performance differences of these models on the typical datasets of OpenHuEval with corresponding datasets in other languages. The results indicate a significant necessity for evaluation and model optimization specifically for Hungarian language and specifics.

Moreover, Large Reasoning Models (LRMs), like o1, mark a new direction in LLM development. Through extensive reasoning, self-reflective negation, and exploring multiple reasoning paths, they greatly improve reasoning abilities, adhering to the test-time scale law—a crucial step toward AGI. Recent studies (Wang et al., 2025) have analyzed these reasoning processes, offering insights for optimization, but have largely focused on English-language contexts, neglecting Hungarian language and specific scenarios. Building on OpenHuEval, we developed the framework for dissecting the reasoning processes of LRMs. Using Hungarian as the example, we uncovered intrinsic patterns of the representative LRMs in non-English languages. These findings provide valuable insights for the research community to further advance the development of LRMs.

In summary, the contributions of this paper include the following three points:
- We developed OpenHuEval, the first bench-
mark for LLMs focusing on the Hungarian language and specifics. OpenHuEval incorporates the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs’ generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations.

- We conducted the comprehensive evaluation of current mainstream LLMs, including traditional LLMs and recently developed LRM s. The results highlight the significant necessity for evaluation and model optimization tailored to Hungarian language and specifics.

- We established the framework for analyzing the thinking processes of the cutting-edge LRM s, revealing the intrinsic patterns and mechanisms of these models in the Hungarian language and specifics, and providing a reference for research related to other non-English languages.

2 OpenHuEval

OpenHuEval is the benchmark specifically designed to evaluate the performance of LLM in handling Hungarian language and specifics. The overview of OpenHuEval is in Figure 1. Examples are shown in Figure 2.

2.1 Hungarian-specific dimensions and OpenHuEval tasks

Inspired by (Liu et al., 2024b; Sun et al., 2024), OpenHuEval encompasses eight Hungarian-Specific Dimensions (HuSpecificDim), as shown in Table 2: Language (\( \mathcal{L} \)), History (\( \mathcal{H} \)), Life, Culture, and Customs (\( \mathcal{LCC} \)), Education and Profession (\( \mathcal{EP} \)), Geography and Place (\( \mathcal{GP} \)), Figure (\( \mathcal{F} \)), Politics, Policy, and Law (\( \mathcal{PPL} \)), and Business and Finance (\( \mathcal{BF} \)). These dimensions comprehensively cover a wide range of scenarios encountered by users when utilizing Hungarian as the query language. As a result, they enable the systematic evaluation of the performance of LLMs in tasks related to the Hungarian.

Bearing the above HuSpecificDim in mind, the first step in building OpenHuEval involves gathering corpora rich in Hungarian specifics. Inspired by (Liu et al., 2024c; Li et al., 2023; He et al., 2024; Arora et al., 2024), we collected data from sources such as Hungarian proverbs, exam questions, forums, and Wikipedia. Through processes including filtering, refinement, construction, and quality assurance, we developed a total of five evaluation tasks comprising 3953 questions in total, as detailed in Table 3. The subsequent sections of this chapter will introduce these tasks and their corresponding datasets in detail.

2.2 Hungarian WildBench

Task Introduction: The Hungarian WildBench (HuWildBench) task aims to evaluate the performance of LLMs in answering various questions arising from the everyday lives of Hungarians. All questions are sourced from Hungary’s well-known forum website\(^1\) (hereinafter referred to as “g13k” for brevity). These questions cover a wide range of topics, including cultural customs, education, tourism, legal regulations, and business and finance, thus reflect real-life issues encountered by Hungarians. Examples of HuWildBench questions are shown in Figure 2 and Table 21. The queries in HuWildBench are user-generated content, therefore their linguistic expressions and question formats tend to be less formal than the structured and polished written language. This poses the realistic challenge for LLMs, as they must adapt to such informal and spontaneous language style. The construction of HuWildBench is detailed in Appendix D.

Metric and judge: We use the WB-Score (Lin et al., 2024) as evaluation metric for HuWildBench, which is calculated in the following manner: We have simultaneously developped the customized checklist for each question during the process of creating each question, to assist the LLM judge in evaluating the responses. Examples of these checklists can be seen in Table 21. Following (Lin et al., 2024), GPT-4o is used as the judge model, which evaluates the quality of each response based on the checklist and provides detailed strengths and weaknesses before assigning a score from 1 to 10. The definition of scores is shown in Table 8 and the judge prompt is detailed in Appendix D.5. Different from (Lin et al., 2024), our final scores are calculated as the average of all test sample scores, with each score multiplied by 10.

2.3 Hungarian SimpleQA

Task Introduction: Hungarian SimpleQA (HuSimpleQA) is designed to assess the ability of LLMs to answer short, fact-seeking questions related to Hungary. Inspired by (Wei et al., 2024)

\footnotetext{1https://www.gyakorikerdesek.hu/, which is similar to https://www.quora.com/ for English-speaking world.}
a kérés az: Mi lesz a jövőben a szociális és gazdasági alapvetőnek mondott panel? A történelem úgy tudja, hogy kb 60 éves élelmezési tervezetlen okok. Magyarországon (és a környező országokban) is rengeteg ember bennél emelkedik. Mi fog történni akkor, ha lakhatatlanságra kezdenek válni? Mi lesz azzal a sok emberrrel? Meg a panelnek?

The question: What will happen in the future to the many people built under socialism?
Description: I understand they are designed for a life cycle of about 60 years. There are a lot of people living in them. Will they become uninhabitable? What will happen to all those people? And the panels?

HuProverbRea-OF
Hungarian Phrase: "Ajándék lóval ne scéd a fogat" and a context using this phrase:
Hungarian Context:
Speaker1: Képzelj, kapitán egy régi bicikli a szomszédodtól ajándékba, de kicsit rozsdás.
Speaker2: Íme, egy régi bicikli, amit ne scéd a fogat.
What does the person mean by using this phrase?

Hungarian Phrase: Don’t look at a gift horse’s teeth and a context using this phrase:
Hungarian Context:
Speaker1: I got a old bicycle from my neighbor as a gift, but it’s a little rusty.
Speaker2: Don’t worry about it! Don’t look at a gift horse’s teeth.
What does the person mean by using this phrase?

HuSimpleQA
Question 1: “Melyik évben alakult a Nyíregyháza Spartacus FC?” Answer: “1958”
Question 2: “In which year was Nyíregyháza FC founded?” Answer: “1958”
Question 3: “Melyik magyar film nyert a 41. FIPRESCI-díjat az 1983-as Cannes-i Nemzetközi Filmfesztiválon?” Answer: “Szerencsét Dániel”
Question 2: “Which Hungarian film won the FIPRESCI Prize at the 1983 Cannes International Film Festival?” Answer: “Lucky Daniel”

Figure 2: Examples of OpenHuEval. The original text is in black, while the translation into English is in blue. In HuMatchingFIB and HuStandardFIB, the blank is highlighted.

<table>
  <tr>
    <th>HuSpecificDim</th>
    <th>Definition</th>
    <th>#Question</th>
  </tr>
  <tr>
    <td>Language(Ł)</td>
    <td>Basic knowledge of the Hungarian language and Hungarian proverbs and sayings</td>
    <td>1333</td>
  </tr>
  <tr>
    <td>History(Ł)</td>
    <td>Historical events and historical development of Hungary</td>
    <td>324</td>
  </tr>
  <tr>
    <td>Life, Culture, and Custom(ŁCC)</td>
    <td>Religion, rituals, culture, holidays, and the daily life of Hungarians</td>
    <td>622</td>
  </tr>
  <tr>
    <td>Education and Profession(ŁP)</td>
    <td>Education system in Hungary and related professions</td>
    <td>282</td>
  </tr>
  <tr>
    <td>Geography and Place(ŁGP)</td>
    <td>Geographical knowledge of Hungary, cities, and locations</td>
    <td>165</td>
  </tr>
  <tr>
    <td>Figure(ŁF)</td>
    <td>Famous figures of Hungary</td>
    <td>409</td>
  </tr>
  <tr>
    <td>Politics, Policy and Law(ŁPPL)</td>
    <td>Politics, policies, and laws of Hungary</td>
    <td>485</td>
  </tr>
  <tr>
    <td>Business and Finance(ŁBF)</td>
    <td>Business and finance in Hungary</td>
    <td>333</td>
  </tr>
</table>

Table 2: The Hungarian-Specific Dimensions (HuSpecificDim).

and (He et al., 2024), we constructed HuSimpleQA based on Hungarian Wikipedia2, with the following key characteristics. Hungarian: The questions in HuSimpleQA are in Hungarian, and they focus on facts specifically related to Hungary. Diverse: The questions in HuSimpleQA cover the eight Hungary-specific dimensions proposed in §2.1. High-quality: The construction process of HuSimpleQA (in Appendix E) includes comprehensive and strict quality control procedures, ensuring the quality and accuracy of the questions. Static: Similar to SimpleQA, the answers to the questions in HuSimpleQA do not change over time, ensuring that the dataset remains evergreen. Easy-to-evaluate: The questions and answers in HuSimpleQA are short and concise, making them ideal for evaluation through LLMs. The examples of HuSimpleQA are shown in Figure 2 and Table 22. The construction of HuSimpleQA is detailed in Appendix E.

Metric and Judge: Following (Wei et al., 2024), we use GPT-4o as a judge to categorize the responses of the LLM to HuSimpleQA into three classes: CORRECT, INCORRECT or NOT_ATTEMPTED. The judge prompt can be found in Appendix E.7, Figure 27.

2.4 Hungarian Proverb Reasoning

Task Introduction: Hungarian Proverb Reasoning (HuProverbRea), which consists of the collection of Hungarian proverbs, idioms, abbreviations, is a task that requires the LLM to understand and reason the meaning of Hungarian proverbs in a specific context. As shown by the examples in Figure 2, LLM is provided with a context in which a Hungarian proverb is used, accompanied by a question: “What does the speaker mean by the saying?”. Then, the LLM is tasked with discerning the speaker’s true intention, either by selecting the correct interpretation from two provided options (2CQ setting), or by directly articulating the speaker’s intended meaning (OE setting). The construction of HuProverbRea is detailed in Appendix F.

Metric and judge: For the 2CQ setting, we simply measure the correct ratio of candidate LLMs. For the OE setting, we adopt GPT-4o as judge to decide if the answer is acceptable. We provide the original proverb, its context and the English expla-

2https://hu.wikipedia.org/
<table>
  <tr>
    <th>Task</th>
    <th>HuSpecificDim</th>
    <th>Judge</th>
    <th>Question type</th>
    <th>#Question</th>
  </tr>
  <tr>
    <td>HuWildBench</td>
    <td>LCC, E.P., PPL, B.F.</td>
    <td>llm.checklist</td>
    <td>OE</td>
    <td>1154</td>
  </tr>
  <tr>
    <td>HuProverbQA</td>
    <td>L.H.LCC,E.P,G.P,P.PPL,B</td>
    <td>llm</td>
    <td>OE</td>
    <td>1293</td>
  </tr>
  <tr>
    <td>HuProverbRea</td>
    <td></td>
    <td>rule.llm</td>
    <td>2CQ/OE</td>
    <td>1135</td>
  </tr>
  <tr>
    <td>HuMatchingFIB</td>
    <td>L, H</td>
    <td>rule</td>
    <td>Matching Filling-in-Blank</td>
    <td>278</td>
  </tr>
  <tr>
    <td>HuStandardFIB</td>
    <td>L, H</td>
    <td>rule.similarity matching</td>
    <td>Standard Filling-in-Blank</td>
    <td>93</td>
  </tr>
</table>

Table 3: Tasks of OpenHuEval

nation of the proverb as references when judging OE responses. Detailed prompt templates are listed in Appendix F.

2.5 Hungarian Matching and Standard Filling-in-Blank

Task Introduction: Hungarian Matching Fill-in-the-Blank (HuMatchingFIB) is a task where key terms in a text are removed, and a pool of candidate words or phrases is provided. This pool includes both correct answers and distractors. The task requires the LLM to choose the most suitable words from the pool to fill in the blanks, thus restoring the full meaning of the text. The example is shown in Figure 2 and Figure 33. HuMatchingFIB effectively tests the LLM’s abilities in understanding information, reasoning within context, and differentiating correct answers from distractors.

In contrast, Hungarian Standard Fill-in-the-Blank (HuStandardFIB) also involves filling in blanks but does not offer a candidate pool. Instead, the model must rely on its internal knowledge and the provided context to complete the text. The examples are shown in Figure 34. Consequently, HuStandardFIB evaluates the LLM’s overall ability to recall knowledge and reason within context.

The constructions of HuMatchingFIB and HuStandardFIB are detailed in Appendix G.

Metric and Judge: In our inference prompts, we explicitly instruct the LLM to generate responses in a specified format. Additionally, we have established a set of well-defined rules to evaluate the correctness of the LLM’s answers for each blank. The detailed format requirements and judgment criteria can be found in the Appendix G.

For both HuMatchingFIB and HuStandardFIB, we evaluate performance at two levels: blank-level and question-level accuracy. Specifically, Acc_b (Blank-level Accuracy) measures the proportion of blanks that the model answers correctly across all questions. On the other hand, Acc_q (Question-level Accuracy) evaluates the proportion of questions that the model answers entirely correctly. A question is only considered correct if all its associated blanks are answered accurately.

3 Experiments and Analysis

3.1 Experimental setup

We evaluated the currently mainstream LLMs, including GPT-4o (Hurst et al., 2024), GPT-4o mini3, Deepseek-V3 (Liu et al., 2024a), Qwen2.5-Instruct (Yang et al., 2024), and Llama-3.1-Instruct (Dubey et al., 2024), as well as the latest Large Reasoning Models (LRMs) such as OpenAI o1-mini (Jaech et al., 2024), QwQ-32B-Preview (Team, 2024b) (abbreviated as QwQ in following text), and Deepseek-R1 (Team, 2024a). Detailed specifications of these models are provided in Table 4.

We used OpenCompass4 in all our experiments. For traditional instruction-based LLMs, we adopted OpenCompass’s default settings for the maximum output length. For Large Reasoning Models, we set the output length to 8192 to ensure sufficient space for reasoning process and to produce a complete final answer, avoiding premature output truncation. For OpenAI models (GPT series and o1-mini), we used their official API with settings following OpenCompass’s default configuration. For Deepseek-V3 and Deepseek-R1, due to the high usage volume of Deepseek’s official API causing instability, we used equivalent API services provided by Alibaba Cloud5 and Silicon Valley Flow6. The settings followed OpenCompass’s configurations, with the temperature set to 0.7. For other models in Table 4, we performed inference locally with NVIDIA A100 GPUs, using LMDeploy7 as the inference backend. The settings followed OpenCompass’s default configuration (Temperature = 1e-6, top_k = 1).

3.2 Overall performance

The overall performance of all LLMs on OpenHuEval is presented in Table 5. It can be observed that across a total of five tasks, Deepseek-R1 ranks first

3We used the gpt-4o-2024-11-20 version for GPT-4o and the gpt-4o-mini-2024-07-18 version for GPT-4o-mini.
4https://github.com/open-compass/opencompass
5https://cn.aliyun.com/
6https://siliconflow.cn/
7https://github.com/InternLM/lmdeploy
<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Reasoning Model</th>
    <th>Open-source</th>
    <th>Inference Method</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>N</td>
    <td>N</td>
    <td>Official API</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>-</td>
    <td>N</td>
    <td>N</td>
    <td>Official API</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>-</td>
    <td>N</td>
    <td>Y</td>
    <td>Alibaba Cloud and SiliconFlow API</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct</td>
    <td>7B, 72B</td>
    <td>N</td>
    <td>Y</td>
    <td>Local GPU</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct</td>
    <td>8B, 70B</td>
    <td>N</td>
    <td>Y</td>
    <td>Local GPU</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>-</td>
    <td>Y</td>
    <td>N</td>
    <td>Official API</td>
  </tr>
  <tr>
    <td>Q4-Q4</td>
    <td>32B</td>
    <td>Y</td>
    <td>Y</td>
    <td>Local GPU</td>
  </tr>
  <tr>
    <td>Deepseek-R1</td>
    <td>-</td>
    <td>Y</td>
    <td>Y</td>
    <td>Alibaba Cloud and SiliconFlow API</td>
  </tr>
</table>

Table 4: LLMs evaluated in our experiments.

in three tasks and achieves top-tier performance in the other two tasks. GPT-4o ranks first in two tasks and second in the remaining three tasks. These results demonstrate the exceptional performance of the two models in Hungary-specific tasks.

Open-source models vs Closed-source models: Among open-source models, Deepseek-R1 stands out, while Deepseek-V3 also demonstrates strong overall performance, ranking highly across all tasks. Llama-3.1-Instruct-70B achieved impressive scores of 93.83% in the HuProverbRea-2CQ task and 36% in the HuSimpleQA task, ranking second only to the closed-source model GPT-4o. This highlights the growing potential of open-source models, led by Deepseek-R1, which are increasingly showing capabilities comparable to closed-source models in Hungarian language tasks.

Traditional LLMs vs. Large Reasoning Models: We compared Traditional LLMs and LRM within the same series. Across five tasks, Deepseek-R1 consistently outperforms Deepseek-V3 in four of them. Specifically, in the HuMatchingFIB task, Deepseek-R1 achieves relative improvements of 12% at the blank level and 7.19% at the question level compared to Deepseek-V3. Similarly, for the HuStandardFIB task, it achieves gains of 10.32% (blank level) and 7.52% (question level). Although Deepseek-R1 performs slightly worse than Deepseek-V3 on the HuProverbRea task, the performance gap is less than 1%. Considering that both Deepseek-R1 and Deepseek-V3 are based on the same pretrained model, the significantly stronger performance of Deepseek-R1 on the Open-HuEval benchmark demonstrates the effectiveness of LRM architectures in Hungarian language tasks and domain-specific scenarios. This result underscores the potential of LRM as a key avenue of exploration in advancing Artificial General Intelligence (AGI).

Model size: From the results, models with larger parameter sizes perform better on OpenHuEval.

![Three pie charts comparing model performance on Open-HuEval and similar datasets, highlighting that most LLMs experience rank changes.](page_1012_186_377_234.png)

Figure 3: Comparison of model performance on Open-HuEval and similar datasets, highlighting that most LLMs experience rank changes.

For example, GPT-4o, Llama-3.1-Instruct-70B, and Qwen2.5-Instruct-72B outperform their smaller counterparts in the same series (GPT-4o-mini, Llama-3.1-Instruct-7B, and Qwen2.5-Instruct-7B) across all tasks.

3.3 Comparison with Existing Benchmarks

We compare the LLM’s performance rankings on the datasets in OpenHuEval and the existing similar datasets:

HuWildBench vs WildBench (Lin et al., 2024): As shown in Figure 3(a), 70% LLMs experienced ranking changes. Detailed results can be found in Table 9.

HuSimpleQA vs SimpleQA (Wei et al., 2024): As shown in Figure 3(b), 70% LLMs experienced ranking changes. Detailed results can be found in Table 12.

HuProverbRea vs MAPS: HuProverbRea was constructed with reference to MAPS (Liu et al., 2024c), which is the proverb reasoning dataset comprising six subsets, each corresponding to a different language: English (en), German (de), Russian (ru), Bengali (bn), Mandarin Chinese (zh), and Indonesian (id). We compared the model performance rankings on HuProverbRea and each subset of MAPS. As shown in Figure 3(c), the percentage of LLMs with ranking changes were: 70% for en, 80% for zh, 90% for ru & bn, and 100% for id & de. Detailed results can be found in Table 15 and Table 16.

These results underscore the importance of evaluating LLMs on Hungarian proverbs and Hungarian-specific questions, highlighting the need for targeted optimization of models to better handle language-specific proverbs and cultural nuances across diverse languages.
<table>
  <tr>
    <th>Model</th>
    <th>HuWildBench<br>WBScore</th>
    <th>HuSimpleQA<br>Acc.</th>
    <th>HuProverbRea<br>Acc. (OE)</th>
    <th>HuProverbRea<br>Acc. (2CQ)</th>
    <th>HuMatchingFIB<br>B acc.</th>
    <th>HuMatchingFIB<br>Q acc.</th>
    <th>HuStandardFIB<br>B acc.</th>
    <th>HuStandardFIB<br>Q acc.</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>81.09</td>
    <td>50.3</td>
    <td>89.16</td>
    <td>95.51</td>
    <td>77.00</td>
    <td>43.88</td>
    <td>57.26</td>
    <td>15.05</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>74.19</td>
    <td>25.56</td>
    <td>84.67</td>
    <td>92.11</td>
    <td>55.68</td>
    <td>18.00</td>
    <td>35.08</td>
    <td>7.53</td>
  </tr>
  <tr>
    <td>QwQ</td>
    <td>58.02</td>
    <td>9.09</td>
    <td>67.49</td>
    <td>84.23</td>
    <td>38.65</td>
    <td>12.23</td>
    <td>6.05</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Deepseek-R1</td>
    <td><b>82.96</b></td>
    <td><b>34.58</b></td>
    <td>82.29</td>
    <td>91.72</td>
    <td><b>80.87</b></td>
    <td><b>47.12</b></td>
    <td><b>61.76</b></td>
    <td><b>17.2</b></td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>78.42</td>
    <td>32.71</td>
    <td>83.26</td>
    <td>92.51</td>
    <td>68.87</td>
    <td>39.93</td>
    <td>51.44</td>
    <td>9.68</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct-70B</td>
    <td>61.78</td>
    <td><b>35.99</b></td>
    <td>80.18</td>
    <td>93.83</td>
    <td>59.56</td>
    <td>24.46</td>
    <td>40.99</td>
    <td>6.45</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct-8B</td>
    <td>53.62</td>
    <td>15.2</td>
    <td>63.35</td>
    <td>73.48</td>
    <td>5.74</td>
    <td>0.72</td>
    <td>16.64</td>
    <td>1.08</td>
  </tr>
  <tr>
    <td>01-mini</td>
    <td>76.43</td>
    <td>15.8</td>
    <td>77.44</td>
    <td>87.67</td>
    <td>60.83</td>
    <td>17.63</td>
    <td>45.25</td>
    <td>13.98</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct-72B</td>
    <td>74.05</td>
    <td>14.9</td>
    <td>77.8</td>
    <td>90.22</td>
    <td>63.8</td>
    <td>24.1</td>
    <td>32.32</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct-7B</td>
    <td>42.01</td>
    <td>5.22</td>
    <td>50.48</td>
    <td>67.05</td>
    <td>31.88</td>
    <td>1.08</td>
    <td>7.43</td>
    <td>0</td>
  </tr>
</table>

Table 5: Overall performance of 10 LLMs on OpenHuEval. The first, second, and third place in each metric are marked with red, green, and blue text, respectively. In the FIB task evaluation metric, B represents the blank level, and Q represents the question level.

4 Framework for Analyzing the Thinking Process of LRM

When responding to the user’s query, the LRM’s response typically consists of two parts: the thinking process and the answer.8 We developed the framework for the in-depth analysis of the LRM’s thinking process on OpenHuEval. For LRM, we chose Deepseek-R1 and QwQ, as these are the only two models with accessible reasoning processes.

4.1 Task Selection

Among the OpenHuEval tasks, we selected HuSimpleQA and HuMatchingFIB as our subjects of study. Unlike recent work (Wang et al., 2025), which focuses solely on math reasoning datasets, the two tasks we selected each have distinctive characteristics: HuSimpleQA assesses the LLM’s ability to recall and retrieve Hungarian-specific knowledge, as well as its awareness of its own knowledge boundaries. HuMatchingFIB involves questions where multiple competitive blanks exist within the same problem, requiring the model to carefully choose which answers to fill in. Therefore, analyzing these tasks allows us to explore the reasoning mechanisms of LRMs in both broader and deeper contexts, providing the research community with more valuable conclusions and insights.

4.2 Analysis on HuSimpleQA

Method: Following (Wang et al., 2025), the LRM’s thinking process can be broken down into “thoughts”. A “thought” refers to the intermediate cognitive step output by a LRM during its thinking process. Throughout the thinking process, the LRM transitions between multiple thoughts, which are typically separated by reflective phrases such as “Alternative”, “Várni”(wait). The examples of the thoughts and the transitions can be found in Figure 40 and Figure 41. Thoughts can be further classified as “correct” or “incorrect”: reasoning along correct thoughts leads to CORRECT responses, while incorrect thoughts result in INCORRECT.

The LRM’s responses to HuSimpleQA have been judged as CORRECT, INCORRECT, or NOT_ATTEMPTED. Then we used GPT-4o to split the thinking process into thoughts (see the prompt in Figure 37 and Figure 38). We evaluated the correctness of each thought (see the prompt in Figure 39), with examples provided Figure 40 and Figure 41 in Appendix H.

Efficiency of thinking process: We measure the length of the process (in terms of token count) and the number of thoughts under the three evaluation outcomes of the responses, as shown in Figure 5. The results indicate that both the reasoning length (in tokens) and the thought count were generally shorter for Deepseek-R1 compared to QwQ. Con-

![Method for analyzing the LRM’s thinking process on HuSimpleQA.](page_1012_320_388_180.png)

Figure 4: Method for analyzing the LRM’s thinking process on HuSimpleQA.

![Average number of tokens and thoughts generated per response on HuSimpleQA.](page_1012_520_388_180.png)

Figure 5: Average number of tokens and thoughts generated per response on HuSimpleQA.
sidering that Deepseek-R1 performs better than QwQ on the HuSimpleQA task, it suggests that Deepseek-R1 achieves its superior performance with relatively lower reasoning overhead.

Confidence in thinking process: For Deepseek-R1, the reasoning length and thought count showed no significant differences across the three types of evaluation outcomes. In contrast, for QwQ, the length and the number of thoughts were significantly higher in the NOT_ATTEMPTED cases compared to the other two types. This observation suggests that, compared to Deepseek-R1, QwQ is less “confident”, which tends to repeatedly attempt generating answers when faced with uncertainty and is more inclined to abstain from answering altogether.

Correct thoughts in INCORRECT responses: In HuSimpleQA, we analyzed the ratio of INCORRECT and NOT_ATTEMPTED responses from Deepseek-R1 and QwQ that contained correct thoughts. We compared these results with those from MATH500-Hard as reported in (Wang et al., 2025), as shown in Figure 6. In the math reasoning task (MATH500-Hard), the significant portion of LRM’s ultimately incorrect responses still included correct thoughts: 72% for Deepseek-R1 and 78% for QwQ contained at least one correct thought. However, in HuSimpleQA, the ratio of INCORRECT responses containing correct thoughts dropped to 46% for Deepseek-R1 and 42% for QwQ. These findings suggest that the reasoning processes of LRMs differ significantly between memory-intensive tasks (HuSimpleQA) and reasoning-intensive tasks (MATH500-Hard), highlighting the need for targeted analysis and optimization.

![A line graph showing the distribution of thought correctness ratio in INCORRECT and NOT_ATTEMPTED responses for HuSimpleQA and MATH500-Hard.](page_109_109_495_384.png)

Figure 6: The distribution of thought correctness ratio in INCORRECT and NOT_ATTEMPTED responses. Red lines for HuSimpleQA and blue lines for MATH500-Hard from (Wang et al., 2025).

![A flowchart diagram showing the method for analyzing the LRM’s thinking process on HuMatchingFIB.](page_624_109_495_384.png)

Figure 7: Method for analyzing the LRM’s thinking process on HuMatchingFIB.

![A screenshot of an example for splitting DeepSeek-R1’s thinking process into segments and categorizing these segments on HuMatchingFIB.](page_624_573_495_384.png)

Figure 8: Example for splitting DeepSeek-R1’s thinking process into segments and categorizing these segments on HuMatchingFIB.

4.3 Analysis on HuMatchingFIB

Method: Unlike HuSimpleQA where each query contains only one question, HuMatchingFIB involves multiple competitive blanks within the same question that need to be filled (see the example in Figure 2 and Figure 33). Based on extensive case studies, we have developed the analytical method to facilitate further in-depth analysis, as illustrated in Figure 7.

Initially, we split the thinking process of LRMs into several segments, categorizing each as introduction, reasoning, review, or summary. Typically, each thinking process begins with an introduction segment, includes several
Figure 9: Tagging results of LRM’s reasoning segments in the correctness and complexity dimensions.

reasoning segments and some review segments in the middle, and concludes with a summary segment. In each reasoning segment, the LRM addresses one or more blanks through reasoning. During the review segment, there is usually a reflection on the completed blanks, with potential revisions to the answers for earlier blanks. Examples of these segments can be found in Figure 8, and detailed definitions are provided in Table 18.

We further tagged the reasoning segments according to the following four dimensions: (Dim1) correctness: Are the answers in this reasoning segment correct? (Dim2) complexity: In this reasoning segment, does the LRM simply assert the answer, or does it involve more complex reasoning? (Dim3) scope: Does this reasoning segment focuses on a single blank, modifies previous blanks, or addresses multiple blanks? (Dim4) language transfer: Does the LRM switch languages within this reasoning segment? The details of the tagging can be found in Appendix H.2.2. Examples of the tagging results can be found in Figure 47 and Figure 48.

Simple Assertion or Complex Thought? As shown in Table 19, the reasoning segments can be categorized into two classes. The first is referred to as Simple Assertion, where LRM directly provides the answer to the blank. The second type is termed Complex Thought, where the segment involves repeated thinking, logical reasoning, hypothesis validation, or other complex processes. Examples can be found in Figure 47 and 48.

We analyzed the correctness of the reasoning segments for both Simple Assertion and Complex Thought (tagging results of dim1), as shown in Figure 9. By comparing the statistical distributions of the Deepseek-R1 and QwQ models, we noted the following: Firstly, the proportion of Simple Assertion segments that are not Completely Correct is quite low (3% for Deepseek-R1 and 6% for QwQ), indicating that both models achieve high accuracy when resolving blanks through Simple Assertion. This suggests that the models’ calibration is relatively reliable, implying that the models “know what they know”. Secondly, the proportion of Simple Assertion segments that are Completely Correct is significantly higher for Deepseek-R1 at 49%, compared to only 31% for QwQ. This difference reflects the performance disparity between the two models in the thinking processes. Thirdly, the correctness for Complex Thought is notably lower than for Simple Assertion, and both models show a higher proportion of cases where no conclusion is reached for the Complex Thought tag. This indicates that when faced with unfamiliar problems, the LRM could actively extend the reasoning and analysis process.

Explicit Translation Insertion (ETI): We observed that in some reasoning segments, when faced with the problem in Hungarian, the LRM first translates the key phrase of the original question into English and then proceeds with analysis and reasoning based on this translation. For example, "... Erőteljes #3# és a költői #4# gazdag használata jellezi. This translates to "It is characterized by strong #3# and rich use of poetic #4#." ...". We refer to this phenomenon as Explicit Translation Insertion (ETI). Statistical analysis shows that ETI occurs in 42.5% of DeepSeek-R1’s reasoning segments, while for QwQ, the proportion is 31.9%. This demonstrates that the LRM can adaptively translate input from the language in which it is relatively less proficient (such as Hungarian) into the more proficient language (such as English) during its thinking process. By scaling up the length of reasoning during the test stage, the LLM can partially overcome the cross-language performance discrepancies typical of traditional instruction-based LLM.

rect is quite low (3% for Deepseek-R1 and 6% for QwQ), indicating that both models achieve high accuracy when resolving blanks through Simple Assertion. This suggests that the models’ calibration is relatively reliable, implying that the models “know what they know”. Secondly, the proportion of Simple Assertion segments that are Completely Correct is significantly higher for Deepseek-R1 at 49%, compared to only 31% for QwQ. This difference reflects the performance disparity between the two models in the thinking processes. Thirdly, the correctness for Complex Thought is notably lower than for Simple Assertion, and both models show a higher proportion of cases where no conclusion is reached for the Complex Thought tag. This indicates that when faced with unfamiliar problems, the LRM could actively extend the reasoning and analysis process.

Explicit Translation Insertion (ETI): We observed that in some reasoning segments, when faced with the problem in Hungarian, the LRM first translates the key phrase of the original question into English and then proceeds with analysis and reasoning based on this translation. For example, "... Erőteljes #3# és a költői #4# gazdag használata jellezi. This translates to "It is characterized by strong #3# and rich use of poetic #4#." ...". We refer to this phenomenon as Explicit Translation Insertion (ETI). Statistical analysis shows that ETI occurs in 42.5% of DeepSeek-R1’s reasoning segments, while for QwQ, the proportion is 31.9%. This demonstrates that the LRM can adaptively translate input from the language in which it is relatively less proficient (such as Hungarian) into the more proficient language (such as English) during its thinking process. By scaling up the length of reasoning during the test stage, the LLM can partially overcome the cross-language performance discrepancies typical of traditional instruction-based LLM.

5 Conclusion

In this paper, we constructed the first benchmark for LLMs focusing on the Hungarian language and its specifics. The results highlight the significant need for evaluation and model optimization tailored to Hungarian language and specifics. We also developed the framework for analyzing the thinking processes of the cutting-edge LMRs. Our work not only advances LLM technology in Hungarian but also provides valuable insights for studying languages of other countries and regions.
6 Limitation

Given the rapid advancements in English evaluation datasets, this research represents only an initial step towards bridging the gap between Hungarian and English evaluation resources. Currently, evaluation datasets for less-resourced languages like Hungarian still lag behind their English counterparts in terms of depth and breadth. Moving forward, we plan to closely monitor developments in English evaluation methodologies, continually refining and enhancing evaluation techniques and datasets for low-resource languages to reduce this disparity.

Moreover, as the field of LLMs evolves rapidly, many promising models, especially those tailored for low-resource languages, remain under-evaluated. Our future goal is to establish a vibrant OpenHuEval community that will regularly update evaluation results for the latest models. This will ensure comprehensive and up-to-date assessments, fostering the optimization and development of models in the low-resource language domain.

7 Ethical Consideration

This work involved human annotation. For all annotators, we explicitly informed them about the use of the data and required them to ensure that the questions included in OpenHuEval do not involve any social bias, ethical issues or privacy concerns during the annotation process.

Acknowledgments

This research was supported by National Key R&D Program of China (2022ZD0160201).

References

David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. 2023. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. arXiv preprint arXiv:2309.07445.

Kabir Ahuja, Harshita Diddie, Rishav Hada, Milli-cent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528.

Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyer, and Eunsol Choi. 2024. Calmqa: Exploring culturally specific long-form question answering across 23 languages. arXiv preprint arXiv:2406.17761.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308.16884.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369.

Yancheng He, Shilong Li, Jaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. 2024. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR.

Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jincheng Liu, Yuzhuang Xu, Jinan Xu, et al. 2024. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936.

Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. 2025. Benchmax: A comprehensive multilingual evaluation suite for large language models. Preprint, arXiv:2502.07346.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Al Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374, Singapore. Association for Computational Linguistics.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. 2024. Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840.

Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Demenoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv preprint arXiv:2307.16039.

Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024a. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.

Jialin Li, Junli Wang, Junjie Hu, and Ming Jiang. 2024b. How well do llms identify cultural unity in diversity? arXiv preprint arXiv:2408.05102.

Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, and Tamás Váradi. 2024. Hulu: Hungarian language understanding benchmark kit. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8360–8371.

Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeez Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2024b. Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art. arXiv preprint arXiv:2406.05930.

Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024c. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. Preprint, arXiv:2309.08591.

Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems, 37:78104–78146.

Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456.

Attila Novák, Borbála Novák, Tamás Zombori, Gergő Szabó, Zsolt Szántó, and Richárd Farkas. 2023. A question answering benchmark database for hungarian. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 188–198.

Mátyás Osváth, Zijian Győző Yang, and Karolina Kósa. 2023. Analyzing narratives of patient experiences: A bert topic modeling approach. Acta Polytechnica Hungarica, 20(7):153–171.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2024. mcsqa: Multilingual commonsense reasoning dataset with unified creation strategy by language models and humans. arXiv preprint arXiv:2406.04215.

Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548.

Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, and Conghui He. 2024. Benchmarking chinese commonsense reasoning of llms: From chinese-specifics to reasoning-memorization correlations. arXiv preprint arXiv:2403.14112.

DeepSeek Team. 2024a. Deepseek-rl-lite-preview is now live: unleashing supercharged reasoning power.

Qwen Team. 2024b. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face.

Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. 2025. Thoughts are all over the place: On the underthinking of ol-like llms. arXiv preprint arXiv:2501.18585.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Da Yin, Hritik Bansal, Masoud Monajatipoor, Lillian Harold Li, and Kai-Wei Chang. 2022. Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models. arXiv preprint arXiv:2205.12247.

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, and Hinrich Schütze. 2024. Turkishmlu: Measuring massive multitask language understanding in turkish. arXiv preprint arXiv:2407.12402.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Yidan Zhang, Boyi Deng, Yu Wan, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, and Jingren Zhou. 2024. P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms. arXiv preprint arXiv:2411.09116.
A Related Works

A.1 Hungarian benchmarks for LLMs from Translation

Directly translating an existing English evaluation dataset into non-English is an effective and straightforward method for constructing non-English evaluation datasets. Many existing multilingual evaluation datasets are constructed using this approach, such as those described in (Ahuja et al., 2023; Huang et al., 2024). Among these, quite a few evaluation datasets include Hungarian as one of the languages, such as (Hu et al., 2020) and (Adelani et al., 2023). (Lai et al., 2023) utilized GPT-3.5 to translate datasets such as ARC (Clark et al., 2018), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2021), and HellaSwag (Zellers et al., 2019) into Hungarian. Belebele (Bandarkar et al., 2023) includes Hungarian as part of its multilingual parallel corpus reading comprehension evaluation dataset. BenchMaX (Huang et al., 2025) is a multilingual parallel corpus evaluation dataset focused on comprehensively assessing LLMs’ capabilities across various generative tasks.

These translation-based parallel corpus evaluation datasets provide a comprehensive assessment of LLMs’ language-agnostic capabilities in Hungarian, including areas like world knowledge, mathematical reasoning, logical reasoning, and code generation. However, they overlook the unique characteristics and linguistic features of Hungarian, such as language nuances, culture, history, and regional context, which are critical for Hungarian users.

A.2 Language specific benchmarks for LLMs

There has been extensive research focusing on evaluating the unique features and capabilities of languages (Liu et al., 2024b). Some studies have constructed benchmarks similar to MMLU (Hendrycks et al., 2020) by collecting exam questions specific to various countries, such as CMMLU (Li et al., 2023), IndoMMLU (Koto et al., 2023), ArabicMMLU (Koto et al., 2024), KMMILU (Son et al., 2024), and TurkishMMLU (Yüksel et al., 2024). Other studies have built evaluation datasets by crawling material and user queries from internet forums, filtering for queries related to linguistic and cultural features, such as the benchmark in (Naous et al., 2023) and CaLMQA (Arora et al., 2024). Some works manually construct evaluation datasets that emphasize linguistic and cultural characteristics, such as BLEnD (Myung et al., 2024), CHARM (Sun et al., 2024), and HuLU (Ligeti-Nagy et al., 2024). Others adopt a “LLM generation combined with human expert modification and review” method to construct culturally characteristic evaluation datasets, such as MAPS (Liu et al., 2024c), mCSQA (Sakai et al., 2024), and ChineseSimpleQA (He et al., 2024). Although Hungarian is rarely covered in these works, they provide crucial inspiration and approaches for our work.

Among these works, MILQA (Novák et al., 2023) is the Hungarian question answering benchmark created mainly following the SQuAD (Rajpurkar et al., 2016). HuLU (Ligeti-Nagy et al., 2024) is a comprehensive Hungarian evaluation benchmark kit 9 that includes a total of seven tasks and corresponding datasets. Of the seven tasks, four are constructed by translating existing English evaluation datasets, and three are manually created based on native Hungarian corpora. However, HuLU only supports multiple-choice and true/false questions, which limits its ability to assess broader LLM capabilities such as language generation, open-domain Q&A, reasoning, and instruction-following.

B Hungarian-Specific Dimensions

OpenHuEval encompasses eight Hungarian-Specific Dimensions (HuSpecificDim), as shown in Table 2: Language (\( \mathcal{L} \)), History (\( \mathcal{H} \)), Life, Culture, and Customs (\( \mathcal{LCC} \)), Education and Profession (\( \mathcal{EP} \)), Geography and Place (\( \mathcal{GP} \)), Figure (\( \mathcal{F} \)), Politics, Policy, and Law (\( \mathcal{PPL} \)), and Business and Finance (\( \mathcal{BF} \)).

To select these Hungarian-Specific Dimensions, e first conducted extensive literature research to summarize the knowledge dimensions in existing cultural benchmarks, as shown in Table 6.

Based on above and by considering the knowledge needs of typical Hungarian users and the content on authoritative Hungarian websites, we established the Hu-specific dimensions for OpenHuEval. We believe these dimensions cover most user needs related to Hungarian specifics.

C Inference Detail

C.1 Default Maximum Output Length for LLM in Opencompass

For traditional instruction-based LLMs, we adopted OpenCompass’s default settings for the

9https://hulu.nytud.hu/
<table>
  <tr>
    <th>Benchmark</th>
    <th>Dimension</th>
  </tr>
  <tr>
    <td>CHARM (Sun et al., 2024)</td>
    <td>History, Traditional Culture and Arts, Daily Life and Customs, Entertainment, Public Figures, Geography, Chinese Language</td>
  </tr>
  <tr>
    <td>CUNIT (Li et al., 2024b)</td>
    <td>Clothing, Food</td>
  </tr>
  <tr>
    <td>BLENd (Myung et al., 2024)</td>
    <td>Food, Sports, Family, Education, Holidays/Celebrations/Leisure, Work-Life</td>
  </tr>
  <tr>
    <td>CultureAtlas (Fung et al., 2024)</td>
    <td>Culture, Holidays, Dining Etiquette, Education, Honorifics, etc.</td>
  </tr>
  <tr>
    <td>GEOMLAMA (Yin et al., 2022)</td>
    <td>Habits and Personal Choices, Cultures and Customs, Policies and Regulations, Geography</td>
  </tr>
</table>

Table 6: The knowledge dimensions present in existing cultural benchmarks.

<table>
  <tr>
    <th>Model</th>
    <th>Max out length</th>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct-7B</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct-72B</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>GPT4-o</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>GPT4o-mini</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct-8B</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct-70B</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>2048</td>
  </tr>
</table>

Table 7: Default maximum output length for LLM in OpenCompass.

![Flowchart showing the construction pipeline of HuWildBench, including steps such as crawling user questions, filtering Hungarian-specific queries, removing duplicates, manual filtering, and creating a checklist.](page_324_1012_797_312.png)

Figure 10: Constuction of HuWildBench.

maximum output length, as shown in Table 7.

D HuWildBench

D.1 Overall Construction Pipeline of HuWildBench

The construction of HuWildBench contains the following steps(see Figure 10):

(1) Crawling: All user queries on the g13k website10 are systematically categorized into a multi-level tag system, which consists of 27 primary tags and 231 secondary tags. We manually reviewed all the secondary tags and selected 37 of them that contain a higher number of questions related to Hungary, such as népszokások (folk customs), egészségügyi-ellátások (healthcare services), and rezsí (overheads). We then crawled user queries under these 37 secondary tags, with a query date range from January 1, 2019, to August 31, 2024, resulting in the dataset \( Q_{crawl} \)(approximately 523K queries).

(2) Filtering for Hungary-specific content: Although these 37 secondary tags are closely related to Hungary, many of the questions still do not focus on Hungary-specific topics. Therefore, we used GPT-4o to classify the questions in \( Q_{crawl} \) (detailed prompt in Figure 13), resulting in a subset of approximately 260K questions, \( Q_{specific} \)

(3) Deduplication: To ensure the diversity of questions, we performed deduplication on the Hungary-related questions within each secondary tag. The detailed process is outlined in Appendix D.2. After deduplication, the number of user questions in \( Q_{dedup} \) was approximately 134K.

(4) Automatic high-quality question filtering: To ensure that only high-quality questions are extracted from the question pool, we designed a comparative-based high-quality question filtering strategy, as detailed in Appendix D.3. After filtering, the resulting set \( Q_{high\_quality} \) contained around 2K questions.

(5) Manual filtering: We hired a group of Hungarian native speakers to further manually review the questions in \( Q_{high\_quality} \). Only questions that met the following two criteria were retained: First, the question should be Hungary-specific and closely related to Hungary. Second, the question must be harmless, meaning it does not contain inappropriate content such as pornography, violence, politics, or taboo topics specific to Hungary. The final set \( Q_{manual\_check} \) consists of 1731 questions.

(6) Checklist construction: Based on WildBench (Lin et al., 2024), we constructed a checklist for each question. The purpose of the checklist is to assist the LLM judger in evaluating the answers. Each item in the checklist queries a specific
aspect of the answer to a question. An example of the checklist can be found in Table 21, and the detailed construction method is provided in Appendix D.4. To ensure the relevance of the checklist items to the questions, we hired a Hungarian native speaker to review the checklist for quality, filtering out non-compliant items and performing deduplication. The filtering criterion was whether the item was suitable as an evaluation dimension for the model’s response. To ensure the reliability of the LLM-as-judge, we filtered out user questions with fewer than 8 checklist items. The final set \( Q_{checklist} \) contains 1154 questions. In the end, we obtained 1154 user questions along with their corresponding checklists.

D.2 Deduplication of similar questions
Since there are similar questions in the results obtained in the previous step, we design a method to remove similar ones. Specifically, we first use the SentenceTransformer (Osváth et al., 2023)\(^{11}\) model to extract the embedding vector of each question. Then, we calculate the cosine similarity between the embedding of each two questions, and choose a threshold between [0.15-0.25] according to the number of questions under each secondary tag. The larger the number of problems, the larger the threshold. Finally, one of the questions whose similarity is less than the threshold is removed, ensuring that the similarities between all questions are greater than the threshold.

D.3 Automatic high-quality question filtering
In order to automate the filtering of high-quality sample pots, we constructed the prompt that allows the GPT-4o to select the two best Hungarian questions out of the five based on the criteria of linguistic complexity, Hungarian relevance, common-sense accuracy, context-dependence, answer diversity, ambiguity, reasoning requirements, socio-ethical considerations, format diversity, and breadth of knowledge and outputs their indexes in JSON format to output their indexes, as shown in Figure 14. Specifically, we first set the criteria for high-quality questions in Prompt. Then we ask GPT-4o to compare the input questions based on the criteria. In order to mitigate the occurrence of some high-quality questions being eliminated prematurely (or vice versa) when all the questions in the same batch are of high quality, we follow the following 3 rules when filtering the high-quality questions: (1) filter 2 high-quality questions from 5 questions at a time, instead of filtering 1 high-quality question directly from 2 questions. (2) use the Swiss system mechanism instead of the knockout mechanism. In each screening round, each question can win in the current round as long as it ensures that it wins in two comparisons, and it will not be eliminated directly because of a failure in one comparison. (3) Our question screening strategy eliminates 65% of the questions in each round, in order to ensure that each secondary label has a sufficient number of high-quality questions. We conducted different elimination rounds for questions under different labels, and finally got about 2K questions.

To ensure the reasonableness and robustness of the standard, we reviewed the results obtained using it. We selected 500 sets of questions filtered by GPT-4o according to this standard (where the model had to select two high-quality questions from each set of five). Human experts annotated these sets, and the overlap rate between the selections of the human experts and the model was 81% (a set was considered overlapping if the two questions chosen by the model matched those chosen by the experts). This demonstrates the effectiveness of the standard for selecting high-quality questions. The model’s recall rate for the questions selected by the human experts was 90%.

The final constructed HuWildBench is shown in Table 21.

D.4 Checklist construction
In the process of building the Checklist, we mainly use large language models to generate it. In order to ensure the diversity of the Checklist and make the judge model can better evaluate the quality of the answers, here we use two non-open source LLM GPT-4o and Claude-3.5, each model generates a list of length 3-5. then we merge the two Checklists into one final Checklist. Checklists are then merged into a final Checklist. ultimately, each problem has a length of 6-10 and a Checklist. The details of our designed Prompt are shown in Figure 15 and the final constructed partial Checklist is shown in Table 21.

D.5 LLM-as-judge
Following WB-Score (Lin et al., 2024), we use GPT-4o as the judge model to evaluate the score of responses generated by LLM. The prompt used in
<table>
  <tr>
    <th>Score</th>
    <th>Definition</th>
  </tr>
  <tr>
    <td>Score 1-2</td>
    <td>The response is very poor and does not make sense at all.</td>
  </tr>
  <tr>
    <td>Score 3-4</td>
    <td>The response is poor and does not help the user solve the problem meaningfully.</td>
  </tr>
  <tr>
    <td>Score 5-6</td>
    <td>The response is fair but has issues (e.g., factual errors, hallucinations, missing key information).</td>
  </tr>
  <tr>
    <td>Score 7-8</td>
    <td>The response is good but could be improved.</td>
  </tr>
  <tr>
    <td>Score 9-10</td>
    <td>The response is perfect and provides helpful information to solve the problem.</td>
  </tr>
</table>

Table 8: Definition of WB-Scores.

<table>
  <tr>
    <th>Rank</th>
    <th>WildBench</th>
    <th>HuWildBench</th>
  </tr>
  <tr>
    <td>1</td>
    <td>DeepSpeed-V3</td>
    <td>DeepSpeed-R1(1)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>GPT-4o</td>
    <td>GPT-4o(-)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>ol-mini</td>
    <td>DeepSpeed-V3(2)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>DeepSpeed-R1</td>
    <td>ol-mini(1)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Qwen2.5-Instruct-72B</td>
    <td>GPT-4o-mini(1)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>GPT-4o-mini</td>
    <td>Qwen2.5-Instruct-72B(1)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>Llama-3.1-Instruct-70B</td>
    <td>Llama-3.1-Instruct-70B(1)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>QwQ</td>
    <td>QwQ(-)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>Qwen2.5-Instruct-7B</td>
    <td>Llama-3.1-Instruct-8B(1)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Llama-3.1-Instruct-8B</td>
    <td>Qwen2.5-Instruct-7B(1)</td>
  </tr>
</table>

Table 9: Performance rankings on WildBench and HuWildBench.

judge is shown in Figure 16.

D.6 Comparing the Performance Ranking on WildBench and HuWildBench

We compare the performance rankings of LLM on HuWildBench and WildBench (Lin et al., 2024), as shown in Table 9.

E HuSimpleQA

E.1 Construction pipeline of HuSimpleQA

The construction of HuSimpleQA consists of the following steps, as shown in Figure 11:

(1) Obtaining corpora rich in Hungary-specific facts and knowledge: First, we chose the Hungarian Wikipedia website.12 as the source of corpus material for question construction. We crawled all the entry pages and extracted their content. Next, we used GPT-4o to classify whether the entries were Hungary-specific, with the prompts detailed in Appendix E.2, Figure 17.13 We then used GPT to extract key information from the content of these entries suitable for question-answering. The extraction prompts are detailed in Appendix E.2, Figure 18. An example of the extracted key information is shown in Figure 19. As a result, we obtained Hungary-specific key information covering the eight distinct dimensions, totaling 4428 pieces of information.

(2) Generating questions and corresponding answers: We used the GPT-4o model to generate open-ended questions and corresponding answers based on the Hungary-specific key information obtained in the previous step. The prompt used is detailed in Appendix E.3, Figure 20. In this step, we generated a total of 9424 questions based on 4K entries. We then classified the generated questions according to the eight Hungary-specific dimensions outlined in Section 2.1, using GPT-4o with the prompt detailed in Appendix E.3, Figure 21.

(3) Automatic quality checking of questions: To ensure the quality of the questions, we used GPT-4o to check and filter the generated questions. We set the following four criteria, retaining only those questions that met all four standards (the corresponding prompt is detailed in Appendix E.4):
- Criterion 1: Hungary-specific: The content of the question-answer pair must align with the eight Hungary-specific dimensions proposed in this paper.
- Criterion 2: Accuracy: The information in selected pages until we reached 8K Hungary-specific entries, at which point we stopped.

![Construction of HuSimpleQA.](page_1012_186_484_627.png)

Figure 11: Construction of HuSimpleQA.
the question-answer pair must align with the entry description and facts, and the answer should not be directly inferable from the question itself.

- Criterion 3: Concise and specific: The question and answer should be clear and concise, with no redundant information. The question should not contain nested sub-questions. The phrasing should be specific and direct, matching the scope of the answer (e.g., for time and location questions, the exact year/month/day/district/city must be specified).

- Criterion 4: Consistency Over Time: The answer should remain consistent over time and not be influenced by future events.

After the automatic checking process, we retained 5503 questions corresponding to 2666 entries.

(4) Manual review of question quality: To further ensure the quality of the questions, we hired Hungarian native speakers to manually review the questions. Annotators checked whether the questions met the four criteria mentioned in Step 3. During the annotation process, each question was assigned to two annotators, who received the questions but not the answers. A question was considered valid and retained only if both annotators agreed that it met all four criteria and that the provided answer matched the original reference answer. Detailed procedures are provided in Appendix E.4. After these four steps, we obtained a total of 1293 questions, with their distribution across the eight Hungary-specific dimensions shown in Table 22.

E.2 Obtaining corpora rich in Hungary-specific facts and knowledge

In the process of filtering Wikipedia entries with Hungarian characteristics, we randomly selected entries and provided both the entries and the first two paragraphs of the main content to GPT-4o (prompt shown in Figure 17) to determine if they were related to Hungary. If the entry was deemed relevant, it was categorized based on the eight characteristic dimensions proposed in this paper. At this stage, an “Others” category was added to ensure the focus on the eight thematic categories and to exclude interference from entries that belonged to other themes. The screening process stopped once the total number of Hungarian characteristic entries reached 8,000.

Due to the uneven distribution of entry themes on Wikipedia, with more data in the categories of figures, geography and place, and history, we filtered the data based on the proportion of themes, ensuring that no single category exceeded 1,000 entries. This resulted in 4428 characteristic entries covering the eight dimensions.

Given the varying lengths of content describing entries on Wikipedia, we aimed to streamline the complexity of constructing subsequent question-answer pairs. To achieve this, we first employed GPT-4o to extract key information from the main text of each entry. This step aims to avoid any deviation from the theme caused by redundant content during the construction of the question-answer pairs (prompt shown in Figure 18). The results of the key information extraction are presented in Figure 19.

E.3 Generating questions and corresponding answers

Based on the key information extracted and the provided entries, we utilized GPT-4o to generate 1-3 Hungarian open-ended question-answer pairs for each entry (prompt details in Figure 20). In total, 9,424 question-answer pairs were generated based on 4,000 entries.

Given that the focus and orientation of the generated question-answer pairs may differ from the original entry categories, this paper employed GPT-4o to reclassify the obtained question-answer pairs, with the corresponding prompt detailed in Figure 21.

E.4 Automatic quality checking of questions

We focused on evaluating the quality of the generated questions from two perspectives: the information contained in the question-answer pairs and the formulation of the questions. The quality assessment was divided into two stages, with each stage generating two evaluation metrics. The first stage focuses on the relevance and correctness of the question information. We provided GPT-4o with the entry, its corresponding key information, and the generated question-answer pairs to verify whether the questions contain Hungarian-specific content and whether the information in the question-answer pairs aligns with the provided background material (prompt shown in Figure 22).

Second, from the perspective of the precision of the question formulation, we only provided GPT-4o with the generated question-answer pairs to simulate real user response scenarios. This step emphasized evaluating whether the questions were based
on objective facts, and whether the descriptions were precise and specific enough to allow independent answering without ambiguity. Additionally, we required that the answers remain unaffected by future events, ensuring consistency across any time period and guaranteeing the long-term validity of the dataset (prompt details in Figure 23).

Based on the results of the above automated quality assessment, we retained only those question-answer pairs that passed all four evaluation criteria, resulting in a final set of 5,503 questions.

E.5 Manual review of question quality

To further ensure the quality of the constructed question-answer pairs, we engaged native Hungarian speakers to review these questions. Each question was independently reviewed by two annotators who could only see the questions and not the reference answers. The annotation process consisted of three main steps.

First, the annotators were required to determine whether the given questions aligned with the eight Hungarian-specific dimensions proposed in this paper. Next, they evaluated whether the questions met the four assessment criteria outlined in Step 3, ensuring that the questions were objectively framed, precisely described, had unique answers, contained correct information, and maintained consistent answers over time. Finally, if a question satisfied all the above criteria, the annotators provided the correct answer. During this process, annotators were permitted to search for relevant information online and provided reference sources for their answers.

To address potential issues such as overly obscure questions or non-fixed answers, we used GPT-4o to verify whether the annotated results matched the generated reference answers. If the annotated answer matched the reference answer, it was labeled as "CORRECT"; otherwise, it was labeled as "INCORRECT" (prompt details in Figure 24). We selected question-answer pairs that both annotators deemed valid, Hungarian-specific, and consistent with the original reference answers as candidates for the HuSimpleQA dataset, resulting in a total of 2134 questions.

Considering that the HuSimpleQA dataset should exhibit diversity and broad coverage, we removed question-answer pairs belonging to the same entry, retaining only one question-answer pair per entry that best met the construction and evaluation criteria. The details of selecting optimal question-answer pairs prompt can be seen in Figure 25.

Through this process, we obtained a total of 1293 pieces of Hungarian open-ended question-answer pairs, with the category distribution shown in Table 22.

E.6 Inference prompt

We constructed prompts in two languages for model inference, as shown in Figure 26, while also instructing the model to provide a confidence score (ranging from 1 to 100)to measure the model’s confidence in its generated answers.

E.7 LLM-as-judge

Following the approach of SimpleQA (Wei et al., 2024), we employed GPT-4o as the judge to evaluate the LLM’s responses. The evaluation criteria for this step were similar to those used in the manual review process of Step 4. In addition to the classification labels CORRECT and INCORRECT, we introduced an additional category, NOT_ATTEMPTED to further assess the model’s ability to respond to questions and the breadth of its knowledge coverage (prompt details in Figure 27).

Based on the results from the judge, we evaluate the performance of the LLM on HuSimpleQA using the following five metrics:

- Correct (CO): The predicted answer completely encompasses the reference answer without any conflicting or contradictory information.
- Not Attempted (NA): The predicted answer does not fully include the reference answer, but there are no contradictions between the two.
- Incorrect (IN): The predicted answer contradicts the reference answer, regardless of whether the contradiction is resolved.
- Correct Given Attempted (CGA): This metric measures the percentage of accurately answered questions out of all attempted questions.
- F-score: This metric calculates the harmonic mean between the proportion of correct answers and the proportion of correct answers among attempted questions.

The formulas for CGA and F-score are as follows:

\[
\text{CGA} = \frac{c}{c + i}
\] (1)

\[
\text{F-Score} = \frac{2}{\frac{c+i}{c} + \frac{c+i+n}{c}} = \frac{2c}{2c + 2i + n}
\] (2)

Here, \( c \) represents the number of CORRECTly answered questions, \( i \) represents the number of
<table>
  <tr>
    <th>Model</th>
    <th>Judge Correct Ratio</th>
  </tr>
  <tr>
    <td>Deepseek-R1</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct-7B</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct-72B</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct-8B</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>Llama-3.1-Instruct-70B</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>QwQ</td>
    <td>0.98</td>
  </tr>
</table>

Table 10: Accuracy of LLM-as-Judge for HuSimpleQA.

INCORRECTly answered questions, and \( n \) represents the number of NOT_ATTEMPTED questions. To verify the effectiveness of LLM-as-judge, we randomly selected 100 samples from each of the 10 evaluated models for inspection, with accuracy exceeding 98% for all models, as shown in Table 10.

E.8 Comparing the Performance Ranking on HuSimpleQA and SimpleQA

We compare the performance rankings of LLM on HuSimpleQA and SimpleQA (Wei et al., 2024), as shown in Table 12.

E.9 Results of Translating HuSimpleQA to English

As shown in Table 5, the current leading LLMs perform poorly on HuSimpleQA. To further analyze and confirm whether these issues stem from the models’ lack of knowledge of the Hungarian language or Hungarian-specific cultural aspects, we translated HuSimpleQA into English and reevaluated the models. The results, presented in Table 13, show that after translation, 2 out of 10 models improved, while 8 experienced a decline in performance, indicating that translation generally worsens performance. This suggests that LLMs have more difficulty understanding Hungarian cultural nuances than the language itself. These findings are consistent with the conclusions in

E.10 Human Performance on SimpleQA and HuSimpleQA

We hire human annotators to complete the tasks in the SimpleQA and HuSimpleQA datasets in order to assess human performance on these datasets. HuSimpleQA was annotated by Hungarian native speakers, while SimpleQA was handled by English native speakers. We used two annotators per dataset, averaging their performance metrics to determine human performance. Annotators were instructed NOT to use external tools like search engines or GPT. For HuSimpleQA and SimpleQA, they were told to answer "I don’t know" if uncertain about a question.

The results for HuSimpleQA and SimpleQA are shown in Table 14. The average absolute performance (CGA) of the ten models is 26.65 vs 19.34, highlighting significant room for improvement in LLM performance on both datasets. Regarding average relative performance to humans, the scores are 0.40 vs 0.57, indicating that LLMs perform much worse on HuSimpleQA compared to SimpleQA. Despite better absolute performance on HuSimpleQA, LLMs are significantly weaker in simple factual ability in Hungarian than in English.

F HuProverbRea

F.1 Construction pipeline of HuProverb

The proverbs in HuProverbRea are from 2 separate sources. The first part, 733 traditional Hungarian proverbs, are collected from the website\(^{14}\), where each proverb is assigned an English or Hungarian explanation. The other 402 proverbs, focusing on abbreviations and Internet slang, are manually collected and explained by native speakers of Hungarian. Inspired by MAPS (Liu et al., 2024c), we adopt a human-in-loop pipeline to generate and refine the context for each Hungarian specific usage, as shown in Figure 12. For each proverb, we first let GPT4 generate a seed context where the proverb is used. Then, we assign it to a Hungarian native

![Construction of HuProverbRea.](page_1012_180_388_388.png)

Figure 12: Construction of HuProverbRea.
<table>
  <tr>
    <th>model</th>
    <th>correct</th>
    <th>incorrect</th>
    <th>not_attempted</th>
    <th>correct given attempted</th>
    <th>F-Score</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>50.3</td>
    <td>40.61</td>
    <td>9.09</td>
    <td>55.33</td>
    <td>52.69</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>24.52</td>
    <td>74.07</td>
    <td>1.42</td>
    <td>24.87</td>
    <td>24.69</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>32.71</td>
    <td>64.08</td>
    <td>3.2</td>
    <td>33.8</td>
    <td>33.24</td>
  </tr>
  <tr>
    <td>QwQ</td>
    <td>9.17</td>
    <td>52.68</td>
    <td>38.15</td>
    <td>14.82</td>
    <td>11.33</td>
  </tr>
  <tr>
    <td>Deepseek-R1</td>
    <td>34.58</td>
    <td>62.15</td>
    <td>3.28</td>
    <td>35.75</td>
    <td>35.15</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>5.29</td>
    <td>84.13</td>
    <td>10.58</td>
    <td>5.92</td>
    <td>5.59</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>15.05</td>
    <td>78.61</td>
    <td>6.33</td>
    <td>16.07</td>
    <td>15.54</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>14.9</td>
    <td>80.25</td>
    <td>4.84</td>
    <td>15.66</td>
    <td>15.27</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>36.36</td>
    <td>61.03</td>
    <td>2.61</td>
    <td>37.34</td>
    <td>36.84</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>16.24</td>
    <td>44.19</td>
    <td>39.57</td>
    <td>26.88</td>
    <td>20.25</td>
  </tr>
</table>

Table 11: Complete results of HuSimpleQA

<table>
  <tr>
    <th>Rank</th>
    <th>SimpleQA</th>
    <th>HuSimpleQA</th>
  </tr>
  <tr>
    <td>1</td>
    <td>GPT-4o</td>
    <td>GPT-4o(1)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Deepseek-R1</td>
    <td>Llama-3.1-Instruct-70B(1)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Llama-3.1-Instruct-70B</td>
    <td>Deepseek-R1(1)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Deepseek-V3</td>
    <td>Deepseek-V3(4)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>OwQ</td>
    <td>GPT-4o-mini(3)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>Llama-3.1-Instruct-8B</td>
    <td>o1-mini(3)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>Qwen2.5-Instruct-72B</td>
    <td>Llama-3.1-Instruct-8B(1)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>GPT-4o-mini</td>
    <td>Qwen2.5-Instruct-72B(1)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>o1-mini</td>
    <td>QwQ(4)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Qwen2.5-Instruct-7B</td>
    <td>Qwen2.5-Instruct-7B(1)</td>
  </tr>
</table>

Table 12: Performance rankings on SimpleQA (Wei et al., 2024) and HuSimpleQA

<table>
  <tr>
    <th>Model</th>
    <th>Original HuSimpleQA</th>
    <th>Translated to EN</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>50.3</td>
    <td>35.96 (<i>1.4</i>,<i>2.9</i>)</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>24.52</td>
    <td>19.72 (<i>1.4</i>,<i>4.80</i>)</td>
  </tr>
  <tr>
    <td>QwQ</td>
    <td>9.17</td>
    <td>12.84 (<i>1.3</i>,<i>3.67</i>)</td>
  </tr>
  <tr>
    <td>Deepseek-R1</td>
    <td>34.58</td>
    <td>32.02 (<i>1.2</i>,<i>2.56</i>)</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>32.71</td>
    <td>29.08 (<i>1.3</i>,<i>3.63</i>)</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>36.36</td>
    <td>23.82 (<i>1.1</i>,<i>12.54</i>)</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>14.9</td>
    <td>14.08 (<i>1.0</i>,<i>8.2</i>)</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>16.24</td>
    <td>14.31 (<i>1.1</i>,<i>9.3</i>)</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>15.05</td>
    <td>8.89 (<i>0.6</i>,<i>1.6</i>)</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>5.29</td>
    <td>13.47 (<i>1.1</i>,<i>10.18</i>)</td>
  </tr>
</table>

Table 13: Results of translating HuSimpleQA to English.

speaker to check whether this context is grammatically correct and the use of slang is appropriate. If not, the annotator is required to manually write down a new context for the saying, which will be sent back to another annotator for inspection again. We continue the above procedures until all contexts pass the quality check. It’s worth noting that each option of the 2CQ setting is manually constructed by a human annotator, and only when it passes the double check of two other annotators could it be considered usable. We choose not to involve LLM in this part because designing correct/incorrect options requires deep understanding of sayings, LLM may generate ambiguous options if it does not understand the proverb used in the context, and such pre-provided ambiguous options may negatively influence the creativity of the annotators. Finally, we obtain 1,135 Hungarian proverbs, each equipped with a context, an English explanation, and two candidate options for question “What does the speaker mean by the saying?”.

F.2 More examples of HuProverbRea
The example of HuProverbRea is shown in Figure 28 and Figure 29. The prompt for judging HuProverbRea is shown in Figure 32. The prompt for model inference on HuProverbRea is shown in Figure 31 and 30.

F.3 Comparing the Performance Ranking on HuProverbRea and MAPS
We compare the performance rankings of LLM on HuProverbRea and MAPS (Liu et al., 2024c), as shown in Table 15 and 16.

F.4 Human Performance on HuProverbRea and MAP(en)
We hire human annotators to complete the tasks in the HuProverbRea and MAP(en) datasets in order to assess human performance on these datasets. HuProverbRea was annotated by Hungarian native speakers, while MAP(en) was handled by English native speakers. We used two annotators per dataset, averaging their performance metrics to determine human performance. Annotators were instructed NOT to use external tools like search engines or GPT. The accuracy for HuProverbRea is 80.44%. The accuracy for MAP(en) is 95.31%.
Furthermore, we calculated the LLMs’ relative performance to human, as shown in Table 17.
The average absolute performance of the ten models is 0.8684 vs 0.9350, indicating that LLMs perform worse on HuProverbRea compared to English proverb reasoning, suggesting room for improvement in Hungarian proverb tasks.
For average relative performance to humans, the scores are 1.08 vs 0.98. This indicates that while LLMs slightly underperform compared to humans
<table>
  <tr>
    <th>Model</th>
    <th>HuSimpleQA-abs</th>
    <th>HuSimpleQA-rel</th>
    <th>SimpleQA-abs</th>
    <th>SimpleQA-rel</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>55.33</td>
    <td>0.85</td>
    <td>38.0</td>
    <td>1.12</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>24.87</td>
    <td>0.38</td>
    <td>8.7</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Deepseek-V3</td>
    <td>33.8</td>
    <td>0.52</td>
    <td>24.9</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>QwQ</td>
    <td>14.82</td>
    <td>0.23</td>
    <td>21.0</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td>Deepseek-R1</td>
    <td>55.75</td>
    <td>0.55</td>
    <td>30.1</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Qwen2.5-TB-Instruct</td>
    <td>5.92</td>
    <td>0.09</td>
    <td>5.3</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>Qwen2.5-TB2-Instruct</td>
    <td>16.07</td>
    <td>0.25</td>
    <td>12.2</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>Llama-3.1-SB-Instruct</td>
    <td>15.66</td>
    <td>0.24</td>
    <td>12.6</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>37.34</td>
    <td>0.58</td>
    <td>29.3</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>26.88</td>
    <td>0.41</td>
    <td>11.3</td>
    <td>0.33</td>
  </tr>
  <tr>
    <th>AVG</th>
    <th>26.65</th>
    <th>0.40</th>
    <th>19.34</th>
    <th>0.57</th>
  </tr>
</table>

Table 14: The results of HuSimpleQA and SimpleQA, abs is the absolute performance of CGA metric, rel is the performance compared to human.

<table>
  <tr>
    <th>Rank</th>
    <th>HuProverbRea</th>
    <th>MAPS (hu)</th>
    <th>MAPS (hu)</th>
    <th>MAPS (hu)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>GPT-4o</td>
    <td>GPT-4o (+)</td>
    <td>GPT-4o (+)</td>
    <td>GPT-4o (+)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Llama-3.1-Instruct-70B</td>
    <td>Llama-3.1-Instruct-70B (+)</td>
    <td>Deepseek-V3 (†)</td>
    <td>Qwen2.5-Instruct-72B (†)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Deepseek-V3</td>
    <td>Qwen2.5-Instruct-72B (†)</td>
    <td>Deepseek-R1 (†)</td>
    <td>Llama-3.1-Instruct-70B (†)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>GPT-4o-mini</td>
    <td>GPT-4o-mini (+)</td>
    <td>Qwen2.5-Instruct-72B (†)</td>
    <td>GPT-4o (+)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Deepseek-R1</td>
    <td>Deepseek-V3 (†)</td>
    <td>01-mini (†)</td>
    <td>Deepseek-V3 (†)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>Qwen2.5-Instruct-72B</td>
    <td>Deepseek-V3 (†)</td>
    <td>GPT-4o (+)</td>
    <td>01-mini (†)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>01-mini</td>
    <td>Qwen2.5-Instruct-7B (†)</td>
    <td>Llama-3.1-Instruct-70B (†)</td>
    <td>Deepseek-R1 (†)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>QwQ</td>
    <td>Llama-3.1-Instruct-R1 (†)</td>
    <td>Llama-3.1-Instruct-R1 (†)</td>
    <td>Llama-3.1-Instruct-R1 (†)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>Llama-3.1-Instruct-SB</td>
    <td>01-mini (†)</td>
    <td>Qwen2.5-Instruct-7B (†)</td>
    <td>Qwen2.5-Instruct-7B (†)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Qwen2.5-Instruct-TB</td>
    <td>QwQ (†,2)</td>
    <td>QwQ (†,2)</td>
    <td>QwQ (†,2)</td>
  </tr>
</table>

Table 15: Performance rankings on HuProverbRea and MAPS (Liu et al., 2024c) (part1/2)

on the MAP(en) dataset, they outperform humans on the HuProverbRea dataset, highlighting their strengths in Hungarian language-related knowledge and reasoning.

G HuMatchingFIB and HuStandardFIB

G.1 Construction of HuMatchingFIB and HuStandardFIB

The questions for both HuMatchingFIB and HuStandardFIB are sourced from the Hungarian National Public Education Portal(NKP)15, a comprehensive platform for cultural funding and support in Hungary. This portal connects artists, cultural organizations, and the public with resources and opportunities to promote Hungarian culture both domestically and internationally. Notably, this website is a government initiative, reflecting the collaborative efforts between the Hungarian government and the European Union, particularly through projects or programs supported by the European Social Fund. After extracting the original questions from the NKP website, we engaged native Hungarian speakers to annotate the data. The annotation process involved manually extracting questions and their corresponding answers16, classifying the questions into appropriate categories, and filtering out questions that required additional modalities such as images, tables, audio, or video. This ensured that only purely language-based questions were retained. Through this process, we obtained 278 questions for the HuMatchingFIB task and 93 questions for the HuStandardFIB task, as shown in Table x.

G.2 More examples of HuMatchingFIB and HuStandardFIB

Examples of questions from HuMatchingFIB and HuStandardFIB are provided in Figure 33 and Figure 34. The prompt for model inference on HuMatchingFIB and HuStandardFIB is shown in Figure 36 and Figure 35.

G.3 Metric and Judge of HuMatchingFIB and HuStandardFIB

HuMatchingFIB employs a rule-based evaluation approach, where the assessment is conducted at two levels: the blank level and the question level (as a single question may contain multiple blanks). The evaluation process is analogous to that of multiple-choice questions, and accuracy (acc) is used as

15https://www.nkp.hu/
16The questions for HuMatchingFIB and HuStandardFIB on the NKP website are not in plain text but are instead presented in interactive modules, and the answers can only be obtained through additional interactive operations. As a result, the commonly used data cleaning and extraction methods for LLM pre-training datasets are unable to accurately extract these questions and their corresponding answers. Consequently, it can be concluded that the likelihood of these questions being incorporated into the LLM pre-training data in their proper format is minimal, thereby significantly reducing the potential risk of data contamination. This ensures the reasonableness and effectiveness of the test sets for HuMatchingFIB and HuStandardFIB.
<table>
  <tr>
    <th>Rank</th>
    <th>HuProverbRea</th>
    <th>MAPS (de)</th>
    <th>MAPS (ru)</th>
    <th>MAPS (zh)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>GPT-4o</td>
    <td>Llama-3.1-Instruct-70B (*)</td>
    <td>Deepseck-V3 (*)</td>
    <td>GPT-4o (*)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Llama-3.1-Instruct-70B</td>
    <td>GPT-4o-mini (*)</td>
    <td>ol-mini</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Deepseck-V3</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
    <td>GPT-4o (*)</td>
    <td>Deepseck-V3 (*)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>GPT-4o-mini</td>
    <td>GPT-4o (*)</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
    <td>ol-mini (*)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Deepseck-R1</td>
    <td>Llama-3.1-Instruct-70B (*)</td>
    <td>GPT-4o-mini (*)</td>
    <td>Llama-3.1-Instruct-70B (*)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>Qwen2.5-Instruct-72B</td>
    <td>Deepseck-V3 (*)</td>
    <td>Deepseck-R1 (*)</td>
    <td>GPT-4o-mini (*)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>ol-mini</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>QwQ</td>
    <td>Qwen2.5-Instruct-72B (*)</td>
    <td>Llama-3.1-Instruct-70B (*)</td>
    <td>Llama-3.1-Instruct-SB (*)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>Llama-3.1-Instruct-SB</td>
    <td>Deepseck-R1 (*)</td>
    <td>Llama-3.1-Instruct-SB (*)</td>
    <td>Deepseck-R1 (*)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Qwen2.5-Instruct-TB</td>
    <td>QwQ (*)</td>
    <td>QwQ (*)</td>
    <td>QwQ (*)</td>
  </tr>
</table>

Table 16: Performance rankings on HuProverbRea and MAPS (Liu et al., 2024c) (part2/2)

<table>
  <tr>
    <th>Model</th>
    <th>HuProverbRea-abs</th>
    <th>HuProverbRea-rel</th>
    <th>MAPS(en)-abs</th>
    <th>MAPS(en)-rel</th>
  </tr>
  <tr>
    <td>Deepseck-V3</td>
    <td>92.51</td>
    <td>1.15</td>
    <td>96.95</td>
    <td>1.02</td>
  </tr>
  <tr>
    <td>Deepseck-R1</td>
    <td>91.72</td>
    <td>1.14</td>
    <td>98.48</td>
    <td>1.03</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>95.51</td>
    <td>1.19</td>
    <td>97.97</td>
    <td>1.03</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>92.16</td>
    <td>1.15</td>
    <td>97.97</td>
    <td>1.03</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>93.83</td>
    <td>1.17</td>
    <td>98.48</td>
    <td>1.03</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>73.48</td>
    <td>0.91</td>
    <td>92.39</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>ol-mini</td>
    <td>87.67</td>
    <td>1.09</td>
    <td>92.89</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>90.22</td>
    <td>1.12</td>
    <td>98.22</td>
    <td>1.03</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>67.05</td>
    <td>0.83</td>
    <td>94.16</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>QwQ</td>
    <td>84.23</td>
    <td>1.05</td>
    <td>67.51</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>AVG</td>
    <td>86.84</td>
    <td>1.08</td>
    <td>93.50</td>
    <td>0.98</td>
  </tr>
</table>

Table 17: The results of HuProverbRea and MAPs(en), abs is absolute performance, rel is the performance compared to human.

the metric to determine performance. The corresponding formula for blank level accuracy is as follows, where c represents the number of correctly predicted blanks in one question, t represents the number of blanks in one question.

\[
\text{Acc}_{\text{blank level}} = \frac{\sum \text{blank}_c}{\sum \text{blank}_t}
\]

HuStandardFIB questions are designed with open-ended reference answers to accommodate variations in part of speech and semantics. We employ a many-to-one fuzzy matching mechanism. Fuzzy matching is a technique that calculates the similarity between strings, allowing for flexibility in matching by considering variations such as typos, synonyms, or different word orders. In this context, the model’s answer is compared against a set of possible reference answers (where multiple correct answers may exist for a single question or blank). If the similarity score between the model’s answer and any of the reference answers exceeds a predefined threshold, the answer is considered correct. This approach is particularly suitable for evaluating open-ended questions where exact matches are often infeasible due to the variability in acceptable responses. The annotator information involved in all tasks of OpenHuEval can be found in Appendix I.

H Analyzing LRM’s thinking process on OpenHuEval

H.1 Analyzing LRM’s thinking process on HuSimpleQA

We use GPT-4o to break down the answers into thoughts. This is done in two steps: the first step is to identify expressions that may be a shift in thought (see the prompt in Figure 37), and the second step is to confirm whether it is indeed a shift in thought (set the prompt in Figure 38). Then, We utilized the LLM to evaluate whether each idea would lead to the correct answer, the prompt is shown in Figure 39. We consider a confident score of 2 as the correct thought.

Examples of the thoughts and corresponding correctness can be found in Figure 40 and Figure 41.

H.2 Analyzing LRM’s thinking process on HuMatchingFIB

H.2.1 Splitting LRM’s thinking process into segments and classifying these segments

We first divided the thinking process of LRMs into multiple segments, with each segment categorized as introduction, reasoning, review, or summary. The definitions of these four segment types are provided in Table 18. Typically, each thinking process begins with an introduction segment, includes several reasoning segments and some review segments in the middle, and concludes with a summary segment. The segmentation
and classification were performed using GPT-4o, with the prompt template detailed in Figures 42, 43 and 44. Examples of the segmentation and classification can be found in Figure 47 and Figure 48.

H.2.2 Tagging the reasoning segments along the dimensions
We further tagged the reasoning segments according to the following four dimensions: (Dim1) correctness: Are the answers in this reasoning segment correct? (Dim2) complexity: In this reasoning segment, does the LRM simply assert the answer, or does it involve more complex reasoning? (Dim3) scope: Does this reasoning segment focuses on a single blank, modifies previous blanks, or addresses multiple blanks? (Dim4) language transfer: Does the LRM switch languages within this reasoning segment? The details of the tagging can be found in Table 19. Examples of the tagging results can be found in Figure 47 and Figure 48.

I Information of the Annotators
We submitted the annotation task online to a professional data annotation company, which organized annotators to complete the annotation work. In the construction phase of OpenHuEval, the annotations were carried out by professional annotators who are native Hungarian speakers. Table 20 shows the number of annotators and the total time spent on each task. All annotators involved in this project hold a bachelor’s or master’s degree, with academic backgrounds in fields such as Social Sciences, Translating and Interpreting, English Studies, and IT Engineering. They all possess the ability to distinguish subtle aspects of the Hungarian language and handle Hungarian-specific knowledge effectively.
<table>
  <tr>
    <th>Segment types</th>
    <th>Definition</th>
  </tr>
  <tr>
    <td>introduction</td>
    <td>typically located at the beginning of the reasoning process; usually consists of the LRM’s brief restatement of the question and the descriptive account of the work it is about to undertake; does not include the actual start of the analysis of the question.</td>
  </tr>
  <tr>
    <td>reasoning</td>
    <td>typically constitutes the main body of the reasoning process; includes the detailed thinking and reasoning steps undertaken by the LRM to solve the fill-in-the-blank questions.</td>
  </tr>
  <tr>
    <td>review</td>
    <td>usually occurs after the reasoning process is essentially complete but before the final output. This section typically includes a review of the entire reasoning process and may contain keywords or phrases such as “Overall,...” or “double check...” .</td>
  </tr>
  <tr>
    <td>summary</td>
    <td>Summarizes the overall content or provides final conclusions, often using phrases like ‘in conclusion’ or ‘overall’.</td>
  </tr>
</table>

Table 18: Definitions of the four categories of segments in the thinking process of LRM on HuMatchingFIB.

<table>
  <tr>
    <th>Dimension</th>
    <th>Description</th>
    <th>Tags</th>
  </tr>
  <tr>
    <td>Dim 1: Correctness</td>
    <td>Are the answers in this reasoning segment correct? Class 4 is used when no conclusion is reached.</td>
    <td>Class 1: Completely Incorrect<br>Class 2: Partially Correct<br>Class 3: Completely Correct<br>Class 4: Non Conclusion</td>
  </tr>
  <tr>
    <td>Dim 2: Complexity</td>
    <td>In this reasoning segment, does the LRM simply assert the answer, or does it involve more complex reasoning?</td>
    <td>Class 1: Simple Assertion<br>Class 2: Complex Thought</td>
  </tr>
  <tr>
    <td>Dim 3: Scope</td>
    <td>Does this reasoning segment focuses on a single blank, modifies previous blanks, or addresses multiple blanks?</td>
    <td>Class 1: Only Current Blank<br>Class 2: Modify Previous Blanks<br>Class 3: Current Blank and Consecutive Blank</td>
  </tr>
  <tr>
    <td>Dim 4: Language Transfer</td>
    <td>Does the LRM switch languages within this reasoning segment (e.g., Hungarian to English)?</td>
    <td>Class 1: Contains Language Transfer<br>Class 2: No Language Transfer</td>
  </tr>
</table>

Table 19: Tagging dimensions of the reasoning segments in LRM’s thinking process on HuMatchingFIB

<table>
  <tr>
    <th>Task</th>
    <th># Anotater</th>
    <th>Total working hours</th>
  </tr>
  <tr>
    <td>HuSimpleQA</td>
    <td>14</td>
    <td>161.9</td>
  </tr>
  <tr>
    <td>HuWildBench</td>
    <td>5</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>HuProverbRea</td>
    <td>15</td>
    <td>118.2</td>
  </tr>
  <tr>
    <td>HuMatchingFIB and HuStandardFIB</td>
    <td>8</td>
    <td>84.5</td>
  </tr>
</table>

Table 20: Information of the Annotators
Hungarian-specific dimensions: CCC
Count: 365

Question Example:
a kérdés az: Mi lesz a jövőben a szocializmus alatt megépült sok panelház?
a leírás: Úgy tudom, hogy kb 60 éves életkorsúra ter- vezik őket. Magyarországon (és a környező országokban is) erőségek ember él benneik. Mi fog történni akkor, ha lakhatat- lanná kezdenek válni? Mi lesz azzal a sok emberrel? Meg a panellokkal?

Translation to EN:
The question is: What will happen in the future to the many panels built under socialism?
The description is: I understand they are designed for a life cycle of about 60 years. There are a lot of people living in them in Hungary (and surrounding countries). What will happen if they start to become uninhabitable? What will happen to all those people? And the panels?

Hungarian-specific dimensions: EP
Count: 201

Question Example:
a kérdés az: A kapitalistai magyarok Ukrajnában oroszul vagy ukránul tanultak meg a 2000-es évek közepén?
a leírás: Mit tanítottak az iskolákban? Mennyire réalis az, hogy valakire szinte semmi se ragad a környezetből? Vannak olyan tömb területek ahol mondjuk egy magyar gyerekeknek egyáltalán nem kell helyi ukránokkal beszéljen? Egyáltalán a helyi ukránok beszélek a 2000-es években?

Translation to EN:
The question is: Did Hungarians in Transcarpathia learn Russian or Ukrainian in Ukraine in the mid-2000s?
The description is: What was taught in schools? How realistic is it that almost nothing sticks to someone from their environment? Are there block areas where, say, a Hungarian child doesn’t have to speak to local Ukrainians at all? Did local Ukrainians even speak Ukrainian in the 2000s?

Hungarian-specific dimensions: FPC
Count: 27

Question Example:
a kérdés az: Mi történt azzal, aki az 50-es években a felhívás ellenére sem jegyezt “önként” békekolcsót? Érthette ezért retorizó az embert?
a leírás: Persze nyilván volt, amilyen “bolondos” idők jártak nálunk akkortájt. Bízatos kikálltottak reakcióinak vagy feszültsé- nak, meg a “népi demokrácia” ellensegének.

Translation to EN:
The question is: What happened to the man who did not “voluntarily” subscribe to a peace charter in the 1950s, despite the call? Could he have been retaliated against for this?
The description is: He must have been branded a reactionary or a fascist or an enemy of ‘people’s democracy’.

Hungarian-specific dimensions: BFP
Count: 289

Question Example:
a kérdés az: Meddig tartható fent Magyarország negatív külk- erskedelmi mérlegé?
a leírás: Nem a haború óta, hanem már 2021 nyarától folyamatosan negatív az ország külkereskedelmi mérlege. Júliusban és augusztusban összesen több, mint 1000 milliárd forintnyi minusz keletkezett. Persze a többi honap nem volt ennyire szörnyű, de ez csak erre az évre már több, mint 2000 milliárd forintnyi minusz keletkezett. Az országban még mindig ész- jódott energiaáramlásként is több, mint 1000 milliárdos negatív mérleg hozható össze 2023-ban. Meddig lehet ezt tovább foly- tani? Meddig elég a devizatarthaték a hiány pótlására?

Translation to EN:
The question is: How long can Hungary maintain a negative trade balance?
The description is: In July and August there was a total deficit of more than HUF 1000 billion. Of course, the other months were not so bad, but for this year alone it is already more than HUF 2000 billion in deficit. Even with unchanged foreign exchange imports and moderating energy prices, a negative balance of more than 1,000 billion in 2023 could be created. How long can this go on? How long will foreign exchange reserves be enough to cover the deficit?

Checklist:
“Does the response provide an analysis of the current condition and expected lifespan of the panel buildings in Hungary and neighboring countries?”,
“Does the response address the expected lifespan of panel buildings and their current age?”,
“Are there any historical or international examples included to illustrate possible outcomes or strategies?”,
“Does the response consider the economic implications of renovating or replacing panel buildings?”,
“Does the response include potential government or private sector plans or policies addressing the future of these buildings and their residents?”,
“Does the answer discuss potential scenarios for when these buildings become uninhabit- able?”,
“Are environmental and urban planning aspects of dealing with aging panel buildings mentioned?”,
“Is there an explanation of possible solutions or government plans for relocating residents?”

Checklist:
“Does the answer provide information on the language predominantly spoken by local Ukrainians in Transcarpathia in the 2000s?”,
“Does the response discuss the social and linguistic dynamics in areas with significant Hungarian populations, including interactions with local Ukrainians?”,
“Does the response clearly explain the educational policies and language of instruction in schools for Hungarians in Transcarpathia during the mid-2000s?”,
“Does the response accurately describe the language of instruction in Transcarpathian Hun- garian schools in the mid-2000s?”,
“Does the response consider the historical and political context of language policies in Ukraine during this period?”,
“Does the response provide insight into whether local Ukrainians predominantly spoke Ukrainian during the 2000s?”,
“Does the response offer a balanced view of cultural and linguistic integration in Tran- scarpathia during the specified period?”,
“Does the answer address the likelihood of a Hungarian child not acquiring any local language skills from their environment?”,
“Does the response discuss the existence of predominantly Hungarian areas where interaction with local Ukrainians might be limited?”

Checklist:
“Does the answer address the political labels mentioned in the description (e.g., ‘reactionary’, ‘fascist’, ‘enemy of people’s democracy’)?”,
“Does the response differentiate between official consequences and social/societal repercus- sions for not subscribing to the peace loan?”,
“Does the response address potential consequences for individuals who did not subscribe to the peace loan, with references to historical examples or documentation?”,
“Does the response provide a balanced view, considering both potential punitive measures and any cases of leniency or exceptions, if applicable?”,
“Is there a clear mention of what ‘békekolcsón’ (peace loan) was and its significance during that time period?”,
“Does the response accurately describe the historical context of the 1950s in Hungary?”,
“Is there an analysis of the societal and governmental attitudes toward dissenters in Hungary during the 1950s, including any possible labels or accusations they might have faced?”,
“Does the response provide specific examples of potential retaliations against those who didn’t subscribe to the peace loan?”

Checklist:
“Does the response analyze Hungary’s current foreign exchange reserves and their sufficiency in covering the trade deficit?”,
“Is there an exploration of historical trends and comparisons to similar situations in other countries to provide context?”,
“Is the impact of energy prices on the trade balance accurately assessed in the response?”,
“Does the response offer a clear and supported prediction or timeframe for how long Hungary can sustain its negative trade balance?”,
“Is there an analysis of the factors affecting Hungary’s foreign exchange reserves and their ability to cover the deficit?”,
“Does the response provide a clear timeline or projection for how long the negative balance can be sustained?”,
“Are there comparisons made to similar situations in other countries or historical precedents in Hungary?”,
“Does the response accurately explain the current state of Hungary’s foreign trade balance?”
<table>
  <tr>
    <th>Hungarian-specific dimensions</th>
    <th>Count</th>
    <th>Question-Answer Pairs</th>
    <th>translation to EN</th>
  </tr>
  <tr>
    <td rowspan="3">L</td>
    <td rowspan="3">11</td>
    <td>Question1: Mit jelent a Kara török eredetű régi magyar személynév?<br>Answer1: fekete<br>Question2: Melyik régi magyar név a Pantaleon megfelelője?<br>Answer2: Pentele</td>
    <td>Question1: What does the old Hungarian personal name Kara of Turkish origin mean?<br>Answer1: black<br>Question2: Which old Hungarian name is the equivalent of Pantaleon?<br>Answer2: Pentele</td>
  </tr>
  <tr>
    <td>Question1: Melyik király nevezte ki Szapolyai Imrét szepesi örököletes főispánná 1465-ben?<br>Answer1: Mátyás király<br>Question2: Melyik várost foglalta el Báthory Gábor 1610. december 11-én?<br>Answer2: Szeben</td>
    <td>Question1: Which king appointed Imre Szapolyai as the hereditary ispán of Szepes in 1465?<br>Answer1: King Matthias<br>Question2: Which city was captured by Gabriel Báthory on December 11, 1610?<br>Answer2: Sibiu</td>
  </tr>
  <tr>
    <td>Question1: Melyik magyar film nyerte el a FIPRESCI-díjat az 1983-as Cannes-i Nemzetközi Filmfesztiválon?<br>Answer1: Szerencsés Dániel<br>Question2: Melyik legendára épít az 'Eredet / Origins' táncjáték?<br>Answer2: Csodaszarvas-legendára</td>
    <td>Question1: Which Hungarian film won the FIPRESCI Prize at the 1983 Cannes International Film Festival?<br>Answer1: Lucky Daniel<br>Question2: Which legend is the 'Origin / Origins' dance play based on?<br>Answer2: Legend of the Miraculous Deer</td>
  </tr>
  <tr>
    <td rowspan="2">LCC</td>
    <td rowspan="2">257</td>
    <td>Question1: Melyik városban alapították a Gandhi Gimnáziumot 1994-ben?<br>Answer1: Pécsen<br>Question2: Melyik évben alapította a Magyar Tudományos Akadémia az Acta Juridica Hungarica folyóiratot?<br>Answer2: 1959</td>
    <td>Question1: In which city was the Gandhi High School founded in 1994?<br>Answer1: Pécs<br>Question2: In which year did the Hungarian Academy of Sciences establish the journal Acta Juridica Hungarica?<br>Answer2: 1959</td>
  </tr>
  <tr>
    <td>Question1: Melyik magyar vármegyében található Nemesmedves?<br>Answer1: Vas vármegyében<br>Question2: Mi a neve Magyarország legmagasabban fekvő csillagvizsgálójának, amely a Piszkés-tetőn található?<br>Answer2: Piszkéstetői Obszervatórium</td>
    <td>Question1: In which Hungarian county is Nemesmedves located?<br>Answer1: Vas county<br>Question2: What is the name of Hungary’s highest observatory, located on Piszkés Peak?<br>Answer2: Piszkés Peak Observatory</td>
  </tr>
  <tr>
    <td rowspan="2">GP</td>
    <td rowspan="2">165</td>
    <td>Question1: Nádasdy Kálmán hányszor kapott Kossuth-díjat élete során?<br>Answer1: Háromszor<br>Question2: Balogh József melyik magyar városban született 1946. április 15-én?<br>Answer2: Nagykanizsán</td>
    <td>Question1: How many times did Kálmán Nádasdy receive the Kossuth Prize during his lifetime?<br>Answer1: Three times<br>Question2: In which Hungarian city was József Balogh born on April 15, 1946?<br>Answer2: Nagykanizsa</td>
  </tr>
  <tr>
    <td>Question1: Melyik szervezet jogkörét vette át a Népgazdasági Tanács 1949. június 11-én?<br>Answer1: Gazdasági Főtanács<br>Question2: Melyik törvénycikk rendelkezett 1878-ban Magyarországon a réz-váltópénz sza- poritásáról?<br>Answer2: 1878. évi VI. törvénycikk</td>
    <td>Question1: Which organization’s authority was taken over by the National Economic Council on June 11, 1949?<br>Answer1: Supreme Economic Council<br>Question2: Which statute regulated the increase of copper coinage in Hungary in 1878?<br>Answer2: Act VI of 1878</td>
  </tr>
  <tr>
    <td rowspan="2">PPL</td>
    <td rowspan="2">186</td>
    <td>Question1: Milyen néven működött az ÉVITERV 1954-től az 1980-as évek elejéig?<br>Answer1: ÉM Szerelőipari Tervező Vállalat<br>Question2: Melyik cég gyártotta a Puli autóti-pust a gyártás kezdeti időszakában?<br>Answer2: HÓDGÉP</td>
    <td>Question1: Under what name did ÉVITERV operate from 1954 to the early 1980s?<br>Answer1: ÉM Installation Industry Design Company<br>Question2: Which company manufactured the Puli car model in the early production period?<br>Answer2: HÓDGÉP</td>
  </tr>
  <tr>
    <td>Question1: Milyen néven működött az ÉVITERV 1954-től az 1980-as évek elejéig?<br>Answer1: ÉM Szerelőipari Tervező Vállalat<br>Question2: Melyik cég gyártotta a Puli autóti-pust a gyártás kezdeti időszakában?<br>Answer2: HÓDGÉP</td>
    <td>Question1: Under what name did ÉVITERV operate from 1954 to the early 1980s?<br>Answer1: ÉM Installation Industry Design Company<br>Question2: Which company manufactured the Puli car model in the early production period?<br>Answer2: HÓDGÉP</td>
  </tr>
</table>

Table 22: Examples of HuSimpleQA. The rightmost column is the English translation of the original OpenHuEval examples, used for visualization.
"**Given the following question, identify whether it has a characteristic related to Hungary. A question is considered to have a Hungarian characteristic if it meets any of the following criteria:
Hungary-Specific Context: The question itself directly references or relates to Hungary. For example, "What is the capital of Hungary?" clearly has a Hungarian characteristic.
Hungary-Specific Answer: The question might not directly reference Hungary, but the answer would vary depending on the country, particularly Hungary. For example, "What is the minimum wage according to labor laws?" The answer would depend on Hungary's laws and practices.
Hungary-Specific Context and Answer: Both the question and the likely answer have strong connections to Hungary. For example, "How do you view Hungary's 2024 foreign policy?" is likely to have both the question and the answer centered on Hungary.
Any Other Model-Identified Hungarian Characteristic: If the model identifies a Hungarian characteristic based on context, culture, or any other relevant factors.
There are several special rules to follow:
The language of the question should not be used as an evidence.
For a question to which the Hungarian answer is not significantly different from the answer of the rest of the world, the question is not considered having a Hungarian characteristic.
If a question only mentions a Hungarian-related term, such as the Hungarian currency, the forint, or a certain place in Hungary, but the question itself is not more related to Hungary's cultural, social, political, economic, military, life, etc., the question is not considered having a Hungarian characteristic.
If the answer to a question is open-ended, for example, "Will you buy a flower for your mom on Mothers' Day?", the question is not considered having a Hungarian characteristic.
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
{
    "Question": "[The original question]",
    "HasHungarianCharacteristic": "yes/no",
    "Reason": "[Explanation for why this question was classified as having a Hungarian characteristic]"
    "Score": "[This score is used to evaluate how relevant this issue is to Hungary, with 0 being the lowest and 10 being the highest.]"
}
The question is: <question>.
Note that each question is composed of a question itself and a question description."**

Figure 13: Prompt template for automatic filtering of user questions related to Hungarian specifics in the construction of HuWildBench.
""""# Instruction
You are an expert responsible for evaluating the capabilities of a language model in handling questions related to the Hungarian language and context.
The questions are sourced from the Gyakori kérdések website, and the objective is to assess the model’s performance by selecting the best questions based on a set of criteria.
You will be given five Hungarian questions.
Based on the 9 criteria listed below, select the two best questions.

## Evaluation Criteria
<begin of evaluation>
1. Linguistic complexity: Does the question contain complex syntactic structures and rich vocabulary, testing the model’s ability to process complex language?
2. Hungarian-specific relevance: Is the question highly relevant to Hungarian culture, society, history, or daily life, testing the model’s understanding of Hungary-specific context?
3. Requirement for common knowledge and factual accuracy: Does the question require knowledge of Hungarian common sense or factual information, allowing for the evaluation of the model’s knowledge base and accuracy?
4. Context dependency: Does the question require the model to understand or infer from the context, testing the model’s ability to use prior or surrounding information?
5. Answer diversity: Does the question allow for multiple reasonable answers, testing the model’s creativity and ability to generate diverse responses?
6. Ambiguity: Does the question contain ambiguity or multiple meanings, testing the model’s ability to handle uncertain or vague information?
7. Reasoning requirement: Does the question require logical reasoning or causal inference, testing the model’s ability to analyze and reason through complex information?
8. Social and ethical considerations: Does the question involve social, ethical, or moral issues, testing the model’s ability to generate responses that align with ethical standards?
9. Format diversity: Does the question come in a unique format (e.g., multiple choice, open-ended, narrative, etc.), testing the model’s ability to handle different types of question formats?
10. Breadth of knowledge: Does the question cover a broad range of knowledge areas (e.g., science, arts, technology), testing the model’s general knowledge across various domains.
<end of evaluation>

## Questions
<begin of questions>
1. <question0>.
2. <question1>.
3. <question2>.
4. <question3>.
5. <question4>.
<end of questions>
Note that each question is composed of a question itself and a question description.

## Output format
Your output should be in JSON format as follows:
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
{
    "question_indices": [a list of the indices of the two best question in int type],
}""""
""""You are a model designed to assist in evaluating responses to questions.
You will receive a question about Hungary, and your task is to provide a list of 3–5 evaluation criteria.
Each item in the list should be a distinct angle for assessing whether the response to the question meets the required standard.
For example, if the question is: "Is a monthly income of $1000 sufficient to cover normal living expenses in the capital city of Hungary?", the list could include criteria such as:
1. Does the response comprehensively outline all relevant living expenses in Budapest?
2. Are the amounts mentioned for each expense aligned with objective facts?
3. Does the response provide an overall conclusion on whether $1000 is enough for living expenses?
Each criterion should assess a different aspect of the response, ensuring no overlap in evaluation angles.
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
{
    "Checklist": "[The evaluation criteria list]"
}
The question is: <question>.
Note that each question is composed of a question itself and a question description.""""
#### Instruction
You are an expert evaluator. Your task is to evaluate the quality of the responses generated by AI models.
We will provide you with the user query and an AI-generated responses.
You should first read the user query and the conversation history carefully for analyzing the task, and then evaluate the quality of the responses based on and rules provided below.

# Conversation between User and AI

## History
<begin_of_history> {history} <end_of_history>

## Current User Query
<begin_of_query> {user_query} <end_of_query>

## AI Response
<begin_of_response> {prediction} <end_of_response>

# Evaluation

## Checklist
<begin_of_checklist> {checklist} <end_of_checklist>
Please use this checklist to guide your evaluation, but do not limit your assessment to the checklist.

## Rules
You should compare the above response based on your analysis of the user queries and the conversation history.
You should first write down your analysis and the checklist that you used for the evaluation, and then provide your assessment according to the checklist.
The scores are in the range of 1~10, where 1 means the response is very poor and 10 means the response is perfect.
Here are more detailed criteria for the scores:

– Score 1-2: The response is very poor and does not make sense at all.
– Score 3-4: The response is poor and does help user solve the problem in a meaningful way.
– Score 5-6: The response is fair but has some issues (e.g., factual errors, hallucinations, missing key information).
– Score 7-8: The response is good enough but could be improved in some ways.
– Score 9-10: The response is perfect and provides helpful information that can help user solve the problem.

## Output Format
First, please output your analysis for the model response, and then summarize your assessment to two aspects: "strengths" and "weaknesses"; Finally, please write down your rating for the assessment.
Please provide your evaluation results in the following json format by filling in the placeholders in []:
```json
{"strengths": "[analysis for the strengths of the response]", "weaknesses": "[analysis for the weaknesses of the response]", "score": "[1-10]"}
```

Figure 16: Prompt template for LLM as judge on HuWildBench (WBScore).
****
- Role: Expert in Hungarian Culture and Data Classification
- Background: You are tasked with classifying data that is deeply related to Hungarian-specific content. This data may involve Hungarian history, culture, art, folklore, language, traditions, tourism, and more. Your expertise is critical to ensuring the classification is precise and adheres strictly to Hungarian cultural relevance.
- Goal: Analyze the detailed description in the input data and categorize it into one of the following nine predefined categories:
1. Language: Content related to Hungarian language, including proverbs, idioms, or linguistic knowledge.
2. History: strictly for content describing specific historical events or developments in Hungary. Examples include wars, revolutions, significant treaties, or influential periods of political or societal change.
3. Life, Culture, and Customs: Hungarian religion, etiquette, cultural practices, holidays, and daily life (including tourism).
4. Education and Profession: Information on Hungary’s education system or associated occupations.
5. Geography and Place: Hungarian geography, city locations, landmarks, and travel-related content.
6. Figure: Notable Hungarian individuals and their achievements.
7. Politics, Policy, and Law: Hungarian political systems, policies, or legal regulations.
8. Business and Finance: Hungarian economy, business practices, or financial systems.
9. Others: Content not relevant to Hungarian culture or not fitting into the above categories.

Constraints:
• Cultural Accuracy: Your classification must be based on an in-depth understanding of Hungarian culture and the context provided in the input. Avoid assumptions or generic classifications that lack cultural alignment.
• Systematic Approach: Follow a logical and consistent process to ensure every input is matched to the most relevant category. If the content cannot be clearly classified into one category, opt for “Others”
• Specificity: Focus on how the content relates explicitly to Hungary. Avoid overgeneralizing or assigning tags that are only loosely connected to the data.

Please classify the following data according to the above requirements and example: <input_question>

Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
{
"cn_specific_label": "[predicted label]"
}
****

Figure 17: Prompt template for selecting Hungarian specific Wikipedia entries in the construction of HuSimpleQA.
!!!!!
As a general knowledge expert, please judge the knowledge value of the material and extract key information from the following descriptive materials. The requirements are as follows:
1. The extracted content is the most critical information of the text description subject. Please extract the core content of the description text in a targeted manner.
2. Please ensure that the extracted information is accurate and unambiguous.
3. The extracted key information is in <language>.
4. The key information extracted should be related to the title corresponding to the material.

[Contextual information]
{
    "title": "704-es busz",
    "content": "# 704-es busz\n\nA 704-es jelzésű elővárosi autóbusz Százhalombatta, DE-Zrt. 2 sz. kapu és Martonvásár, vasútállomás között közlekedik. A járatot a Volánbusz üzemelteti.\n\n## Megállóhelyei",
}
[Extract key information]
{
    "key_info": "1. **Üzemeltetési útvonal**: A 704-es autóbuszjárat Százhalombatta DE-Zrt. 2-es kapuja és Martonvásár vasútállomása között közlekedik.\n 2. **Üzemeltető cég**: A járatot a Volánbusz üzemelteti."
}

[Contextual information]
{
    "title": <title>,
    "content": <content>
}

Please respond strictly in JSON format. Do not include any additional text outside the JSON structure:
{
    "key_info": "[the key information extracted from the given Contextual material]"
}
!!!!!

Figure 18: Prompt template for extracting key information from the wikipedia entries in the construction of HuSimpleQA.
{
"title": "Bodor Anikó",
"content": "# Bodor Anikó\nBodor Anikó (Zenta, 1941. június 15. – Zenta, 2010. július 9.) vajdasági népzenekutató, tanár.\n## Életrajz\n\n1960-ban a zentai gimnáziumban érettségizett, majd 1969-ig jogi tanulmányokat folytatott Újvidéken és Zágrábban. 1966-1972 között zenetudományi, művészettörténeti tanulmányokat folytatott Stockholmban és Uppsalában, ahol zenetudományi diplomát szerzett. Tanulmányait 1976-1980 között a belgrádi Zeneakadémia etnomuzikológia szakán folytatta, ahol 1984-ben a népzenetudományok magisztere lett.1972 és 1973 között az Újvidéki Rádió és Televízió munkatársa, 1975 és 1995 között a zentai alsófokú zeneiskola tanára, 1995-től a Zentai Városi Múzeum munkatársa.\n\nMunkásságát bizonyítja a több mint félszáz tanulmány, nagyobb cikk és népzenei kiadvány (könyv, kotta, lemez). Ezek közül a legnagyobb horderejű az őt könyvre tervezett Vajdasági magyar népdalok című sorozata volt, amelyből eddig négy kötet jelent meg. Szerkesztésében készült el a Daloló vajdasági fiatalok és a Vajdasági elő magyar népzene című népzenei lemezsorozat. A Délvidéki Népzenei Archívum létrehozója és gondozója volt.\n## Főbb művei\n\n-Hallottatok-e hírét? (1977)\n-Tiszából a Dunába folyik a víz (1978)\n-sajtó alá rendezte a Gombos és Doroszló népzenéje (1982), Az aldunai székelyek népdalai (1984) és A drávaszögi magyarok dalai (1989) c. könyveket.\n-A szlavóniai szigetmagyarság népdalai I. (Kiss Lajossal, 1990)\n-Vajdasági magyar népdalok I. (1997)\n-Vajdasági magyar népdalok II. (1999)\n-Vajdasági magyar népdalok III. (2003)\n-Vajdasági magyar népdalok IV. (2008)",
"key_info": "1. **Születési és halálzási adatok**: Bodor Anikó 1941. június 15-én született Zentán és 2010. július 9-én hunyt el Zentán.\n2. **Szakmai tevékenység**: Vajdasági népzenekutató és tanár.\n3. **Tanulmányok**: Jogot tanult Újvidéken és Zágrábban, zenetudományt és művészettörténetet Stockholmban és Uppsalában, etnomuzikológiát a belgrádi Zeneakadémián.\n4. **Munkásság**: Több mint félszáz tanulmány és népzenei kiadvány szerzője, a Vajdasági magyar népdalok című sorozatból négy kötet jelent meg.\n5. **Fontos művek**: 'Hallottatok-e hírét?' (1977), 'Tiszából a Dunába folyik a víz' (1978), 'Vajdasági magyar népdalok' sorozat (1997, 1999, 2003, 2008)."
}

Figure 19: Example of the extracted key information from the Hungarian wikipedia entries. (HuSimpleQA)
As a general knowledge expert, please generate 1 to 3 factual open-ended questions with their corresponding answers, based on the specified knowledge material. Ensure the questions meet the following criteria:

1. Content Relevance:
   • The question content should be related to the title corresponding to the key information. Only objective knowledge should be tested, such as the life story of important historical figures, information about important events, leaders of important events, or important attributes of certain objects and concepts. Do not test irrelevant information.
   • Minimize questions that are based solely on time and place, and instead, focus on unique and detailed aspects of the subject matter.

2. Clarity and Scope:
   • Each question is an independent and unambiguous question and can be answered independently without the help of other materials.
   • The question stem must specify the scope of the answer Avoid broad or open-ended questions. Ensure answer is clear and objective, avoiding subjective speculation.
   • For example, instead of asking 'hol találkozott Barack és Michelle Obama' (for which could have multiple answers 'Chicago' or 'a Sidney & Austin ügyvédi irodában'), questions had to specify 'melyik városban' or 'melyik cégnél'. Another common example is that instead of asking simply 'mikor' or 'melyik időpontban' (meaning "when" or "what time"), the question should ask 'melyik évben' or 'melyik napon' (meaning "which year" or "which day").
   • Answer should be brief, without additional explanations or redundancy. For example, if the question asks about someone's occupation, the answer should be simply 'tanár' ("teacher") not 'Ő tanár' (He is a teacher).

3. Consistency over Time:
   • Ensure that reference answers do not change over time. Try to avoid generating content that will change due to the progress of historical research, entertainment works, construction and updates of transportation roads, etc.
   • For example, instead of broadly asking 'ki Meredith pára a Grey's Anatomy-ban', which could change as new seasons are produced, questions about TV shows, movies, video games, and sports typically require specifying a point in time (e.g., 'ki Meredith pára a Grey's Anatomy 13. évadában?')

4. Question type: The questions should be open-ended, with a clear problem description and answer.

5. Moderate difficulty: Ensure the questions have appropriate readability and difficulty, allowing for clear differentiation between correct answers while maintaining accuracy.

6. Distinct Knowledge Points:
   • For each material, generate 1 to 3 questions and answers, ensuring the knowledge points being tested are distinct and do not overlap. Each question should offer a unique perspective and related answer.
   • All questions should be related to Hungarian-specific knowledge, reflecting aspects of Hungarian history, culture, geography, economy, figure, education or other uniquely Hungarian topics.

7. Language: The questions and answers are in Hungarian.

Example1:
[Input title and key information]:
{
    "title": "2004-es Formula-1 magyar nagydíj",
    "key_info": "1. A 2004-es Formula-1 magyar nagydíj a 2004-es világbajnokság tizenharmadik futama volt, amelyet 2004. augusztus 15-én rendeztek meg a Hungaroringen. Ez volt a 19. Formula-1-es futam Magyarországon.\n2. Michael Schumacher..."
}
[questions generated based on the information]
{
    "1": {
        "question": "A 2004-es Formula-1-es Magyar Nagydíj hányadik Forma-1-es versenyvolt Magyarországon?",
        "answer": "Michael Schumacher"
    },
    "2": {
        "question": "Milyen büntetést kapott Felipe Massa a 2004-es Formula-1-es Magyar Nagydijon a motorcsere miatt?",
        "answer": "Tizhelyes rajtbüntetés",
    },
    "3": {
        "question": "A 2004-es Formula-1-es Magyar Nagydíj a 2004-es világbajnokság melyik futama volt?",
        "answer": "tizenharmadik futam"
    }
}

Example2:
...
Please strictly follow the above requirements to generate the questions and answers in Json format, Do not add extra irrelevant format or content.
[Input title and key information]:
{
    "title": <title>,
    "key_info": <key_info>
}
[questions generated based on the information]
{
    "1": {
        "question": "str",
        "answer": "str"
    },
    "2": {
        "question": "str",
        "answer": "str"
    },
    ...
}...
****
- Role: Hungarian featured Content Identification Expert
- Background: Your role is to classify given questions and answers, determining whether they are related to Hungary and identifying their specific category.
- Goals:
  1. Determine whether given question and answer is related to Hungary.
  2. If related, identify the category it belongs to, and assign the appropriate label from the predefined list of categories.
- Classification Categories:
  1. Language: Content related to the Hungarian language, including proverbs, idioms, or linguistic knowledge.
  2. History: Content strictly describing specific historical events or developments in Hungary, such as wars, revolutions, significant treaties, or influential political or societal periods.
  3. Life, Culture, and Customs: Information about Hungarian religion, etiquette, cultural practices, holidays, daily life, and tourism.
  4. Education and Profession: Details about Hungary’s education system or associated professions.
  5. Geography and Place: Content about Hungary’s geography, cities, landmarks, or travel-related topics.
  6. Figure: Information about notable Hungarian individuals and their achievements.
  7. Politics, Policy, and Law: Information about Hungary’s political systems, policies, or legal regulations.
  8. Business and Finance: Content related to Hungary’s economy, business practices, or financial systems.
  9. Others: Content unrelated to Hungarian culture or not fitting into the above categories.
- Constraints:
  - Irrelevance: Only classify the content of question and answer related to Hungary. If the content is in Hungarian but unrelated to Hungary or is generic, classify it as unrelated.
  - Strict adherence to categories: Ensure consistent and accurate classification according to the nine dimensions.
  - Unclear content: For texts that cannot be clearly categorized, assign them to the “Others” category.
  - Each question can only have one category label.

Example1:
Input:
{
  "question": "Milyen posztumusz díjat kapott Fehér Sándor hegedűművész 2013. január 10-én?",
  "answer": "Magyar Civil Becsületrend",
}
Output:
{
  "hu_related":"True",
  "question_specific_label": "Figure"
}

Example2:
…
Please strictly follow the above format classify given questions and answers, do not add extra irrelevant format or content.
Input:
{
  "question": <question>,
  "answer": <answer>
}

Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
Output:
{
  "hu_related": [If question and answer are related to Hungarian characteristics, enter "True". Otherwise, enter "False"]
  "question_specific_label": [Predicated label should be chosen from the above nine categories. If there is an exception or it cannot be judged, set the string to an empty string.]
}
****

Figure 21: Prompt template for categorizing the generated question-answer pairs in the construction of HuSimpleQA.
"""
- Role: Hungarian Content Review Expert
- Background Information:
You need to determine whether an open-ended question and its answer are relevant to Hungarian characteristics and align with the provided background information. The given information includes various aspects about Hungary, including history, culture, language, geography, people, law, economy and more. If the question and answer involve content related to Hungary, you need to ensure the content is accurate and consistent with the background information.

- Task:
1. Determine relevance to Hungarian characteristics:
   - Ensure that the question and answer relate to the given title and key information, particularly with regard to Hungarian history, life, culture, customs, people, geography, politics, economy, education, etc. If the content does not align with Hungarian characteristics, it is considered irrelevant.
   - If the question is in Hungarian but unrelated to Hungarian characteristics, it is also considered irrelevant.
2. Ensure consistency with background information:
   - Verify that the question and answer are not only relevant to Hungary but also fit the background information provided. For example, Hungarian historical events should match the correct time and facts, and cultural references should align with actual Hungarian culture.
   - Ensure that the question description and answer are consistent with the information provided in the materials, without any deviation or omission
3. Appropriate difficulty:
   - The question should not be overly simple, and the answer should not be directly obvious from the question itself.

Example1:
Input: # This question is irrelevant to Hungary characteristics.
{
    "title": "Ipari Termékosztályozás",
    "key_info":"1. **Definíció**: Az Ipari Termékosztályozás (ITO) egy hierarchikus statisztikai osztályozás, amely az Eurostat PRODCOM jegyzékének hazai sajátosságokkal kiegészített változata, és ipari termékek és szolgáltatások gazdasági megfigyelésére használják.\n2. **Struktúra**: Az ITO kód 12 számjegyből áll, amelyek a TEAR08, TESZOR, PRODCOM, és KSH által képzett kódok kombinációjából állnak.\n3. **Történet**: Az ITO 2008. január 1-jén lépett hatályba, elődje a Belföldi Termékosztályozás (BTO) volt, amely 2007. december 31-ig volt érvényben.\n4. **Jogszabályok**: Az ITO-ra vonatkozó jogszabályok közé tartozik a 6/2018. (III.12.) Kmt rendelet, a Bizottság 2017/2119 rendelete, a Bizottság 912/2004/EK rendelete, a Bizottság 1209/2014/EU rendelete, az Európai Parlament és a Tanács 451/2008/EK rendelete, az Európai Parlament és a Tanács 1893/2006/EK rendelete, a 16/2011. (V. 16.) KIM rendelet, a Bizottság 2017/2119 rendelete, és a Tanács 3924/91/EK rendelete.",
    "question": "Melyik évben lépett hatályba az Ipari Termékosztályozás (ITO)?",
    "answers": "2008"
}
Output:
{
    "question_hu_relevant" : "fail",
    "answer_hu_correct": "pass"
}

Example2:
...
Please strictly follow the above format to judge the quality of question and answer, do not add extra irrelevant format or content.
Input:
{
    "title": <title>,
    "key_info": <key_info>,
    "question": <question>,
    "answer": <answer>,
}
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure:
Output:
{
    "question_hu_relevant": [If the question and answer is relevant to Hungarian characteristics, enter "pass". Otherwise, enter "fail"],
    "answer_hu_correct": [If the question and answer is consistent to key information, enter "pass". Otherwise, enter "fail"]
}
"""
"""
- Role: Hungarian Content Review Expert
- Background Information: You are responsible for evaluating whether an open-ended question and its corresponding answer meet the following standards:
1. Conciseness:
   - The question and answer should be clear, concise, and to the point. Avoid unnecessary details or redundant descriptions.
   - The content should focus on the core information, providing a precise answer without extraneous information.
   - For example, if the question asks about someone’s occupation, the answer should be simply 'tanár' ("teacher") not 'Ő tanár' (He is a teacher).
2. Single Question:
The question should contain only one query. Nested or multiple sub-questions within a single question are not allowed.
3. Specificity:
   - The question must be precise and targeted. Avoid broad open-ended questions like 'Miért' or "Milyen hatása volt?" which require analysis or subjective answers.
   - Questions should focus on factual, specific knowledge that leads to straightforward answers.
4. Clarity in Range:
   - The question must clearly indicate the exact range of possible answers.
   - For time-related questions, **do not use vague terms like 'Mikor' ('When'/‘what time’) in question**. Instead, specify “év” (year), “hónap” (month), or “nap” (day), not just “mikor”, to avoid ambiguous questions due to unclear time references. Ensure that the time units in both the question and answer are consistent (e.g., "melyik nap", "melyik év", "melyik hónap") (meaning "which year" or "which day").
   - For example, the question should not use a vague formulation like "mikor indította" but should be more precise, such as: "Melyik évben indította el...".
   - For location-related questions, the question should pinpoint a specific location (e.g., 'város', 'ország', 'régio') (meaning city, country, or region) rather than a vague 'hol' (where').
5. Consistency Over Time:
   - The question and answer should focus on historical facts or specific events that have already occurred. Avoid questions about ongoing trends, future predictions, or scenarios that could change over time.
   - Answers should remain consistent regardless of future developments.
   - For example, avoid asking about dynamic, ongoing series like 'Ki Meredith partnere a Grey’s Anatomy-ban?' Instead, use a **fixed time reference** such as 'Ki Meredith partnere a Grey’s Anatomy 13. évadában?'
Example1:
Input:
{
    "question": "Melyik törvénycikk szabályozta 1886-ban Magyarországon a trachoma terjedésének meggátolását?",
    "answer": "1886. évi V. törvénycikk",
}
Outputs:
{
    "question_hu_qualified" : "pass",
    "question_hu_time_consistency": "pass"
}

Example2:
...
Please strictly follow the above requirements to review the question and answer, do not add extra irrelevant format or content.
Input:
{
    "question": <question>,
    "answer": <answer>,
}
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure:
Output
{
    "question_hu_qualified": [If the question or answer are concise, specific and only have one question, enter "pass". Otherwise, enter "fail"],
    "question_hu_time_consistency": [If the question or answer do not change over time, enter "pass". Otherwise, enter "fail"]
}
"""
---
- Role: Hungarian factual Knowledge open-ended questions Review Expert
- Background: Hungarian General Knowledge open-ended question involves many aspects of Hungarian history, culture, geography, politics, people, etc., and a professional review expert is needed to ensure the accuracy of the answers.
- Goals: According to the given questions and reference answers, accurately judge whether the answers entered by the user are correct, and give the evaluation results: "CORRECT", "INCORRECT".
- Evaluation criteria for the given answer to be correct:
  • Predict answer fully contain the important information in the gold target.
  • Predict answer do not contain any information that contradicts the gold target.
  • Only syntactic matching matters; capitalization, punctuation, grammar, and order don’t matter.
  • Hedging and guessing are permissible, provided that the gold target is fully included and the response contains no incorrect information or contradictions.
  • The answer must be consistent with the scope of the question. For example, if the question asks “on which day was someone born,” the answer must specify the exact date, such as “January 3, 1997.”

- Workflow:
  1. Receive questions, reference answers, and user answers.
  2. Compare the reference answers and user answers to determine whether they are consistent.
  3. Based on the judgment results, output the corresponding evaluation results.
- Constraints:
  • For grading questions where the gold target is a number, the predicted answer needs to be correct to the last significant figure in the gold answer. For example, consider a question “Hány látogató érkezett Magyarországra 2024-ben?” with gold target “440k”.
    • Predicted answers “440k”, “444k”, and “435k” are all CORRECT.
    • Predicted answers “400k” and “413k” are INCORRECT.
  • The gold target may contain more information than the question. In such cases, the predicted answer only needs to contain the information that is in the question.
    • For example, consider the question “Where was The Martian filmed to represent the NASA Johnson Space Center?” with the gold target “Budapesti Bálna (HU)”. Either “Budapesti Bálna” or Bálna (építmény) would be considered a CORRECT answer.
  • Do not punish predicted answers if they omit information that would be clearly inferred from the question.
    • For example, consider the question “Melyik városban található a Magyar Nemzeti Bank székhelye?” with the gold target “Budapest, Magyarország”. The predicted answer “Budapest” would be considered CORRECT even though it does not include “Magyarország”.
    • Consider the question “Milyen díjat nyert a ‘Napsugár’ című magyar film a 2018-as Velencei Filmfesztiválon?” with the gold target “Legjobb Forgatókönyv Díja”. The predicted answer “Legjobb Forgatókönyv” would be considered CORRECT, because “díj” is presumed in the question.
    • For the question “Mi Dominik Szoboszlai magassága méterben?” with the gold target “1,85 m”, the predicted answer “1,85” would be considered CORRECT, because meters is specified in the question.
    • For the question “Mi Magyarország első női miniszterelnökének neve?” with the gold target “Wittner Mária”, the predicted answer “Mária” would be considered CORRECT, because the last name is presumed.
  • Do not punish for typos in people’s names if it’s clearly the same name.
    • For example, if the gold target is “Nagy László”, you can consider the following predicted answers as correct: “László Nagy”, “NAGY LÁSZLÓ”, or “nagy lászló”.

Example1:
Input:
{
  "question": "Melyik városban született Tisza Lajos 1832. szeptember 12-én?",
  "gold target": "Nagyváradon",
  "predicted answer": "Nagyváradon született Tisza Lajos 1832. szeptember 12-én."
}
Output: # Although the answer is long, it accurately answers the question
{
  "evaluation": "CORRECT"
}

Example2:
...
Please strictly follow the above example and requirements, evaluate the following answer.
Input:
{
  "question": <question>,
  "gold target": <std_answer>
  "predicted answer": <pred_answer>
}
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
Output:
{
  "evaluation":"CORRECT"/"INCORRECT"
}
---
***
- Role: Question Screening Expert
- Goals: Screen out the most suitable questions from multiple Hungarian general knowledge questions, ensuring that the questions and answers meet the following standards:
    1. Relevance to Hungarian Characteristics: Ensure that the question and answer are related to Hungarian history, culture, geography, etc.
    2. Appropriate Difficulty: The question should not be overly simple, and the answer should not be immediately obvious.
    3. Conciseness: The question and answer should be clear and to the point, avoiding unnecessary details.
    4. Single Question: Each question should contain only one query, no sub-questions.
    5. Specificity: The question should be precise and not too broad. Avoid vague, open-ended questions.
    6. Clear Range: For time or location-related questions, avoid vague inquiries like “Mikor” (When), as they do not provide a clear timeframe. Instead, ensure the question explicitly asks for a specific year, month, day, or a defined period.
    7. Historical Consistency: Focus on fixed, historical facts and events. Avoid questions about ongoing trends or future scenarios.
    8. Time/Geography-Specific Queries: If a question includes specific time limitations (such as year, month, or specific period) or specific geographic or personal details, the answer should be considered fixed and not subject to change over time.
        • This is especially important for questions related to transportation, geography, historical landmarks, and iconic structures.
- Constrains:
    1. Selecting the Best-Matching Question and Answer: From a group of questions, select the question and answer that best meet the criteria and mark it as 1. All other questions in the group should be marked as 0. In a group, there may be at most one question that is selected, but it is also possible that none of the questions meet the requirements.
    2. Consistent Evaluation Results: The number of evaluation results must match the number of input questions. Ensure that for every question, there is a corresponding evaluation result.
    3. Limit Time-Related Questions for Answer Diversity: Avoid selecting too many questions that focus on specific time-related aspects, such as the year an event occurred or a person’s birth year. Aim to ensure that the questions generate a diverse range of answers.

Example1:
Input:
{
    "question1": "Milyen feltételek szükségesek a bírósági titkári kinevezéshez?",
    "answer1": "Cselekvőképesség, magyar állampolgárság, jogi egyetemi diploma és jogi szakvizsga, érvényes pályaalkalmassági vizsgálat, érvényes hatósági erkölcsi bizonyítvány",
    "question2": "Melyik törvény szabályozza az igazságügyi alkalmazottak jogállását Magyarországon?",
    "answer2": "Az 1997. évi LXVIII. törvény (Iasz.)"
}
Output: # Question 1 is too broad and does not meet the requirement of asking and answering concisely and objectively.
{
    "is_selected": [0,1]
}

Example2:
...
Input:
<input_question>

Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
{
    "is_selected": [evaluation results]
}
***
HuSimpleQA Inference Prompt

"""Kérdés: {question} Kérjük, magyar nyelven adja meg a legjobb választ erre a kérdésre, és 0-tól 100-ig terjedő pontszámmal jelezze, hogy bizik a válaszában. Kérjük, válaszát a következő JSON formátumban adja meg:
{
    "answer": "Az Ön válasza itt",
    "confidence_score": szám
}"""

Translation to EN

"""Question: {question} Please provide your best answer to this question in Hungarian and indicate your confidence in your answer using a score from 0 to 100. Please provide your response in the following JSON format:
{
    "answer": "Your answer here",
    "confidence_score": number
}"""
JUDGE_PROMPT = {
    'system_prompt':
    """
    - Role: Hungarian General Knowledge Quiz Review Expert
    - Background: Hungarian General Knowledge Quiz involves many aspects of Hungarian history, culture, geography, politics, people etc., and a professional review expert is needed to ensure the accuracy of the answers.
    - Goals: According to the given questions and reference answers, accurately judge whether the answers entered by the user are correct, and give the evaluation results: "CORRECT", "INCORRECT", "NOT_ATTEMPTED".
    - Evaluation criteria:
        - "Correct":
            - Predict answer fully contain the important information in the gold.target.
            - Predict answer do not contain any information that contradicts the gold.target.
            - Only semantic meaning matters; capitalization, punctuation, grammar, and order don’t matter.
        - Hedging and guessing are permissible, provided that the gold.target is fully included and the response contains no incorrect information or contradictions.
            - The answer must be consistent with the scope of the question. For example, if the question asks “on which day was someone born,” the answer must specify the exact date, such as “January 3, 1997.”
        - If the attempt is not made, the answer should be marked as "NOT_ATTEMPTED".
        - Questions that the user has not attempted to answer should be marked as "NOT_ATTEMPTED".
        - The important information in the gold.target is not included in the answer.
        - No statements in the answer contradict the gold.target.
    - Workflow:
        1. Receive questions, reference answers, and user answers.
        2. Compare the reference answers and user answers to determine whether they are consistent.
        3. Based on the judgment results, output the corresponding evaluation results.
    - Constraints:
        - For grading questions where the gold.target is a number, the predicted_answer needs to be correct to the last significant figure in the gold answer. For example, consider a question “Hány látogató érkezett Magyarországra 2024-ben?” with gold_target “446k”.
            - predicted_answers “44k”, “444k”, and “435k” are all CORRECT.
            - predicted_answers “400k” and “413k” are INCORRECT.
            - predicted_answers “Körülbelül 400k” and “Több mint 300k” are considered NOT_ATTEMPTED because they neither confirm nor contradict the gold.target.
        - The gold target may contain more information than the question. In such cases, the predicted_answer only needs to contain the important information in the question.
            - For example, consider the question “Where was The Martian filmed to represent the NASA Johnson Space Center?” with the gold_target “Budapesti Bálna (HU)”. Either “Budapesti Bálna” or Bálna (építmény) would be considered a CORRECT answer.
            - Do not punish predicted_answers if they omit information that would be clearly inferred from the question.
        - For example, consider the question “Melyik városban található a Magyar Nemzeti Bank székhelye?” with the gold_target “Budapest, Magyarország”. The predicted_answer “Budapest” would be considered CORRECT, even though it does not include “Magyarország”.
        - Consider the question “Milyen díjat nyert a ‘Napsugár’ című magyar film a 2018-as Velencei Filmfesztiválon?” with the gold_target “Legjobb Forgatókönyv Díja”. The predicted_answer “Legjobb Forgatókönyv” would be considered CORRECT, because “díj” is presumed in the question.
        - For the question “Mi Dohnák Szoboszlai magassága méterben?” with the gold_target “1,85 m”, the predicted_answer “1,85” would be considered CORRECT, because nothing is specified in the question.
        - For the question “A Magyarország első női sportolóinak névét” with the gold_target “Wittner Mária”, the predicted_answer “Mária” would be considered CORRECT, because the last name is presumed.
        - Do not punish for typos in people’s names if it’s clearly the same name.
        - For example, if the gold target is “Nagy László”, you can consider the following predicted_answers as correct: “László Nagy”, “NAGY LASZLÓ”, or “nagy lászló”.
    Example:
    {
        "question": "Melyik törvény foglalkozik a találmányok szabadalmi oltalmával az 1969-es jogalkotásban?",
        "gold_target": "1969. évi II. törvény",
        "predicted_answer": "Nem áll rendelkezésre internetes keresés, így nem tudom megválaszolni a kérdést. Azonban 1969-ben valóban elfogadták a szabadalmi védelmi törvényt."
    }
    Output:
    {
        "evaluation": "NOT_ATTEMPTED"
    }
    Example2:
    ...
    ...
    ...
    'user_prompt':
    """Please strictly follow the above example and requirements, evaluate the following answer. Input:
    {{
    "question": {question},
    "gold_target": {answer},
    "predicted_answer": {pred_answer}
    }}
    Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
    Output:
    {{
    "evaluation":"Correct"/"Incorrect"/"NOT_ATTEMPTED"
    }}
    """
}
HuProverbRea-2CQ-Query:
You are a language expert specialized in Hungarian. Given a Hungarian phrase:

###############################################
Hungarian Phrase:
'Aki á-t mond, mondjon bé-t is.'
###############################################

and a context using this phrase:

###############################################
Hungarian Context:
Speaker1: Azt gondolom, abbahagyon a szakácsképző tanfolyamot. Valamennyire már úgyis tudok főzni.
Speaker2: Hiszen már két évét elvégeztél és már csak egy van hátra, most akarod feladni? Ez így nem helyes, aki á-t mond, mondjon bé-t is.
###############################################

What does the person mean by using this phrase? Please select one correct answer from the following two options:

###############################################
Options:
Option 1: ha te kezdted, viseld tetted következményeit!
Option 2: Ha azt mondod "a", mondd "b".
###############################################

You should only answer the option number, '1' or '2'. Do not output any other content other than the option number. Your answer:

Translation to EN:
You are a language expert specialized in Hungarian. Given a Hungarian phrase:

###############################################
Hungarian Phrase:
'He who says a, should also say b.'
###############################################

and a context using this phrase:

###############################################
Hungarian Context:
Speaker1: I think I'm going to quit the cooking course. I can already cook to some extent.
Speaker2: You've already completed two years and only one left, so you want to give up now? Isn't that right, he who says a, should also say b.
###############################################

What does the person mean by using this phrase? Please select one correct answer from the following two options:

###############################################
Options:
Option 1: if you start, you'll suffer the consequences of your actions!
Option 2: If you say "a", say "b".
###############################################

You should only answer the option number, '1' or '2'. Do not output any other content other than the option number. Your answer:

Figure 28: Example of HuProverbRea (2CQ). The left is the original example in OpenHuEval, the right is the English translation for visualization.

HuProverbRea-OE-Query
You are a language expert specialized in Hungarian. Given a Hungarian phrase:

###############################################
Hungarian Phrase:
'Ajándék lónak ne nézd a fogát.'
###############################################

and a context using this phrase:

###############################################
Hungarian Context:
Speaker1: 'Képtél, kaptam egy régi biciklit a szomszédunktól ajándékba, de kicsit rozsdás.'
Speaker2: 'Ne aggódj emiatt! Ajándék lónak ne nézd a fogát.'
###############################################

What does the person mean by using this phrase? Please do not just explain the meaning of the proverb itself, you should describe the true intention of the person who said the proverb (not the other person talking to him) based on the context. Please answer concisely in one sentence:

Translation to EN:
You are a language expert specialized in Hungarian. Given a Hungarian phrase:

###############################################
Hungarian Phrase:
'Don’t look at a gift horse’s teeth.'
###############################################

and a context using this phrase:

###############################################
Hungarian Context:
Speaker1: 'Imagine, I got an old bicycle from my neighbor as a gift, but it’s a little rusty.'
Speaker2: 'Don’t worry about it! Don’t look at a gift horse’s teeth.'
###############################################

What does the person mean by using this phrase? Please do not just explain the meaning of the proverb itself, you should describe the true intention of the person who said the proverb (not the other person talking to him) based on the context. Please answer concisely in one sentence:

Figure 29: Example of HuProverbRea (OE). The left is the original example in OpenHuEval, the right is the English translation for visualization.
"""
You are a language expert specialized in Hungarian. Given a Hungarian phrase:
#############################
Hungarian Phrase:
--------------------------
'{hu_text}'
#############################
and a context using this phrase:
#############################
Hungarian Context:
--------------------------
{context}
#############################
What does the person mean by using this phrase? Please do not just explain the meaning of the proverb itself, you should describe the true intention of the person who said the proverb (not the other person talking to him) based on the context. Please answer concisely in one sentence:
"""
Figure 30: Prompt template for model inference on HuProverbRea (OE).

"""
You are a language expert specialized in Hungarian. Given a Hungarian phrase:
#############################
Hungarian Phrase:
--------------------------
'{hu_text}'
#############################
and a context using this phrase:
#############################
Hungarian Context:
--------------------------
{context}
#############################
What does the person mean by using this phrase? Please select one correct answer from the following two options:
#############################
Options:
--------------------------
Option 1: {option1}
Option 2: {option2}
#############################
You should only answer the option number, '1' or '2'. Do not output any other content other than the option number.
Your answer:
"""
Figure 31: Prompt template for model inference on HuProverbRea (2CQ).
'en_system':""" Please act as an impartial judge specialized in Hungarian language and culture. Given a Hungarian saying, a context using that saying, and two analyses explaining 'what does the person mean by using that saying in the context?', please decide whether the given two analyses express the same meaning. If they reflect the same understanding of the saying's meaning, you should answer YES. If they are based on different interpretations of the saying, you should answer NO. Do not output anything other than 'YES' or 'NO'. Avoid any position biases and ensure that the order in which the analyses were presented does not influence your decision. Do not allow the length of the analyses to influence your judge, focus on their core meanings and their understandings of the Hungarian saying.""""
'en_user':"""[The start of Hungarian saying]
{proverb}
[The end of Hungarian saying]

[The start of the context]
{conversation}
[The end of the context]

[The start of the first analysis]
{answer}
[The end of the first analysis]

[The start of the second analysis]
{raw_pred}
[The end of the second analysis]
"""
'Your decision:

Figure 32: Prompt template for LLM as judge on HuProverbRea.

HuMatchingFIB-Hungarian
{
    "q_main": "Válaszd ki a legördülő listából, hogy melyik fogalom illik a hiányos mondatokba!\nA faj azon egyedeit, melyek tényleges szaporodási közösségeket alkotnak, #0# nevezzük.\nAz #1# mindazoknak a hatásoknak az összessége, melyek ténylegesen hatnak az élőlényekre.\nA populáció méretét jellemző egyik legfontosabb sajátosság a/az #2#.\nTerület- vagy térfigetegységre vonatkoztatott egyedszám a/az #3#. #4# tényező azon területen, amelyen belül az élőlények életműködésének feltételei vannak a #4#.\nJellemzően az a környezeti tényező határozza meg a populáció elterjedését, amelyre nézve az adott faj szűk tűrésű, ezt nevezzük úgy, hogy #5#.",
    "options": [
        "A.környezet",
        "B.tűrképesség",
        "C.egyedsűrűség",
        "D.egyedszám",
        "E.korlátozó tényező",
        "F.populációának"
    ],
    "std_ans": [
        "#0#",
        "#1#A",
        "#2#0",
        "#3#C",
        "#4#B",
        "#5#E"
    ]
}

Translation to EN
{
    "q_main": "Select from the dropdown list which concept fits into the incomplete sentences!\nThe individuals of a species that form an actual reproductive community are called #0#.\nThe #1# is the totality of all effects that actually influence living organisms.\nOne of the most important characteristics describing the size of a population is the #2#.\nThe number of individuals per unit area or volume is the #3#.\nThe range of an environmental factor within which living organism exhibit life processes is the #4#. Implicitly, the environmental factor that determines the distribution of a population is the one for which the species has a narrow tolerance, and this is called the #5#.",
    "options": [
        "A.environment",
        "B.tolerance",
        "C.population density",
        "D.population size",
        "E.limiting factor",
        "F.population"
    ],
    "std_ans": [
        "#0#",
        "#1#A",
        "#2#0",
        "#3#C",
        "#4#B",
        "#5#E"
    ]
}

Figure 33: Example of HuMatchingFIB. The left is the original example in OpenHuEval, the right is the English translation for visualization.
HuStandardFIB-Hungarian
{
    "q_main": "Találd ki a leírások alapján, hogy kiről vagy mireől van szó! Írd be a meghatározások után a megfelelő kifejezéseket!",
    "std_ans": [
        "#0#Mánuel;Manuelcsászár",
        "#1#Kancellária",
        "#2#Anonymus",
        "#3#Írásbeliség",
        "#4#Jegyző",
        "#5#Székesfehérvár;Fehérvár"
    ],
    "formatted_q_sub": [
        "A.Az #0# udvarában nevelkedett ifjúkorában III. Béla:",
        "B.A királyi adminisztráció céljából létrehozott intézményrendszer: #1#",
        "C.Feltehetően ő volt III. Béla jegyzője: #2#",
        "D.1181-ben tette általánossá III. Béla a hivatali ügyintézésben: #3#",
        "E.#4# fogalmazta meg a hivatalos íratokat, okleveleket;",
        "F.Ebben a városban temették el III. Bélát: #5#"
    ],
    "formatted_std_ans": [
        "#0#Mánuel;Manuelcsászár",
        "#1#Kancellária",
        "#2#Anonymus",
        "#3#Írásbeliség",
        "#4#Jegyző",
        "#5#Székesfehérvár;Fehérvár"
    ]
}

Translation to EN
{
    "q_main": "Based on the descriptions, guess who or what is being referred to! Enter the appropriate terms after the definitions!",
    "std_ans": [
        "#0#Manuel;Emperor Manuel",
        "#1#chancellery",
        "#2#Anonymus",
        "#3#written records",
        "#4#scribe",
        "#5#Székesfehérvár;Fehérvár"
    ],
    "formatted_q_sub": [
        "A. In the court of #0#, Béla III spent his youth:",
        "B. The institutional system created for royal administration: #1#",
        "C. He was likely the scribe of Béla III: #2#",
        "D. In 1181, Béla III made this mandatory in official proceedings: #3#",
        "E. #4# was responsible for drafting official documents and charters:",
        "F. The city where Béla III was buried: #5#"
    ],
    "formatted_std_ans": [
        "#0#Manuel;Emperor Manuel",
        "#1#chancellery",
        "#2#Anonymus",
        "#3#written records",
        "#4#scribe",
        "#5#Székesfehérvár;Fehérvár"
    ]
}

Figure 34: Example of HuStandardFIB. The left is the original example in OpenHuEval, the right is the English translation for visualization.

"""
The following questions are in Hungarian language on {hu_specific_dim}, please read the questions, and try to fill in the blanks in the question list. Please organize the answer in a list. An example:
{
    "instruction": "Írd be a megfelelő meghatározás mellé a fogalmat!",
    "questions": ["A.A szerzetesi közösségek szabályzatának elnevezése latinul: #0#", "B.Az első ún. kolduló rend: #1#", "C.A szerzetesek által kézzel másolt mű: #2#", "D.Papi nőtlenség: #3#", "E.A pápát megválasztó egyházi méltóságok: #4#", "F.A bencés rend megújítása ebben a kolostorban kezdődött a 10. században: #5#"],
}
The answers are:
{
    "answers": ["#0#regula", "#1#ferencesrend", "#2#kódex", "#3#cöllibátus", "#4#biborosok", "#5#Cluny"]
}
Now try to answer the following questions, your response should be in a JSON format. Contain the "answers" like the case given above.
The questions are:
{
    "instruction": {instruction},
    "questions": {questions},
}
"""

Figure 35: Prompt template for model inference on HuStandardFIB.
"""You are a native Hungarian teacher. The following question is in Hungarian language on {hu_specific_dim}.
Please read the question, and choose the appropriate option from the provided "options" list to fill in each blanks in the text based on the context. Read the entire text, then fill in the blanks. Some options can be selected repeatedly. Please organize the answer in a list. An example:
{
    "question": "Egészítsd ki a Janus Pannonius életére vonatkozó rövid szöveget! Segítségként használd az internetet! Vigyazz, nem minden szót kell felhasználnod!\nJanus Pannonius nem csupán költőként volt jelentős személyisége kora Magyarországnak. #0# unokaöccsekét a politikából is hamar kivetette a részét #1# tanulmányai után pécsi #2# lett, majd a királyné mellett #3#. Fokincstartóként és a #4# báni cím elnyerésével komoly politikai karrierét futott be Mátyás király udvarában. A királlyal megromló kapcsolata miatt részt vett a #5# elleni összeesküvésben, ezért menekülnie kellett. Ez, és az akkor már súlyosbodó betegsége okozta halálát #6#.",
    "options": ["A.érsék", "B.szilavón", "C.Vitéz János", "D.püspök", "E.főpohártnok", "F.Ulászló", "G.1474-ben", "H.főkancellár", "I.Itáliai", "J.Kinizsi Pál", "K.Kálmán", "L.1472-ben", "M.Prágai", "N.Mátyás"],
},
The answer is:
{
    "answer": ["#0#C", "#1#I", "#2#D", "#3#H", "#4#B", "#5#N", "#6#L"]
}
Now try to answer the following question, your response should be in a JSON format. Contain the "answer" like the case given above.
The question and options are:
{
    "question": {question},
    "options": {options},
}"""
Figure 36: Prompt template for model inference on HuMatchingFIB.
"""
Given a thinking process for answering a question, follow these steps to extract contrastive expressions from the answer text:
1. Identify the Primary Language:
   • First, determine the primary language of the answer text. The language could be English, Hungarian, or any other language.
2. Extract Contrastive Words, Phrases, or Expressions:
   • Identify all the phrases that express a shift in opinion, explanation, or answer, phrases that signal a contrast or change in direction.
   • For English: “However,” “but,” “On the other hand,” “Although,” “Nevertheless,” “Yet,” “Despite,” “In contrast,” “Instead,” “Even though.”
   • For Hungarian: “azonbAzonbanan,” “De,” “Másrészt,” Ellentétben,” “Pedig,” “MEGIS,” “Bár,”
- Requirements:
  1. Identify and list all the contrastive words or phrases that indicate a shift in meaning, thought, or direction.
  2. These expressions should be **at the beginning of a sentence** to signal a shift.
  3. Keep the original text’s meaning and context intact.
  4. Ensure to maintain the original capitalization of the words (e.g., “However” vs. “however”).
  5. Provide a clear list of all the identified contrast words or phrases.

Input text:
{
    "question": <question>,
    "answer": <answer>,
}
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure. The output should also include the detected language type.
Output:
{
    "language": "<detected_language>",
    "shift_expression":[list]
}
"""

Figure 37: Prompt template for splitting the LRM’s thinking process into thoughts on HuSimpleQA (step 1/2).
"""
Given the thinking process, identify all the phrases that express a shift in opinion, explanation, or answer, i.e., phrases that signal a contrast or change in direction (commonly known as “contradiction,” “contrast,” or “transition” phrases). For each identified phrase, wrap it in the format <shift_word_X>word<shift_word_X>, where X is the sequential number for each occurrence of the phrase. Notice that the text provided may be in either Hungarian or English.
You should provide a list of the sequence numbers corresponding to the phrases that convey a shift in meaning.
If there is no "<shift_word_X>" tag in text, return an empty list.

Input text:
<input_text>
Please respond strictly in JSON format. Do not include any additional text outside the JSON structure.
Output:
{
    "shift_word_ids": [list of sequential numbers of shifting phrases]
}
"""

Figure 38: Prompt template for splitting the LRM’s thinking process into thoughts on HuSimpleQA (step 2/2).

"""
Question Q = <question>
Splited Thought S = <split_solutions>
Correct Answer A = <expected_answer>

1. Please analyze the relevance between the splited thought S and the questin Q, and conduct some verifications to check the correctness of the solution itself. Please think step by step to give an explanation **EXPLANATION**.

2. If you think the splited thought S can lead to the orrect answer A of the problem P, please stick to the line of thinking without deviation and carry it through to completion. If you think it cannot yield the correct answer or you’re not sure, don’t force yourself to give an answer and generate **None**.

3. Please tell me honestly how confident you are that you can solve the problem P correctly based on the the splited thought S. Out of 2, please generate your confidence score **CONFIDENT SCORE**.

Please output **EXPLANATION** and **CONFIDENT SCORE** according to the following JSON format:
{
    "expalanation": "EXPLANATION",
    "confident_score": "CONFIDENT SCORE"
}
"""

Figure 39: Prompt template for evaluating the correctness of the thoughts of the LRM’s thinking process on HuSimpleQA.
Question: Melyik hivatal látja el a Hamisítás Elleni Nemzeti Testület titkársági feladatait?
Which office handles the secretarial tasks of the National Board Against Counterfeiting?
Correct Answer: Szellemi Tulajdon Nemzeti Hivatala
National Office of Intellectual Property

Thought1
Okay, let's tackle this question. The user is asking which office handles the secretarial tasks for the National Anti-Forgery Committee (Hamisítás Elleni Nemzeti Testület) in Hungary. I need to figure out the correct answer and assess my confidence. First, I recall that in Hungary, various government bodies have specific roles. The National Anti-Forgery Committee is likely related to financial or document security. I remember that the Hungarian National Bank ...the secretariat duties could be under a different department.

Thought2
Alternatively, the Government Office (Kormányhivatal) usually handles administrative tasks for various committees.

Thought3
Wait, the National Tax and Customs Administration (Nemzeti Adó- és Vámhivatal, NAV) deals with tax, customs, and financial fraud. Since counterfeiting currency is a big part of their work, maybe NAV supports the committee

Thought5
But given the context, NAV seems correct

Predicted Answer: A Hamisítás Elleni Nemzeti Testület titkársági feladatait a Nemzeti Adó- és Vámhivatal (NAV) látja el.
The secretariat tasks of the National Body Against Counterfeiting are carried out by the National Tax and Customs Administration (NAV).

Figure 40: Example of the thoughts in Deepseek-R1’s thinking process on HuSimpleQA. ✓ and ✗ indicate the correctness of the thoughts. The original text is in black, while the translation into English is in blue.
Question: Melyik magyar rendező filmjében nyújtott alakításáért kapott Helen Mirren különdíjat a 2012-es Európai Filmdíjakon?
For her performance in which Hungarian director’s film did Helen Mirren receive a special award at the 2012 European Film Awards?
Correct Answer: Szabó István (Istvan Szabo)

Thought1
Melyik magyar rendező filmjében nyújtott alakításáért kapott Helen Mirren különdíjat a 2012-es Európai Filmdíjakon? Nagyon nehéz kérdés, nem vagyok teljesen biztos a válaszban, de megpróbálom kitalálni. Először is, Helen Mirren egy angol nagyjátékostermek, de számos filmben játszott, így valószínűleg valami brit vagy Európa kapcsolatú filmben játszott, amelyet magyar rendező forgattat.
In which Hungarian director’s film did Helen Mirren receive a special award at the 2012 European Film Awards? This is a very difficult question, and I’m not completely sure of the answer, but I’ll try to figure it out. First of all, Helen Mirren is a renowned English actress who has appeared in many films, so it’s likely that she acted in a British or Europe-related film directed by a Hungarian director.

Thought2
De a kérdésről szóló részlet szerint a magyar rendező filmjében nyújtott alakításáért kapta a díjat. Hmmm, ki lehetne a magyar rendező? Tudom, hogy néhány magyar rendező sikert ért el nemzetközi filmekben, de pontosan nem emlékszem rájuk most. Lehet Béla Tarr?
But according to the detail of the question, she received the award for her performance in a film by a Hungarian director. Hmm, who could the Hungarian director be? I know that some Hungarian directors have achieved success in international films, but I can’t exactly recall them right now. Could it be Béla Tarr?

Thought20
De most, a fejemben, nem emlékszem rá. Így, a válaszom a kérdésre, hogy sajnálom, de nem tudom, melyik magyar rendező filmjében nyújtott alakításáért kapott Helen Mirren különdíjat a 2012-es Európai Filmdíjakon.
But right now, I can’t remember it. So, my answer to the question is: Sorry, I don’t remember which Hungarian director’s film Helen Mirren received the special award for her performance at the 2012 European Film Awards.

Predicted Answer: Sajnálom, de nem emlékszem arra a filmben, ahol Helen Mirren nyújtott alakításáért kapott különdíjat a 2012-es Európai Filmdíjakon, és melyik magyar rendező műve volt.
Sorry, but I don’t remember the film where Helen Mirren received a special award for her performance at the 2012 European Film Awards, nor which Hungarian director’s work it was.

Figure 41: Example of the thoughts in QWQ’s thinking process on HuSimpleQA. ✓ and × indicate the correctness of the thoughts. The original text is in black, while the translation into English is in blue.
You are a semantic paragraph segmentation expert, responsible for dividing the thinking process content I provide to you (generated by a large language model) into paragraphs. The content of the thinking process pertains to the reasoning and solving process of fill-in-the-blank questions in Hungarian.
The background of the thinking process content is the reasoning and solving process for fill-in-the-blank questions in the Hungarian version. I need you to segment the original complete thought process content into multiple paragraphs and assign each paragraph a tag strictly limited to the categories: "Introduction", "Reasoning", "Review", and "Summary", based on its content. Below, I will describe the characteristics of these four types in detail and provide examples for reference. In most cases, the thinking process content is presented in English, with a very small portion in Hungarian. You can apply the same logic for segmentation. Please note that no additional content should be added or removed from the original thinking process content;
Additionally, there should be no overlap between the divided paragraphs.

Segment 1: Introduction
Description: The introduction is typically located at the beginning of the thinking process content. It usually consists of the large language model's brief restatement of the problem and a descriptive account of the work it is about to undertake. It does not include the actual start of the analysis of the problem.
Such statements may generally include the following:
(1) Alright, I have this history question to complete. It’s about the concept of royal power and political systems in Western Europe, specifically in England and France during a certain period. I need to fill in the blanks using the provided options. Let’s see, there are nine blanks, and I have nine options to choose from, but some might be used more than once, though the example didn't specify that. I'll approach this step by step.
(2) I have this task to complete a diagram by dragging expressions to their corresponding numbers. The expressions are...
(3) I have this task here. I need to find the odd one out from each group of words. Each group has words that belong to one part of speech, except for one word that doesn’t fit in that group. I need to identify the odd one out and state its part of speech.
(4) I'm going to answer this question about the Csőrsz-ditch. I need to decide whether each statement is true or false based on the information provided and any knowledge I have about the topic. Let's go through each one step by step.
(5) I'm going to try to fill in the blanks in this text. It seems like a story about someone exploring unknown places, maybe flying or something like that. I have a list of options to choose from, and I need to pick the right ones to complete the sentence properly. I should pay attention to the context and make sure the words fit grammatically and make sense in the story.

Segment 2: Reasoning (Important)
Description: The reasoning process typically constitutes the main body of the thinking process content. It includes the detailed thinking and reasoning steps undertaken by the large language model to solve the fill-in-the-blank questions. You should collect, as thoroughly and sequentially as possible, the content that you identify as part of the "reasoning".
The use of '\n\n' paragraph separators may serve as a suitable paragraph division choice, but please note that answer-related statements may also utilize '\n\n' for line breaks or section divisions. Exercise judgment to distinguish between these usages. Paragraphs in the Reasoning section should neither be excessively brief nor unduly lengthy.

Segment 3: Review (Important)
Description: The review usually occurs after the reasoning process is essentially complete but before the summary. This section typically includes a review of the entire reasoning process and may contain keywords or phrases such as "Overall, ..." or "double check..."
Please note that not all thinking process content necessarily includes a review content; in some cases, the reasoning process may be directly followed by the summary. In such instances, you can refer to the example response format provided.
Segment 4: Summary
Description: The summary is generally the model's ultimate output, i.e., the part where the model provides the summary after completing all the reasoning in the thinking process. It may also be presented at the very beginning of the thinking process, in which case it might be directly displayed in a JSON format, requiring your judgment. It typically includes some indicative phrases, such as "...final answer..." or "...final choices...". However, please note that content containing "...summarize..." may not necessarily be the summary; it could be part of the intermediate reasoning process. Be sure to distinguish such content and exclude it from the summary.
The statements in the summary may generally include the following:
(1) So, my final answer is:
{
"answer": ["#0#B", "#1#F", "#2#H", "#3#D", "#4#G", "#5#H", "#6#C", "#7#C"]
}
(2) I'll present this in the required JSON format.
**Final Answer**
\[ \boxed{ \{ "answer": [ "#0#A", "#1#G", "#2#C", "#3#I", "#4#B", "#5#C", "#6#H", "#7#E", "#8#F" ] \} } \]
(3) So, the final answer should be:
{
"answer": ["#0#L", "#1#H", "#2#A", "#3#I", "#4#E", "#5#C", "#6#K", "#7#M", "#8#B", "#9#M", "#10#D", "#11#F"]
}

The input content you receive is after [input thinking process content] and you should response strictly in the provided format. The specific content should be added after the [Your segmentation results] field and must in JSON format:
[input thinking process content]
COTs content
[Your segmentation results]
{
    'segment':[
        {
            'text': content you regard as "Introduction",
            'category': "Introduction"
        },
        {
            'text': content you regard as "Reasoning",
            'category': "Reasoning"
        },
        ...
        {
            'text': content you regard as "Review",
            'category': "Review"
        },
        {
            'text': content you regard as "Summary",
            'category': "Summary"
        }
    ]
}

Figure 43: Prompt template for splitting LRM’s thinking process into segments and categorizing these segments on HuMatchingFIB. (part 2/3).
Some specific examples are as follows:
[input thinking process content]
...
[Your segmentation results]
{
    'segment':[
        {
            'text': "<think>Okay, let me try to figure out how to answer this. The task is to fill in the blanks in the Hungarian text about morphemes using the given options. Let's look at each blank step by step.\n\n",
            'category': "Introduction"
        },
        {
            'text': "First, the question starts by defining a morpheme as the smallest linguistic unit with its own meaning. So blank #0# should be \"unit!\", which in Hungarian is \"egység\". Checking the options, H is \"egység\",",
            'category': "Reasoning"
        },
        {
            'text': "Then #1# is \"meaning\", which is \"jelentése\" (option B).\n\n",
            'category': "Reasoning"
        },
        ...
        {
            'text': "Putting it all together:\n\n#0: H (egység)\n#1: B (jelentése)\n#2: A (toldalék)\n#3: C (egyszerű)\n#4: D (összetett)\n#5: F (képző)\n#6: G (jel)\n#7: E (rag)</think>\n    \\"answer\": [\"#0#H\", \\"#1#B\", \\"#2#A\", \\"#3#C\", \\"#4#D\", \\"#5#F\", \\"#6#G\", \\"#7#E\"]\n",
            'category': "Summary"
        }
    ]
}

[input thinking process content]
...
[Your segmentation results]
...

Now, the target content you need to split is as follows. Please provide your standardized answer after [Your segmentation results] in JSON format:
[input thinking process content]
{Raw_COT}
[Your segmentation results]

Figure 44: Prompt template for splitting LRM’s thinking process into segments and categorizing these segments on HuMatchingFIB. (part 3/3).
Role
You are a text judgement and reasoning expert.

Task Description
Your task is to perform multidimensional classification of the output from a reasoning model. The model's output has been segmented into multiple segments (Introduction, Reasoning, Review, final_answer), among which there are Reasoning-type segments. You need to classify these Reasoning segments based on the following four dimensions:

Dimensions and Classification Rules:

Dimension 1: Correctness of the Result
Based on the options, the standard answer (std_ans) and the model's answer (model_ans), determine whether the result in each Reasoning segment is correct. The classification is as follows:
Class 1: Completely Incorrect
All blank-filling results in the segment do not match the standard answer.
Class 2: Partially Correct
Some blank-filling results in the segment match the standard answer, while others do not.
Class 3: Completely Correct
All blank-filling results in the segment match the standard answer.
Class 4: Non Conclusion
No conclusion has been provided yet.
Constraints:
If the segment involves multiple blanks, compare each result with the standard answer.
If the segment does not explicitly mention the blank-filling results, infer based on the context.

Dimension 2: Reasoning Complexity
Determine whether the reasoning process in each segment is a simple assertion or involves complex thinking. The classification is as follows:
Class 1: Simple Assertion
The segment directly provides the answer without detailed reasoning.
Class 2: Complex Thought
The segment includes repeated thinking, logical reasoning, hypothesis validation, or other complex processes.
Constraints:
If the segment contains keywords such as: "Wait, perhaps...", "I need to consider...", "Alternatively...", "Hmm, maybe...", "Let me think..." classify it as "Complex Thought."
If the segment only directly provides the answer (e.g., "#1# is H.508"), classify it as "Simple Assertion."

Dimension 3: Reasoning Scope
Determine whether the reasoning in each segment involves modifying any previously solved blanks. The classification is as follows:
Class 1: Only Current Blank
The segment only provides an answer for the unresolved blank and does not modify previously solved blanks.
Class 2: Modify Previous Blanks
The segment not only provides an answer for the unresolved blank but also modifies or corrects previously solved blanks.
Class 3: Current Blank and Consecutive Blank
The segment provides an answer for the current unresolved blank and also addresses consecutive blanks, either by solving them or making adjustments.
Constraints:
If the segment explicitly mentions modifying previously solved blanks (e.g., "Wait, I need to change #2# to..."), classify it as "Modify Previous Blanks."
If the segment only focuses on the current blank, classify it as "Only Current Blank."
If the segment addresses both the current blank and consecutive blanks, classify it as "Current Blank and Consecutive Blank."

Figure 45: Prompt template for tagging the reasoning segments of LRM’s thinking process along the four dimensions on HuMatchingFIB (part 1/2).
Dimension 4: Language Transfer
Determine whether each Reasoning segment includes the process of translating Hungarian into English. The classification is as follows:
Class 1: Contains Language Transfer
The segment includes a translation process similar to:
“Érőteljes #3# és a költői #4# gazdag használata jellemzi.”
This translates to “It is characterized by strong #3# and rich use of poetic #4#.”
Class 2: No Language Transfer
The segment does not include the above translation process.
Constraints:
If the segment contains an explicit translation process (e.g., “This translates to…”), classify it as “Contains Language Transfer.”
If the segment only uses Hungarian or English without translation, classify it as “No Language Transfer.”

Example Input:
{
    "options": [...],
    "std_ans": [...],
    "model_ans": [...],
    "segments": [
        {
            'text': ...
            'category': "Introduction"
        },
        {
            'text': ...
            'category': "Reasoning"
        },
        ...
    ]
}

Example Output:{
    "segments": [
        {
            'text': ...
            'category': "Introduction"
        },
        {
            'text': ...
            'category': "Reasoning",
            'Dimension1': "Non Conclusion",
            'Dimension2': "Complex Thought",
            'Dimension3': "Only Current Blank",
            'Dimension4': "No Language Transfer",
        },
        ...
        {
            'text': ...
            'category': "Reasoning",
            'Dimension1': "Completely Correct",
            'Dimension2': "Complex Thought",
            'Dimension3': "Current Blank and Consecutive Blank",
            'Dimension4': "No Language Transfer",
        },
        ...
    ]
}

Notes
1. Ensure that every Reasoning segment is classified, and the classification results are logical.
2. If a dimension cannot be clearly classified for a segment, infer the most appropriate category based on the context.
3. The output must be in JSON format and include classification results for all four dimensions.

Figure 46: Prompt template for tagging the reasoning segments of LRM’s thinking process along the four dimensions on HuMatchingFIB (part 2/2).
[
    {
        "text": "<think>Okay, let me try to work through this problem step by step. So, the task is to fill in the blanks in the Hungarian text about manipulation techniques in media. Each blank corresponds to a term from the provided options. Let's look at each numbered section and see which option fits best.",
        "category": "Introduction"
    },
    ...
    {
        "text": "#4 is \"Egy gondolat bánt engemet...\"", described as a "zaklatott menetű költemény\" (disturbed flow poem). The options include rapszódia (E), which is a rhapsody, often passionate or irregular in form. That seems to fit, so #4 is E.",
        "category": "Reasoning",
        "Dimension1": "Completely Correct",
        "Dimension2": "Simple Assertion",
        "Dimension3": "Only Current Blank",
        "Dimension4": "Contains Language Transfer"
    },
    {
        "text": "#5 is \"Szeptember végén\", a love poem to his wife. The options left include A (dal), C (elégia), etc. But \"szerelmi vallomás\" (love confession) might be a song or lyric poem. Dal (A) is possible, but I'm not certain. Wait, the example says \"szerelmi vallomása\", which could be a lyric poem. Dal (song) is a common form for love poems. So #5 is A.",
        "category": "Reasoning",
        "Dimension1": "Completely Correct",
        "Dimension2": "Complex Thought",
        "Dimension3": "Only Current Blank",
        "Dimension4": "Contains Language Transfer"
    },
    ...
    {
        "text": "Let me confirm:\n\n- #0: János vítez → G (elbeszélő költemény)\n- #1: Helység kalapácsa → H (vigéposz)\n- #2: Borozó → A (dal)\n- #3: Felhők ciklus verse → C (elégia)\n- #4: Egy gondolat... → E (rapszódia)\n- #5: Szeptember végén → A (dal)\n- #6: Tigris és hiéna → B (regény)\n- #7: A hóhér kötele → D (dráma)",
        "category": "Review"
    },
    ...
    {
        "text": "{\n    \"answer\": [\n        \"#0#G\",\n        \"#1#H\",\n        "\"#2#A\",\n        "\"#3#C\",\n        "\"#4#E\",\n        "\"#5#A\",\n        "\"#6#B\",\n        "\"#7#D\"\n    ]\n}",
        "category": "Summary"
    }
]

Figure 47: Example of splitting Deepseek-RL’s thinking process into segments, categorizing these segments, and tagging the reasoning segments on HuMatchingFIB. The categorizing results are in green and the tagging results are in red.
[
    {
        "text": "Alright, I have this task to fill in the blanks in a text about bank cards using the provided options. The question is in Hungarian, and I need to make sure that the words I choose fit correctly in the context. Let's see, the text is about bank cards being a substitute for cash, and how they can be used for purchases, withdrawing money, and checking balances. It also mentions that the card belongs to the issuing bank, which can reclaim it in case of non-payment or cancel its use.",
        "category": "Introduction",
    },
    ...
    {
        "text": "Moving on to the third blank: \"Segítségével a vásárláson kívül még pénzt lehet felvenni a bankjegykiadó automatából, vagy tájékozódni lehét a #2# egyenlegeről.\" This translates to \"With its help, besides shopping, you can also withdraw money from an ATM or check your... balance.\" So, it's about checking a balance related to something. Looking at the options, \"bankszámla\" means \"bank account,\" which makes sense because you check your account balance. \"Megtakarításunk\" means \"our savings,\" which could also be related, but \"bankszámla\" is more precise. So, I'll go with \"A.\tbankszámla.\"",
        "category": "Reasoning",
        "Dimension1": "Completely Correct",
        "Dimension2": "Complex Thought",
        "Dimension3": "Only Current Blank",
        "Dimension4": "Contains Language Transfer"
    },
    ...
    {
        "text": "Fifth and final blank: \"...illetve a #4# letiltani.\" This means \"and to cancel its use.\" So, it's about the bank being able to cancel the card's usage. Looking at the options, \"használatát\" means \"its usage,\" which fits perfectly here. So, \"E.\thasználatát\" is the right choice.",
        "category": "Reasoning",
        "Dimension1": "Completely Correct",
        "Dimension2": "Simple Assertion",
        "Dimension3": "Only Current Blank",
        "Dimension4": "Contains Language Transfer"
    },
    ...
    {
        "text": "Let me double-check all the choices:\n\n- #0# F.\tfizetési (payment)\n\n- #1# B.\tkészpénzzel (with cash)\n\n- #2# A.\tbankszámla (bank account)\n\n- #3# G.\tkibocsátó (issuing)\n\n- #4# E.\thasználatát (its usage)\n\nSeems logical and consistent with the context provided. I don't think I need to make any changes.",
        "category": "Review"
    },
    {
        "text": "**Final Answer**\n\n\\boxed{ \\text{#0#F, #1#B, #2#A, #3#G, #4#E} } \\}",
        "category": "Summary"
    }
]

Figure 48: Example of splitting QwQ’s thinking process into segments, categorizing these segments, and tagging the reasoning segments on HuMatchingFIB. The categorizing results are in green and the tagging results are in red.

Reviewer paper summaries: The goal of the paper is to study the performance on LLMs on Hungarian. The paper points out that Hungarian performance of LLMs is not well-studied and previous works have limited themselves to automated translations which could miss out on linguistic and cultural nuances. The previous works also largely limit themselves to multiple choice questions, which leaves out richer, generation-based tasks. The paper curates a dataset from various online Hungarian resources, e.g., the Hungarian version of Quora. Results show that the frontier LLMs still have some way to go until they achieve satisfactory performance.

The paper introduces OpenHuEval, a Hungarian-language-focused benchmark designed to evaluate large language model (LLM) capabilities on Hungarian-specific linguistic and cultural tasks. OpenHuEval employs real user queries, generative task evaluation, and the LLM-as-a-judge approach. It explicitly incorporates Hungarian-specific dimensions, covering a variety of language nuances to offer a thorough assessment. The paper provides comparative analyses of LLM performance on selected OpenHuEval benchmarks against English-centric benchmarks, highlighting the performance differences between open-source and closed-source models. Furthermore, the authors propose novel methodologies to analyze and interpret the reasoning processes and confidence levels of recently developed large reasoning models (LRMs), identifying intrinsic reasoning patterns, such as explicit translation insertion, particularly in non-English scenarios.
Summary of strengths: 1. Lack of comprehensive multilingual evaluations of LLMs is indeed an issues so it is great that the paper makes a contribution in this area.
2. The dataset curation process is quite through, though it would be nice to move important details from the appendix to the main paper.

s1. The study provides evidence supporting the need for language-specific benchmarks by evaluating and comparing performance across a range of tasks involving both open-source and closed-source LLMs.

s2. The dataset used in OpenHuEval includes authentic, informal, user-generated queries collected from Hungarian online platforms, alongside formal data sources such as Hungarian Wikipedia articles and examination questions, thus covering diverse linguistic and cultural contexts.

s3. The selected tasks are designed to capture different aspects of the models' comprehension capabilities, utilizing a diverse set of evaluation metrics.
Summary of weaknesses: 1. The dataset filtering protocol in Appendix B can use more discussion. It is not clear how robust the GPT4o prompts in Figure 10 were, how the criteria were 1-10 were selected and the precise phrasing was decided upon.
2. The paper is sometimes difficult to understand because it relegates important details to the appendix, e.g., datasets filtering procedure in Section 2. I would suggest bringing the filtering criteria to the main paper. The precise prompts can still stay in the appendix.
3. At many places, the GPT4o model is used as a judge. The paper should clarify if the performance of GPT-4o was evaluated using humans.
4. The abstract and intro hint at an apples-to-apples English vs. Hungarian comparison. That is, the reader might think that the paper asks the same questions in two languages and compares the performance. That is not the case, e.g., in Section 3.3 and Table 5. It would be great to clarify that up front.

w1. Comparing HuProverbRea with MAPS, without first analyzing MAPS’s nature, could introduce implicit biases arising from differences in the datasets. These biases may prevent a direct comparison of proverb reasoning. A similar concern applies when comparing HuSimpleQA with SimpleQA

w2. GPT-4o was extensively used in the creation of the OpenHuEval datasets. Re-evaluating GPT-4o on this data (Table 12) risks circularity and potential bias resulting from data leakage, unfairly favoring it over other models.
Comments, suggestions and typos: 1. Section 2.1: How were the dimensions like Language, History, Life, etc were chosen? Are there any salient dimensions that the paper left out?
2. Section 2.2: Since the posts are scraped from an online forum, there might be issues with quality, e.g., non-serious posts, toxicity and bias. What kind of quality control mechanism did the paper employ? I see that Figure 10 contains a list of issues that GPT4o filters. But some of the items in the list require the model to reason about complex issues, e.g., social and ethical considerations. How performant was GPT4o on these issues? Was there any prompt tuning required? How did the questions that GPT4o rejected look like?
3. The selection criteria in Figure 10 is mostly about questions. Did the paper also test the quality of the answers?
4. Line 331: What was the default length?
5. Line 417: Regarding low performance on BF, PPL and EP, is the issue that the model does not know the Hungarian language, or the Hungarian culture? In other words, would the model perform similarly worse if the same questions were answered in English?
6. “revealing the intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example”. This seems to be a bit of an over claim. If Hungarian is the only non-English language the paper studied, the claim should be limited to Hungarian. There is no guarantee that all non-English languages will behave like Hungarian.

Suggestions:
1. Compare DeepSeek R1 with its Open AI counterpart models: O1 and O3-mini-high
2. Fine-tune LLMs on the Hungarian language then evaluate them on OpenHuEval and the other existing benchmarks (as in section 3.4)

Typos :
1. Line 059: we introduces → we introduce
4. Line 549: repeatedd → repeated 
5. Line 563: clonclusion → conclusion
6. Line 564: reasoniing → reasoning 
7. Appendix B - Figure 6: querys → queries
Overall assessment scores: 3.5

3.5