MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset
Authors: Sagi Shaier, George Arthur Baker, Chiranthan Sridhar, Lawrence Hunter, Katharina von der Wense
Accepted at: acl25

MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset

Sagi Shaier†   George Arthur Baker†   Chiranthan Sridhar†
Lawrence E Hunter†   Katharina von der Wense†◊
†University of Colorado Boulder
†University of Chicago, Department of Pediatrics
◊Johannes Gutenberg University Mainz
E-mail: {sagi.shaier, george.baker, chiranthan.sridhar, katharina.kann}@colorado.edu

Abstract
Language models (LMs) have excelled in various broad domains. However, to ensure their safe and effective integration into real-world educational settings, they must demonstrate proficiency in specific, granular areas of knowledge. Existing cloze-style benchmarks, commonly used to evaluate LMs’ knowledge, have three major limitations. They: 1) do not cover the educational domain; 2) typically focus on low-complexity, generic knowledge or broad domains, which do not adequately assess the models’ knowledge in specific subjects; and 3) often rely on templates that can bias model predictions. Here, we introduce MALAMUTE, a multilingual, template-free, and highly granular probing dataset comprising expert-written, peer-reviewed probes from 71 university-level textbooks across three languages (English, Spanish, and Polish). MALAMUTE is the first education-based cloze-style dataset. It covers eight domains, each with up to 14 subdomains, further broken down into concepts and concept-based prompts, totaling 33,361 university curriculum concepts and 116,887 prompts. MALAMUTE’s fine granularity, educational focus, and inclusion of both sentence-level and paragraph-level prompts make it an ideal tool for evaluating LMs’ course-related knowledge. Our evaluation of masked and causal LMs on MALAMUTE shows that despite overall proficiency, they have significant gaps in knowledge when examined closely on specific subjects, hindering their safe use in classrooms and underscoring the need for further development. Code and data can be found at https://github.com/Shaier/MALAMUTE.

1 Introduction
Language models (LMs) have shown remarkable abilities in various domains (OpenAI, 2023a,b; Touvron et al., 2023). However, to ensure their safe and effective integration into real-world educational settings, they must demonstrate proficiency in specific,

Example MALAMUTE probes:
An atom is the smallest unit of [...] (Chemistry)
Treasury bills are short-term debt [...] (Finance)
[...] such as 2x – 9, is called a binomial. (Algebra)

<table>
  <tr>
    <th>3 Languages</th>
    <th>8 Domains</th>
    <th>55 Subdomains</th>
  </tr>
  <tr>
    <td>33,361 Concepts</td>
    <td>116,887 Probes</td>
    <td></td>
  </tr>
</table>

Figure 1: MALAMUTE: a highly granular cloze-style dataset for educational knowledge probing. This dataset assesses language models’ knowledge across three languages with fine-grained detail. Bold strings indicate predictable text. Example prompts shown for 3 out of the 55 subdomains.

granular areas of knowledge (Xu et al., 2020). For instance, a LM may excel in general knowledge of mathematics, but struggle with the nuances of calculus or algebra. Hence, we need precise and targeted evaluation methods for LMs’ knowledge in these critical areas.

The need for such evaluations is pressing, as LMs are increasingly being considered for use in classrooms, where their performance can have a direct and profound impact on student learning outcomes (Jošt et al., 2024; Kumar et al., 2023). Moreover, LMs are being explored for a range of educational applications, including adaptive learning (Weijers et al., 2024), intelligent tutoring systems (Nye et al., 2023), and automated grading (Xie et al., 2024b). In order to ensure that these systems are both effective and safe, we must develop evaluation methods that can accurately assess their knowledge in specific, curriculum-aligned areas.

Currently, evaluating LMs’ knowledge is a significant research focus. Two prominent evaluation methods are question answering (QA) (Shaier
et al., 2024b; Li et al., 2023a; Xie et al., 2024a) and cloze-style (CS) probing (Petroni et al., 2019; Sung et al., 2021). CS probing, in particular, has gained popularity, where models receive a prompt with a masked portion, such as “Dante was born in [MASK],” and are tasked with predicting the masked string. This approach measures models’ lower bound of knowledge (Petroni et al., 2019).

However, existing CS benchmarks have limitations. Firstly, there is a notable lack of CS datasets that are specifically designed for educational settings. Secondly, the prompts in existing datasets typically focus on low-complexity, generic knowledge or broad domains, which do not adequately capture models’ knowledge in specific subjects (Petroni et al., 2019; Keleg and Magdy, 2023; Kassner and Schütze, 2020; Kassner et al., 2020; Dhingra et al., 2022). Furthermore, most existing cloze-style probing datasets rely on expert-written templates, which have been shown to bias model predictions (Cao et al., 2021a; Petroni et al., 2019; Sung et al., 2021; Shaier et al., 2024a).

To address these limitations, we introduce MALAMUTE, a MultilinguAI, highLy grAnular, teMplate-free, edUcation-based probing daTasEt. Derived from 71 university-level textbooks across three languages (English, Spanish, and Polish), MALAMUTE covers eight domains, each with up to fourteen subdomains, broken down into individual curriculum concepts and concept-centric prompts. With 33,361 curriculum concepts and 116,887 expert-written, peer-reviewed CS prompts, MALAMUTE offers an unprecedented level of granularity and alignment with educational curricula, making it an ideal tool for evaluating LMs’ knowledge in educational contexts. See Figure 1 for examples. MALAMUTE is also the first probing dataset to include both sentence-level and paragraph-level prompts, which provides a more comprehensive evaluation of LMs’ knowledge, establishing a new higher lower-bound of knowledge for masked LMs (MLMs). Lastly, MALAMUTE also comes with a number of parallel domains, subdomains, and concepts across multiple languages.

We evaluate MALAMUTE on 8 causal LMs (CLMs) and 5 MLMs and highlight the need for granular datasets: despite many models’ overall proficiency on the dataset, they have significant gaps in knowledge when examined closely on specific subjects, hindering their safe use in classrooms and underscoring the need for further development.

2 Related Work

2.1 Models’ Parametric Knowledge

LMs have been shown to contain a substantial amount of factual knowledge within their parameters, as demonstrated by Petroni et al. (2019); Sung et al. (2021); Roberts et al. (2020). This embedded knowledge enables LMs to perform tasks requiring extensive information, like open-domain question answering, without needing external databases or knowledge sources (Roberts et al., 2020). As models’ size increase, their capacity to store and retrieve vast amounts of information increases significantly (OpenAI, 2023a,b), which also allow them to encompass a wide range of knowledge types, including factual data (Petroni et al., 2019; Shaier et al., 2023b), linguistic nuances (Zhang and Choi, 2021), cultural references (Zhou et al., 2024; Keleg and Magdy, 2023), and domain-specific information (Sung et al., 2021).

Evaluating the amount and types of knowledge within these models is crucial, as it ensures that they can be effectively and reliably applied to various tasks. This assessment helps identify gaps and biases, informing improvements in training and deployment (Shaier et al., 2023a). Moreover, understanding models’ knowledge scope is crucial for ethical considerations, such as preventing misinformation and ensuring factual language generation (Wang et al., 2023; Shaier et al., 2024d).

2.2 Knowledge Probing

Knowledge probing is an approach to assess knowledge implicitly embedded in LMs’ weights. Various methods are utilized to evaluate this parametric knowledge, including closed-book QA (Shaier et al., 2024b; Li et al., 2023a; Xie et al., 2024a) and CS prompts (Petroni et al., 2019; Sung et al., 2021). Closed-book QA tests models without access to external resources, which can further be separated into those that use multiple-choice QA, or free-form QA, while CS prompts require completion of missing segments within a given context. Evaluators may further employ template-based or template-free probes to gauge comprehension, where the latter provide a more accurate assessment of LMs knowledge (Shaier et al., 2024a).

2.3 Cloze-style Knowledge Probing

CS knowledge probing has experienced a surge in popularity in recent years, resulting in the development of over 80 distinct datasets to date, span-
ning multiple languages and domains. CS probing involves presenting models with prompts that have masked portions, such as “Dante was born in [MASK]”, and tasking them with predicting the correct completion (Petroni et al., 2019, 2020; Zhou et al., 2024; Talmor et al., 2020; Sung et al., 2021; Kassner and Schütze, 2020; Kassner et al., 2020; Dhingra et al., 2022). This method is used to measure models’ lower bound of knowledge. (Petroni et al., 2019; Jiang et al., 2020).

While existing datasets span diverse domains, such as law (Chalkidis et al., 2023), geography (Bhandari et al., 2023), biomedicine (Sung et al., 2021; Shaier et al., 2024a; Meng et al., 2022), and general knowledge (Petroni et al., 2019; Cao et al., 2021b; He et al., 2024), they have several limitations. Firstly, the prompts in existing datasets typically focus on low-complexity, generic knowledge or broad domains, which do not adequately capture models’ knowledge in specific subjects. Secondly, most existing cloze-style probing datasets rely on expert-written templates, which have been shown to bias model predictions. Furthermore, there is a notable lack of CS datasets that are specifically designed for educational settings.

2.4 Educational Knowledge Probing

As LMs are increasingly being integrated into classrooms (Kazemitabaar et al., 2024; Cao et al., 2023), it is essential to have a reliable educational knowledge-probing dataset to assess their performance, which can significantly impact student learning outcomes. However, current educational probing datasets have several limitations.

One major concern is that some datasets rely on expert-written templates (Koto et al., 2024), which can introduce bias into model predictions. Furthermore, these datasets usually have a limited scope, focusing on high-level subjects (such as American Government and U.S. history (Ciosici et al., 2021)) rather than providing the fine-grained, concept-level details that are necessary for in-depth knowledge. This is particularly problematic in education, where many concepts comprise various sub-domains that require comprehensive assessment.

Another limitation is that these datasets have a limited number of prompts, restricting their ability to comprehensively evaluate LMs. Additionally, they are not based on CS prompts, instead relying on either multiple-choice QA or closed-book settings. However, multiple-choice QA is a significantly less challenging task due to the limited number of options that can skew knowledge assessment results and can be subject to models’ biases towards lettered answer options in causal setups (Zheng et al., 2023). Closed-book settings, on the other hand, are not suitable for MLMs without additional fine-tuning or mask tokens, which can lead to grammatical issues. Moreover, evaluating free-form text generation often relies on human assessment to fully capture its correctness, as automatic metrics may not fully capture the nuances of the generated output, which can be time-consuming and subjective. Hence, there is a pressing need for a more comprehensive and nuanced CS educational knowledge-probing dataset. A summary of such probing datasets can be found in Table 1.

3 MALAMUTE

In this section, we introduce MALAMUTE, the first CS educational knowledge probing dataset. We also describe our systematic and scalable approach to generating high-quality probing prompts directly from reputable academic sources. See Table 2 for prompt examples from each of the 8 domains and 55 subdomains in MALAMUTE.

3.1 Data Source: OpenStax

OpenStax (OpenStax, 2012), a project led by Rice University, is dedicated to creating and distributing high-quality, peer-reviewed, and open-source textbooks for popular university-level general education courses. These textbooks are made available under a Creative Commons license, allowing for free use and adaptation.

The OpenStax library currently contains an impressive collection of 70 English textbooks, 11 Spanish textbooks, and 6 Polish textbooks. All textbooks can be accessed in multiple formats: print, PDF, and interactive web pages. Moreover, the library is continuously expanding, with new subjects, textbook improvements, and language translations being added regularly. This ongoing development ensures that our dataset will remain extensible and adaptable in the future.

The textbooks are structured in a hierarchical system, with eight main categories (or “domains”): Business, College Success, Computer Science, Humanities, Math, Nursing, Science, and Social Sciences. Each domain is further divided into subjects, which we refer to as “subdomains”. These subdomains roughly correspond to the content of a single general education college course. Finally, each
<table>
  <tr>
    <th>Dataset</th>
    <th>Template-free Educational Focus</th>
    <th>Cloze-style Domains</th>
    <th>Prompts</th>
    <th>Languages</th>
    <th>Domain Granularity</th>
  </tr>
  <tr>
    <td>LAMA (Petroni et al., 2019)</td>
    <td>Both</td>
    <td>No</td>
    <td>Yes</td>
    <td>1*</td>
    <td>51k</td>
    <td>1</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>IndicGLUE† (Kakwani et al., 2020)</td>
    <td>Yes</td>
    <td>No</td>
    <td>Yes</td>
    <td>1*</td>
    <td>239k</td>
    <td>11</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>mLAMA (Kassner et al., 2021)</td>
    <td>No</td>
    <td>No</td>
    <td>Yes</td>
    <td>1*</td>
    <td>855k</td>
    <td>53</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>MedLAMA (Meng et al., 2022)</td>
    <td>No</td>
    <td>No</td>
    <td>Yes</td>
    <td>1</td>
    <td>19k</td>
    <td>1</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>BioLAMA (Sung et al., 2021)</td>
    <td>No</td>
    <td>No</td>
    <td>Yes</td>
    <td>1</td>
    <td>49k</td>
    <td>1</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>LegalLAMA (Chalkidis et al., 2023)</td>
    <td>Yes</td>
    <td>No</td>
    <td>Yes</td>
    <td>1</td>
    <td>27k</td>
    <td>1</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>NumerSense (Lin et al., 2020)</td>
    <td>No</td>
    <td>No</td>
    <td>Yes</td>
    <td>8</td>
    <td>3k</td>
    <td>1</td>
    <td>Medium</td>
  </tr>
  <tr>
    <td>LEFT (Ciosici et al., 2021)</td>
    <td>Yes</td>
    <td>Yes</td>
    <td>No</td>
    <td>2</td>
    <td>1k</td>
    <td>1</td>
    <td>Medium</td>
  </tr>
  <tr>
    <td>EXAMS† (Hardalov et al., 2020)</td>
    <td>Yes</td>
    <td>Yes</td>
    <td>No</td>
    <td>3</td>
    <td>14k</td>
    <td>16</td>
    <td>High</td>
  </tr>
  <tr>
    <td>ArabicMMLU†† (Koto et al., 2024)</td>
    <td>No</td>
    <td>Yes</td>
    <td>No</td>
    <td>5</td>
    <td>15k</td>
    <td>2</td>
    <td>High</td>
  </tr>
  <tr>
    <td>MALAMUTE</td>
    <td>Yes</td>
    <td>Yes</td>
    <td>Yes</td>
    <td>8</td>
    <td>116k</td>
    <td>3</td>
    <td>Very High</td>
  </tr>
</table>

Table 1: Comparison of recent knowledge probing datasets. Low domain granularity means a dataset is not divided into domains, though it may have multiple tasks or sources. Medium granularity involves several domains without subdomains. High granularity includes domains and subdomains, while very high granularity adds a further division into concepts. * These datasets cover broad general knowledge and do not correspond to a specific domain. † multiple-choice QA.

subdomain contains numerous concepts, which are linked to specific sections within the relevant textbooks where those concepts are discussed.

3.2 Dataset Creation

3.2.1 Data Extraction

We collect MALAMUTE by first retrieving a list of textbook URLs from the OpenStax sitemap. We then drop remedial instances such as earlier editions and high-school level textbooks, as their contents are largely redundant, and compile a list of index pages. Next, we iterate over the vocabulary terms in the index pages and scrape the contents of the pages associated with the hypertext reference attributes. For each page, we save any parent object of a span with the attribute “data-type” equal to “term”, which generally corresponds to the surrounding paragraph. We exclude rare cases that result in prompts consisting of only the concept itself, primarily occurring in tables, which would be inappropriate for cloze-style prompts. The term spans we select are chosen by domain-expert authors of the textbooks and undergo multiple rounds of peer review, with educational value and informativeness as the primary criteria. This makes them ideal candidates for knowledge probing prompts.

3.2.2 Cloze-style Prompts Creation

While existing benchmark largely focus on sentence-level prompts, MALAMUTE is the first CS dataset that is composed of both sentence-level and paragraph-level prompts.

Paragraph-level Prompts We take each unique term in each paragraph and replace every occurrence of that term with the “[MASK]” string. We store the original term as a label and hide all other occurrences of the term in the same paragraph by replacing them with the “[HIDDEN]” string. This results in prompts that provide more contextual information about the term.

Sentence-level Prompts We split each paragraph into individual sentences using the NLTK library. Then, we select only the sentences that contain the term we are interested in. This results in a more challenging prompt, as it focuses on a single sentence and provides less contextual information.

3.2.3 Filtration

To ensure the quality of our prompts, we developed a comprehensive filtration methodology, which is described in detail in Appendix B. This methodology involves a combination of regular expressions and POS tagging to remove prompts that do not meet our quality standards, as determined through a thorough manual review of the prompt set.

3.3 Quality Control

To ensure the quality of our generated prompts, we conducted a thorough quality control process, which is described in detail in Appendix C. In brief, Graduate student annotators with academic backgrounds spanning MALAMUTE’s diverse subject areas, as evident from their academic transcripts, manually reviewed a representative sample of 495 English prompts. The results showed that the prompts demonstrated a high level of quality, with 97.6% of paragraph-level prompts and 99.1% of sentence-level prompts meeting grammatical standards, and 92.9% and 93.9%, respectively, effectively conveying the intended concept.
<table>
  <tr>
    <th>Domain</th>
    <th>Subdomain</th>
    <th>Prompt</th>
    <th>Concept</th>
  </tr>
  <tr>
    <td rowspan="12">Math</td>
    <td>Pre-Algebra</td>
    <td>What is the [MASK] of a number?</td>
    <td>absolute value</td>
  </tr>
  <tr>
    <td>Elementary Algebra</td>
    <td>In [MASK], we express x>3 as (3,∞).</td>
    <td>interval notation</td>
  </tr>
  <tr>
    <td>Intermediate Algebra</td>
    <td>If we reverse the x and y in the function and then solve for y, we get our [MASK].</td>
    <td>inverse function</td>
  </tr>
  <tr>
    <td>College Algebra</td>
    <td>A polynomial containing two terms, such as 2x + 9, is called a [MASK].</td>
    <td>binomial</td>
  </tr>
  <tr>
    <td>Trigonometry</td>
    <td>An [MASK] is an imaginary number.</td>
    <td>imaginary number</td>
  </tr>
  <tr>
    <td>Precalculus</td>
    <td>The [MASK] can be used to determine whether a graph represents a function.</td>
    <td>vertical line test</td>
  </tr>
  <tr>
    <td>Calculus I *</td>
    <td>If f' is decreasing over I, we say f is [MASK] over I.</td>
    <td>concave down</td>
  </tr>
  <tr>
    <td>Calculus II *</td>
    <td>The net change theorem considers integral of a [MASK].</td>
    <td>rate of change</td>
  </tr>
  <tr>
    <td>Calculus III *</td>
    <td>A [MASK] is a value of x for which f(x) = 0.</td>
    <td>orthogonal vectors</td>
  </tr>
  <tr>
    <td>Introductory Statistics</td>
    <td>A [MASK] is the number of times a value of the data occurs.</td>
    <td>frequencymode</td>
  </tr>
  <tr>
    <td>Business Statistics *</td>
    <td>The [MASK] is the most frequent value.</td>
    <td>mode</td>
  </tr>
  <tr>
    <td>Statistics *</td>
    <td>A [MASK] provides a way of portraying data that can facilitate calculating probabilities.</td>
    <td>two-way table</td>
  </tr>
  <tr>
    <td>Contemporary Math</td>
    <td>A [MASK] is any voting method that satisfies the Condorcet criterion.</td>
    <td>Condorcet method</td>
  </tr>
  <tr>
    <td rowspan="14">Science</td>
    <td>Genetics of Biology</td>
    <td>All the organisms living together within a specific area are collectively called a [MASK].</td>
    <td>population</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>There are very few living species of [MASK]: the platypus and four species of echidnas [...]</td>
    <td>monotremes</td>
  </tr>
  <tr>
    <td>Microbiology</td>
    <td>The reactions of nitrogen fixation occur in specialized cells called [MASK].</td>
    <td>heterocysts</td>
  </tr>
  <tr>
    <td>Chemistry Atoms First</td>
    <td>The number of protons in the nucleus of an atom is its [MASK].</td>
    <td>atomic number</td>
  </tr>
  <tr>
    <td>Chemistry *</td>
    <td>An [MASK] is a group of atoms that can participate in a chemical change.</td>
    <td>molecule</td>
  </tr>
  <tr>
    <td>Organic Chemistry</td>
    <td>[MASK] completed with proteins provides the physical makeup of the ribosomes.</td>
    <td>Ribosomal RNA</td>
  </tr>
  <tr>
    <td>Anatomy</td>
    <td>The psoas major and iliacus make up the [MASK].</td>
    <td>iliopsoas group</td>
  </tr>
  <tr>
    <td>Astronomy</td>
    <td>[MASK], the closest planet to the Sun [...]</td>
    <td>Mercury</td>
  </tr>
  <tr>
    <td>College Physics</td>
    <td>Physicists refer to the temperature at the end of the 19th century is called [MASK].</td>
    <td>critical physics</td>
  </tr>
  <tr>
    <td>Physics I *</td>
    <td>The boiling point of water is 100 °C for the Celsius scale, and its unit is the [MASK].</td>
    <td>Celsius</td>
  </tr>
  <tr>
    <td>Physics II *</td>
    <td>For elliptical orbits, the point of closest approach of a planet to the Sun is called the [MASK].</td>
    <td>perihelion</td>
  </tr>
  <tr>
    <td>Physics III *</td>
    <td>A special type of potential difference is known as [MASK].</td>
    <td>electromotive force</td>
  </tr>
  <tr>
    <td>Electromagnetism</td>
    <td>Electric current is the flow of [MASK] through a conductor.</td>
    <td>quantum electrodynamics</td>
  </tr>
  <tr>
    <td rowspan="5">Humanities</td>
    <td>U.S. History</td>
    <td>The end of the Civil War saw the beginning of the [MASK] era [...]</td>
    <td>Reconstruction</td>
  </tr>
  <tr>
    <td>Writing</td>
    <td>[MASK] reference to the source of information used in a writer's research.</td>
    <td>Citation</td>
  </tr>
  <tr>
    <td>Philosophy</td>
    <td>German philosopher Friedrich [MASK] famously declared that “God is dead”</td>
    <td>Nietzsche</td>
  </tr>
  <tr>
    <td>World History I</td>
    <td>Contact between [MASK] and the Sin began in the 590s and lasted for decades.</td>
    <td>Goguryeo</td>
  </tr>
  <tr>
    <td>World History II</td>
    <td>In the 1860s, the U.S. Congress passed the [MASK] Act, also known as FWA in [MASK].</td>
    <td>Landsman</td>
  </tr>
  <tr>
    <td rowspan="17">Business</td>
    <td>Introduction to Business</td>
    <td>[MASK] are the individuals or groups to whom a business has a responsibility.</td>
    <td>Stakeholders</td>
  </tr>
  <tr>
    <td>Financial Accounting</td>
    <td>The [MASK] of cost allocation assumes that the last units purchased are the first units sold.</td>
    <td>last-in, first-out method</td>
  </tr>
  <tr>
    <td>Managerial Accounting</td>
    <td>The person overview of all the accounting and financial systems is the [MASK].</td>
    <td>Chief Financial Officer</td>
  </tr>
  <tr>
    <td>Business Ethics</td>
    <td>An [MASK] manager is someone who is a code of ethics in his hiring.</td>
    <td>integrity</td>
  </tr>
  <tr>
    <td>Organizational Behavior</td>
    <td>Once acceptable behavioral criteria have been specified, a [MASK] can be done.</td>
    <td>performance audit</td>
  </tr>
  <tr>
    <td>Finance</td>
    <td>[MASK] are short-term debt instruments issued by the federal government.</td>
    <td>Treasury bills</td>
  </tr>
  <tr>
    <td>Business Law</td>
    <td>Another remedy as [MASK], which would terminate the right of a partnership to exist.</td>
    <td>dissolution</td>
  </tr>
  <tr>
    <td>Intellectual Property</td>
    <td>The [MASK] is the right to reproduce, distribute, or display a work.</td>
    <td>disfranchisement</td>
  </tr>
  <tr>
    <td>Marketing</td>
    <td>[MASK] happens when those being observed aren’t aware that they are being watched.</td>
    <td>Unobtrusive observation</td>
  </tr>
  <tr>
    <td>Management</td>
    <td>[MASK] are desired goals, objectives, or end states that individuals wish to pursue.</td>
    <td>Terminal values</td>
  </tr>
  <tr>
    <td>Economics</td>
    <td>We call a firm’s first stock sale to the public an [MASK].</td>
    <td>initial public offering</td>
  </tr>
  <tr>
    <td>Macroeconomics ◯</td>
    <td>The simplest example of a rate of return is the [MASK].</td>
    <td>interest rate</td>
  </tr>
  <tr>
    <td>Microeconomics ◯</td>
    <td>In the case of [MASK], a firm produces all of the output in a market.</td>
    <td>monopoly</td>
  </tr>
  <tr>
    <td>Entrepreneurship</td>
    <td>A [MASK] is a company that does not allow members of the investing public to own stock.</td>
    <td>privately held corporation</td>
  </tr>
  <tr>
    <td>Nursing</td>
    <td>Nutrition</td>
    <td>Any change in sodium and [MASK] can dramatically affect the cells of the brain.</td>
    <td>water balance</td>
  </tr>
  <tr>
    <td rowspan="2">Computer Science</td>
    <td>Python Programming</td>
    <td>[MASK] is the task of arranging elements in a sequence in ascending or descending order.</td>
    <td>Sorting</td>
  </tr>
  <tr>
    <td>Workplace Software</td>
    <td>The first modern popular operating system to emerge during this time was the [MASK], first developed in 2005 and later acquired by Google.</td>
    <td>Android operating system</td>
  </tr>
  <tr>
    <td rowspan="5">Social Sciences</td>
    <td>American Government</td>
    <td>The [MASK] limits the ability of the government to control or restrict religious practices.</td>
    <td>free exercise clause</td>
  </tr>
  <tr>
    <td>Anthropology</td>
    <td>Another group, called “untouchables” or [MASK] are outside the scheme of varnas.</td>
    <td>dalits</td>
  </tr>
  <tr>
    <td>Political Science</td>
    <td>The French philosophers [MASK] argued that animals were “political animals” [...]</td>
    <td>Animals</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>The behavior caused by the external stimulus is called the [MASK].</td>
    <td>conditioned response</td>
  </tr>
  <tr>
    <td>Sociology</td>
    <td>The power in an [MASK] is held by a small, elite group.</td>
    <td>oligarchy</td>
  </tr>
  <tr>
    <td rowspan="2">College Success</td>
    <td>College Success</td>
    <td>If we rely too heavily on assumptions, we may be buying into [MASK], or generalizations</td>
    <td>Stereotypes</td>
  </tr>
  <tr>
    <td>College Success Concise</td>
    <td>One particular studying technique is called [MASK], which calls for students to mix [...]</td>
    <td>interleaving</td>
  </tr>
</table>

Table 2: Examples of probes from each of the English textbooks. Books that are also in Polish are marked with ◯, where books that are also in Spanish are marked with *.

This rigorous quality control process verified that the generated prompts meet the desired standards, providing a robust foundation for evaluating the performance of LLMs.

3.4 MALAMUTE Statistics

Our methodology yields a substantial dataset of 116,887 unique prompts, covering 33,361 distinct concepts across 71 textbooks and three languages. The language distribution of the prompts is: 100,258 prompts are in English, 8,147 in Polish, and 8,482 in Spanish. Domain-level statistics are provided in Table 3.

4 Experiments

4.1 Models

We evaluate a set of five MLMs: BERT-base (Devlin et al., 2019), DistilBERT-base (Sanh et al., 2020), and SciBERT, a variant pretrained on scientific text (Beltagy et al., 2019). Additionally, we include two multilingual models, XLM-RoBERTa-large (Conneau et al., 2020) and mBERT-base (Devlin et al., 2019), which are specifically designed to handle a wide range of languages.

We also assess eight CLMs: GPT-4o, GPT-4o-mini, and GPT-4-turbo (Achiam et al., 2023), which are state-of-the-art generative models; Llama-3.1-405B-Instruct (Dubey et al., 2024), a powerful model designed for instruction-based tasks; Bloom-7B (Muenninghoff et al., 2023) and Mistral-7B-Instruct (Jiang et al., 2023), which are trained for cross-lingual tasks; and Llama-2-7B-Chat (Touvron et al., 2023) and Llama-3-8B-Instruct (Llama3), both pretrained on multilingual data, with a focus on both general and scientific domains.
<table>
  <tr>
    <th>Domain</th>
    <th>#Books</th>
    <th>#Prompts</th>
    <th>#Concepts</th>
    <th>Words (Pg.)</th>
    <th>Words (Sent.)</th>
  </tr>
  <tr>
    <td rowspan="7">EN</td>
    <td>Business</td>
    <td>14</td>
    <td>18,373</td>
    <td>7,923</td>
    <td>105.27</td>
    <td>22.84</td>
  </tr>
  <tr>
    <td>College Success</td>
    <td>2</td>
    <td>206</td>
    <td>115</td>
    <td>104.09</td>
    <td>18.60</td>
  </tr>
  <tr>
    <td>Computer Science</td>
    <td>2</td>
    <td>3,682</td>
    <td>1,447</td>
    <td>93.75</td>
    <td>18.44</td>
  </tr>
  <tr>
    <td>Humanities</td>
    <td>5</td>
    <td>18,622</td>
    <td>6,342</td>
    <td>114.47</td>
    <td>24.78</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>13</td>
    <td>7,103</td>
    <td>3,384</td>
    <td>55.17</td>
    <td>18.44</td>
  </tr>
  <tr>
    <td>Nursing</td>
    <td>1</td>
    <td>3,934</td>
    <td>1,567</td>
    <td>96.85</td>
    <td>23.32</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>13</td>
    <td>35,095</td>
    <td>15,602</td>
    <td>116.55</td>
    <td>21.87</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td>5</td>
    <td>13,696</td>
    <td>5,792</td>
    <td>139.82</td>
    <td>26.31</td>
  </tr>
  <tr>
    <td rowspan="3">PL</td>
    <td>Science</td>
    <td>3</td>
    <td>3,428</td>
    <td>1,684</td>
    <td>101.33</td>
    <td>16.60</td>
  </tr>
  <tr>
    <td>Business</td>
    <td>2</td>
    <td>1,660</td>
    <td>771</td>
    <td>115.07</td>
    <td>20.25</td>
  </tr>
  <tr>
    <td>Social Sciences</td>
    <td>1</td>
    <td>3,059</td>
    <td>1,481</td>
    <td>93.74</td>
    <td>17.19</td>
  </tr>
  <tr>
    <td rowspan="2">ES</td>
    <td>Math</td>
    <td>6</td>
    <td>3,481</td>
    <td>1,512</td>
    <td>61.14</td>
    <td>19.84</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>4</td>
    <td>5,001</td>
    <td>2,388</td>
    <td>128.64</td>
    <td>23.45</td>
  </tr>
</table>

Table 3: Statistics of MALAMUTE. Pg and Sent refer to the average #words in paragraph and sentence level.

This comprehensive evaluation includes a range of models from both the MLM and CLM categories, pretrained on general and scientific data to ensure broad coverage of linguistic and domain-specific tasks.

4.2 Evaluation Metric

MLM Evaluation: Unlike previous work, which mostly focuses on single-token mask prediction (Zhong et al., 2021; Petroni et al., 2019; Sung et al., 2021; Bouraoui et al., 2019), we adopt a more comprehensive approach by predicting multi-token entities. Following Kassner et al. (2021), we frame the task as entity ranking, but with a key difference: we do not limit predictions to specific entity types. This is because entity type information is often scarce or unreliable, and using external resources to classify entity types may introduce inaccuracies. Instead, following Shaier et al. (2024a), we allow our models to predict any entity from the dataset’s entity list. To evaluate the predictions, we use the top-k accuracy metric, as suggested by Sung et al. (2021); Shaier et al. (2024a). This metric assigns a score of 1 if the correct entity is among the top k predicted entities, and 0 otherwise. Due to the complex relationships between entities (N-to-M connections), we report accuracy at three different levels: Acc@1, Acc@5, and Acc@10.

CLM Evaluation: Assessing the accuracy of CLM-generated answers is a complex task, especially since a proportion of responses are lengthy and explanatory. As a result, the traditional exact match metric commonly used in question-answering tasks is inadequate. We therefore adopt best sub-span accuracy, following Kandpal et al. (2023); Mallen et al. (2023), which is equal to 1 if the gold label appears in the generated answer and 0 otherwise. To account for different capitalization possibilities in labels and outputs, we cast both to lowercase before scoring the outputs.

To mitigate the known sensitivity of CLMs to minor input variations (Jia and Liang, 2017; Shaier et al., 2024c), we employ three diverse prompt designs (detailed in Appendix A), in line with established CLM probing research (Taylor et al., 2022; Li et al., 2023b; Kalo, 2022; Nayak, 2023). Notably, previous studies have shown that varying prompt designs, such as providing in-context examples, can significantly enhance performance (Brown et al., 2020). To this end, one of our prompts includes five in-context examples. To maintain consistency with the MLM input, we opt for a zero-shot evaluation, where no examples or additional context are provided with the other two prompts.

Given the substantial resources required to evaluate the larger models (GPT-4 and Llama 405B), we first assess the three prompts using the smaller models on a representative sample of the data. Our results show that the in-context learning prompt performs best. We therefore evaluate the larger models on this prompt on the vast English data, and all models on the remaining languages data.

5 Results & Discussion

Due to space constraints, we provide a summary of our results in Table 4. See full presentation of models’ results in Appendix D.

5.1 Model-wise Analysis

For CLMs, Llama 3.1 405b Instruct performs best in all languages. On English sentence-level, it scores 57.5%, while GPT-4o-mini scores the
lowest at 49.5%. On paragraph-level, Llama 3.1 405b Instruct scores 67.0%. Similar results are seen in Spanish (Llama 3.1: 49.5%, GPT-4o-mini: 37.4%) and Polish (Llama 3.1: 39.6%, GPT-4o-mini: 27.0%).

For MLM, XLM-RoBERTa excels on English sentence- and paragraph-level, with top-1, top-5, and top-10 scores of 23.8%, 45.6%, and 54.2%, respectively. mBERT performs the worst, with scores of 12.5%, 33.4%, and 41.9%, respectively. XLM-RoBERTa dominates on Spanish and Polish sentence- and paragraph-level, with scores of 17.8%, 40.5%, and 49.9% on Spanish, and similar results on Polish.

While these models show promise, their educational knowledge is limited, and their performance varies greatly between languages, particularly in CLMs. This language gap is problematic in educational settings, where students from diverse linguistic backgrounds need equal access to educational resources. Many students speak languages other than English, and the significant performance disparity between languages may exacerbate existing educational inequalities.

For instance, students who speak Spanish or Polish may not receive the same level of educational support as their English-speaking peers, simply because the models are less proficient in their native languages. Notably, this gap is smaller in MLM, but it is unclear if multilingual models perform equally well across all subjects due to the larger English dataset. Ultimately, bridging this language gap is crucial to ensuring that AI-powered educational tools are inclusive, equitable, and effective for all students.

5.2 Prompt Design and Evaluation

One striking observation is that the performance of most CLMs varies significantly depending on the prompt used for evaluation. For instance, on the Spanish portion, Llama 3’s performance on Business Statistics diverges drastically across different prompt settings (see Table 8): it achieves a score of 23.0% when evaluated using paragraph-level prompts with setting 3, but only 5.0% when evaluated with prompt 1. This notable discrepancy underscores the importance of evaluating CLMs using a diverse range of prompts to gain a comprehensive understanding of their knowledge.

5.3 Sentence-level vs. Paragraph-level Performance

As discussed in Petroni et al. (2019), CS probing evaluates the lower bound of LMs’ knowledge. Notably, Shaier et al. (2024a) find that using template-free prompts instead of template-based prompts can reveal a higher lower bound of knowledge, as templates can bias model predictions.

Building upon this insight, MALAMUTE introduces a significant innovation by providing a comprehensive set of both sentence-level and paragraph-level template-free prompts. This enables a more nuanced evaluation of LMs’ knowledge in diverse contextual settings. Our experimental findings demonstrate that, across nearly all subdomains and models, paragraph-level prompts consistently yield higher scores for both CLMs and MLMs. This suggests that the provision of additional context can substantially enhance the evaluation of knowledge and masked concepts.

Furthermore, MALAMUTE’s extensive collection of parallel sentence and paragraph-level prompts allows us to draw a more definitive conclusion: LMs may possess a significantly higher level of knowledge than previously estimated. Notably, when the same concept is being masked, our results show that models exhibit a substantially higher likelihood of correct prediction when more context is provided. This finding has important implications for the development of more accurate and comprehensive knowledge evaluation frameworks for LMs.

5.4 Granularity Analysis

The MALAMUTE benchmark represents a significant milestone in probing datasets, offering an unprecedented level of granularity that enables a fine-grained examination of LMs’ strengths and weaknesses. This level of detail is crucial for developing effective educational systems that interact with students across diverse levels and domains.

For instance, our analysis of CLMs reveals that the top-performing model, Llama 3.1 405b Instruct, excels in subjects like Concepts of Biology, American Government, and Anatomy with scores approaching 80% (see Table 5). However, it struggles with subjects like Calculus 3, Marketing, and Business Law, where its scores plummet to around 50%. Similarly, MLMs demonstrate varying degrees of proficiency across subjects, with some showing significant knowledge gaps.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">English</th>
    <th colspan="2">Spanish</th>
    <th colspan="2">Polish</th>
  </tr>
  <tr>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
  </tr>
  <tr>
    <td>GPT-4-turbo</td>
    <td>55.1</td>
    <td>62.2</td>
    <td>35.8</td>
    <td>43.4</td>
    <td>22.9</td>
    <td>31.3</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>49.5</td>
    <td>59.8</td>
    <td>30.1</td>
    <td>39.6</td>
    <td>18.6</td>
    <td>28.1</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>53.4</td>
    <td>63.5</td>
    <td>34.5</td>
    <td>42.7</td>
    <td>27.7</td>
    <td>38.3</td>
  </tr>
  <tr>
    <td>Llama-3.1-405B-Instruct</td>
    <td>57.5</td>
    <td><b>67.0</b></td>
    <td>42.4</td>
    <td><b>51.4</b></td>
    <td>27.8</td>
    <td><b>39.7</b></td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>[15.4, 32.6, 40.4]</td>
    <td>[16.7, 34.6, 41.1]</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DistilBERT</td>
    <td>[13.0, 32.3, 41.8]</td>
    <td>[14.2, 35.1, 43.8]</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>XLM-RoBERTa</td>
    <td>[18.7, 40.0, 50.3]</td>
    <td>[<b>23.8</b>, 45.6, <b>54.2</b>]</td>
    <td>[12.5, 31.1, 42.7]</td>
    <td>[<b>17.8</b>, <b>40.5</b>, <b>49.9</b>]</td>
    <td>[11.2, 29.4, 38.1]</td>
    <td>[<b>29.9</b>, <b>51.7</b>, <b>59.4</b>]</td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>[9.3, 27.4, 36.8]</td>
    <td>[12.5, 33.4, 41.9]</td>
    <td>[9.1, 28.4, 38.1]</td>
    <td>[13.3, 36.3, 45.5]</td>
    <td>[4.4, 17.4, 24.2]</td>
    <td>[16.1, 37.3, 45.7]</td>
  </tr>
  <tr>
    <td>SciBERT</td>
    <td>[18.5, 40.9, 49.6]</td>
    <td>[22.6, <b>45.7</b>, 53.9]</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 4: All models’ overall accuracies (percentages) on the English, Spanish, and, Polish portions of the dataset. Results are shown as [prompt 1, prompt 2, prompt 3] for Causal LMs, and [top1, top5, top10] for Masked LMs. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each setting, language, and model type is highlighted in bold font.

The granular nature of MALAMUTE is essential because it allows us to uncover hidden patterns and biases in model performance. While models may appear successful at first glance, a closer inspection can reveal significant gaps in their knowledge. For example, SciBERT, the overall top-performing MLM, owes its success largely to its exceptional performance in scientific subjects, with average scores of around 60% in Math and Science. However, when we drill down to specific subjects like Philosophy, World History, or Anthropology, its scores drop to around 30%. Furthermore, despite its strong performance in certain science domains, SciBERT’s knowledge in Astronomy is found to be lacking.

The insights provided by MALAMUTE underscore the critical need for more targeted and fine-grained evaluation and development of LMs. By recognizing the specific areas where models excel and struggle, we can focus our efforts on creating more effective, well-rounded models that can support students across diverse educational domains.

5.5 MLM vs. CLM Performance

Notably, many MLMs outperform many CLMs across all languages and subjects, which is a surprising finding given the significant size difference between the two types of models. Specifically, the smallest CLM (with 7B parameters) is more than 10 times larger than the largest MLM (XLM-RoBERTa). However, it is essential to acknowledge that this comparison is not apples-to-apples, as the two types of models are evaluated differently. MLMs are evaluated on their ability to predict masked tokens, which has a relatively limited number of possible options, whereas CLMs are tasked with generating free-form text, which has an exponentially larger solution space.

Furthermore, the entity ranking evaluation method used for MLMs is simpler and more straightforward compared to the sub-span accuracy method used for CLMs. This raises important questions: (1) Can we directly compare the knowledge captured by MLMs and CLMs, given their fundamentally different evaluation criteria? and (2) What alternative evaluation methods, yet to be explored, could provide a more accurate assessment of each model type’s knowledge? We present these findings and questions for the reader’s consideration, acknowledging the complexity and nuance of comparing these two types of LMs.

6 Conclusion

We introduce MALAMUTE, a highly-granular, multilingual, and template-free educational probing dataset that exposes significant knowledge gaps in existing LMs. As LMs are increasingly considered for classroom use, where their performance can directly impact student outcomes, it is essential to develop evaluation methods that assess knowledge in specific, curriculum-aligned areas. MALAMUTE highlights the need for more specialized benchmarks that go beyond general knowledge and ensure models can support, rather than hinder, student learning. We hope MALAMUTE inspires the development of more educational benchmarks, driving the field toward safer, more reliable LMs for educational contexts. By addressing these gaps, we can better align language models with the needs of educational systems and improve their utility in real-world applications.
Limitations

We acknowledge that student materials are subject to change over time. However, we have mitigated this issue by using OpenStax, a constantly updated and expanding library of open educational resources. With new subjects, textbook improvements, and language translations being added regularly, our dataset will remain adaptable and extensible in the future.

Ethics Statement

We developed MALAMUTE to promote the creation of safer educational systems. We believe it is essential to rigorously evaluate language models on real-world student materials before deploying them in actual educational settings, to ensure they can accurately and reliably support student learning.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. Preprint, arXiv:1903.10676.

Prabin Bhandari, Antonios Anastasopoulos, and Dieter Pfoser. 2023. Are large language models geospatially knowledgeable? In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, SIGSPATIAL ’23, New York, NY, USA. Association for Computing Machinery.

Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2019. Inducing relational knowledge from bert. Preprint, arXiv:1911.12753.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021a. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1860–1874, Online. Association for Computational Linguistics.

Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021b. Knowledgeable or educated guess? revisiting language models as knowledge bases. Preprint, arXiv:2106.09231.

Jie Cao, Rachel Dickler, Marie Grace, Jeffrey B Bush, Alessandro Roncone, Leanne M Hirshfield, Marilyn A Walker, and Martha S Palmer. 2023. Designing an ai partner for jigsaw classrooms. In Workshop on Language-Based AI Agent Interaction with Children (AIAIC’2023).

Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. 2023. LeXFiles and LegalLAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.

Manuel Ciosici, Joe Cecil, Dong-Ho Lee, Alex Hedges, Marjorie Freedman, and Ralph Weischedel. 2021. Perhaps PTLMs should go to school – a task to assess open book and closed book QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6104–6111, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427–5444, Online. Association for Computational Linguistics.

Qiyuan He, Yizhong Wang, and Wenya Wang. 2024. Can language models act as knowledge bases at scale? Preprint, arXiv:2402.14273.

Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Preprint, arXiv:1911.12543.

Gregor Jošt, Viktor Taneski, and Sašo Karakatić. 2024. The impact of large language models on programming education and student learning outcomes. Applied Sciences, 14(10):4115.

Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961, Online. Association for Computational Linguistics.

Jan-Christoph Kalo. 2022. [link].

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR.

Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online. Association for Computational Linguistics.

Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. Are pretrained language models symbolic reasoners over knowledge? Preprint, arXiv:2006.10413.

Nora Kassner and Hinrich Schütze. 2020. Negated and misprised probes for pretrained language models: Birds can talk, but cannot fly. Preprint, arXiv:1911.03343.

Majeed Kazemitaabar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–20.

Amr Keleg and Walid Magdy. 2023. DLAMA: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6245–6266, Toronto, Canada. Association for Computational Linguistics.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. 2024. Arabicmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840.

Harsh Kumar, Ilya Musabirov, Mohi Reza, Jiakai Shi, Anastasia Kuzminykh, Joseph Jay Williams, and Michael Liut. 2023. Impact of guidance and interaction strategies for llm use on learner performance and perception. arXiv preprint arXiv:2310.13712.

Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023a. Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1774–1793, Toronto, Canada. Association for Computational Linguistics.

Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Xu Sun, Lingpeng Kong, and Qi Liu. 2023b. Can language models understand physical concepts? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11843–11861, Singapore. Association for Computational Linguistics.

Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Association for Computational Linguistics.
Llama3. Introducing meta llama 3: The most capable openly available llm to date.

Linbao Luo, Trang Vu, Dinh Phung, and Reza Haf. 2023. Systematic assessment of factual knowledge in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13272–13286, Singapore. Association for Computational Linguistics.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822.

Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, and Nigel Collier. 2022. Rewire-then-probe: A contrastive recipe for probing biomedical knowledge of pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4798–4810, Dublin, Ireland. Association for Computational Linguistics.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. Preprint, arXiv:2211.01786.

Anmol Nayak. 2023. [link].

B Nye, Dillon Mee, and Mark G Core. 2023. Generative large language models for dialog-based tutoring: An early consideration of opportunities and concerns. In AIED Workshops.

OpenAI. 2023a. Chatgpt: Optimizing language models for dialogue.

OpenAI. 2023b. Gpt-4 technical report. Preprint, arXiv:2303.08774.

OpenStax. 2012. Openstax. Accessed: 2024-06-09. Originally launched in 2012. Continuously updated.

Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. Preprint, arXiv:2005.04611.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.

Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint, arXiv:1910.01108.

Sagi Shaier, Kevin Bennett, Lawrence Hunter, and Katharina Kann. 2023a. Emerging challenges in personalized medicine: Assessing demographic effects on biomedical question answering systems. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 540–550, Nusa Dua, Bali. Association for Computational Linguistics.

Sagi Shaier, Kevin Bennett, Lawrence Hunter, and Katharina von der Wense. 2024a. Comparing template-based and template-free language model probing. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 766–776, St. Julian’s, Malta. Association for Computational Linguistics.

Sagi Shaier, Lawrence Hunter, and Katharina Kann. 2023b. Who are all the stochastic parrots imitating? they should tell us! In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 113–120, Nusa Dua, Bali. Association for Computational Linguistics.

Sagi Shaier, Lawrence Hunter, and Katharina von der Wense. 2024b. Desiderata for the context use of question answering systems. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 777–792, St. Julian’s, Malta. Association for Computational Linguistics.

Sagi Shaier, Lawrence Hunter, and Katharina Wense. 2024c. It is not about what you say, it is about how you say it: A surprisingly simple approach for improving reading comprehension. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8292–8305, Bangkok, Thailand. Association for Computational Linguistics.

Sagi Shaier, Ari Kobren, and Philip V. Ogren. 2024d. Adaptive question answering: Enhancing language model proficiency for addressing knowledge conflicts with source citations. In Proceedings of the
2024 Conference on Empirical Methods in Natural Language Processing, pages 17226–17239, Miami, Florida, USA. Association for Computational Linguistics.

Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4723–4734, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. olimpics – on what language model pre-training captures. Preprint, arXiv:1912.13283.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. Preprint, arXiv:2211.09085.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. Preprint, arXiv:2310.07521.

Ruben Weijers, Gabrielle Fidelis de Castilho, Jean-François Godbout, Reihaneh Rabbany, and Kellin Pelrine. 2024. Quantifying learning-style adaptation in effectiveness of LLM teaching. In Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024), pages 112–118, St. Julians, Malta. Association for Computational Linguistics.

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024a. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. Preprint, arXiv:2305.13300.

Wenjing Xie, Juxin Niu, Chun Jason Xue, and Nan Guan. 2024b. Grade like a human: Rethinking automated assessment with large language models. arXiv preprint arXiv:2405.19694.

Dongfang Xu, Peter Jansen, Jaycie Martin, Zhengnan Xie, Vikas Yadav, Harish Tayyar Madabushi, Oyvind Tafjord, and Peter Clark. 2020. Multi-class hierarchical question classification for multiple choice science exams. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5370–5382, Marseille, France. European Language Resources Association.

Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations.

Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Computational Linguistics.

Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, and Daniel Hershcovich. 2024. Does mapo tofu contain coffee? probing llms for food-related cultural knowledge. Preprint, arXiv:2404.06833.
A  Prompts for Causal LMs

We evaluate the performance of our CLMs on the MALAMUTE dataset using each of the following three prompts. We represent each masked prompt as \( p_{ti} \).

A.1 Prompt 1
The following prompt is based on Taylor et al. (2022); Li et al. (2023b)’s work

\[
p_{ti}
\]

A.2 Prompt 2
The following prompt is based on Luo et al. (2023)’s work

Please predict the missing words to complete the following sentence based on the facts in the real-world. The missing words are represented by [MASK]. Please return the missing words only. \( p_{ti} \)

A.3 Prompt 3
We drew inspiration from the work of Nayak (2023) and created a modified prompt. Specifically, we augmented the original prompt by incorporating five in-context examples, which are designed to provide additional context and guidance.

You are a helpful, respectful and honest assistant. Your answers should be crisp, short and not repetitive. Here are 5 examples: Example 1: Prompt: Francesco Bartolomeo Conti was born in [MASK]. Answer: Florence. Example 2: Prompt: English bulldog is a subclass of [MASK]. Answer: dog. Example 3: Prompt: The official language of Mauritius is [MASK]. Answer: English. Example 4: Prompt: Nicotine binds to [MASK]. Answer: CHRNA4. Example 5: Prompt: Hepatitis has symptoms such as [MASK]. Answer: Abdominal pain. Prompt: \( p_{ti} \)

B  Filtration Methodology

To ensure the quality of our prompts, we have developed a comprehensive filtration methodology that involves a combination of regular expressions and POS tagging. This methodology enables us to remove prompts that do not meet our quality standards, as determined through a thorough manual review of the prompt set.

Our filtration methodology begins with the application of regular expressions to remove prompts that exhibit commonly seen issues. Specifically, we remove prompts that start with the following words: "this", "that", "and", "those", "these", "such", "we", "also", "it", "for example", "even", "they", "here", "another", "hence", "therefore", "neither", and "either". We also remove prompts that have digits as labels, as well as those that contain fewer than five words. Additionally, we eliminate prompts that refer to visual aids, including "graphs", "images", "figures", "plots", "example", "diagram", "chart", "photograph", and "photo".

Furthermore, we remove prompts that start with or contain action words, including: "explain", "describe", "analyze", "discuss", "interpret", "summarize", "define", "identify", "clarify", "elucidate", "examine", "determine", "decide", "solve", "calculate", "evaluate", "assess", "judge", "compare", "contrast", "distinguish", "differentiate", "write", "create", "design", "develop", "formulate", "compose", "draw", "sketch", "graph", "plot", "take", "find", "state", "prove", "illustrate", "represent", "model", "simulate", "demonstrate", "show", "indicate", "point", "outline", "develop", "expand", "elaborate", "justify", "argue", "debate", "review", "revise", "edit", "redraft", "reorganize", "rephrase", "paraphrase", "translate", "transcribe", "record", "document", "report", "present", "display", "exhibit", "construct", "build", "make", "produce", "generate", "develop", "plan", "organize", "schedule", "coordinate", "arrange", "implement", "execute", "enact", "establish", "institute", "initiate", "begin", "start", "continue", "resume", "complete", "finish", and "conclude".

We also use regular expressions to remove parenthesized text that appears immediately after or at the end of a label, such as "(AC)" in "audit committee (AC)". However, we apply this step with caution, ensuring that the parenthesized text is separated from the label by a white space character. This constraint is necessary to avoid damaging prompts in subjects like mathematics and computer science, where parentheses are frequently used to denote other concepts, such as mathematical notation (e.g., "f(x)") or programming language syntax (e.g., "upper()").

Our comprehensive filtration methodology enables us to remove low-quality prompts and ensure that our dataset meets the highest standards of quality and accuracy.
C Quality Control Methodology

To ensure the high quality of the generated prompts, we conducted a rigorous quality control process. This process involved manually reviewing a representative sample of 495 English prompts, consisting of 9 prompts from each of the 55 subdomains. The sample included a mix of sentence-level and paragraph-level prompts, with a ratio of 2:1, to provide a comprehensive representation of our dataset.

Three graduate student annotators, each holding an undergraduate degree and possessing in-depth knowledge of MALAMUTE’s diverse subject areas, evaluated the prompts based on a standardized set of criteria. Their knowledge and expertise in these areas are evident from their academic transcripts, which demonstrate a strong foundation in the subject matter. This evaluation was designed to assess two key aspects of the prompts: (1) grammatical correctness, and (2) the effective description of concept properties, which enables the inference of the label from the context.

The results of our quality control process revealed an impressive level of quality in the generated prompts. Among the paragraph-level prompts, a substantial 97.6% were found to be grammatically correct, while a significant 92.9% effectively conveyed the intended concept. Similarly, the sentence-level prompts demonstrated a high level of quality, with 99.1% meeting grammatical standards and 93.9% successfully demonstrating the concept. It is important to note that in many probing datasets, entities are often related to numerous other entities (N-to-M connections), which can result in multiple possible answers (Sung et al., 2021; Shaier et al., 2024a). Our results indicate that while there is some variation in quality, the overwhelming majority of prompts are suitable for knowledge probing applications.

It is worth noting that our approach to selecting annotators was deliberate. We chose to select annotators who were non-experts in any particular field of study, rather than domain-specific experts. This decision was motivated by two key considerations. Firstly, given the highly granular nature of MALAMUTE’s knowledge, it would be impractical to find and engage a large number of annotators with expertise in each of the domains, subdomains, and concepts. Secondly, and more importantly, LLMs are trained on general data and are, in that sense, also general and "non-experts." Therefore, it is more suitable to evaluate the performance of models against humans who possess general knowledge, but also have a focus on MALAMUTE’s highly granular knowledge in their studies.

By conducting this rigorous quality control process, we were able to verify that the generated prompts meet the desired standards of quality, and are suitable for knowledge probing applications. While there may be some variation in quality, the overwhelming majority of prompts demonstrate a high level of quality, providing a robust foundation for evaluating the performance of LLMs.

C.1 Annotation Prompts

For each prompt, annotators were instructed to evaluate two aspects in response to the following prompt: “For each prompt evaluate whether: The prompt is grammatically correct; The label is specific to the prompt, i.e. the label can be inferred from the context and there aren’t many possible alternative answers. Use web resources such as Google search, Wikipedia, etc., as needed.” Annotators were encouraged to use their discretion and consider all relevant information when making their judgments. Importantly, there were no time constraints imposed on the annotation task, allowing annotators to take as much time as necessary to provide accurate assessments.

D Additional Results

The complete results of our evaluation are presented in the following tables. Table 5 shows the performance of CLMs on English data. Table 6 presents the results of MLMs evaluated on English at the sentence level, while Table 7 provides their performance at the paragraph level. For Spanish, Tables 8 and 10 summarize the results of CLMs, and Tables 12 details the MLM evaluations. Similarly, for Polish, Tables 11 and 9 report the performance of CLMs, and Table 13 presents the results of MLMs.

E Error Analysis

E.1 Inter-annotator Agreement

To ensure the reliability and consistency of our dataset, we conducted a thorough analysis of inter-annotator agreement. For that, we evaluated the extent to which our three annotators concurred on the two key metrics of interest: specificity and grammaticality of the prompts. More specifically, we report 3-way unanimous percent agreement\(^1\) on

\(^1\)https://shorturl.at/em3FD
<table>
  <tr>
    <th rowspan="2">Subdomain</th>
    <th colspan="4">GPT-4-Turbo</th>
    <th colspan="4">Causal LMs</th>
    <th colspan="4">Llama 3.1 405b Instruct</th>
  </tr>
  <tr>
    <th>S</th><th>P</th><th>S</th><th>P</th><th>S</th><th>P</th><th>S</th><th>P</th><th>S</th><th>P</th><th>S</th><th>P</th>
  </tr>
  <tr>
    <td>Prealgebra</td><td>59.4</td><td>61.9</td><td>59.4</td><td>59.6</td><td>59.0</td><td>63.5</td><td>64.1</td><td>70.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Elementary Algebra</td><td>48.5</td><td>53.0</td><td>49.1</td><td>54.3</td><td>45.5</td><td>55.3</td><td>55.7</td><td>61.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Intermediate Algebra</td><td>49.1</td><td>55.4</td><td>46.0</td><td>54.4</td><td>50.1</td><td>52.9</td><td>54.1</td><td>60.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>College Algebra</td><td>51.0</td><td>53.3</td><td>49.7</td><td>58.8</td><td>53.1</td><td>58.8</td><td>58.6</td><td>63.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Trigonometry</td><td>49.8</td><td>55.9</td><td>49.5</td><td>56.2</td><td>51.6</td><td>61.3</td><td>56.3</td><td>64.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Precalculus *</td><td>46.4</td><td>49.9</td><td>44.9</td><td>49.8</td><td>46.6</td><td>52.2</td><td>54.9</td><td>59.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Calculus I *</td><td>47.3</td><td>51.6</td><td>39.3</td><td>50.0</td><td>48.2</td><td>49.5</td><td>56.2</td><td>58.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Calculus II *</td><td>46.2</td><td>51.0</td><td>41.9</td><td>45.2</td><td>45.2</td><td>50.2</td><td>48.1</td><td>53.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Calculus III *</td><td>36.6</td><td>39.1</td><td>37.8</td><td>37.7</td><td>37.8</td><td>43.9</td><td>45.1</td><td>49.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Intro. Statistics</td><td>41.9</td><td>59.1</td><td>41.9</td><td>60.6</td><td>43.5</td><td>56.2</td><td>53.2</td><td>68.6</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Business Statistics *</td><td>44.1</td><td>59.2</td><td>45.4</td><td>57.8</td><td>46.1</td><td>59.7</td><td>52.0</td><td>64.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Statistics</td><td>41.2</td><td>54.6</td><td>42.5</td><td>55.3</td><td>41.2</td><td>53.3</td><td>55.0</td><td>61.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Contemporary Math</td><td>51.8</td><td>62.0</td><td>50.0</td><td>62.3</td><td>49.6</td><td>63.0</td><td>56.7</td><td>65.0</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Math Avg.</td><td>47.2</td><td>54.3</td><td>46.0</td><td>54.0</td><td>47.5</td><td>55.4</td><td>54.6</td><td>61.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Concepts of Biology</td><td>71.2</td><td>79.9</td><td>67.0</td><td>78.7</td><td>72.7</td><td>82.4</td><td>73.9</td><td>83.7</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Biology</td><td>66.0</td><td>74.2</td><td>61.3</td><td>74.1</td><td>68.5</td><td>76.6</td><td>68.1</td><td>78.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Microbiology</td><td>56.0</td><td>62.6</td><td>50.4</td><td>60.4</td><td>56.0</td><td>64.8</td><td>58.1</td><td>68.1</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Chemistry Atoms First</td><td>64.4</td><td>68.7</td><td>61.0</td><td>69.3</td><td>66.0</td><td>72.9</td><td>68.8</td><td>77.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Chemistry *</td><td>64.3</td><td>73.3</td><td>50.0</td><td>73.3</td><td>64.3</td><td>75.6</td><td>67.9</td><td>77.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Organic Chemistry</td><td>61.2</td><td>64.9</td><td>56.9</td><td>65.3</td><td>65.4</td><td>69.6</td><td>65.6</td><td>72.0</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Anatomy</td><td>62.4</td><td>72.5</td><td>58.0</td><td>72.9</td><td>66.1</td><td>77.8</td><td>65.7</td><td>79.0</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Astronomy</td><td>66.3</td><td>73.5</td><td>57.6</td><td>67.5</td><td>63.9</td><td>72.5</td><td>66.9</td><td>76.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>College Physics</td><td>57.2</td><td>62.6</td><td>51.5</td><td>62.6</td><td>55.6</td><td>63.5</td><td>59.0</td><td>66.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics</td><td>62.5</td><td>67.6</td><td>60.3</td><td>68.3</td><td>61.9</td><td>69.2</td><td>66.2</td><td>72.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics I o *</td><td>40.1</td><td>49.2</td><td>37.2</td><td>47.0</td><td>38.9</td><td>49.1</td><td>43.0</td><td>53.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics II o *</td><td>49.9</td><td>58.4</td><td>43.5</td><td>56.7</td><td>50.4</td><td>56.4</td><td>54.2</td><td>61.6</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics III o *</td><td>48.4</td><td>53.3</td><td>43.9</td><td>52.0</td><td>49.5</td><td>55.6</td><td>57.3</td><td>60.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Science Avg.</td><td>59.2</td><td>66.2</td><td>53.7</td><td>65.2</td><td>59.9</td><td>68.2</td><td>62.7</td><td>71.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>U.S. History</td><td>65.8</td><td>74.4</td><td>57.0</td><td>72.2</td><td>64.8</td><td>74.8</td><td>68.3</td><td>77.0</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Writing</td><td>46.8</td><td>56.6</td><td>41.1</td><td>51.3</td><td>49.0</td><td>58.6</td><td>48.4</td><td>58.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Philosophy</td><td>59.6</td><td>53.8</td><td>50.8</td><td>64.3</td><td>57.4</td><td>69.3</td><td>59.9</td><td>72.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>World History I</td><td>58.7</td><td>64.6</td><td>48.2</td><td>61.7</td><td>57.8</td><td>64.7</td><td>59.5</td><td>67.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>World History II</td><td>65.7</td><td>70.2</td><td>58.5</td><td>67.2</td><td>64.7</td><td>70.3</td><td>66.6</td><td>72.9</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Humanities Avg.</td><td>59.3</td><td>66.8</td><td>51.3</td><td>63.1</td><td>58.7</td><td>67.5</td><td>60.5</td><td>69.7</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Intro. to Business</td><td>43.8</td><td>52.2</td><td>36.2</td><td>48.8</td><td>42.7</td><td>52.2</td><td>45.1</td><td>56.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Financial Accounting</td><td>47.4</td><td>54.0</td><td>44.7</td><td>57.5</td><td>48.4</td><td>56.3</td><td>50.8</td><td>62.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Managerial Accounting</td><td>40.9</td><td>50.1</td><td>37.4</td><td>50.4</td><td>42.9</td><td>52.2</td><td>43.1</td><td>59.0</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Business Ethics</td><td>51.9</td><td>63.1</td><td>45.0</td><td>57.5</td><td>50.4</td><td>57.1</td><td>54.7</td><td>66.4</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Og. Behavior</td><td>31.2</td><td>47.8</td><td>27.7</td><td>42.5</td><td>30.7</td><td>47.8</td><td>34.7</td><td>51.6</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Finance</td><td>44.3</td><td>51.0</td><td>39.9</td><td>49.5</td><td>43.8</td><td>50.7</td><td>48.5</td><td>58.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Business Law</td><td>41.7</td><td>51.3</td><td>41.7</td><td>52.5</td><td>43.8</td><td>52.8</td><td>48.2</td><td>57.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Intellectual Property</td><td>47.2</td><td>53.8</td><td>42.7</td><td>48.7</td><td>44.9</td><td>52.1</td><td>44.9</td><td>55.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Marketing</td><td>38.9</td><td>46.8</td><td>33.7</td><td>43.4</td><td>38.5</td><td>47.2</td><td>39.8</td><td>48.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Management</td><td>41.4</td><td>53.4</td><td>40.1</td><td>53.0</td><td>41.4</td><td>53.0</td><td>41.9</td><td>57.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Economics</td><td>46.8</td><td>58.2</td><td>45.0</td><td>57.6</td><td>47.7</td><td>58.8</td><td>53.0</td><td>66.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Macroeconomics o</td><td>56.9</td><td>63.1</td><td>54.3</td><td>64.0</td><td>57.8</td><td>64.7</td><td>61.2</td><td>70.4</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Microeconomics o</td><td>59.7</td><td>69.6</td><td>61.3</td><td>73.4</td><td>62.9</td><td>70.9</td><td>71.0</td><td>78.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Entrepreneurship</td><td>46.4</td><td>54.5</td><td>39.0</td><td>48.3</td><td>46.7</td><td>54.2</td><td>48.4</td><td>57.9</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Business Avg.</td><td>45.6</td><td>54.9</td><td>42.1</td><td>53.4</td><td>45.9</td><td>55.0</td><td>48.8</td><td>60.7</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Nutrition</td><td>43.0</td><td>48.0</td><td>39.0</td><td>46.4</td><td>42.1</td><td>48.3</td><td>43.7</td><td>52.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Nursing Avg.</td><td>43.0</td><td>48.0</td><td>39.0</td><td>46.4</td><td>42.1</td><td>48.3</td><td>43.7</td><td>52.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Python Programming</td><td>46.0</td><td>55.2</td><td>46.8</td><td>56.5</td><td>31.0</td><td>55.2</td><td>46.1</td><td>61.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Workshop Software</td><td>37.4</td><td>52.9</td><td>34.4</td><td>49.9</td><td>38.1</td><td>54.6</td><td>39.6</td><td>59.7</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Computer Science Avg.</td><td>41.7</td><td>54.0</td><td>40.6</td><td>53.2</td><td>44.6</td><td>54.9</td><td>44.8</td><td>60.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>American Government</td><td>66.0</td><td>76.1</td><td>61.0</td><td>72.7</td><td>64.9</td><td>75.8</td><td>70.6</td><td>80.2</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Anthropology</td><td>55.4</td><td>59.7</td><td>44.3</td><td>52.1</td><td>55.9</td><td>61.0</td><td>57.8</td><td>65.5</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Political Science</td><td>58.6</td><td>64.6</td><td>52.0</td><td>59.6</td><td>57.5</td><td>64.9</td><td>59.8</td><td>67.7</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Psychology o</td><td>66.4</td><td>73.7</td><td>62.9</td><td>74.6</td><td>68.4</td><td>78.1</td><td>67.2</td><td>76.0</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Sociology</td><td>58.4</td><td>68.9</td><td>49.2</td><td>66.1</td><td>60.1</td><td>71.3</td><td>59.9</td><td>73.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Social Sciences Avg.</td><td>61.0</td><td>68.6</td><td>53.9</td><td>65.0</td><td>61.4</td><td>70.2</td><td>63.1</td><td>72.9</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>College Success</td><td>46.5</td><td>60.9</td><td>37.2</td><td>65.6</td><td>44.2</td><td>62.5</td><td>41.9</td><td>57.8</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Col. Suc. Concise</td><td>40.4</td><td>53.8</td><td>40.4</td><td>61.5</td><td>42.6</td><td>51.9</td><td>44.7</td><td>67.3</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>College Success Avg.</td><td>43.4</td><td>57.4</td><td>38.8</td><td>63.6</td><td>43.4</td><td>57.2</td><td>43.3</td><td>62.6</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Overall</td><td>51.6</td><td>59.7</td><td>47.4</td><td>58.5</td><td>52.0</td><td>60.7</td><td>55.4</td><td>65.3</td><td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 5: Causal models’ accuracies (percentages) on the English portion of the dataset. Books that are also in Polish are marked with ◦, where books that are also in Spanish are marked with ★, and their results can be seen in Tables T1, T2. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.
<table>
  <tr>
    <th>Subdomain</th>
    <th>BERT</th>
    <th>DistilBERT</th>
    <th>XLM-RoBERTa</th>
    <th>mBERT</th>
    <th>SciBERT</th>
  </tr>
  <tr>
    <td>Prealgebra</td>
    <td>135.2, 37.6, 58.1 | [14.7, 36.4, 47.9]</td>
    <td>[19.4, 37.8, 48.5]</td>
    <td>[18.5, 28.5, 36.9]</td>
    <td>23.5, 48.4, 57.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Elementary Algebra</td>
    <td>[10.3, 35.8, 49.7]</td>
    <td>[10.3, 35.8, 49.7]</td>
    <td>[10.3, 35.8, 49.7]</td>
    <td>[10.3, 35.8, 49.7]</td>
    <td>[10.3, 35.8, 49.7]</td>
  </tr>
  <tr>
    <td>Intermediate Algebra</td>
    <td>[10.4, 27.6, 37.4]</td>
    <td>[12.3, 31.3, 42.9]</td>
    <td>[13.5, 30.7, 39.3]</td>
    <td>[16.1, 28.8, 36.2]</td>
    <td>[14.7, 41.7, 50.3]</td>
  </tr>
  <tr>
    <td>College Algebra</td>
    <td>[9.7, 33.1, 42.1]</td>
    <td>[9.7, 36.6, 47.6]</td>
    <td>[7.6, 31.7, 42.8]</td>
    <td>[4.1, 23.4, 30.3]</td>
    <td>[20.0, 53.8, 63.4]</td>
  </tr>
  <tr>
    <td>Trigonometry</td>
    <td>[10.1, 28.5, 34.3]</td>
    <td>[9.7, 29.6, 36.3]</td>
    <td>[11.2, 27.8, 37.9]</td>
    <td>[4.3, 18.4, 27.4]</td>
    <td>[20.2, 41.5, 52.0]</td>
  </tr>
  <tr>
    <td>PreCalculus +</td>
    <td>[6.3, 27.8, 40.0]</td>
    <td>[9.4, 31.2, 45.1]</td>
    <td>[10.4, 31.2, 45.1]</td>
    <td>[10.4, 31.2, 45.1]</td>
    <td>[10.4, 31.2, 45.1]</td>
  </tr>
  <tr>
    <td>Calculus I</td>
    <td>[10.6, 30.0, 36.2]</td>
    <td>[11.9, 35.0, 43.8]</td>
    <td>[13.8, 35.0, 44.4]</td>
    <td>[10.0, 28.7, 39.4]</td>
    <td>[12.8, 45.0, 54.4]</td>
  </tr>
  <tr>
    <td>Calculus II *</td>
    <td>[13.4, 36.6, 47.0]</td>
    <td>[14.8, 38.4, 45.7]</td>
    <td>[12.8, 36.6, 47.6]</td>
    <td>[9.1, 28.7, 37.3]</td>
    <td>[23.2, 50.0, 61.0]</td>
  </tr>
  <tr>
    <td>Calculus III *</td>
    <td>[16.1, 40.3, 50.8]</td>
    <td>[16.1, 40.3, 50.8]</td>
    <td>[16.1, 40.3, 50.8]</td>
    <td>[16.1, 40.3, 50.8]</td>
    <td>[16.1, 40.3, 50.8]</td>
  </tr>
  <tr>
    <td>Intro. Statistics</td>
    <td>[16.2, 41.8, 52.1]</td>
    <td>[16.2, 43.8, 55.0]</td>
    <td>[16.2, 43.8, 55.0]</td>
    <td>[10.0, 38.8, 48.8]</td>
    <td>[28.7, 70.0, 80.0]</td>
  </tr>
  <tr>
    <td>Business Statistics *</td>
    <td>[18.7, 44.7, 56.0]</td>
    <td>[21.1, 49.2, 62.0]</td>
    <td>[18.3, 47.5, 62.0]</td>
    <td>[9.5, 35.9, 45.4]</td>
    <td>[25.4, 54.6, 66.9]</td>
  </tr>
  <tr>
    <td>Contemporary Math</td>
    <td>[11.5, 35.2, 44.1]</td>
    <td>[11.7, 37.5, 48.6]</td>
    <td>[14.4, 36.0, 48.6]</td>
    <td>[8.1, 28.5, 39.7]</td>
    <td>[23.7, 52.8, 62.0]</td>
  </tr>
  <tr>
    <td>Concepts of Biology</td>
    <td>[11.4, 22.3, 36.6]</td>
    <td>[11.4, 22.3, 36.6]</td>
    <td>[11.4, 22.3, 36.6]</td>
    <td>[6.7, 23.4, 32.7]</td>
    <td>[26.9, 53.4, 64.8]</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>[11.4, 22.3, 36.6]</td>
    <td>[11.4, 22.3, 36.6]</td>
    <td>[11.4, 22.3, 36.6]</td>
    <td>[6.7, 23.4, 32.7]</td>
    <td>[26.9, 53.4, 64.8]</td>
  </tr>
  <tr>
    <td>Microbiology</td>
    <td>[13.8, 8.4, 11.0]</td>
    <td>[13.3, 10.1, 14.2]</td>
    <td>[5.8, 16.5, 22.7]</td>
    <td>[2.2, 11.8, 17.5]</td>
    <td>[14.4, 32.8, 41.7]</td>
  </tr>
  <tr>
    <td>Chemistry Atoms First</td>
    <td>[14.1, 31.6, 39.0]</td>
    <td>[10.7, 32.4, 41.1]</td>
    <td>[14.5, 38.9, 49.5]</td>
    <td>[9.2, 27.5, 38.1]</td>
    <td>[20.3, 48.1, 56.9]</td>
  </tr>
  <tr>
    <td>Chemistry *</td>
    <td>[32.1, 64.3, 67.9]</td>
    <td>[25.8, 57.1, 82.1]</td>
    <td>[32.1, 78.6, 96.4]</td>
    <td>[21.4, 64.3, 92.9]</td>
    <td>[35.7, 71.4, 89.3]</td>
  </tr>
  <tr>
    <td>Organic Chemistry</td>
    <td>[12.2, 28.6, 36.5]</td>
    <td>[10.5, 28.6, 36.5]</td>
    <td>[12.2, 36.2, 42.9]</td>
    <td>[8.5, 29.9, 33.7]</td>
    <td>[15.4, 48.0, 57.1]</td>
  </tr>
  <tr>
    <td>Astronomy</td>
    <td>[12.7, 21.3, 31.5]</td>
    <td>[12.7, 21.3, 31.5]</td>
    <td>[12.7, 21.3, 31.5]</td>
    <td>[12.7, 21.3, 31.5]</td>
    <td>[12.7, 21.3, 31.5]</td>
  </tr>
  <tr>
    <td>Astronomy</td>
    <td>[12.0, 27.4, 33.1]</td>
    <td>[7.2, 22.4, 29.2]</td>
    <td>[10.7, 27.7, 36.6]</td>
    <td>[10.5, 25.3, 32.8]</td>
    <td>[10.3, 23.0, 30.3]</td>
  </tr>
  <tr>
    <td>College Physics</td>
    <td>[11.4, 26.5, 33.5]</td>
    <td>[10.7, 30.1, 39.3]</td>
    <td>[12.8, 33.1, 41.1]</td>
    <td>[6.3, 23.8, 31.7]</td>
    <td>[19.9, 46.6, 57.0]</td>
  </tr>
  <tr>
    <td>Physics I</td>
    <td>[18.3, 35.6, 46.8]</td>
    <td>[18.3, 35.6, 46.8]</td>
    <td>[18.3, 35.6, 46.8]</td>
    <td>[18.3, 35.6, 46.8]</td>
    <td>[18.3, 35.6, 46.8]</td>
  </tr>
  <tr>
    <td>Physics I o *</td>
    <td>[13.7, 26.3, 34.9]</td>
    <td>[12.4, 29.6, 35.4]</td>
    <td>[18.7, 35.2, 45.6]</td>
    <td>[10.4, 28.4, 35.9]</td>
    <td>[21.3, 45.3, 51.9]</td>
  </tr>
  <tr>
    <td>Physics II o *</td>
    <td>[12.9, 30.3, 38.7]</td>
    <td>[11.6, 32.4, 40.8]</td>
    <td>[15.7, 36.9, 44.6]</td>
    <td>[10.8, 30.1, 39.9]</td>
    <td>[12.6, 47.4, 54.5]</td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>[12.9, 27.0, 32.6]</td>
    <td>[10.9, 28.3, 37.0]</td>
    <td>[14.6, 34.8, 43.3]</td>
    <td>[8.6, 27.1, 36.1]</td>
    <td>[20.9, 45.6, 54.0]</td>
  </tr>
  <tr>
    <td>U.S. History</td>
    <td>[16.3, 29.2, 34.6]</td>
    <td>[15.9, 29.4, 35.2]</td>
    <td>[18.3, 32.1, 39.5]</td>
    <td>[14.0, 25.9, 32.4]</td>
    <td>[18.4, 15.2, 21.6]</td>
  </tr>
  <tr>
    <td>Writing</td>
    <td>[17.6, 27.9, 33.1]</td>
    <td>[17.6, 27.9, 33.1]</td>
    <td>[17.6, 27.9, 33.1]</td>
    <td>[17.6, 27.9, 33.1]</td>
    <td>[17.6, 27.9, 33.1]</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>[17.6, 27.9, 33.1]</td>
    <td>[17.6, 27.9, 33.1]</td>
    <td>[17.6, 31.0, 39.0]</td>
    <td>[11.3, 28.1, 35.1]</td>
    <td>[19.8, 22.5, 28.7]</td>
  </tr>
  <tr>
    <td>World History 1</td>
    <td>[13.1, 24.4, 28.9]</td>
    <td>[7.6, 18.7, 24.9]</td>
    <td>[9.0, 21.0, 27.0]</td>
    <td>[7.8, 21.1, 30.4]</td>
    <td>[3.0, 8.8, 12.7]</td>
  </tr>
  <tr>
    <td>World History 2</td>
    <td>[23.6, 40.3, 47.9]</td>
    <td>[10.5, 26.3, 34.7]</td>
    <td>[20.0, 38.0, 46.3]</td>
    <td>[15.9, 35.9, 45.2]</td>
    <td>[18.8, 19.2, 24.9]</td>
  </tr>
  <tr>
    <td>Humanities Avg.</td>
    <td>[17.2, 30.1, 35.1]</td>
    <td>[11.3, 25.5, 32.8]</td>
    <td>[17.1, 32.1, 40.0]</td>
    <td>[12.0, 27.0, 34.6]</td>
    <td>[18.1, 17.8, 23.6]</td>
  </tr>
  <tr>
    <td>Intro. to Business</td>
    <td>[12.6, 28.6, 35.1]</td>
    <td>[11.9, 26.4, 34.3]</td>
    <td>[11.9, 26.4, 34.3]</td>
    <td>[11.9, 26.4, 34.3]</td>
    <td>[11.9, 26.4, 34.3]</td>
  </tr>
  <tr>
    <td>Financial Accounting</td>
    <td>[14.6, 36.3, 42.9]</td>
    <td>[11.4, 37.1, 48.3]</td>
    <td>[22.6, 50.6, 63.1]</td>
    <td>[9.1, 27.4, 36.9]</td>
    <td>[16.0, 38.0, 45.1]</td>
  </tr>
  <tr>
    <td>Managarial Accounting</td>
    <td>[21.3, 41.9, 51.9]</td>
    <td>[19.8, 42.6, 54.7]</td>
    <td>[31.4, 56.2, 66.7]</td>
    <td>[14.7, 31.8, 43.0]</td>
    <td>[21.3, 45.7, 53.1]</td>
  </tr>
  <tr>
    <td>Business Ethics</td>
    <td>[10.2, 25.9, 32.9]</td>
    <td>[9.5, 24.9, 35.2]</td>
    <td>[16.7, 38.4, 49.1]</td>
    <td>[6.2, 19.2, 23.9]</td>
    <td>[14.0, 33.1, 42.9]</td>
  </tr>
  <tr>
    <td>Finance</td>
    <td>[27.0, 39.3, 44.8]</td>
    <td>[21.8, 40.5, 46.8]</td>
    <td>[29.0, 54.6, 63.1]</td>
    <td>[13.1, 31.3, 43.7]</td>
    <td>[18.7, 38.9, 46.8]</td>
  </tr>
  <tr>
    <td>Business Law</td>
    <td>[14.6, 46.1, 64.0]</td>
    <td>[12.4, 38.2, 59.6]</td>
    <td>[21.3, 49.4, 66.3]</td>
    <td>[7.9, 32.6, 44.9]</td>
    <td>[9.0, 38.2, 57.3]</td>
  </tr>
  <tr>
    <td>Intellectual Property</td>
    <td>[9.0, 25.5, 25.2]</td>
    <td>[7.9, 20.2, 26.5]</td>
    <td>[20.2, 38.3, 44.4]</td>
    <td>[5.4, 15.5, 21.3]</td>
    <td>[19.3, 20.5, 25.9]</td>
  </tr>
  <tr>
    <td>Marketing</td>
    <td>[21.3, 41.9, 51.9]</td>
    <td>[19.8, 42.6, 54.7]</td>
    <td>[31.4, 56.2, 66.7]</td>
    <td>[14.7, 31.8, 43.0]</td>
    <td>[21.3, 45.7, 53.1]</td>
  </tr>
  <tr>
    <td>Economics</td>
    <td>[19.1, 33.1, 42.4]</td>
    <td>[32.1, 42.9]</td>
    <td>[17.2, 40.9, 50.5]</td>
    <td>[5.2, 23.9, 31.6]</td>
    <td>[15.8, 38.6, 48.5]</td>
  </tr>
  <tr>
    <td>Macroeconomics ◯</td>
    <td>[24.1, 58.6, 74.1]</td>
    <td>[18.1, 56.4, 64.7]</td>
    <td>[26.7, 59.5, 69.8]</td>
    <td>[7.8, 43.1, 55.2]</td>
    <td>[29.3, 60.3, 70.7]</td>
  </tr>
  <tr>
    <td>Microeconomics ◯</td>
    <td>[27.4, 64.5, 83.9]</td>
    <td>[24.1, 61.7, 75.8]</td>
    <td>[30.6, 66.1, 82.3]</td>
    <td>[12.9, 53.2, 64.5]</td>
    <td>[38.7, 71.0, 85.5]</td>
  </tr>
  <tr>
    <td>Entrepreneurship</td>
    <td>[10.0, 24.3, 30.1]</td>
    <td>[9.5, 24.9, 35.2]</td>
    <td>[16.7, 38.4, 49.1]</td>
    <td>[6.2, 19.2, 23.9]</td>
    <td>[14.0, 33.1, 42.9]</td>
  </tr>
  <tr>
    <td>Business Avg.</td>
    <td>[15.8, 35.3, 42.5]</td>
    <td>[13.1, 28.1, 35.1]</td>
    <td>[15.8, 35.3, 42.5]</td>
    <td>[10.8, 27.1, 34.6]</td>
    <td>[18.1, 17.8, 23.6]</td>
  </tr>
  <tr>
    <td>Nutrition</td>
    <td>[15.8, 17.5, 20.7]</td>
    <td>[16.7, 16.7, 20.7]</td>
    <td>[10.7, 27.5, 35.9]</td>
    <td>[12.0, 16.7, 19.7]</td>
    <td>[18.5, 9.8, 21.7]</td>
  </tr>
  <tr>
    <td>Nursing Avg.</td>
    <td>[9.8, 15.7, 20.1]</td>
    <td>[6.7, 16.3, 21.6]</td>
    <td>[10.7, 25.5, 35.9]</td>
    <td>[2.0, 10.6, 16.7]</td>
    <td>[18.6, 39.8, 51.3]</td>
  </tr>
  <tr>
    <td>Python Programming</td>
    <td>[17.6, 15.9, 20.1]</td>
    <td>[6.8, 17.6, 20.7]</td>
    <td>[20.1, 36.5, 46.2]</td>
    <td>[10.0, 15.3, 23.5]</td>
    <td>[16.4, 39.0, 34.0]</td>
  </tr>
  <tr>
    <td>Workplace Software</td>
    <td>[8.1, 17.2, 21.6]</td>
    <td>[6.5, 16.7, 22.0]</td>
    <td>[17.6, 36.3, 45.1]</td>
    <td>[4.2, 13.0, 18.5]</td>
    <td>[9.4, 22.8, 29.3]</td>
  </tr>
  <tr>
    <td>Computer Science Avg.</td>
    <td>[17.8, 16.6, 20.9]</td>
    <td>[6.7, 17.1, 21.4]</td>
    <td>[18.9, 36.4, 45.7]</td>
    <td>[4.1, 14.2, 21.0]</td>
    <td>[12.9, 26.9, 32.0]</td>
  </tr>
  <tr>
    <td>American Government</td>
    <td>[22.8, 33.1, 38.3]</td>
    <td>[10.4, 23.3, 31.0]</td>
    <td>[23.4, 38.4, 45.5]</td>
    <td>[15.8, 28.0, 33.4]</td>
    <td>[17.0, 21.0, 23.3]</td>
  </tr>
  <tr>
    <td>Anthropology</td>
    <td>[12.3, 23.0, 29.0]</td>
    <td>[9.5, 21.2, 28.1]</td>
    <td>[13.0, 26.0, 32.8]</td>
    <td>[8.6, 21.6, 28.0]</td>
    <td>[8.2, 19.1, 24.9]</td>
  </tr>
  <tr>
    <td>Political Science</td>
    <td>[12.3, 23.0, 29.0]</td>
    <td>[9.5, 21.2, 28.1]</td>
    <td>[13.0, 26.0, 32.8]</td>
    <td>[8.6, 21.6, 28.0]</td>
    <td>[8.2, 19.1, 24.9]</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>[12.9, 26.4, 32.0]</td>
    <td>[11.3, 26.8, 33.6]</td>
    <td>[18.9, 39.0, 48.0]</td>
    <td>[16.7, 20.6, 28.7]</td>
    <td>[12.2, 42.6, 51.0]</td>
  </tr>
  <tr>
    <td>Sociology</td>
    <td>[19.1, 39.3, 48.9]</td>
    <td>[16.8, 41.8, 54.0]</td>
    <td>[27.9, 52.5, 63.9]</td>
    <td>[9.9, 28.6, 38.4]</td>
    <td>[17.2, 39.5, 49.8]</td>
  </tr>
  <tr>
    <td>Social Sciences Avg.</td>
    <td>[17.2, 30.9, 37.6]</td>
    <td>[11.9, 28.6, 36.1]</td>
    <td>[21.3, 39.5, 47.9]</td>
    <td>[10.8, 23.1, 32.6]</td>
    <td>[12.7, 28.1, 35.1]</td>
  </tr>
  <tr>
    <td>College Success</td>
    <td>[41.9, 60.5, 74.1]</td>
    <td>[44.2, 62.8, 74.4]</td>
    <td>[41.9, 76.7, 88.4]</td>
    <td>[27.9, 51.5, 69.8]</td>
    <td>[123.3, 55.8, 67.4]</td>
  </tr>
  <tr>
    <td>College Success Concise</td>
    <td>[40.4, 57.4, 70.2]</td>
    <td>[29.8, 51.7, 70.2]</td>
    <td>[46.8, 72.3, 87.2]</td>
    <td>[17.0, 38.3, 59.6]</td>
    <td>[27.7, 53.5, 63.8]</td>
  </tr>
  <tr>
    <td>College Success Avg.</td>
    <td>[41.1, 58.0, 72.1]</td>
    <td>[37.0, 58.7, 72.1]</td>
    <td>[41.3, 74.5, 87.8]</td>
    <td>[23.4, 58.9, 64.7]</td>
    <td>[123.3, 55.8, 65.5]</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>[117.4, 326.0, 404.1]</td>
    <td>[130.1, 325.2, 413.8]</td>
    <td>[187.4, 401.5, 503.1]</td>
    <td>[97.2, 274.3, 368.3]</td>
    <td>[183.7, 409.3, 496.0]</td>
  </tr>
</table>

Table 6: Masked models’ accuracies (percentages) on the English portion of the dataset using sentence-level prompts. Results are shown as [top1, top5, top10]. Books that are also in Polish are marked with ◯, where books that are also in Spanish are marked with *. The highest result for each subdomain across both sentence and paragraph level Tables (Table 6 and Table 7) is highlighted in bold font.
<table>
  <tr>
    <th>Subdomain</th>
    <th colspan="6">Mask LMS</th>
  </tr>
  <tr>
    <th></th>
    <th>BERT</th>
    <th>DistilBERT</th>
    <th>XLM-RoBERTa</th>
    <th>mBERT</th>
    <th>SciBERT</th>
  </tr>
  <tr><td>Prealgebra</td><td>[13.7, 35.0, 45.6]</td><td>[8.2, 37.5, 44.1]</td><td>[15.0, 37.0, 46.8]</td><td>[18.7, 26.9, 37.0]</td><td>[22.3, 47.2, 54.7]</td></tr>
  <tr><td>Elementary Algebra</td><td>[8.2, 34.3, 44.9]</td><td>[11.1, 33.7, 44.6]</td><td>[15.8, 34.8, 41.7]</td><td>[18.2, 32.3, 34.7]</td><td>[14.0, 57.9, 59.0]</td></tr>
  <tr><td>Intermediate Algebra</td><td>[11.6, 28.4, 38.2]</td><td>[9.8, 32.1, 43.3]</td><td>[11.3, 29.4, 39.1]</td><td>[7.6, 26.3, 34.6]</td><td>[18.7, 44.0, 55.0]</td></tr>
  <tr><td>College Algebra</td><td>[11.6, 35.2, 44.2]</td><td>[11.3, 38.7, 47.7]</td><td>[11.6, 28.1, 42.2]</td><td>[4.5, 23.1, 30.7]</td><td>[23.6, 50.3, 59.8]</td></tr>
  <tr><td>Trigonometry</td><td>[8.9, 39.8, 37.4]</td><td>[7.6, 32.0, 41.1]</td><td>[9.9, 37.3, 36.3]</td><td>[4.2, 22.2, 28.1]</td><td>[17.7, 42.1, 53.2]</td></tr>
  <tr><td>PreCalculus*</td><td>[8.6, 36.5, 42.7]</td><td>[7.5, 28.9, 39.7]</td><td>[7.7, 32.4, 38.1]</td><td>[4.1, 27.2, 31.4]</td><td>[16.4, 44.8, 59.4]</td></tr>
  <tr><td>Calculus I*</td><td>[12.6, 26.6, 41.1]</td><td>[11.1, 39.7, 47.8]</td><td>[11.5, 31.5, 43.4]</td><td>[5.4, 31.2, 39.7]</td><td>[27.5, 56.5, 62.7]</td></tr>
  <tr><td>Calculus II*</td><td>[14.2, 33.7, 39.5]</td><td>[13.4, 36.4, 43.7]</td><td>[16.5, 42.5, 48.7]</td><td>[13.8, 34.9, 39.8]</td><td>[25.4, 47.1, 56.3]</td></tr>
  <tr><td>Calculus III*</td><td>[11.8, 29.1, 37.4]</td><td>[9.0, 32.5, 40.5]</td><td>[13.8, 35.6, 45.5]</td><td>[11.4, 31.1, 36.7]</td><td>[23.2, 48.4, 57.4]</td></tr>
  <tr><td>Intro. Statistics</td><td>[11.5, 36.8, 41.5]</td><td>[11.4, 36.0, 46.3]</td><td>[11.3, 33.8, 40.6]</td><td>[8.4, 36.5, 47.4]</td><td>[15.1, 61.1, 63.0]</td></tr>
  <tr><td>Biostatistics*</td><td>[15.4, 38.4, 41.4]</td><td>[14.0, 37.3, 46.4]</td><td>[11.5, 33.8, 40.6]</td><td>[7.8, 21.0, 28.2]</td><td>[14.2, 61.1, 62.9]</td></tr>
  <tr><td>Statistics*</td><td>[13.5, 38.2, 42.8]</td><td>[13.2, 39.5, 48.7]</td><td>[11.7, 44.7, 55.3]</td><td>[11.8, 36.2, 46.1]</td><td>[30.9, 57.2, 66.8]</td></tr>
  <tr><td>Contemporary Math</td><td>[18.9, 46.7, 54.6]</td><td>[19.4, 51.4, 62.8]</td><td>[26.6, 51.6, 58.6]</td><td>[13.4, 40.7, 48.6]</td><td>[29.8, 60.3, 71.5]</td></tr>
  <tr><td>Math Avg.</td><td>[11.4, 34.0, 41.9]</td><td>[12.2, 36.5, 43.6]</td><td>[16.4, 37.8, 47.8]</td><td>[8.7, 30.2, 37.9]</td><td>[24.5, 51.2, 60.6]</td></tr>
  <tr><td>Concepts of Biology</td><td>[11.6, 28.7, 39.9]</td><td>[12.6, 33.3, 42.0]</td><td>[20.2, 41.5, 49.5]</td><td>[16.3, 31.8, 39.9]</td><td>[31.8, 62.2, 72.0]</td></tr>
  <tr><td>Botany</td><td>[11.0, 25.4, 34.2]</td><td>[10.9, 28.3, 36.6]</td><td>[9.4, 29.0, 39.9]</td><td>[9.3, 29.7, 38.0]</td><td>[16.4, 46.1, 58.6]</td></tr>
  <tr><td>Microbiology</td><td>[14.0, 9.5, 12.3]</td><td>[13.4, 11.0, 15.5]</td><td>[8.9, 22.6, 29.6]</td><td>[13.7, 19.2, 21.4]</td><td>[16.9, 38.3, 47.4]</td></tr>
  <tr><td>Chemistry Aoms First</td><td>[15.4, 32.2, 40.0]</td><td>[13.1, 36.7, 45.2]</td><td>[22.2, 47.2, 56.3]</td><td>[15.4, 39.1, 49.4]</td><td>[27.2, 56.0, 64.4]</td></tr>
  <tr><td>Chemistry*</td><td>[28.8, 55.6, 62.2]</td><td>[28.8, 51.1, 66.7]</td><td>[42.2, 73.3, 82.2]</td><td>[26.7, 60.0, 75.6]</td><td>[46.7, 60.0, 77.8]</td></tr>
  <tr><td>Organic Chemistry</td><td>[17.2, 34.7, 45.1]</td><td>[16.8, 34.6, 41.5]</td><td>[24.5, 53.9, 65.1]</td><td>[15.5, 35.3, 42.9]</td><td>[31.1, 55.0, 64.9]</td></tr>
  <tr><td>Astrophysics</td><td>[17.2, 34.7, 45.1]</td><td>[16.8, 34.6, 41.5]</td><td>[24.5, 53.9, 65.1]</td><td>[15.5, 35.3, 42.9]</td><td>[31.1, 55.0, 64.9]</td></tr>
  <tr><td>Astronomy</td><td>[13.5, 28.9, 34.6]</td><td>[8.9, 29.6, 32.5]</td><td>[10.0, 34.1, 43.4]</td><td>[10.2, 37.0, 38.9]</td><td>[12.9, 29.8, 35.9]</td></tr>
  <tr><td>College Physics</td><td>[11.4, 30.3, 37.8]</td><td>[12.3, 32.7, 41.7]</td><td>[17.1, 39.4, 47.5]</td><td>[17.9, 29.6, 38.5]</td><td>[24.8, 52.8, 63.1]</td></tr>
  <tr><td>Physics I*</td><td>[17.4, 37.2, 43.7]</td><td>[17.5, 37.8, 48.6]</td><td>[17.1, 42.0, 48.9]</td><td>[14.5, 37.5, 47.5]</td><td>[24.1, 53.8, 63.0]</td></tr>
  <tr><td>Physics II*</td><td>[17.5, 35.5, 38.0]</td><td>[12.9, 32.7, 41.8]</td><td>[13.3, 33.6, 41.0]</td><td>[18.8, 35.5, 47.7]</td><td>[25.1, 48.8, 59.1]</td></tr>
  <tr><td>Physics III*</td><td>[14.9, 30.9, 37.9]</td><td>[14.9, 35.4, 42.0]</td><td>[19.7, 41.9, 50.3]</td><td>[11.8, 34.6, 42.5]</td><td>[30.9, 51.5, 58.7]</td></tr>
  <tr><td>Science Avg.</td><td>[11.3, 28.3, 34.2]</td><td>[12.4, 31.0, 39.5]</td><td>[19.6, 41.0, 49.2]</td><td>[11.7, 37.6, 41.4]</td><td>[25.8, 50.3, 59.4]</td></tr>
  <tr><td>U.S. History</td><td>[19.3, 33.3, 38.0]</td><td>[19.4, 37.8, 44.1]</td><td>[26.1, 43.5, 51.5]</td><td>[19.8, 36.7, 43.3]</td><td>[14.4, 27.0, 32.8]</td></tr>
  <tr><td>Writing*</td><td>[9.8, 24.1, 31.7]</td><td>[8.5, 22.7, 31.5]</td><td>[8.0, 21.5, 30.7]</td><td>[7.6, 20.9, 30.2]</td><td>[8.1, 20.3, 31.8]</td></tr>
  <tr><td>Philosophy</td><td>[19.6, 32.8, 37.7]</td><td>[13.2, 32.8, 41.0]</td><td>[22.4, 38.5, 46.6]</td><td>[17.7, 37.8, 46.4]</td><td>[14.0, 30.2, 38.2]</td></tr>
  <tr><td>World History 1</td><td><b>15.0</b>, 27.5, 34.1</td><td><b>9.3</b>, 20.3, 30.6</td><td><b>13.8</b>, 25.9, 34.3</td><td><b>11.5</b>, <b>29.3</b>, <b>37.3</b></td><td><b>14.5</b>, <b>21.1</b>, <b>32.3</b></td></tr>
  <tr><td>World History 2</td><td><b>25.4</b>, 45.0, 50.7</td><td><b>12.6</b>, 32.5, 42.4</td><td><b>24.5</b>, <b>54.5</b>, <b>53.7</b></td><td><b>19.3</b>, <b>42.3</b>, <b>51.1</b></td><td><b>19.8</b>, <b>28.4</b>, <b>32.7</b></td></tr>
  <tr><td>Humanities Avg.</td><td>[11.7, 33.5, 40.7]</td><td>[12.9, 30.8, 39.1]</td><td>[21.7, 39.2, 47.2]</td><td>[13.7, 41.2, 42.2]</td><td>[10.8, 24.3, 30.8]</td></tr>
  <tr><td>Intro. To Business</td><td>[11.4, 31.0, 37.7]</td><td>[11.0, 28.1, 35.7]</td><td>[13.8, 32.6, 45.0]</td><td>[10.2, 28.4, 33.2]</td><td>[16.0, 42.4, 53.1]</td></tr>
  <tr><td>Financial Accounting</td><td>[10.7, 34.7, 42.4]</td><td>[9.0, 30.1, 36.7]</td><td>[11.8, 28.5, 38.1]</td><td>[9.3, 28.6, 33.9]</td><td>[11.9, 34.2, 40.1]</td></tr>
  <tr><td>Managerial Accounting</td><td>[16.5, 41.8, 51.1]</td><td>[11.7, 46.6, 55.7]</td><td>[29.1, 60.3, 71.1]</td><td>[14.4, 39.2, 50.1]</td><td>[23.3, 49.4, 58.7]</td></tr>
  <tr><td>Business Ethics</td><td>[26.2, 46.2, 54.2]</td><td>[22.3, 47.2, 56.1]</td><td>[39.5, 62.8, 70.8]</td><td>[18.9, 41.5, 51.5]</td><td>[28.2, 49.8, 58.5]</td></tr>
  <tr><td>Org. Behavior</td><td>[14.4, 31.3, 37.6]</td><td>[13.3, 31.1, 40.6]</td><td>[29.8, 52.4, 61.5]</td><td>[13.5, 33.1, 42.7]</td><td>[28.4, 50.3, 57.5]</td></tr>
  <tr><td>Finance</td><td>[16.5, 41.8, 51.1]</td><td>[11.7, 46.6, 55.7]</td><td>[29.1, 60.3, 71.1]</td><td>[14.4, 39.2, 50.1]</td><td>[23.3, 49.4, 58.7]</td></tr>
  <tr><td>Business Law</td><td>[24.2, 37.5, 44.0]</td><td>[18.2, 38.4, 47.2]</td><td>[30.8, 52.8, 63.3]</td><td>[15.2, 34.0, 44.0]</td><td>[19.6, 44.9, 52.5]</td></tr>
  <tr><td>Intellectual Property</td><td>[17.6, 42.9, 55.5]</td><td>[14.3, 38.7, 53.8]</td><td>[23.5, 49.6, 63.0]</td><td>[9.2, 37.8, 50.4]</td><td>[17.6, 42.9, 54.6]</td></tr>
  <tr><td>Marketing</td><td>[10.7, 21.9, 27.1]</td><td>[8.3, 21.4, 25.7]</td><td>[13.6, 43.2, 48.4]</td><td>[8.7, 20.0, 24.8]</td><td>[11.1, 24.7, 29.3]</td></tr>
  <tr><td>Management</td><td>[29.9, 52.4, 57.5]</td><td>[25.3, 56.3, 64.2]</td><td>[25.6, 52.8, 73.0]</td><td>[27.3, 56.3, 59.1]</td><td>[15.9, 58.5, 63.0]</td></tr>
  <tr><td>Finance</td><td>[29.8, 38.7, 45.8]</td><td>[12.8, 36.3, 43.5]</td><td>[23.8, 48.9, 61.2]</td><td>[13.4, 48.9, 63.3]</td><td>[35.1, 79.8, 80.6]</td></tr>
  <tr><td>Macroeconomics*</td><td>[28.1, 66.2, 70.5]</td><td>[27.3, 60.4, 71.2]</td><td>[34.5, 66.9, 78.4]</td><td>[17.3, 48.9, 63.3]</td><td>[35.1, 79.8, 80.6]</td></tr>
  <tr><td>Microeconomics*</td><td>[29.1, 65.8, 78.5]</td><td>[26.6, 64.8, 81.0]</td><td>[38.0, 74.7, 79.7]</td><td>[26.6, 63.5, 72.2]</td><td>[43.0, 79.7, 86.1]</td></tr>
  <tr><td>Entrepreneurship</td><td>[11.5, 26.1, 33.1]</td><td>[8.8, 29.4, 32.6]</td><td>[23.8, 48.2, 51.9]</td><td>[7.2, 29.9, 30.4]</td><td>[8.9, 26.6, 23.8]</td></tr>
  <tr><td>Finance Avg.</td><td>[14.9, 34.0, 41.7]</td><td>[12.7, 34.2, 39.9]</td><td>[18.0, 49.8, 54.9]</td><td>[12.4, 38.1, 42.5]</td><td>[16.4, 43.2, 53.7]</td></tr>
  <tr><td>Nutrition</td><td>[11.6, 17.5, 21.1]</td><td>[11.7, 17.6, 22.9]</td><td>[12.8, 22.7, 27.5]</td><td>[11.4, 28.8, 31.1]</td><td>[14.7, 20.0, 21.0]</td></tr>
  <tr><td>Nursing</td><td>[17.8, 17.5, 22.4]</td><td>[17.1, 7.5, 23.5]</td><td>[14.4, 58.2, 41.5]</td><td>[1.4, 17.0, 20.0]</td><td>[21.0, 43.2, 43.5]</td></tr>
  <tr><td>Python Programming*</td><td>[18.3, 15.2, 17.6]</td><td>[7.8, 15.0, 20.7]</td><td>[20.9, 40.6, 51.4]</td><td>[15.2, 14.2, 18.6]</td><td>[16.0, 30.1, 33.9]</td></tr>
  <tr><td>Workplace Software*</td><td>[19.3, 22.0, 27.2]</td><td>[7.4, 21.2, 26.6]</td><td>[22.7, 45.1, 53.1]</td><td>[5.4, 19.3, 26.0]</td><td>[10.3, 30.5, 38.7]</td></tr>
  <tr><td>Computer Science Avg.</td><td>[18.8, 16.6, 22.1]</td><td>[7.8, 16.1, 23.6]</td><td>[21.8, 42.9, 52.3]</td><td>[13.6, 18.8, 23.3]</td><td>[11.3, 30.2, 36.3]</td></tr>
  <tr><td>American Government</td><td>[25.5, 38.7, 43.6]</td><td>[11.0, 32.3, 39.6]</td><td>[29.9, 47.2, 53.8]</td><td>[19.1, 38.0, 41.3]</td><td>[10.4, 24.2, 30.9]</td></tr>
  <tr><td>Anthropology</td><td>[14.0, 26.6, 31.0]</td><td>[10.7, 26.3, 33.4]</td><td>[19.7, 35.4, 40.8]</td><td>[16.2, 29.4, 36.7]</td><td>[18.2, 27.9, 33.9]</td></tr>
  <tr><td>Physical Science*</td><td>[9.8, 17.5, 31.3]</td><td>[8.8, 14.3, 31.7]</td><td>[11.5, 31.5, 39.1]</td><td>[9.1, 27.3, 36.4]</td><td>[28.7, 52.2, 59.4]</td></tr>
  <tr><td>Psychology</td><td>[16.5, 29.2, 34.8]</td><td>[14.4, 31.5, 39.1]</td><td>[15.9, 36.9, 54.2]</td><td>[11.2, 31.6, 40.9]</td><td>[28.7, 52.2, 59.4]</td></tr>
  <tr><td>Sociology</td><td>[21.5, 49.1, 54.8]</td><td>[20.1, 51.1, 60.4]</td><td>[37.6, 64.1, 70.9]</td><td>[16.5, 46.1, 55.2]</td><td>[28.0, 55.9, 63.9]</td></tr>
  <tr><td>Social Sciences Avg.</td><td>[119.6, 33.9, 44.6]</td><td>[14.4, 34.8, 42.4]</td><td>[28.3, 48.2, 54.8]</td><td>[18.5, 33.0, 42.8]</td><td>[18.2, 37.8, 44.1]</td></tr>
  <tr><td>College Success</td><td>[30.0, 62.5, 68.1]</td><td>[42.2, 62.5, 68.8]</td><td>[57.8, 82.8, 93.8]</td><td>[26.6, 57.8, 71.9]</td><td>[45.3, 67.2, 71.9]</td></tr>
  <tr><td>College Success Concise</td><td>[18.5, 65.4, 80.9]</td><td>[13.6, 65.9, 71.2]</td><td>[12.7, 78.8, 90.4]</td><td>[30.8, 59.6, 75.0]</td><td>[10.8, 59.6, 69.2]</td></tr>
  <tr><td>College Success Avg.*</td><td>[167.2, 64.0, 69.0]</td><td>[138.4, 57.5, 60.0]</td><td>[157.8, 80.8, 92.1]</td><td>[125.7, 58.7, 63.5]</td><td>[110.3, 61.4, 60.7]</td></tr>
  <tr><th>Overall</th><th>[167.7, 34.5, 41.7]</th><th>[18.2, 35.1, 43.8]</th><th>[23.8, 45.6, 54.2]</th><th>[12.5, 31.4, 41.9]</th><th>[22.7, 45.7, 55.0]</th></tr>
</table>

Table 7: Masked models’ accuracies (percentages) on the English portion of the dataset using paragraph-level prompts. Results are shown as [top1, top5, top10]. Books that are also in Polish are marked with ◯, where books that are also in Spanish are marked with *. The highest result for each subdomain across both sentence and paragraph level Tables (Table 6 and Table 7) is highlighted in bold font.
<table>
  <tr>
    <th>Subdomain</th>
    <th colspan="2">Bloom</th>
    <th colspan="2">Llama2</th>
    <th colspan="2">Causal LMs</th>
    <th colspan="2">Mistral</th>
  </tr>
  <tr>
    <th></th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
  </tr>
  <tr>
    <td>PreCalculus</td>
    <td>[2.0, 1.7, 0.7]</td>
    <td>[3.0, 1.7, 1.0]</td>
    <td>[4.0, 2.7, 3.1]</td>
    <td>[3.3, 1.5, 3.0]</td>
    <td>[7.7, 8.7, 16.3]</td>
    <td>[5.3, 5.5, 14.7]</td>
    <td>[8.3, 7.0, 8.7]</td>
    <td>[4.7, 4.7, 6.7]</td>
  </tr>
  <tr>
    <td>Calculus I</td>
    <td>[3.2, 2.7, 4.5]</td>
    <td>[4.3, 5.8, 4.7]</td>
    <td>[5.9, 4.1, 5.4]</td>
    <td>[3.2, 1.1, 2.5]</td>
    <td>[6.3, 5.0, 8.6]</td>
    <td>[7.2, 6.1, 15.2]</td>
    <td>[10.0, 4.5, 5.4]</td>
    <td>[7.6, 3.2, 4.7]</td>
  </tr>
  <tr>
    <td>Calculus II</td>
    <td>[2.0, 1.0, 0.9]</td>
    <td>[5.2, 3.0, 4.4]</td>
    <td>[0.0, 1.0, 2.0]</td>
    <td>[3.0, 0.7, 1.5]</td>
    <td>[1.0, 2.0, 5.1]</td>
    <td>[4.4, 5.2, 7.4]</td>
    <td>[2.0, 2.0, 3.0]</td>
    <td>[3.0, 5.9, 5.2]</td>
  </tr>
  <tr>
    <td>Calculus III</td>
    <td>[1.2, 1.2, 2.4]</td>
    <td>[3.3, 3.0, 3.0]</td>
    <td>[2.0, 0.8, 1.6]</td>
    <td>[3.3, 0.7, 1.7]</td>
    <td>[4.0, 6.1, 8.1]</td>
    <td>[9.0, 5.3, 11.0]</td>
    <td>[4.5, 4.0, 3.2]</td>
    <td>[5.7, 3.7, 4.3]</td>
  </tr>
  <tr>
    <td>Business Statistics</td>
    <td>[4.3, 2.9, 1.4]</td>
    <td>[4.0, 7.0, 6.0]</td>
    <td>[5.8, 0.0, 2.9]</td>
    <td>[6.0, 0.0, 3.0]</td>
    <td>[8.7, 7.2, 20.3]</td>
    <td>[5.0, 5.0, 23.0]</td>
    <td>[13.0, 5.8, 5.8]</td>
    <td>[8.0, 5.0, 5.0]</td>
  </tr>
  <tr>
    <td>Statistics</td>
    <td>[5.1, 4.0, 1.5]</td>
    <td>[5.1, 4.0, 1.5]</td>
    <td>[5.1, 4.0, 1.5]</td>
    <td>[5.1, 4.0, 1.5]</td>
    <td>[8.7, 7.2, 20.3]</td>
    <td>[5.0, 5.0, 23.0]</td>
    <td>[13.0, 5.8, 5.8]</td>
    <td>[8.0, 5.0, 5.0]</td>
  </tr>
  <tr>
    <td>Math Avg.</td>
    <td>[2.6, 2.0, 2.7]</td>
    <td>[3.9, 4.1, 3.9]</td>
    <td>[3.8, 1.1, 2.2]</td>
    <td>[3.8, 0.9, 2.0]</td>
    <td>[6.9, 6.1, 12.7]</td>
    <td>[6.7, 6.1, 14.5]</td>
    <td>[8.4, 5.9, 5.2]</td>
    <td>[6.4, 4.5, 5.3]</td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>[7.3, 3.0, 6.7]</td>
    <td>[9.3, 7.7, 6.7]</td>
    <td>[6.0, 4.3, 8.3]</td>
    <td>[8.3, 1.0, 4.0]</td>
    <td>[10.7, 13.0, 19.3]</td>
    <td>[9.2, 13.3, 24.0]</td>
    <td>[13.7, 9.3, 11.0]</td>
    <td>[9.3, 5.7, 7.0]</td>
  </tr>
  <tr>
    <td>Physics I</td>
    <td>[3.0, 1.0, 3.3]</td>
    <td>[4.3, 3.0, 3.7]</td>
    <td>[5.0, 2.0, 6.3]</td>
    <td>[5.0, 1.7, 2.3]</td>
    <td>[8.7, 7.3, 14.3]</td>
    <td>[10.3, 8.3, 14.0]</td>
    <td>[6.3, 5.3, 6.3]</td>
    <td>[4.7, 3.7, 4.3]</td>
  </tr>
  <tr>
    <td>Physics II</td>
    <td>[3.3, 3.0, 9.0]</td>
    <td>[4.0, 4.3, 4.7]</td>
    <td>[4.0, 4.3, 6.7]</td>
    <td>[4.3, 3.0, 5.0]</td>
    <td>[6.7, 8.3, 16.3]</td>
    <td>[10.0, 12.0, 20.3]</td>
    <td>[6.3, 10.7, 10.7]</td>
    <td>[9.7, 7.0, 6.3]</td>
  </tr>
  <tr>
    <td>Physics III</td>
    <td>[4.0, 1.7, 5.0]</td>
    <td>[3.0, 2.5, 2.7]</td>
    <td>[4.0, 4.7, 9.3]</td>
    <td>[4.7, 2.3, 3.0]</td>
    <td>[7.7, 11.3, 15.7]</td>
    <td>[7.0, 10.3, 17.0]</td>
    <td>[10.0, 10.7, 12.0]</td>
    <td>[6.0, 6.3, 9.3]</td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>[4.2, 2.2, 6.0]</td>
    <td>[5.2, 2.5, 6.5]</td>
    <td>[4.8, 3.8, 7.7]</td>
    <td>[5.1, 2.0, 5.8]</td>
    <td>[8.4, 10.0, 16.4]</td>
    <td>[9.2, 11.2, 18.8]</td>
    <td>[9.1, 10.0, 10.6]</td>
    <td>[7.4, 5.7, 8.7]</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>[3.3, 2.1, 4.0]</td>
    <td>[4.4, 4.2, 4.2]</td>
    <td>[4.2, 2.2, 5.0]</td>
    <td>[4.3, 1.4, 2.6]</td>
    <td>[7.5, 7.7, 14.2]</td>
    <td>[7.1, 8.0, 16.2]</td>
    <td>[8.3, 5.3, 7.1]</td>
    <td>[6.8, 4.9, 6.0]</td>
  </tr>
</table>

Table 8: Causal models’ accuracies (percentages) on the Spanish portion of the dataset. Results are shown as [prompt 1, prompt 2, prompt 3]. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.

<table>
  <tr>
    <th>Subdomain</th>
    <th colspan="2">Bloom</th>
    <th colspan="2">Llama2</th>
    <th colspan="2">Causal LMs</th>
    <th colspan="2">Mistral</th>
  </tr>
  <tr>
    <th></th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
  </tr>
  <tr>
    <td>Physics I</td>
    <td>[0.0, 0.0]</td>
    <td>[0.0, 0.0]</td>
    <td>[0.0, 0.3, 0.3]</td>
    <td>[1.0, 0.3, 0.0]</td>
    <td>[1.0, 1.7, 4.7]</td>
    <td>[1.7, 0.3, 4.0]</td>
    <td>[1.3, 1.0, 1.0]</td>
    <td>[1.7, 0.7, 0.0]</td>
  </tr>
  <tr>
    <td>Physics II</td>
    <td>[0.3, 0.0]</td>
    <td>[0.3, 0.0]</td>
    <td>[1.0, 0.1, 1.3]</td>
    <td>[0.7, 0.0, 1.7]</td>
    <td>[0.7, 0.0, 1.7]</td>
    <td>[1.7, 2.3, 8.3]</td>
    <td>[1.0, 0.7, 0.7]</td>
    <td>[2.3, 1.0, 2.3]</td>
  </tr>
  <tr>
    <td>Physics III</td>
    <td>[0.3, 0.0]</td>
    <td>[0.3, 0.0]</td>
    <td>[1.0, 1.3, 1.3]</td>
    <td>[1.0, 1.3, 1.3]</td>
    <td>[1.3, 1.7, 1.3]</td>
    <td>[1.7, 1.3, 1.3]</td>
    <td>[1.7, 1.3, 1.3]</td>
    <td>[1.7, 1.3, 1.3]</td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>[0.2, 0.0, 0.0]</td>
    <td>[0.2, 0.1, 0.1]</td>
    <td>[0.2, 0.0, 0.1]</td>
    <td>[0.2, 0.0, 0.1]</td>
    <td>[0.2, 0.2, 0.4]</td>
    <td>[0.4, 0.5, 0.8]</td>
    <td>[0.8, 0.8, 0.8]</td>
    <td>[0.8, 0.8, 0.8]</td>
  </tr>
  <tr>
    <td>Macroeconomics</td>
    <td>[0.0, 0.0, 0.0]</td>
    <td>[0.0, 0.7, 0.3]</td>
    <td>[1.7, 1.7, 2.7]</td>
    <td>[1.7, 1.0, 0.3]</td>
    <td>[1.7, 2.3, 6.0]</td>
    <td>[1.3, 2.3, 6.3]</td>
    <td>[1.0, 1.7, 3.7]</td>
    <td>[1.0, 1.7, 2.7]</td>
  </tr>
  <tr>
    <td>Microeconomics</td>
    <td>[0.0, 0.0, 0.0]</td>
    <td>[2.0, 1.0, 1.0]</td>
    <td>[1.7, 0.7, 1.0]</td>
    <td>[1.7, 0.7, 1.0]</td>
    <td>[1.7, 0.7, 2.0]</td>
    <td>[2.3, 2.3, 5.0]</td>
    <td>[0.7, 1.3, 3.0]</td>
    <td>[2.0, 2.0, 2.3]</td>
  </tr>
  <tr>
    <td>Business Avg.</td>
    <td>[0.0, 0.0, 0.0]</td>
    <td>[1.4, 0.8, 0.5]</td>
    <td>[1.0, 1.3, 1.9]</td>
    <td>[2.7, 0.8, 0.7]</td>
    <td>[1.4, 2.5, 5.3]</td>
    <td>[2.8, 2.5, 5.7]</td>
    <td>[10.8, 1.8, 3.4]</td>
    <td>[2.9, 1.9, 2.3]</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>[0.0, 0.0]</td>
    <td>[0.0, 0.0]</td>
    <td>[1.0, 0.0, 1.3]</td>
    <td>[2.0, 1.0, 2.7]</td>
    <td>[1.3, 2.7, 9.0]</td>
    <td>[3.0, 3.0, 9.0]</td>
    <td>[1.0, 1.7, 4.0]</td>
    <td>[2.3, 2.0, 4.0]</td>
  </tr>
  <tr>
    <td>Social Sciences Avg.</td>
    <td>[0.0, 0.0, 0.0]</td>
    <td>[0.0, 0.0, 0.0]</td>
    <td>[1.0, 0.0, 1.3]</td>
    <td>[2.0, 1.0, 2.7]</td>
    <td>[1.3, 2.7, 9.0]</td>
    <td>[3.0, 3.0, 9.0]</td>
    <td>[1.0, 1.7, 4.0]</td>
    <td>[2.3, 2.0, 4.0]</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>[0.1, 0.0, 0.0]</td>
    <td>[0.5, 0.3, 0.2]</td>
    <td>[1.1, 0.6, 1.3]</td>
    <td>[1.1, 0.5, 1.0]</td>
    <td>[1.1, 2.2, 5.6]</td>
    <td>[2.2, 2.0, 6.3]</td>
    <td>[1.3, 1.2, 2.3]</td>
    <td>[2.4, 1.3, 2.1]</td>
  </tr>
</table>

Table 9: Causal models’ accuracies (percentages) on the Polish portion of the dataset. Results are shown as [prompt 1, prompt 2, prompt 3]. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.

<table>
  <tr>
    <th>Subdomain</th>
    <th colspan="2">GPT-4-turbo</th>
    <th colspan="2">GPT-4-mini</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">Llama 3.1 405b Instruct</th>
  </tr>
  <tr>
    <th></th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
  </tr>
  <tr>
    <td>PreCalculus</td>
    <td>33.5</td>
    <td>37.5</td>
    <td>25.8</td>
    <td>35.0</td>
    <td>29.1</td>
    <td>36.3</td>
    <td>43.4</td>
    <td>48.8</td>
  </tr>
  <tr>
    <td>Calculus I</td>
    <td>29.4</td>
    <td>37.9</td>
    <td>25.8</td>
    <td>32.5</td>
    <td>27.1</td>
    <td>38.3</td>
    <td>30.3</td>
    <td>46.6</td>
  </tr>
  <tr>
    <td>Calculus II</td>
    <td>24.2</td>
    <td>26.7</td>
    <td>17.2</td>
    <td>26.7</td>
    <td>21.2</td>
    <td>23.0</td>
    <td>27.3</td>
    <td>36.3</td>
  </tr>
  <tr>
    <td>Calculus III</td>
    <td>25.5</td>
    <td>33.8</td>
    <td>20.6</td>
    <td>25.6</td>
    <td>27.5</td>
    <td>32.0</td>
    <td>31.6</td>
    <td>41.5</td>
  </tr>
  <tr>
    <td>Business Statistics</td>
    <td>42.0</td>
    <td>42.0</td>
    <td>36.2</td>
    <td>31.8</td>
    <td>34.8</td>
    <td>42.0</td>
    <td>47.8</td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>Statistics</td>
    <td>30.4</td>
    <td>42.0</td>
    <td>32.5</td>
    <td>37.4</td>
    <td>30.4</td>
    <td>37.4</td>
    <td>37.2</td>
    <td>49.2</td>
  </tr>
  <tr>
    <td>Math Avg.</td>
    <td>30.8</td>
    <td>36.6</td>
    <td>26.4</td>
    <td>33.0</td>
    <td>28.4</td>
    <td>34.8</td>
    <td>36.3</td>
    <td>46.6</td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>46.4</td>
    <td>56.5</td>
    <td>39.2</td>
    <td>51.0</td>
    <td>46.5</td>
    <td>56.0</td>
    <td>53.1</td>
    <td>63.5</td>
  </tr>
  <tr>
    <td>Physics I</td>
    <td>30.7</td>
    <td>40.4</td>
    <td>27.3</td>
    <td>39.1</td>
    <td>30.1</td>
    <td>41.6</td>
    <td>34.8</td>
    <td>44.7</td>
  </tr>
  <tr>
    <td>Physics II</td>
    <td>36.8</td>
    <td>46.7</td>
    <td>26.5</td>
    <td>43.1</td>
    <td>36.2</td>
    <td>47.1</td>
    <td>44.9</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>Physics III</td>
    <td>38.1</td>
    <td>46.2</td>
    <td>32.2</td>
    <td>39.9</td>
    <td>36.4</td>
    <td>44.6</td>
    <td>44.3</td>
    <td>51.8</td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>38.0</td>
    <td>47.5</td>
    <td>32.0</td>
    <td>43.8</td>
    <td>37.3</td>
    <td>47.3</td>
    <td>44.3</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>33.7</td>
    <td>41.0</td>
    <td>28.6</td>
    <td>37.4</td>
    <td>31.9</td>
    <td>39.8</td>
    <td>39.5</td>
    <td>49.5</td>
  </tr>
</table>

Table 10: Causal models’ accuracies (percentages) on the Spanish portion of the dataset. Averages are book-level. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.

<table>
  <tr>
    <th>Subdomain</th>
    <th colspan="2">GPT-4-turbo</th>
    <th colspan="2">GPT-4-mini</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">Llama 3.1 405b Instruct</th>
  </tr>
  <tr>
    <th></th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
    <th>S</th>
    <th>P</th>
  </tr>
  <tr>
    <td>Physics I</td>
    <td>19.7</td>
    <td>29.5</td>
    <td>14.6</td>
    <td>21.8</td>
    <td>22.0</td>
    <td>32.7</td>
    <td>23.3</td>
    <td>36.2</td>
  </tr>
  <tr>
    <td>Physics II</td>
    <td>21.8</td>
    <td>30.7</td>
    <td>17.0</td>
    <td>25.8</td>
    <td>25.7</td>
    <td>35.3</td>
    <td>29.3</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Physics III</td>
    <td>26.9</td>
    <td>30.7</td>
    <td>20.0</td>
    <td>23.3</td>
    <td>30.8</td>
    <td>39.2</td>
    <td>28.8</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>22.2</td>
    <td>30.3</td>
    <td>19.4</td>
    <td>29.4</td>
    <td>26.2</td>
    <td>35.7</td>
    <td>27.8</td>
    <td>39.4</td>
  </tr>
  <tr>
    <td>Macroeconomics</td>
    <td>22.3</td>
    <td>28.6</td>
    <td>20.1</td>
    <td>28.6</td>
    <td>25.7</td>
    <td>35.0</td>
    <td>27.0</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>Microeconomics</td>
    <td>22.0</td>
    <td>33.1</td>
    <td>18.6</td>
    <td>30.2</td>
    <td>26.6</td>
    <td>35.7</td>
    <td>24.0</td>
    <td>41.7</td>
  </tr>
  <tr>
    <td>Business Avg.</td>
    <td>22.2</td>
    <td>30.9</td>
    <td>18.2</td>
    <td>27.0</td>
    <td>26.9</td>
    <td>36.8</td>
    <td>27.2</td>
    <td>39.6</td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>23.3</td>
    <td>32.8</td>
    <td>20.0</td>
    <td>32.4</td>
    <td>30.4</td>
    <td>42.8</td>
    <td>28.7</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Social Sciences Avg.</td>
    <td>23.3</td>
    <td>32.8</td>
    <td>20.0</td>
    <td>32.4</td>
    <td>30.4</td>
    <td>42.8</td>
    <td>28.7</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>22.7</td>
    <td>30.9</td>
    <td>18.2</td>
    <td>27.0</td>
    <td>26.9</td>
    <td>36.8</td>
    <td>27.2</td>
    <td>39.6</td>
  </tr>
</table>

Table 11: Causal models’ accuracies (percentages) on the Polish portion of the dataset. Averages are book-level. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.
<table>
  <tr>
    <th rowspan="2">Subdomain</th>
    <th colspan="4">XLM-RoBERTa</th>
    <th colspan="4">Mask LMs</th>
    <th colspan="4">mBERT</th>
  </tr>
  <tr>
    <th>S</th><th>P</th><th>S</th><th>P</th>
    <th>S</th><th>P</th><th>S</th><th>P</th>
    <th>S</th><th>P</th><th>S</th><th>P</th>
  </tr>
  <tr>
    <td>Pre calculus</td>
    <td>[76, 24.0, 32.2]</td><td>[121, 29.9, 38.5]</td><td>[45, 23.0, 29.8]</td><td>[72, 27.2, 34.3]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Calculus I</td>
    <td>[16.7, 37.6, 47.1]</td><td>[18.1, 48.4, 54.5]</td><td>[11.8, 33.9, 40.7]</td><td>[15.2, 41.2, 46.6]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Calculus II</td>
    <td>[18.8, 37.4, 47.1]</td><td>[21.2, 48.4, 54.5]</td><td>[11.8, 33.9, 40.7]</td><td>[15.2, 41.2, 46.6]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Calculus III</td>
    <td>[8.5, 32.0, 37.2]</td><td>[15.5, 35.7, 44.2]</td><td>[11.3, 26.7, 38.9]</td><td>[13.1, 36.0, 44.8]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Business Statistics</td>
    <td>[11.6, 31.9, 40.9]</td><td>[21.0, 52.0, 66.0]</td><td>[11.6, 42.0, 59.4]</td><td>[16.0, 51.0, 69.0]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Statistics</td>
    <td>[9.9, 33.5, 46.1]</td><td>[15.5, 38.7, 50.8]</td><td>[7.3, 26.7, 39.1]</td><td>[10.1, 32.4, 42.4]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Math Avg.</td>
    <td>[12.1, 32.2, 45.5]</td><td>[17.4, 41.8, 50.7]</td><td>[9.3, 30.1, 41.2]</td><td>[13.0, 38.0, 47.8]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>[12.8, 29.6, 40.5]</td><td>[22.0, 44.0, 53.9]</td><td>[9.0, 27.6, 37.1]</td><td>[15.7, 35.3, 47.1]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Phys I</td>
    <td>[12.8, 29.6, 40.5]</td><td>[22.0, 44.0, 53.9]</td><td>[9.0, 27.6, 37.1]</td><td>[15.7, 35.3, 47.1]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics II</td>
    <td>[14.2, 31.1, 40.0]</td><td>[18.4, 37.1, 47.1]</td><td>[19.6, 25.6, 39.9]</td><td>[14.3, 32.6, 39.4]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics III</td>
    <td>[13.5, 28.2, 37.3]</td><td>[16.4, 37.5, 48.9]</td><td>[9.5, 27.7, 33.7]</td><td>[15.7, 35.1, 43.3]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>[13.2, 29.4, 38.5]</td><td>[18.2, 38.6, 48.5]</td><td>[8.9, 25.7, 33.5]</td><td>[11.8, 33.7, 42.1]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>[12.5, 31.1, 42.7]</td><td>[17.8, 40.5, 49.9]</td><td>[9.1, 28.4, 38.1]</td><td>[11.5, 36.3, 45.5]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 12: Masked models’ accuracies (percentages) on the Spanish portion of the dataset. Results are shown as [top1, top5, top10]. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.

<table>
  <tr>
    <th rowspan="2">Subdomain</th>
    <th colspan="4">XLM-RoBERTa</th>
    <th colspan="4">Mask LMs</th>
    <th colspan="4">mBERT</th>
  </tr>
  <tr>
    <th>S</th><th>P</th><th>S</th><th>P</th>
    <th>S</th><th>P</th><th>S</th><th>P</th>
    <th>S</th><th>P</th><th>S</th><th>P</th>
  </tr>
  <tr>
    <td>Physics I</td>
    <td>[18.3, 26.8, 36.8]</td><td>[25.5, 48.3, 54.5]</td><td>[14.1, 18.3, 24.1]</td><td>[11.5, 30.6, 43.1]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics II</td>
    <td>[11.6, 29.1, 36.8]</td><td>[28.0, 54.6, 52.6]</td><td>[15.4, 18.6, 24.0]</td><td>[16.0, 34.0, 42.5]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Physics III</td>
    <td>[12.3, 21.7, 39.9]</td><td>[32.0, 54.1, 61.0]</td><td>[7.3, 19.3, 30.8]</td><td>[19.9, 43.0, 49.1]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Science Avg.</td>
    <td>[11.0, 29.4, 37.8]</td><td>[28.9, 49.0, 56.0]</td><td>[5.6, 19.7, 27.0]</td><td>[16.5, 38.5, 45.4]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Macroeconomics</td>
    <td>[11.4, 34.1, 43.9]</td><td>[26.8, 56.0, 65.3]</td><td>[13.4, 17.0, 23.1]</td><td>[15.7, 37.2, 47.6]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Microeconomics</td>
    <td>[12.2, 30.2, 40.1]</td><td>[33.9, 55.9, 64.8]</td><td>[13.2, 16.2, 23.9]</td><td>[15.0, 34.9, 47.8]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Business Avg.</td>
    <td>[11.8, 32.1, 42.0]</td><td>[30.4, 56.0, 65.0]</td><td>[5.3, 16.6, 23.6]</td><td>[14.3, 36.0, 47.7]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Psychology</td>
    <td>[10.4, 23.0, 30.9]</td><td>[23.3, 51.4, 58.1]</td><td>[12.8, 12.3, 17.2]</td><td>[18.1, 36.8, 42.5]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Social Sci Avg.</td>
    <td>[10.4, 23.0, 30.9]</td><td>[23.3, 51.4, 58.1]</td><td>[12.8, 12.3, 17.2]</td><td>[18.1, 36.8, 42.5]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>[11.2, 29.4, 38.1]</td><td>[29.9, 51.7, 59.4]</td><td>[4.4, 17.4, 24.2]</td><td>[16.1, 37.3, 45.7]</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 13: Masked models’ accuracies (percentages) on the Polish portion of the dataset. Results are shown as [top1, top5, top10]. S=sentence-level prompts; P=paragraph-level prompts. The highest result for each subdomain is highlighted in bold font.

a sample of 495 prompts (9 from each subdomain) for True/False ratings of grammaticality and specificity between annotators. Simple agreement was chosen over a Kappa metric due to the "paradox of high agreement" from which Kappa metrics suffer under heavy class imbalance; in our case, since "yes" ratings make up over 90% and 97% in our specificity and grammaticality ratings respectively, Kappa scores would be misleading.

The results show a high level of agreement, with all three annotators reaching the same conclusion in 97.1% of cases for grammaticality judgments, indicating a large consensus on the linguistic accuracy of the prompts. Similarly, our annotators demonstrated a strong agreement on specificity judgments, concurring in 90.1% of cases, which underscores the consistency of their evaluations regarding the degree to which the prompts require precise and detailed responses. These findings provide strong evidence of the high quality and reliability of our dataset.

Reviewer paper summaries: The paper introduces MALAMUTE, a novel multilingual and highly granular cloze-style dataset designed specifically for evaluating the knowledge of large language models (LLMs) in educational contexts. Unlike existing benchmarks that often focus on general or low-complexity knowledge, MALAMUTE targets specific and detailed areas of education across eight domains, including mathematics, science, social sciences, business, humanities, and more. It comprises expert-written and peer-reviewed probes derived from 71 university-level textbooks in three languages: English, Spanish, and Polish. The dataset contains 33,361 curriculum concepts and 116,887 prompts, which are further categorized into sentence-level and paragraph-level prompts.
The authors evaluate both masked and causal LMs on MALAMUTE to assess their proficiency in subject-specific knowledge. Despite the overall impressive abilities of these models, the evaluation reveals significant gaps when closely examined on particular subjects, highlighting the need for further development before LMs can be safely integrated into educational settings.

The paper introduces MALAMUTE, a multilingual, template-free, and highly granular probing dataset. MALAMUTE is composed of expert-written, peer-reviewed probes derived from 71 university-level textbooks across three languages (English, Spanish, and Polish). It is the first education-based cloze-style dataset, covering eight domains with up to 14 subdomains each, further broken down into concepts and concept-based prompts, totaling 33,361 university curriculum concepts and 116,887 prompts.
Summary of strengths: 1. Targeted Evaluation: MALAMUTE addresses the limitations of existing benchmarks by focusing on fine-grained, domain-specific knowledge, making it a valuable tool for assessing LLMs' capabilities in an educational context.

2. Multilingual Scope: By covering three different languages, MALAMUTE provides insights into how LLMs perform across linguistic boundaries, adding depth to the evaluation process.

3. Comprehensive Coverage: With content sourced from 71 textbooks and spanning multiple academic disciplines, MALAMUTE offers a broad yet detailed assessment of LLMs’ knowledge.

1. Comprehensive and Granular Dataset: MALAMUTE is a highly detailed and extensive dataset that covers a wide range of educational domains and subdomains. With 33,361 university curriculum concepts and 116,887 prompts, it provides a thorough and nuanced tool for evaluating language models' knowledge in specific, granular areas. 

2. Multilingual and Template-Free Design: The dataset is multilingual, including probes in English, Spanish, and Polish, which broadens its applicability and relevance across different linguistic contexts.
Summary of weaknesses: 1. Potential Bias in Probes: Although the dataset aims to be template-free, the subjective nature of writing and reviewing prompts could introduce biases that affect the evaluation outcomes.

1. On line 047, “For instance, an LM may excel in general knowledge of mathematics, but struggle with the nuances of calculus or algebra.” Please provide a citation to support this.

2. While the MALAMUTE dataset contains a large number of concepts and prompts across multiple domains and subdomains, it is largely based on textbook content, which may not adequately cover non-textbook forms of educational knowledge such as hands-on practice, laboratory skills, and case studies.

3. During the quality control process, although the quality of the prompts was assessed through manual review, the review was based on a sample of only 495 English prompts. This is a relatively small sample size and may not fully reflect the quality of the entire dataset.

4. The paper mainly focuses on annotator consistency in the error analysis section but does not analyze the specific causes of model prediction errors in sufficient depth. For example, the patterns and trends of model errors across domains, subdomains, or concepts were not explored in detail, nor was the relationship between model errors and factors such as dataset characteristics and prompt design analyzed. This makes it difficult to gain insight into the weaknesses of the model in the assessment of educational knowledge and thus to provide more targeted recommendations for model improvement and optimization.

5. I don't quite understand why the dog image in Figure 1 is included.
Comments, suggestions and typos: See Weaknesses.

see Summary Of Weaknesses
Overall assessment scores: 3.5

2.5