On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures
Authors: Minh Duc Bui, Kyung eun Park, Goran Glavaš, Fabian David Schmidt, Katharina von der Wense
Accepted at: acl25

On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures

Minh Duc Bui△ Kyung Eun Park♣ Goran Glavaš◊
Fabian David Schmidt◊ Katharina von der Wense▼▲
△Johannes Gutenberg University Mainz, Germany ♣University of Mannheim, Germany
◊Center For Artificial Intelligence and Data Science, University of Würzburg, Germany
▲University of Colorado Boulder, USA
{minhducbui, k.vonderwense}@uni-mainz.de kyung.eun.park@uni-mannheim.de {fabian.schmidt, goran.glavas}@uni-wuerzburg.de

Abstract
Measurement systems (e.g., currencies) differ across cultures, but the conversions between them are well defined so that humans can state facts using any measurement system of their choice. Being available to users from diverse cultural backgrounds, large language models (LLMs) should also be able to provide accurate information irrespective of the measurement system at hand. Using newly compiled datasets we test if this is the case for seven open-source LLMs, addressing three key research questions: (RQ1) What is the default system used by LLMs for each type of measurement? (RQ2) Do LLMs’ answers and their accuracy vary across different measurement systems? (RQ3) Can LLMs mitigate potential challenges w.r.t. underrepresented systems via reasoning? Our findings show that LLMs default to the measurement system predominantly used in the data. Additionally, we observe considerable instability and variance in performance across different measurement systems. While this instability can in part be mitigated by employing reasoning methods such as chain-of-thought (CoT), this implies longer responses and thereby significantly increases test-time compute (and inference costs), marginalizing users from cultural backgrounds that use underrepresented measurement systems.

1 Introduction
The user base of large language models (LLMs) expands to people from a wide range of cultural backgrounds, with each person bringing distinct perspectives to their interactions with these ubiquitous systems. Intuitively, users expect that data can be correctly presented in ways that align with their cultural context, including measurement systems that they are used to. For instance, individuals accustomed to the metric system might find it challenging to interpret imperial units.

Prior work shows that LLMs encode factual information, including plethora of measurement values, within their parameters (Petroni et al., 2019; Geva et al., 2023; Allen-Zhu and Li, 2024, inter alia). However, while an LLM may accurately present facts within one cultural context (presumably the one that dominated their respective training corpora), an inclusive system should be able to do so in all contexts. By failing to do so, LLMs would disadvantage users from cultural backgrounds that are less represented in their training data. This raises a critical, yet underinvestigated question: can LLMs effectively generalize across socio-cultural contexts, offering same levels of factual correctness irrespective of the concrete culturally-bound factors such as measurement systems? Or do they exhibit biases that hinder sociocultural adaptability, preventing them from equitably providing accurate information to all users?

In this work, we demonstrate that LLMs are less reliable when the users’ measurement system differs from the dominant system of the LLM. We find that LLMs typically default to the measurement that fits the cultural context of the data, e.g., reporting fiscal values in USD as well as length and

![A diagram showing a measurement system with four questions about ground beef prices and GDP per capita in different measurement systems, and two robots, one correct and one incorrect.](page_728_355_627_246.png)

Figure 1: We test whether LLMs can translate facts across different measurement systems and show that LLMs often struggle to provide accurate information for cultural contexts underrepresented in training data.
weight in the metric or imperial system, depending on the context of the data. For instance LLMs report food prices of US goods in USD normalized per pounds. The main issue arises when users naively prompt the models with systems that differ from the dominant one, as LLMs often fail to successfully translate values from one measurement into another (e.g., monetary amounts from USD to SDG). While advanced prompting techniques such as chain-of-thought reasoning largely mitigate these conversion errors (e.g., from 72% to almost no difference for currency for the largest LLMs in our evaluation), we need to note that this improvement comes at a substantial cost. Under CoT, models produce much longer, more verbose responses as they triangulate the conversion process step-by-step. For example, obtaining the correct value for the GDP of a country in Eritrean Nakfa can increase the inference cost by up to 300%1 compared to requesting the same value in US Dollars. Such issues have severe implications for users from marginalized cultural backgrounds, who are meant to benefit the most from equitable access to language technology. When LLMs misrepresent or inaccurately convert critical measurements (e.g., wrongly measuring monetary value of consumer goods), they alienate users by failing to align with their cultural context. Moreover, as these users frequently come from economically disadvantaged regions, they are disproportionately affected by the increased inference costs associated with extended model responses.

We address the following research questions in this work (together with respective contributions).

(RQ1) What measurement do LLMs default to?
We curate datasets that span diverse cultural contexts, including fiscal data, food prices, and city distances across different countries and regions. We find that LLMs default to the measurement that fits the cultural context of the data. When queried on distances between German and US cities, they provide estimates in the metric and imperial units, respectively. Likewise, LLMs express fiscal data in USD, the common convention for financial data.

(RQ2) Do LLMs’ answers and their accuracy vary across measurement systems?
We test whether varying measurement systems affects factual accuracy of answers provided by LLMs (as illustrated in Figure 1). We find that LLMs exhibit significant variability in behavior depending on the measurement system embedded in the prompt. Models perform best when the query aligns with the measurement system that fits the cultural context of the data (e.g., provided by a location or entity mentions). For instance, models show a strong decline in accuracy when asked about food prices in pounds for a country that uses kilograms.

(RQ3) Can LLMs mitigate these challenges with reasoning?
We find that LLMs often struggle to generate accurate facts across different measurement systems. Nevertheless, very large LLMs (70B parameters) can successfully perform sequential single-hop inferences, starting with the default measurement and then converting the value to the target unit. This exemplifies that LLMs necessitate additional guidance to reliably triangulate to the correct value in the target measurement. Explicitly prompting LLMs to reason across measurement systems similarly stabilizes performance. Yet, such reasoning remains anchored in the default measurement, increasing inference costs for queries reflecting non-default cultural contexts.

2 Measurement Systems & Datasets

2.1 Measurement Systems
Each dataset is rooted in a specific cultural context. For instance, maps often use local units of measurement, international fiscal data such as GDP is typically reported in USD, and food prices are commonly expressed in local currencies. While previous studies have explored the geographical biases in LLMs’ factual retrieval (Manvi et al., 2024; Moayeri et al., 2024), there has been limited investigation into whether these models can effectively adapt facts to diverse socio-cultural contexts. For example, can a model adapt fiscal data to a different currency or report city distances using an alternative length system? To address this gap, we focus on three measurement systems: currency, length systems, and weight systems. In Appendix A.1, we describe each system in depth and report how often each remains in use nowadays.

2.2 Datasets
Table 1 provides an overview of the measurement systems employed in each dataset, including details on their total sizes and representative examples. This dataset is released under the

1Our estimates are based on API pricing from https://groq.com/pricing/ for Llama 3.3 70B.
<table>
  <tr>
    <th></th>
    <th>Fiscal Data</th>
    <th>City Distances</th>
    <th>Food Prices per Weight</th>
  </tr>
  <tr>
    <th>Measurement System Units</th>
    <td>Currency<br>USD, EUR, VND, KRW, CNY, JPY, ...</td>
    <td><i>Length Systems</i><br>Kilometer (Metric), Mile (Imperial), Cape Foot (South African), Wa (Thai), Li (Chinese), ...<br>10</td>
    <td><i>Weight Systems</i><br>Kilogram (Metric), Pound (Imperial), Jin (Chinese), Geun (Korean), Catty/Kati (Malaysian), ...<br>10</td>
  </tr>
  <tr>
    <th>Number of Systems</th>
    <td>112</td>
    <td>31200 (Kilometer), 7800 (Mile)<br>3120 (Kilometer), 780 (Mile)</td>
    <td>24130 (Kilogram), 3690 (Pound)<br>2413 (Kilogram), 369 (Pound)<br><b>What is the price of one Korean Geun of steak (beef) in Nigeria, in Nigerian Naira (NGN)?</b></td>
  </tr>
  <tr>
    <th>Total Samples (Facts × Systems)</th>
    <td>16576</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Number of Facts</th>
    <td>148</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Example Question</th>
    <td><i>What is GDP per capita for Australia in 2021 in JPY?</i></td>
    <td><i>What is the distance between New York and Los Angeles in Li (Chinese)?</i></td>
    <td></td>
  </tr>
</table>

Table 1: Measurement System Datasets: Measurement system and units, as well as sample size by dataset type. For further information about each measurement system, see Appendix Table 5.

CC BY-NC-ND 4.0 license. For complete details on usage terms and proper attribution, please refer to Appendix A.2. Code and dataset are available at https://github.com/MinhDucBui/MeasurementSystemBias.

Fiscal Data. To analyze currency systems, we collect international fiscal data focused on GDP per capita. We obtain the figures for 148 countries from the World Bank (The World Bank, 2024) and specifically use 2021 data to align with the models’ knowledge cutoff, while at the same time minimizing recency biases (Moayeri et al., 2024). We obtain the average monthly exchange rates for 2021 from the International Monetary Fund to convert these fiscal values into other currencies (International Monetary Fund, 2024). To mitigate the impact of exchange rate fluctuations, we select the most favorable rate within that year, ensuring optimal performance. Note that fiscal data is typically reported in USD (Krugman, 1984).

Food Prices per Weight: Kilogram System. We differentiate between countries that use the “Kilogram” (metric) system and the “Pound” (imperial) system. In countries that use the “Kilogram” system, we collect food commodity prices for items such as maize, rice, beans, and sugar reported per kilogram (e.g., price of 1 kg of rice), expressed in the respective local currency. This data stems from World Food Program via HDX (World Food Program, 2025), covers 76 countries, and spans from 2010 to 2021.

Food Prices per Weight: Pound System. We obtain food price data for United States from the Bureau of Labor Statistics (U.S. Bureau of Labor Statistics, 2025). This dataset includes retail prices for a wide range of items, including fruits, meats, and other commodities, expressed in pound (e.g., the price of 1 pound of chicken).

City Distances: Kilometer System. We begin by selecting four countries in which the “Kilometer” (metric) system is the primary measurement system: Germany, Russia, China and Japan. For each country, we randomly select 40 cities from among its 80 largest cities. We then compute the pairwise distances between all selected cities within each country, resulting in 3,120 distance measurements in total. The shortest straight-line distances between cities were calculated using the Haversine formula, with city coordinates sourced from SimpleMaps (2025).

City Distances: Mile System. For the United States, where the “Mile” (imperial) system applies, we likewise compile a list of 80 largest cities by population and calculate the shortest road distances as for the “Kilometer” system.

3 Stability across Measurement Systems

We begin our investigation of RQ1 by analyzing what measurement systems LLMs default to when prompted without any instruction on the use of measurement. Next, we investigate RQ2 by examining whether LLM behavior shifts when instructed to use a measurement system different from its default and whether this affects performance.

3.1 Experimental Setup

Models. We test instruct-finetuned LLMs from different model families and of varying sizes. Specifically, we probe Qwen2.5 (72B, 7B) (Owen et al., 2025), Llama 3.3 (72B), Llama 3.1 (72B, 8B) (Grattafiori et al., 2024) and Aya Expanse (32B, 8B) (Dang et al., 2024). We group the models by
<table>
  <tr>
    <th>System</th>
    <th>Large</th>
    <th>Small</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th colspan="4">Currency (Fiscal Data)</th>
  </tr>
  <tr>
    <td>Local Currency</td>
    <td>18%</td>
    <td>16%</td>
    <td>17%</td>
  </tr>
  <tr>
    <td><i>Other</i></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>= USD</td>
    <td><b>76%</b></td>
    <td><b>82%</b></td>
    <td><b>80%</b></td>
  </tr>
  <tr>
    <td>= Rest</td>
    <td>6%</td>
    <td>2%</td>
    <td>3%</td>
  </tr>
  <tr>
    <th colspan="4">Weight Systems (Food Prices)</th>
  </tr>
  <tr>
    <th>Data: Kilogram</th>
    <td>100%</td>
    <td>100%</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>Pound</td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td><i>Geun, Jin, ...</i></td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <th>Data: Pound</th>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>Kilogram</td>
    <td>100%</td>
    <td>100%</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>Pound</td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td><i>Geun, Jin, ...</i></td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <th colspan="4">Length Systems (City Distances)</th>
  </tr>
  <tr>
    <th>Data: Kilometer</th>
    <td>100%</td>
    <td>100%</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>Mile</td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td><i>Cape Foot, Wa, ...</i></td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <th>Data: Mile</th>
    <td>37%</td>
    <td>22%</td>
    <td>27%</td>
  </tr>
  <tr>
    <td>Kilometer</td>
    <td>63%</td>
    <td>78%</td>
    <td>73%</td>
  </tr>
  <tr>
    <td><i>Cape Foot, Wa, ...</i></td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
</table>

Table 2: LLMs’ Default Measurement Systems: We report the total percentage of measurement systems that LLMs default to when given the freedom to choose.

their size into ‘Large’ (>=70B) and ‘Small’ (<70B). For more information, see Appendix B.2.

Prompts. To first isolate the effect of different measurement systems, we use English prompts in our experiments, ruling out the language of the prompt as a confounding factor. In §3.3, we also explore multilingual prompts. For robustness, we generate three prompt variants for each task. In each variant, the model is instructed to adhere to a specific measurement system, as illustrated by the following example:

Estimate the GDP per capita for Germany in the year 2021, expressed in the currency ERN. Your answer must only include the GDP per capita [...]

We refer to Appendix B.1 for further details. Additionally, we specify the original name and country of origin for certain length and weight systems.

3.2 RQ1: Default Systems

We first analyze which measurement system the model defaults to. To that end, we let the model generate its response without the instruction to output the fact in a specific measurement.

Results. Table 2 summarizes the findings on default systems. For fiscal data, LLMs predominantly default to using USD (80% on average), and almost exclusively rely on either USD or local currency (17% on average). For weight measurements, LLMs strictly employ either the imperial or metric system, in line with the data context. In the case of length data, distances in Germany, Japan, Russia and China are reported solely in the metric system, whereas, for city distances in the USA, 73% are expressed in the imperial system.

Discussion. Our observations suggest that LLMs tend to adopt measurement systems that mirror the conventions present in the data, often defaulting to Western standards, such as using USD for fiscal figures. This trend seems to reflect the broader influence of the West, emerging not through any direct global mandate but through market dynamics shaped by Western forces2 (Krugman, 1984). Interestingly, Large and Small LLMs exhibit very similar behavior, with US city distances as the only (partial) exception: Small LLMs default more consistently (+15%) to the American cultural context, i.e., the imperial system.

3.3 RQ2: Disparity & Performance across Measurement Systems

We evaluate the stability of LLMs’ responses across measurement systems by explicitly instructing them to produce outputs in a specified system. For each dataset, the default system is the one which LLMs used most frequently in §3.2 (Table 2). We then use the performance with the default system as a baseline for the alternative systems.

Evaluation Metric. We evaluate a model’s performance by comparing its predictions to the ground truth. Because the magnitude of answers varies considerably, we resort to an inverse mean absolute percentage deviation (MAPD). To mitigate the impact of extreme predictions—which LLMs may occasionally produce—we cap the deviation at a maximum of 100%. The final metric is defined as follows:

\[
APD_i^k = \min \left( \left| \frac{p(x_i, f^k)^z - y_i^z}{y_i^z} \right|, 1 \right),
\]

\[
\text{MAPD}^k = 100 \times \left( 1 - \frac{1}{N} \sum_{i=1}^N APD_i^k \right).
\]

2For example, before World War I, the pound sterling was the leading international currency; later, the dollar and the pound shared this role, and eventually the dollar dominated during the Bretton Woods era.
Figure 2: Disparity and Performance of LLMs Across Different Measurement Systems: We report the performance in terms of MAPD (cf. §3.3). The best-performing system (higher is better) for each model is highlighted with bold points. Fully filled points indicate statistically significant improvement over the second-best system, whereas half-filled points do not. We report the raw numbers and their significance in Appendix Table 6.

Here, \( p(x_i, f^k) \) is the model’s prediction for sample \( i \) using system \( k \), and \( p(x_i, f^k)^z \) represents its conversion to measurement system \( z \). The ground truth value for sample \( i \), also expressed in system \( z \), is denoted by \( y_i^z \). We note that higher values indicate better performance. An inverse MAPD of 76.0 indicates approximately that the model’s estimates deviate from the ground truth by 24%.

We test whether the behavior of two systems differs significantly by comparing their paired absolute percentage deviation distributions over samples. We employ the non-parametric Wilcoxon Signed-Rank Test (Wilcoxon, 1945), which establishes whether the median difference between the paired values significantly deviates from zero. We use a high significance threshold of \( p = 0.001 \).

Results. Table 2 summarizes our findings. On average across all models, the default system consistently delivers the best performance. Among 35 model–dataset combinations, it outperforms all others in 28 cases, with statistically significant gains over the second-best system in 22 of those. For example, in the fiscal dataset, using USD yields the highest performance for every single model (average MAPD of 63%). Similarly, in the weight and length system dataset, models overwhelmingly yield optimal results with the default system. To illustrate, Llama 3.3 70B achieves a 57% MAPD with the default “Kilogram” system, but its performance drops significantly to 29% when using the “Pound” system. In the length dataset, Llama 3.3 70B achieves a 75% MAPD on cities with the default “Kilometer” system and shows a 4-point performance drop when evaluated with the “Mile” system. Only for US cities, we observe a smaller average difference from the “Kilometer” system and with Qwen2.5 7B being the only model to show a statistically significant improvement when using this alternative system. While other models also perform better with certain alternative systems, their improvements are not statistically significant. Finally, we observe uneven performance drops across alternative measurement systems, as reflected in the drop of average scores.

Discussion. In sum, we find that current LLMs are not stable across different measurement sys-
Figure 3: Currency Performance Mapped and Categorized by Income Groups: We visualize the performance for all currencies—averaged across large models (over 70B parameters)—on a world map, with countries categorized by income levels.

tems; LLMs yield the best performance when used with their default system and generally suffer significant and pronounced drops in performance for alternative systems. Such issues have severe implications for users from marginalized cultural backgrounds, who stand to benefit the most from equitable access to language technology. When LLMs misrepresent or inaccurately convert measurements, e.g., wrongly state the value of consumer goods in local currency, they alienate users by failing to correctly adapt to their cultural context.

Analysis: Currency Bias. We visualize currency performance of our biggest LLMs (>=70B) on a world map, associating each country with its respective currency. Additionally, we group countries by income levels provided by the World Bank (Fantom and Serajuddin, 2016). We immediately observe obvious disparities across income groups. LLMs achieve an average performance of 43% with currencies from high-income countries, which is an impressive 34% better than with currencies from low-income countries. This constitutes a clear bias that favors the richest (i.e., their currencies) and affects the poorest. As already highlighted in §3.3, using non-default systems adversely affects LLMs’ performance: this analysis reveals that this effect disproportionately disadvantages low-income countries.

Analysis: Multilingual Setting. We further examine whether language affects our findings by aligning the prompt language with the measurement system used. For instance, when using the Korean currency, we analyze the performance gap between the Korean currency and USD (default measurement system) under Korean prompt language. For each dataset, we select two languages from among Turkish (TR), Korean (KR), Chinese (ZH), and Japanese (JA)—representing high- to medium-resource languages. For each language, a native speaker reviews our prompts to ensure correctness. We report the results in Table 3.

<table>
  <tr>
    <th>Setting</th>
    <th>Llama 3.3 70B</th>
    <th>Qwen2.5 72B</th>
    <th>Aya 32B</th>
  </tr>
  <tr>
    <th colspan="4">Currency (Fiscal Data)</th>
  </tr>
  <tr>
    <th colspan="4">Default=USD</th>
  </tr>
  <tr>
    <td>Δ<sub>KR</sub> (KRW)</td>
    <td>-37.89*</td>
    <td>-27.43*</td>
    <td>-36.36*</td>
  </tr>
  <tr>
    <td>Δ<sub>TR</sub> (TRY)</td>
    <td>-57.01*</td>
    <td>-42.22*</td>
    <td>-49.13*</td>
  </tr>
  <tr>
    <th colspan="4">Weight System (Food Prices)</th>
  </tr>
  <tr>
    <th colspan="4">Default=Kilogram</th>
  </tr>
  <tr>
    <td>Δ<sub>KR</sub> (Geun)</td>
    <td>-21.99*</td>
    <td>-31.12*</td>
    <td>-22.11*</td>
  </tr>
  <tr>
    <td>Δ<sub>ZH</sub> (Jin)</td>
    <td>-25.39*</td>
    <td>-26.79*</td>
    <td>-11.33*</td>
  </tr>
  <tr>
    <th colspan="4">Default=Pound</th>
  </tr>
  <tr>
    <td>Δ<sub>KR</sub> (Geun)</td>
    <td>-3.70</td>
    <td>-12.54*</td>
    <td>-39.14*</td>
  </tr>
  <tr>
    <td>Δ<sub>ZH</sub> (Jin)</td>
    <td>-1.31</td>
    <td>-20.25*</td>
    <td>-2.29</td>
  </tr>
  <tr>
    <th colspan="4">Length System (City Distances)</th>
  </tr>
  <tr>
    <th colspan="4">Default=Kilometer</th>
  </tr>
  <tr>
    <td>Δ<sub>JA</sub> (Ken)</td>
    <td>-80.09*</td>
    <td>-75.71*</td>
    <td>-73.90*</td>
  </tr>
  <tr>
    <td>Δ<sub>ZH</sub> (LI)</td>
    <td>-20.19*</td>
    <td>-13.18*</td>
    <td>-15.03*</td>
  </tr>
  <tr>
    <th colspan="4">Default=Mile</th>
  </tr>
  <tr>
    <td>Δ<sub>JA</sub> (Ken)</td>
    <td>-89.68*</td>
    <td>-86.97*</td>
    <td>-84.98*</td>
  </tr>
  <tr>
    <td>Δ<sub>ZH</sub> (LI)</td>
    <td>-30.33*</td>
    <td>-35.26*</td>
    <td>-35.45*</td>
  </tr>
</table>

Table 3: Aligning Language with Measurement System: We compare performance between the default system and the system aligned with the prompt language (e.g., KRW vs. USD for Korean prompts). The language is shown as a subscript, and the aligned system in brackets. Red cells mark significant differences.

Income Country | MAPD
<table>
  <tr>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>High Income</td>
    <td>42.80%</td>
    <td></td>
  </tr>
  <tr>
    <td>Upper Middle Income</td>
    <td>19.79%</td>
    <td></td>
  </tr>
  <tr>
    <td>Lower Middle Income</td>
    <td>14.31%</td>
    <td></td>
  </tr>
  <tr>
    <td>Low Income</td>
    <td>9.03%</td>
    <td></td>
  </tr>
</table>
is aligned, significant drops persist in nearly all cases, highlighting the significant instability of measurement systems independent of language in almost all cases.

Figure 4: Visualization of Multi-Hop Reasoning: Answering w.r.t. an alternative measurement system requires multi-hop reasoning, consisting of the following steps: (i) retrieving the answer in the default system, (ii) retrieving the conversion rate between the default and alternative system, and (iii) combine the results of (i) and (ii) with arithmetic reasoning.

4 RQ3: Multi-Hop Reasoning

We first explain how answering questions w.r.t. alternative measurement systems represents a multi-hop reasoning problem and then explore whether the LLMs’ performance changes with (i) a sequence of prompts that aim at individual reasoning steps and (ii) with chain-of-thought prompts, i.e., by explicitly asking the LLMs to provide the explanation for their final answer.

Multi-Hop Questions. We adopt the definition and notation from Mavi et al. (2024): A multi-hop question (MHQ) is a question that cannot be answered using a single document or passage (context); instead, it requires combining information from multiple sources. For example, a question like "What is the capital of the country whose currency has the highest exchange rate against the US dollar?" necessitates gathering and synthesizing information from several contexts.

Formally, let \( \mathbb{C} \) denote the set of all contexts, \( \mathcal{Q} \) the set of all questions, and \( \mathcal{A} \) the set of all possible answers. For a question \( q \in \mathcal{Q} \) and a subset of related contexts \( C \subseteq \mathbb{C} \), the task is to approximate a function

\[
f : \mathcal{Q} \times 2^{\mathbb{C}} \to \mathcal{A} \cup \{ \Phi \},
\]

such that

\[
f(q, C) = \begin{cases}
a \in \mathcal{A} & \text{if there exists a subset } \\
& P_q = \{p_1, \ldots, p_k\} \subseteq C, \\
& \text{with } k > 1, \text{ such} \\
& \text{that } P_q \models (a \text{ answers } q), \\
\Phi & \text{otherwise.}
\end{cases}
\]

where \( \models \) denotes entailment and \( \Phi \) is returned when \( q \) is unanswerable using \( C \). The requirement that \( k > 1 \) ensures that the question is not solvable in a single hop (i.e., with \( k = 1 \), the task reduces to traditional QA).

When we ask questions of the form “*What is <Fact> in <System>*?”, where *<Fact>* represents information inherently associated with a particular measurement system (e.g., GDP is intrinsically tied to currency) and the system is not the default one for the model, we are effectively asking the model to solve a multi-hop reasoning task consisting of the following steps:

1. *Source Fact Retrieval*: Extract the fact from its original measurement system (the source system, \( S_{src} \)). This constitutes a single-hop operation using one context, \( p_1 \).
2. *Conversion Retrieval*: Obtain the information required to convert the fact from the source system \( S_{src} \) to the target system \( S_{trg} \). This is another single-hop operation that relies on a different context, \( p_2 \).
3. *Arithmetic Reasoning*: Apply the conversion information to the retrieved fact. This requires arithmetic reasoning and may require one or more additional contexts \( \{p_3, \ldots\} \).

In this scenario, the model must integrate multiple contexts \( \{p_1, p_2, p_3, ...\} \) to arrive at an answer, which aligns with our multi-hop definition. Figure 4 provides a visual overview of this process.

Reasoning Chains. Recognizing measurement system questions as multi-hop reasoning problems, we next investigate two strategies that push LLMs to reason: (1) *Sequential Single Hops* explicitly breaks down the problem for the LLM into individual reasoning steps and (2) *Chain-of-Thought (CoT)* prompts implicitly nudge the LLMs to reason by requiring an explanation for the answer.

*Sequential Single Hops (Seq)*. We first ask the LLMs to solve individual hops sequentially. To this end, we structure single-hop queries as illustrated in Figure 4. We feed the hops to the LLM in sequence, integrating the answer from the previous step into the next prompt: e.g., after obtaining the answer to “*What is the exchange rate from USD to
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">Small Models</th>
    <th colspan="4">Large Models</th>
    <th rowspan="2">Avg. Cost</th>
  </tr>
  <tr>
    <th>Avg. Performance</th>
    <th>Default Sys.</th>
    <th>Alternative Systems</th>
    <th>Avg. Performance</th>
    <th>Default Sys.</th>
    <th>Alternative Systems</th>
    <th>Alternative Sys.</th>
  </tr>
  <tr>
    <td>NoReas.</td>
    <td>ΔNoReas.</td>
    <td>ΔSeq.</td>
    <td>ΔCoT</td>
    <td>NoReas.</td>
    <td>ΔNoReas.</td>
    <td>ΔSeq.</td>
    <td>ΔCoT</td>
    <td>ΔSeq.</td>
    <td>ΔCoT</td>
  </tr>
  <tr>
    <th colspan="10">Default: USD (Currency Systems)</th>
  </tr>
  <tr>
    <td>53.43</td>
    <td>-77.45%</td>
    <td>-31.32%</td>
    <td>-44.15%</td>
    <td>74.67</td>
    <td>-72.45%</td>
    <td>-0.12%</td>
    <td>+0.79%</td>
    <td>+189%</td>
    <td>+302%</td>
  </tr>
  <tr>
    <th colspan="10">Default: Kilogram (Weight Systems)</th>
  </tr>
  <tr>
    <td>32.43</td>
    <td>-33.11%</td>
    <td>-17.92%</td>
    <td>-13.88%</td>
    <td>54.33</td>
    <td>-59.13%</td>
    <td>-4.82%</td>
    <td>-8.05%</td>
    <td>+191%</td>
    <td>+206%</td>
  </tr>
  <tr>
    <th colspan="10">Default: Pound (Weight Systems)</th>
  </tr>
  <tr>
    <td>59.08</td>
    <td>-33.97%</td>
    <td>-19.10%</td>
    <td>-33.31%</td>
    <td>75.61</td>
    <td>-29.25%</td>
    <td>-4.72%</td>
    <td>-9.39%</td>
    <td>+199%</td>
    <td>+187%</td>
  </tr>
  <tr>
    <th colspan="10">Default: Kilometer (Length Systems)</th>
  </tr>
  <tr>
    <td>57.00</td>
    <td>-63.64%</td>
    <td>-31.82%</td>
    <td>-41.74%</td>
    <td>72.84</td>
    <td>-57.93%</td>
    <td>-7.88%</td>
    <td>-3.98%</td>
    <td>+180%</td>
    <td>+210%</td>
  </tr>
  <tr>
    <th colspan="10">Default: Mile (Length Systems)</th>
  </tr>
  <tr>
    <td>68.13</td>
    <td>-74.36%</td>
    <td>-34.55%</td>
    <td>-49.95%</td>
    <td>92.46</td>
    <td>-73.13%</td>
    <td>-7.18%</td>
    <td>-7.50%</td>
    <td>+197%</td>
    <td>+185%</td>
  </tr>
</table>

Table 4: Performance and Cost Increases across Reasoning Strategies: *Performance*: We report the percentage change relative to the default system (*Default Sys.*) without reasoning (*NoReas.*, cf. §3.3). We highlight cells in red when the performance drop exceeds 10%. *Cost Increase*: We report the additional cost incurred compared to using no reasoning.

VND in 2021?” we incorporate it into the subsequent query: “*Convert <number> USD to VND.*”

*Chain-of-Thought (CoT)*. We explicitly prompt the LLM to detail its intermediate steps leading to the final answer. Specifically, we prompt the model with: “*What is the GDP of Germany in ERN? Explain first. Be short.*”

**Evaluation.** We report the change in MAPD relative to the default measurement system prompted without any reasoning (as discussed in Section 3.3), as shown in Table 4. Additionally, we examine the cost associated with these reasoning methods. For this purpose, we refer to the pricing provided by an LLM API provider\footnote{https://groq.com/pricing/} for the LLaMA 3.3 70B family and report the cost increase relative to the no-reasoning baseline (right part of Table 4). We further examine which measurement systems, in addition to the targeted system, are referenced in the CoT reasoning (see Figure 5).

**Results.** Small LLMs experience a significant performance drop and show considerable instability, often failing to reason effectively, whereas Large models maintain stability across different measurement systems, with only slight performance deviations from the baseline. However, the cost of reasoning with Large models is considerably higher compared to the no-reasoning baseline. Sequential incur a cost up to 199% of the cost without reasoning, whereas CoT on average increases the test-time compute cost to 300% for currency conversions and around 200% for weight and length system conversions.

**Discussion.** Prompting the model, explicitly (Seq) or implicitly (CoT) to reason does not benefit Small models. For Large models, reasoning does stabilize the performance to the level of the default measurement system without reasoning. This improvement comes at a significant cost: using reasoning increases test-time compute and, in turn, expenses by 180% to 300%, depending on the dimension of cultural context. While reasoning is an effective solution, its **increased cost disproportionately affects users who rely on alternative measurement systems rather than the LLM’s default**. This discrepancy arises as models triangulate the conversion process step-by-step from the default measurement system. When faced with non-default measurement systems, **LLMs engage in additional reasoning steps to convert the result from the default system** (e.g., LLMs start from USD for currency conversion in 91% of the cases; see Figure 5), which demands additional computation. Consequently, users with cultural contexts that are misaligned with the LLMs’ default (Anglo-centric context) face higher costs—this is yet another obstacle that disproportionately hinders access to state-of-the-art language technology for the least developed parts of the world.
Figure 5: Frequency of Measurement Systems as Starting Points in CoT Reasoning. We count how often each measurement system is chosen as the starting point for the model’s CoT reasoning and report the proportion. We average this proportion across all models.

5 Related Work

Biases in Factual Retrieval. Recent research has examined LLMs’ factual recall (Mallen et al., 2023; Kandpal et al., 2023; Sun et al., 2024; Tang et al., 2023, inter alia). While these models often provide correct answers, they can hallucinate (Huang et al., 2025). Studies also show that recall accuracy correlates with an entity’s popularity—with higher error rates for lesser-known subjects (Mallen et al., 2023; Sun et al., 2024)—and reveal geographical biases along regional and income lines (Moayeri et al., 2024; Manvi et al., 2024).

In contrast to prior work, our study shifts attention to the systems representing facts, noting that while measurement systems may influence depiction, the underlying facts remain unchanged. Park et al. (2022) demonstrate that small encoder models struggle with comparisons across different measurement systems. Additionally, Dinu et al. (2020) explicitly incorporates measurement systems into translation models. However, unlike these studies, our work focuses on decoder-based LLMs, emphasizing factual retrieval rather than simple comparisons or translations.

Cultural Biases in LLMs. Kirk et al. (2024) underscore the disparities in how LLMs are used across diverse cultural contexts. However, numerous studies have quantified and identified cultural biases in LLMs and their multimodal extensions (Cao et al., 2024; Shi et al., 2024; Myung et al., 2024; Bui et al.; 2024, inter alia). The consensus is that LLMs tend to align more closely with Western values, which limits their effectiveness in cross-cultural contexts. For instance, Lee et al. (2024) demonstrate that current LLMs achieve higher accuracies on hate speech labels derived from the Anglosphere. In our work, we further show that the same pattern models also favor measurement systems predominantly used in the Western world.

Multi-Hop Reasoning. Research on multi-hop reasoning has produced numerous datasets and analyses (Yang et al., 2018; Fang et al., 2020; Chen et al., 2020; Trivedi et al., 2022, inter alia). Recent work has shifted toward implicit multi-hop questions that require models to recall and integrate facts without explicit reasoning steps (Press et al., 2023; Biran et al., 2024; Li et al., 2024; Yang et al., 2024). Although LLMs can connect internal knowledge, they still struggle when key reasoning details are absent (Press et al., 2023; Dziri et al., 2023). Moreover, while prior work mainly examines concatenated, independent facts, our analysis shows that natural multi-hop questions from measurement systems present unique challenges.

6 Conclusion

In this work, we show that LLMs exhibit significant variability in performance based on the measurement system embedded in the prompt for factual retrieval. Specifically, performance declines when the model encounters non-default measurement systems. We show that reasoning (e.g., CoT) improves the model’s ability to handle alternative systems questions but it leads to an increase in test-time compute, which disproportionately affects users whose cultural contexts are misaligned with the LLMs’ defaults.

Limitations

In this study, we first prompt LLMs in English to examine how measurement systems affect factual retrieval accuracy. However, the interplay between language and measurement systems and its impact on performance remains an important avenue for future research. Users from underrepresented communities are much more likely to interact with dialogue systems in their native languages. Nonetheless, investigating how well LLMs generalize across measurement systems in English represents a ‘best-case’ scenario, given that these models perform best in English. Consequently, our core
results would be even more pronounced in other languages, where LLMs are likely to mistranslate measurements more frequently and significantly, and incur higher inference costs due to tokenization into more subwords.

Similarly, while our investigation examines a selected subset of measurement systems and units, we acknowledge the challenges of accessing comprehensive data—especially regarding systems still actively used within marginalized groups. Despite these constraints, our work successfully highlights key dynamics and interactions. We view this as a promising starting point that opens up opportunities for expanding the scope of research to include a broader array of measurement systems.

Finally, we acknowledge that there are more effective tools for retrieving facts and performing unit conversions than solely relying on LLMs, such as RAG systems or dedicated conversion tools. However, we emphasize that users are more inclined to interact with a simple LLM chatbot for these seemingly straightforward tasks rather than resorting to more complex tools.

Ethical Statement

The sources of our data, including the World Bank, IMF, and HDX, may reflect biases inherent in data collection methodologies, economic models, or reporting structures. These biases could influence the outcomes of analyses based on the dataset. Any model trained or analyzed solely on our dataset may inherit assumptions and limitations from the data sources.

We use AI assistants, specifically o3-mini, to help edit sentences in our paper writing.

Acknowledgments

The work of Minh Duc Bui and Katharina von der Wense was funded by the Carl Zeiss Foundation, grant number P2021-02-014 (TOPML project).

References

Zeyuan Allen-Zhu and Yuanzhi Li. 2024. Physics of language models: Part 3.1, knowledge storage and extraction. Preprint, arXiv:2309.14316.

Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. 2024. Hopping too late: Exploring the limitations of large language models on multi-hop queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14113–14130, Miami, Florida, USA. Association for Computational Linguistics.

Minh Duc Bui, Katharina von der Wense, and Anne Lauscher. 2024. Multi3hate: Multimodal, multilingual, and multicultural hate speech detection with vision-language models. Preprint, arXiv:2411.03888.

Yong Cao, Yova Kementchedjhieva, Ruixiang Cui, Antonia Karamolegkou, Li Zhou, Megan Dare, Lucia Donatelli, and Daniel Hershcovich. 2024. Cultural adaptation of recipes. Transactions of the Association for Computational Linguistics, 12:80–99.

Wenhui Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036, Online. Association for Computational Linguistics.

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kubik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

Georgiana Dinu, Prashant Mathur, Marcello Federico, Stanislas Lauly, and Yaser Al-Onaizan. 2020. Joint translation and unit conversion for end-to-end localization. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 265–271, Online. Association for Computational Linguistics.

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: limits of transformers on compositionality. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.

Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuhang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8823–8838, Online. Association for Computational Linguistics.

Neil Fantom and Umar Serajuddin. 2016. The world bank’s classification of countries by income. Policy Research Working Paper 7528, World Bank. License: CC BY 3.0 IGO.

Mor Geva, Jasmin Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in
Natural Language Processing, pages 12216–12235, Singapore. Association for Computational Linguistics.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).

International Monetary Fund. 2024. International Financial Statistics: Exchange Rates. Accessed: 2024-02-13.

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.

Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. In Advances in Neural Information Processing Systems, volume 37, pages 105236–105344. Curran Associates, Inc.

Paul R. Krugman. 1984. The international role of the dollar: Theory and prospect. In Exchange Rate Theory and Practice, pages 261–278. National Bureau of Economic Research (NBER).

Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. Exploring cross-cultural differences in English hate speech annotations: From dataset construction to analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4205–4224, Mexico City, Mexico. Association for Computational Linguistics.

Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024. Understanding and patching compositional reasoning in LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9668–9688, Bangkok, Thailand. Association for Computational Linguistics.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.

Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. 2024. Large language models are geographically biased. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.

Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2024. Multi-hop question answering. Preprint, arXiv:2204.09140.

Mazda Moayeri, Elham Tabassi, and Soheil Feizi. 2024. Worldbench: Quantifying geographic disparities in llm factual recall. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24, page 1211–1228, New York, NY, USA. Association for Computing Machinery.

Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoti, Eunsu Kim, Carla Perez-Almendros, Abinev Ali Ayele, Victor Gutierrez Basulto, Yazmin Ibanez-Garcia, Hwaran Lee, Shamsudeen H Muhammad, Kiwoong Park, Anar Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, and 3 others. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. In Advances in Neural Information Processing Systems, volume 37, pages 78104–78146. Curran Associates, Inc.

Sungjin Park, Seungwoo Ryu, and Edward Choi. 2022. Do language models understand measurements? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1782–1792, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics.

Qwen, ., An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin
Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Abreu De Paula, and Diyi Yang. 2024. CultureBank: An online community-driven knowledge base towards culturally aware language technologies. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4996–5025, Miami, Florida, USA. Association for Computational Linguistics.

SimpleMaps. 2025. World cities database. https://simplemaps.com/data/world-cities. Accessed: 2025-05-28.

Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. Head-to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 311–325, Mexico City, Mexico. Association for Computational Linguistics.

Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11626–11644, Toronto, Canada. Association for Computational Linguistics.

The World Bank. 2024. GDP per capita (current US). Accessed: 2024-02-13.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

U.S. Bureau of Labor Statistics. 2025. Consumer price index (cpi). Accessed: 2025-02-13.

Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80–83.

World Food Program. 2025. Consumer price index (cpi). Accessed: 2025-02-13.

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210–10229, Bangkok, Thailand. Association for Computational Linguistics.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.

A Dataset

A.1 Measurement Systems

The exhaustive list of measures used in the dataset are shown in Table 7.

For ease of understanding, we additionally summarize the usage context and prevalence of each system in Table 5.

A.2 Terms of Use

Our research is conducted in the public interest under GDPR, meeting the criteria for substantial academic research. To respect all data providers, our published work is licensed under CC BY-NC-ND 4.0. We use datasets with the following licenses:

World Bank Group: The fiscal data is provided under an Access to Information policy and distributed as CC-BY 4.0.

Exchange Rates: Data from the IMF is available for download, extraction, and use under its open terms.

Food Prices: HDX and World Food Programme data use the Creative Commons Attribution for Intergovernmental Organisations license, while U.S. food prices from the Bureau of Labor Statistics are in the public domain.

City Distances: Data from the Simplemaps domain is included in approximate form for reproducibility, with proper attribution as required by their terms.

B Experimental Setup

B.1 Prompts

We report all prompts used in this paper.

Base Prompts (No Reasoning). We create three base prompt variants for the no reasoning part:

• Estimate the <MEASUREMENT> in year <YEAR>. Your answer must only include the <MEASUREMENT> as a decimal number without any abbreviations and explanation. Final number:

• Approximate the <MEASUREMENT> in year <YEAR>. Your answer must only include the <MEASUREMENT> as a decimal
<table>
  <tr>
    <th>System</th>
    <th>Usage Description</th>
    <th>Usage Level</th>
  </tr>
  <tr>
    <th colspan="3">Currencies</th>
  </tr>
  <tr>
    <td>ALL</td>
    <td>Currencies are used in every day life</td>
    <td><i>High</i></td>
  </tr>
  <tr>
    <th colspan="3">Weight System</th>
  </tr>
  <tr>
    <td>Kilogram<br>Pound<br>Geun (Korea)<br>Arrâtel (Portuguese)<br>Funt (Russia)<br>Peitha<br>Troy pound<br>Jin (Mainland China)<br>Catty/Kati (Malaysian)<br>Sër (Pakistani)</td>
    <td>Official SI base unit of mass<br>Imperial/US customary unit<br>Korean traditional unit; still used in markets for meat.<br>Historical Portuguese unit; obsolete.<br>Used historically in local trade and marketplaces; obsolete.<br>Burmese traditional unit, still used in Myanmar for local trade.<br>Specialized unit used for precious metals like gold and silver.<br>Used in markets, and for measuring gold, silver and Chinese medicines.<br>Traditional weight still used in some local markets in Southeast Asia.<br>Traditional unit used historically in the Indian subcontinent; mostly obsolete.</td>
    <td><i>High</i><br><i>High</i><br>Medium<br>Low<br>Low<br>Medium<br>Medium<br>Low<br>Low</td>
  </tr>
  <tr>
    <th colspan="3">Length Systems</th>
  </tr>
  <tr>
    <td>Kilometer<br>Mile<br>Li (China)<br>Ken (Japan)<br>Eski Mil (Turkish)<br>Vyersta (Russian)<br>Cape Foot (South Africa)<br>Depa (Indonesian)<br>Argentine league (legua)<br>Wa (Thai)</td>
    <td>Used in most countries as the official system.<br>Used primarily in the US for everyday measurements.<br>Traditional Chinese unit of length, historically used in land measurement.<br>Traditional Japanese unit of length, commonly used in architecture.<br>Ottoman-era unit, obsolete in modern Turkey.<br>Historical Russian unit; obsolete.<br>Colonial-era unit, obsolete in contemporary use.<br>Traditional unit based on armspan, now rarely used.<br>Historical unit used for long distances in rural Argentina; obsolete.<br>Traditional Thai unit still used for land measurement.</td>
    <td><i>High</i><br><i>High</i><br>Low<br>Medium<br>Low<br>Low<br>Low<br>Low<br>Low<br>Medium</td>
  </tr>
</table>

Table 5: Overview of diverse measurement systems categorized by type (Currency, Weight, and Length), including their usage descriptions and estimated level of usage. We indicate <i>High</i> for systems that are officially used in the country, <i>Medium</i> for those that are not official but still used in specific local contexts, and <i>Low</i> for systems that are largely obsolete in everyday usage and are considered historical units.

number without any abbreviations and explanation. Final number:

• Guess the <MEASUREMENT> in the year <YEAR>. Your answer must only include the <MEASUREMENT> as a decimal number without any abbreviations and explanation. Final number:

Base Prompts (CoT). To enable Chain-of-Thought (CoT) reasoning, we modify each base prompt. Specifically, we replace the instruction "Your answer must only include the <MEASUREMENT> as a decimal number without any abbreviations and explanation. Final number:" with "Explain first and then give your final answer with 'Answer: <Your answer in decimal number>'. Be short:"

Variable Definitions. For different measurement systems, the variable <MEASUREMENT> is defined as follows:

• Currency: <MEASUREMENT> = GDP per capita of <COUNTRY> in the currency <CURRENCY>

• Weight Systems: <MEASUREMENT> = the price of <FOOD> per <WEIGHTSYSTEM> in <COUNTRY>

• Length System: <MEASUREMENT> = distance between cities <CITY1> and <CITY2> in <COUNTRY> expressed in <LENGTH SYSTEM>

B.2 Model Details

We employ the Hugging Face implementation for all models, which are run on two A100 80GB GPUs. Outputs are generated with a limit of 40 new tokens per request, and each large LLM processes a dataset in approximately 1.5 hours with a batch size of 64. For CoT prompting, we increase the token limit, which extends the processing time to nearly 24 hours.
<table>
  <tr>
    <th></th>
    <th>Llama 3.3</th>
    <th>Llama 3.1</th>
    <th>Qwen2.5</th>
    <th>Aya</th>
    <th>Average</th>
  </tr>
  <tr>
    <th></th>
    <th>70B</th>
    <th>70B</th>
    <th>8B</th>
    <th>72B</th>
    <th>7B</th>
    <th>32B</th>
    <th>8B</th>
  </tr>
  <tr>
    <th colspan="9">Currency (Fiscal Data)</th>
  </tr>
  <tr>
    <th>Default = USD</th>
    <td><b>71.51*</b></td>
    <td><b>75.99</b></td>
    <td><b>31.58*</b></td>
    <td><b>76.51*</b></td>
    <td><b>65.01*</b></td>
    <td><b>63.08*</b></td>
    <td><b>54.05*</b></td>
    <td><b>62.53*</b></td>
  </tr>
  <tr>
    <td>EUR</td>
    <td>62.45</td>
    <td>75.48</td>
    <td>2.84</td>
    <td>62.34</td>
    <td>60.34</td>
    <td>58.55</td>
    <td>49.79</td>
    <td>53.11</td>
  </tr>
  <tr>
    <td>AUD</td>
    <td>61.55</td>
    <td>50.99</td>
    <td>6.66</td>
    <td>60.88</td>
    <td>48.34</td>
    <td>55.08</td>
    <td>50.08</td>
    <td>47.65</td>
  </tr>
  <tr>
    <td>GBP</td>
    <td>59.00</td>
    <td>56.24</td>
    <td>9.20</td>
    <td>62.76</td>
    <td>38.32</td>
    <td>52.67</td>
    <td>49.40</td>
    <td>46.80</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>65.73</td>
    <td>45.59</td>
    <td>6.57</td>
    <td>60.00</td>
    <td>39.68</td>
    <td>51.63</td>
    <td>47.87</td>
    <td>45.30</td>
  </tr>
  <tr>
    <td>SGD</td>
    <td>54.99</td>
    <td>30.16</td>
    <td>8.14</td>
    <td>60.94</td>
    <td>45.94</td>
    <td>58.84</td>
    <td>49.40</td>
    <td>44.06</td>
  </tr>
  <tr>
    <td>NZD</td>
    <td>63.59</td>
    <td>29.40</td>
    <td>7.96</td>
    <td>54.41</td>
    <td>9.48</td>
    <td>59.24</td>
    <td>51.18</td>
    <td>39.32</td>
  </tr>
  <tr>
    <td>LOCAL</td>
    <td>58.55</td>
    <td>60.13</td>
    <td>8.08</td>
    <td>49.49</td>
    <td>23.67</td>
    <td>42.27</td>
    <td>28.64</td>
    <td>38.69</td>
  </tr>
  <tr>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
    <td>(all)</td>
  </tr>
  <tr>
    <td>UZS</td>
    <td>6.15</td>
    <td>0.38</td>
    <td>0.00</td>
    <td>1.44</td>
    <td>0.61</td>
    <td>0.13</td>
    <td>0.01</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>SDG</td>
    <td>2.76</td>
    <td>0.18</td>
    <td>0.11</td>
    <td>2.88</td>
    <td>0.10</td>
    <td>0.17</td>
    <td>0.17</td>
    <td>0.91</td>
  </tr>
  <tr>
    <th colspan="9">Weight System (Food Prices)</th>
  </tr>
  <tr>
    <th>Default=Kilogram</th>
    <td><b>56.94*</b></td>
    <td><b>52.83*</b></td>
    <td><b>11.33</b></td>
    <td><b>53.22*</b></td>
    <td><b>42.67*</b></td>
    <td><b>43.99*</b></td>
    <td><b>31.74</b></td>
    <td><b>41.82</b></td>
  </tr>
  <tr>
    <td>Petitha</td>
    <td>46.48</td>
    <td>8.05</td>
    <td>5.82</td>
    <td>48.69</td>
    <td>32.43</td>
    <td>21.46</td>
    <td>34.73</td>
    <td>28.24</td>
  </tr>
  <tr>
    <td>Sér</td>
    <td>41.28</td>
    <td>17.49</td>
    <td>10.85</td>
    <td>26.31</td>
    <td>30.10</td>
    <td>24.44</td>
    <td>28.81</td>
    <td>25.61</td>
  </tr>
  <tr>
    <td>Pound</td>
    <td>29.04</td>
    <td>28.44</td>
    <td>14.68</td>
    <td>28.81</td>
    <td>27.56</td>
    <td>26.43</td>
    <td>23.40</td>
    <td>25.48</td>
  </tr>
  <tr>
    <td>Catty</td>
    <td>23.95</td>
    <td>10.90</td>
    <td>7.57</td>
    <td>33.71</td>
    <td>32.27</td>
    <td>26.40</td>
    <td>23.13</td>
    <td>22.56</td>
  </tr>
  <tr>
    <td>Geun</td>
    <td>26.52</td>
    <td>11.91</td>
    <td>10.04</td>
    <td>24.36</td>
    <td>31.05</td>
    <td>25.54</td>
    <td>24.58</td>
    <td>22.00</td>
  </tr>
  <tr>
    <td>Funt (Russian)</td>
    <td>19.19</td>
    <td>16.79</td>
    <td>7.51</td>
    <td>24.06</td>
    <td>29.57</td>
    <td>29.39</td>
    <td>24.48</td>
    <td>21.57</td>
  </tr>
  <tr>
    <td>Jin</td>
    <td>17.16</td>
    <td>7.86</td>
    <td>8.29</td>
    <td>28.27</td>
    <td>28.85</td>
    <td>29.88</td>
    <td>15.57</td>
    <td>19.41</td>
  </tr>
  <tr>
    <td>Arrâtel</td>
    <td>20.55</td>
    <td>11.88</td>
    <td>11.04</td>
    <td>26.39</td>
    <td>20.83</td>
    <td>23.74</td>
    <td>20.03</td>
    <td>19.21</td>
  </tr>
  <tr>
    <td>Troy Pound</td>
    <td>6.21</td>
    <td>7.61</td>
    <td><b>15.56</b></td>
    <td>7.58</td>
    <td>18.97</td>
    <td>18.77</td>
    <td>17.09</td>
    <td>13.11</td>
  </tr>
  <tr>
    <th>Default=Pound</th>
    <td><b>72.85</b></td>
    <td><b>73.68</b></td>
    <td><b>62.20*</b></td>
    <td><b>80.29*</b></td>
    <td><b>62.27*</b></td>
    <td><b>66.61*</b></td>
    <td><b>45.25</b></td>
    <td><b>66.16</b></td>
  </tr>
  <tr>
    <td>Funt (Russian)</td>
    <td>68.15</td>
    <td>69.61</td>
    <td>55.22</td>
    <td>74.82</td>
    <td>56.25</td>
    <td>61.25</td>
    <td><b>46.61</b></td>
    <td>61.70</td>
  </tr>
  <tr>
    <td>Kilogram</td>
    <td><b>75.32</b></td>
    <td><b>74.74</b></td>
    <td>45.12</td>
    <td>60.82</td>
    <td>54.66</td>
    <td>42.11</td>
    <td>44.56</td>
    <td>56.76</td>
  </tr>
  <tr>
    <td>Catty</td>
    <td>68.54</td>
    <td>70.46</td>
    <td>34.57</td>
    <td>66.56</td>
    <td>49.91</td>
    <td>47.11</td>
    <td>36.49</td>
    <td>53.38</td>
  </tr>
  <tr>
    <td>Arrâtel</td>
    <td>51.63</td>
    <td>63.14</td>
    <td>38.99</td>
    <td>71.05</td>
    <td>59.68</td>
    <td>48.24</td>
    <td>31.09</td>
    <td>51.98</td>
  </tr>
  <tr>
    <td>Geun</td>
    <td>51.59</td>
    <td>62.01</td>
    <td>33.56</td>
    <td>55.71</td>
    <td>46.59</td>
    <td>32.37</td>
    <td>36.24</td>
    <td>45.44</td>
  </tr>
  <tr>
    <td>Jin</td>
    <td>49.62</td>
    <td>63.81</td>
    <td>22.85</td>
    <td>55.12</td>
    <td>33.75</td>
    <td>38.81</td>
    <td>25.99</td>
    <td>41.42</td>
  </tr>
  <tr>
    <td>Sér</td>
    <td>54.83</td>
    <td>42.14</td>
    <td>6.95</td>
    <td>46.84</td>
    <td>33.07</td>
    <td>36.11</td>
    <td>42.07</td>
    <td>37.43</td>
  </tr>
  <tr>
    <td>Troy Pound</td>
    <td>10.70</td>
    <td>30.20</td>
    <td>57.83</td>
    <td>25.12</td>
    <td>58.28</td>
    <td>32.14</td>
    <td>42.14</td>
    <td>36.63</td>
  </tr>
  <tr>
    <td>Petitha</td>
    <td>34.35</td>
    <td>30.16</td>
    <td>2.77</td>
    <td>26.23</td>
    <td>25.88</td>
    <td>22.74</td>
    <td>34.31</td>
    <td>25.21</td>
  </tr>
  <tr>
    <th colspan="9">Length System (City Distances)</th>
  </tr>
  <tr>
    <th>Default=Kilometer</th>
    <td><b>74.92*</b></td>
    <td><b>74.85</b></td>
    <td><b>51.95*</b></td>
    <td><b>68.74*</b></td>
    <td><b>59.26</b></td>
    <td><b>66.66*</b></td>
    <td><b>50.14*</b></td>
    <td><b>63.79</b></td>
  </tr>
  <tr>
    <td>Mile</td>
    <td>70.81</td>
    <td>72.42</td>
    <td>36.25</td>
    <td>64.39</td>
    <td>56.25</td>
    <td>58.14</td>
    <td>26.27</td>
    <td>54.93</td>
  </tr>
  <tr>
    <td>Li</td>
    <td>60.75</td>
    <td>55.83</td>
    <td>12.30</td>
    <td>56.92</td>
    <td>38.05</td>
    <td>46.94</td>
    <td>41.76</td>
    <td>44.65</td>
  </tr>
  <tr>
    <td>Vyersta</td>
    <td>57.88</td>
    <td>56.47</td>
    <td>13.88</td>
    <td>47.28</td>
    <td>38.80</td>
    <td>38.33</td>
    <td>38.55</td>
    <td>41.60</td>
  </tr>
  <tr>
    <td>Eski Mil</td>
    <td>4.78</td>
    <td>7.64</td>
    <td>0.11</td>
    <td>10.07</td>
    <td>35.81</td>
    <td>22.41</td>
    <td>15.24</td>
    <td>13.72</td>
  </tr>
  <tr>
    <td>Argentine League</td>
    <td>8.58</td>
    <td>2.55</td>
    <td>0.20</td>
    <td>11.71</td>
    <td>35.71</td>
    <td>8.60</td>
    <td>0.41</td>
    <td>9.68</td>
  </tr>
  <tr>
    <td>Cape Foot</td>
    <td>43.77</td>
    <td>4.41</td>
    <td>0.00</td>
    <td>7.79</td>
    <td>1.37</td>
    <td>1.61</td>
    <td>1.51</td>
    <td>8.64</td>
  </tr>
  <tr>
    <td>Wa</td>
    <td>3.65</td>
    <td>2.72</td>
    <td>0.10</td>
    <td>2.03</td>
    <td>0.42</td>
    <td>0.37</td>
    <td>0.79</td>
    <td>1.44</td>
  </tr>
  <tr>
    <td>Depa</td>
    <td>0.32</td>
    <td>3.77</td>
    <td>0.14</td>
    <td>1.71</td>
    <td>0.23</td>
    <td>0.50</td>
    <td>0.99</td>
    <td>1.10</td>
  </tr>
  <tr>
    <td>Ken</td>
    <td>0.27</td>
    <td>0.70</td>
    <td>0.07</td>
    <td>0.60</td>
    <td>4.72</td>
    <td>0.27</td>
    <td>0.67</td>
    <td>1.04</td>
  </tr>
  <tr>
    <th>Default=Mile</th>
    <td><b>94.54*</b></td>
    <td><b>92.72</b></td>
    <td><b>80.96</b></td>
    <td><b>90.11*</b></td>
    <td><b>58.40</b></td>
    <td><b>77.25</b></td>
    <td><b>55.92</b></td>
    <td><b>78.56</b></td>
  </tr>
  <tr>
    <td>Kilometer</td>
    <td>91.35</td>
    <td>90.31</td>
    <td>78.40</td>
    <td>83.88</td>
    <td><b>65.10*</b></td>
    <td>77.24</td>
    <td><b>57.84</b></td>
    <td>77.73</td>
  </tr>
  <tr>
    <td>Vyersta</td>
    <td>75.83</td>
    <td>80.50</td>
    <td>10.43</td>
    <td>73.97</td>
    <td>45.05</td>
    <td>53.45</td>
    <td>53.66</td>
    <td>56.13</td>
  </tr>
  <tr>
    <td>Li</td>
    <td>61.57</td>
    <td>56.29</td>
    <td>27.66</td>
    <td>55.53</td>
    <td>41.82</td>
    <td>47.46</td>
    <td>40.84</td>
    <td>47.31</td>
  </tr>
  <tr>
    <td>Eski Mil</td>
    <td>4.78</td>
    <td>7.12</td>
    <td>11.84</td>
    <td>33.70</td>
    <td>62.27</td>
    <td>21.59</td>
    <td>14.74</td>
    <td>22.29</td>
  </tr>
  <tr>
    <td>Cape Foot</td>
    <td>43.72</td>
    <td>25.95</td>
    <td>0.01</td>
    <td>0.84</td>
    <td>0.12</td>
    <td>0.39</td>
    <td>0.10</td>
    <td>10.16</td>
  </tr>
  <tr>
    <td>Argentine League</td>
    <td>4.24</td>
    <td>2.89</td>
    <td>0.34</td>
    <td>1.87</td>
    <td>47.48</td>
    <td>2.02</td>
    <td>0.61</td>
    <td>8.49</td>
  </tr>
  <tr>
    <td>Ken</td>
    <td>0.27</td>
    <td>12.56</td>
    <td>0.03</td>
    <td>0.26</td>
    <td>0.15</td>
    <td>0.76</td>
    <td>0.75</td>
    <td>2.11</td>
  </tr>
  <tr>
    <td>Wa</td>
    <td>5.15</td>
    <td>5.48</td>
    <td>0.19</td>
    <td>0.52</td>
    <td>0.17</td>
    <td>0.31</td>
    <td>0.27</td>
    <td>1.73</td>
  </tr>
  <tr>
    <td>Depa</td>
    <td>0.22</td>
    <td>5.46</td>
    <td>0.14</td>
    <td>0.26</td>
    <td>0.12</td>
    <td>0.40</td>
    <td>0.38</td>
    <td>1.00</td>
  </tr>
</table>

Table 6: Disparity and Performance of LLMs Across Different Measurement Systems: The first row (shaded in gray) within each group represents the model’s default measurement system. We report the performance in terms of MAPD (cf. §3.3). The best performance in each column (higher is better) are shown in bold. Disparity: The best values are additionally marked with *, if statistically significant differences (\( p < 0.001 \)) compared to the second best system exists. Cells highlighted in red indicate cases where a significant difference was observed.
<table>
  <tr>
    <th>Currency code</th>
    <th>Description</th>
    <th>Conversion (to USD)</th>
    <th>Currency code</th>
    <th>Description</th>
    <th>Conversion (to USD)</th>
  </tr>
  <tr>
    <td>AED</td>
    <td>United Arab Emirates Dirham</td>
    <td>3.6725</td>
    <td>KMF</td>
    <td>Comorian Franc</td>
    <td>418.0677243</td>
  </tr>
  <tr>
    <td>AFN</td>
    <td>Afghan Afghani</td>
    <td>77.4148667</td>
    <td>KRW</td>
    <td>South Korean Won</td>
    <td>1149.108333</td>
  </tr>
  <tr>
    <td>ALL</td>
    <td>Albanian Lek</td>
    <td>104.0547222</td>
    <td>KWD</td>
    <td>Kuwaiti Dinar</td>
    <td>0.301641667</td>
  </tr>
  <tr>
    <td>AMD</td>
    <td>Armenian Dram</td>
    <td>503.610556</td>
    <td>KZT</td>
    <td>Kazakhstani Tenge</td>
    <td>425.9513889</td>
  </tr>
  <tr>
    <td>ARS</td>
    <td>Argentine Peso</td>
    <td>93.1869444</td>
    <td>LBP</td>
    <td>Lebanese Pound</td>
    <td>1597.5</td>
  </tr>
  <tr>
    <td>AUD</td>
    <td>Australian Dollar</td>
    <td>1.33732752</td>
    <td>LKR</td>
    <td>Sri Lankan Rupee</td>
    <td>201.3890667</td>
  </tr>
  <tr>
    <td>AZN</td>
    <td>Azerbaijani Manat</td>
    <td>1.7</td>
    <td>LYD</td>
    <td>Libyan Dinar</td>
    <td>4.523994444</td>
  </tr>
  <tr>
    <td>BAM</td>
    <td>Bosnia-Herzegovina Convertible Mark</td>
    <td>1.6673125</td>
    <td>MAD</td>
    <td>Moroccan Dirham</td>
    <td>17.66964278</td>
  </tr>
  <tr>
    <td>BDT</td>
    <td>Bangladeshi Taka</td>
    <td>85.24412794</td>
    <td>MDG</td>
    <td>Malagasy Ariary</td>
    <td>3854.072778</td>
  </tr>
  <tr>
    <td>BGN</td>
    <td>Bulgarian Lev</td>
    <td>1.661877778</td>
    <td>MKD</td>
    <td>Macedonian Denar</td>
    <td>52.55698899</td>
  </tr>
  <tr>
    <td>BHD</td>
    <td>Bahraini Dinar</td>
    <td>0.376</td>
    <td>MMK</td>
    <td>Myanmar Kyat</td>
    <td>1610.540741</td>
  </tr>
  <tr>
    <td>BIF</td>
    <td>Burundi Franc</td>
    <td>1978.428889</td>
    <td>MOP</td>
    <td>Macanese Pataca</td>
    <td>8.098944444</td>
  </tr>
  <tr>
    <td>BND</td>
    <td>Brunel Dollar</td>
    <td>1.345794444</td>
    <td>MUR</td>
    <td>Mauritian Rupee</td>
    <td>41.72418333</td>
  </tr>
  <tr>
    <td>BOB</td>
    <td>Bolivian Boliviano</td>
    <td>6.91</td>
    <td>MXN</td>
    <td>Mexican Peso</td>
    <td>20.654375</td>
  </tr>
  <tr>
    <td>BRL</td>
    <td>Brazilian Real</td>
    <td>5.368375</td>
    <td>MYR</td>
    <td>Malaysian Ringgit</td>
    <td>4.134145833</td>
  </tr>
  <tr>
    <td>BWP</td>
    <td>Botswana Pula</td>
    <td>11.19985833</td>
    <td>MZM</td>
    <td>Mozambican Metical</td>
    <td>66.0258</td>
  </tr>
  <tr>
    <td>BYN</td>
    <td>Belarusian Rouble</td>
    <td>2.533175</td>
    <td>NGN</td>
    <td>Nigerian Naira</td>
    <td>406.2341667</td>
  </tr>
  <tr>
    <td>BZD</td>
    <td>Belize Dollar</td>
    <td>2</td>
    <td>NIO</td>
    <td>Nicaraguan Cordoba</td>
    <td>35.20105833</td>
  </tr>
  <tr>
    <td>CAD</td>
    <td>Canadian Dollar</td>
    <td>1.251053333</td>
    <td>NOK</td>
    <td>Norwegian Krone</td>
    <td>8.631944444</td>
  </tr>
  <tr>
    <td>CDF</td>
    <td>Congo Franc</td>
    <td>1899.948356</td>
    <td>NPR</td>
    <td>Nepalese Rupee</td>
    <td>113.7686111</td>
  </tr>
  <tr>
    <td>CHF</td>
    <td>Swiss Franc</td>
    <td>0.9150375</td>
    <td>NZD</td>
    <td>New Zealand Dollar</td>
    <td>1.416864444</td>
  </tr>
  <tr>
    <td>CLP</td>
    <td>Chilean Peso</td>
    <td>779.482222</td>
    <td>OMR</td>
    <td>Oman Rial</td>
    <td>0.3845</td>
  </tr>
  <tr>
    <td>CNY</td>
    <td>Chinese Yuan</td>
    <td>6.457722222</td>
    <td>PAB</td>
    <td>Panamanian Balboa</td>
    <td>1</td>
  </tr>
  <tr>
    <td>COP</td>
    <td>Colombian Peso</td>
    <td>3784.35806</td>
    <td>PEN</td>
    <td>Peruvian Nuevo Sol</td>
    <td>3.292661111</td>
  </tr>
  <tr>
    <td>CRC</td>
    <td>Costa Rican Colon</td>
    <td>625.5777777</td>
    <td>PHP</td>
    <td>Philippine Peso</td>
    <td>49.43672222</td>
  </tr>
  <tr>
    <td>CVE</td>
    <td>Cape Verdean Escudo</td>
    <td>93.765085</td>
    <td>PKR</td>
    <td>Pakistani Rupee</td>
    <td>164.1562869</td>
  </tr>
  <tr>
    <td>CZK</td>
    <td>Czech Republic Koruna</td>
    <td>21.67958333</td>
    <td>PLN</td>
    <td>Polish Zloty</td>
    <td>3.889841667</td>
  </tr>
  <tr>
    <td>DJF</td>
    <td>Djiboutian Franc</td>
    <td>177.721</td>
    <td>PYG</td>
    <td>Paraguayan Guarani</td>
    <td>6676.519722</td>
  </tr>
  <tr>
    <td>DKK</td>
    <td>Danish Krona</td>
    <td>6.3171375</td>
    <td>QAR</td>
    <td>Qatari Rial</td>
    <td>3.64</td>
  </tr>
  <tr>
    <td>DOP</td>
    <td>Dominican Peso</td>
    <td>57.25494444</td>
    <td>RON</td>
    <td>Romanian Leu</td>
    <td>4.192530556</td>
  </tr>
  <tr>
    <td>DZD</td>
    <td>Algerian Dinar</td>
    <td>135.6026</td>
    <td>RSD</td>
    <td>Serbian Dinar</td>
    <td>99.94511944</td>
  </tr>
  <tr>
    <td>EGP</td>
    <td>Egyptian Pound</td>
    <td>15.65245278</td>
    <td>RUB</td>
    <td>Russian Ruble</td>
    <td>73.53408333</td>
  </tr>
  <tr>
    <td>ETB</td>
    <td>Ethiopian Birr</td>
    <td>44.22141041</td>
    <td>RWF</td>
    <td>Rwandan Franc</td>
    <td>991.6860625</td>
  </tr>
  <tr>
    <td>EUR</td>
    <td>Euro</td>
    <td>0.84978685</td>
    <td>SAR</td>
    <td>Saudi Riyal</td>
    <td>2.71</td>
  </tr>
  <tr>
    <td>GBP</td>
    <td>British Pound Sterling</td>
    <td>0.72827027</td>
    <td>SDG</td>
    <td>Sudanese Pound</td>
    <td>299.4778056</td>
  </tr>
  <tr>
    <td>GEL</td>
    <td>Georgian Lari</td>
    <td>3.2525</td>
    <td>SEK</td>
    <td>Swedish Krona</td>
    <td>8.649225</td>
  </tr>
  <tr>
    <td>GHS</td>
    <td>Ghanaian Cedi</td>
    <td>5.851411111</td>
    <td>SGD</td>
    <td>Singapore Dollar</td>
    <td>1.345016667</td>
  </tr>
  <tr>
    <td>GNF</td>
    <td>Guinea Franc</td>
    <td>9933.2195</td>
    <td>SYP</td>
    <td>Syrian Pound</td>
    <td>125.76</td>
  </tr>
  <tr>
    <td>GTQ</td>
    <td>Guatemalan Quetzal</td>
    <td>7.7412875</td>
    <td>THB</td>
    <td>Thai Baht</td>
    <td>32.01021944</td>
  </tr>
  <tr>
    <td>HKD</td>
    <td>Hong Kong Dollar</td>
    <td>7.77677778</td>
    <td>TND</td>
    <td>Tunisian Dinar</td>
    <td>2.035069444</td>
  </tr>
  <tr>
    <td>HNL</td>
    <td>Honduran Lempira</td>
    <td>24.0525</td>
    <td>TRY</td>
    <td>Turkish Lira</td>
    <td>9.408511111</td>
  </tr>
  <tr>
    <td>HUF</td>
    <td>Hungarian Forint</td>
    <td>304.9938889</td>
    <td>TTD</td>
    <td>Trainidad and Tobago Dollar</td>
    <td>5.474011111</td>
  </tr>
  <tr>
    <td>IDR</td>
    <td>Indonesian Rupiah</td>
    <td>14333.63958</td>
    <td>TWD</td>
    <td>New Taiwan Dollar</td>
    <td>28.05294444</td>
  </tr>
  <tr>
    <td>ILS</td>
    <td>Israeli New Sheqel</td>
    <td>2.324777778</td>
    <td>TZS</td>
    <td>Tanzanian Shilling</td>
    <td>2296.881064</td>
  </tr>
  <tr>
    <td>INR</td>
    <td>Indian Rupee</td>
    <td>73.82430833</td>
    <td>UAH</td>
    <td>Ukrainian Hryvnia</td>
    <td>27.27136389</td>
  </tr>
  <tr>
    <td>IQD</td>
    <td>Iraqi Dinar</td>
    <td>149</td>
    <td>UGX</td>
    <td>Ugandan Shilling</td>
    <td>3601.599952</td>
  </tr>
  <tr>
    <td>IRR</td>
    <td>Iranian Rial</td>
    <td>42000</td>
    <td>USD</td>
    <td>US Dollar</td>
    <td>1</td>
  </tr>
  <tr>
    <td>ISK</td>
    <td>Icelandic Krona</td>
    <td>125.9425</td>
    <td>UYU</td>
    <td>Uruguayan Peso</td>
    <td>43.51508333</td>
  </tr>
  <tr>
    <td>JMD</td>
    <td>Jamaican Dollar</td>
    <td>50.36073333</td>
    <td>UZS</td>
    <td>Uzbekistan Som</td>
    <td>1064.024833</td>
  </tr>
  <tr>
    <td>JOD</td>
    <td>Jordanian Dinar</td>
    <td>0.71</td>
    <td>VND</td>
    <td>Vietnamese Dong</td>
    <td>21.17712222</td>
  </tr>
  <tr>
    <td>JPY</td>
    <td>Japanese Yen</td>
    <td>109.6466667</td>
    <td>XAF</td>
    <td>CFA Franc BEAC</td>
    <td>557.4236325</td>
  </tr>
  <tr>
    <td>KES</td>
    <td>Kenyan Shilling</td>
    <td>110.2038889</td>
    <td>XOF</td>
    <td>CFA Franc BCEAO</td>
    <td>557.4236325</td>
  </tr>
  <tr>
    <td>KHR</td>
    <td>Cambodian Riel</td>
    <td>4097.125</td>
    <td>ZAR</td>
    <td>South African Rand</td>
    <td>14.953069</td>
  </tr>
</table>

<table>
  <tr>
    <th>Measure</th>
    <th>Description</th>
    <th>Conversion (in KG)</th>
    <th>Measure</th>
    <th>Description</th>
    <th>Conversion (in KG)</th>
  </tr>
  <tr>
    <td>Kilogram (kg)</td>
    <td></td>
    <td>1</td>
    <td>Geun (Korea)</td>
    <td></td>
    <td>0.6</td>
  </tr>
  <tr>
    <td>Pound</td>
    <td></td>
    <td>0.453592</td>
    <td>Troy Ounce</td>
    <td></td>
    <td>0.03724</td>
  </tr>
  <tr>
    <td>Aristol (Portuguese)</td>
    <td></td>
    <td>0.485</td>
    <td>Funt (Russia)</td>
    <td></td>
    <td>0.049917</td>
  </tr>
  <tr>
    <td>catty/Kati (Malaysian)</td>
    <td></td>
    <td>0.60479</td>
    <td>Ser (80 Tola, Pakistani)</td>
    <td></td>
    <td>0.93310</td>
  </tr>
  <tr>
    <td>Jin (Mainland China)</td>
    <td></td>
    <td>0.5</td>
    <td>Peittha (Burmese)</td>
    <td></td>
    <td>1.63293</td>
  </tr>
</table>

<table>
  <tr>
    <th>Length System</th>
    <th>Description</th>
    <th>Conversion (in KM)</th>
    <th>Length System</th>
    <th>Description</th>
    <th>Conversion (in KM)</th>
  </tr>
  <tr>
    <td>Kilometer (km)</td>
    <td></td>
    <td>1</td>
    <td>Mile (mi)</td>
    <td></td>
    <td>1.60934</td>
  </tr>
  <tr>
    <td>Li (China)</td>
    <td></td>
    <td>0.5</td>
    <td>Eski Mil (Turkish)</td>
    <td></td>
    <td>1.89435</td>
  </tr>
  <tr>
    <td>Vyersta (Russia)</td>
    <td></td>
    <td>1.0668</td>
    <td>Ken (Japan)</td>
    <td></td>
    <td>0.0181818</td>
  </tr>
  <tr>
    <td>Cape Foot (South Africa)</td>
    <td></td>
    <td>0.0003148584</td>
    <td>Depa (Indonesian)</td>
    <td></td>
    <td>0.0016</td>
  </tr>
  <tr>
    <td>Argentine league (legua)</td>
    <td></td>
    <td>5.572</td>
    <td>Wa (Thai)</td>
    <td></td>
    <td>0.002</td>
  </tr>
</table>

Table 7: Measurement System Description. Description of measures used in the dataset.

Reviewer paper summaries: This paper investigates the default choices of measurement systems across LLMs, the performance of LLMs across different measurement systems, as well as the potential benefit of reasoning-based prompting to mitigate disparate performance across measurement systems.
Summary of strengths: 1. This is a very focused research paper, presenting a clearly defined contribution to LLM research. I personally appreciate this kind of research.

2. The paper is well-written and easy to follow. I really enjoyed reading it.
Summary of weaknesses: I really enjoyed reading this paper and would very much like to see it published. However, I have two concerns, which for me are fundamental to address before I can recommend it for acceptance.

1. The use of only English prompts. You argued "To isolate the effect of different measurement systems, we use English prompts in all our experiments, ruling out the language of the prompt as a confounding factor." I do not agree with this argument. By using only English prompts, you're ignoring the confounding effect of language, not at all controlling for it. You also made that point that LLMs should be culturally adaptable and thus be fair to users from different cultures. I agree with this, but I also believe that users from different cultures would interact with LLMs mostly in their own native languages, thus investigating the performance of LLM on a specific measurement system using the respective culture's language is a more relevant and faithful to this concrete use scenario of LLMs.

I understand that it would be a lot of work to include non-English languages in your experiments. May I suggest that you pick just two languages as a sensitivity analysis to show if language does matter in this case. 

2. In Table 3 regarding Length System (City Distances), the performance of non-western measurement systems seems really bad. Do you have an explanation for it? I have a Chinese background and when I looked at the "Chinese (Zhang)" example from Table 1, it took me a while to figure out what Zhang refers to. Honestly, I have not really seen Zhang being used as a measurement unit in China; I mostly came across its use in ancient Chinese texts. This left me wondering whether the bad performance of the non-western systems is because these systems are not really used anymore in non-western cultures either. could you verify this? I don't know about the Japanese and Russian cases. Also, I'm wondering, if this is the case, whether it is then still relevant to look at these somewhat antiquated measurement systems, since probably very few users use them.

-----------------
Authors addressed both my concerns during the rebuttal period and I therefore adjusted my scores accordingly.
Comments, suggestions and typos: NA
Overall assessment scores: 4