SlowGuess/NAACL_2022_OCR huggingface dataset:
Kabir Ahuja<sup>1</sup> Monjit Choudhury<sup>2</sup> Sandipan Dandapat<sup>2</sup>
<sup>1</sup> Microsoft Research, India
2 Microsoft IDC
{t-kabirahuja,monojitc,sadandap}@microsoft.com
Borrowing ideas from Production functions in micro-economics, in this paper we introduce a framework to systematically evaluate the performance and cost trade-offs between machine-translated and manually-created labelled data for task-specific fine-tuning of massively multilingual language models. We illustrate the effectiveness of our framework through a case-study on the TyDIQA-GoldP dataset. One of the interesting conclusions of the study is that if the cost of machine translation is greater than zero, the optimal performance at least cost is always achieved with at least some or only manually-created data. To our knowledge, this is the first attempt towards extending the concept of production functions to study data collection strategies for training multilingual models, and can serve as a valuable tool for other similar cost vs data trade-offs in NLP.
Transformer based Massively Multilingual Language Models (MMLMs) such as mBERT (Devlin et al., 2019), XLM-RoBERTa (Conneau et al., 2020) and mT5 (Xue et al., 2021) are surprisingly effective at zero-shot cross-lingual transfer (Pires et al., 2019; Wu and Dredze, 2019). However, while zero-shot transfer is effective, often the performances across different languages is not consistent. Low-resource languages (Wu and Dredze, 2020) and the languages that are typologically distant from the pivot language (Lauscher et al., 2020) are known to benefit the least from zero-shot transfer, which can often be mitigated by using target-language specific labelled data for the task during fine-tuning.
One common approach for collecting such data in the target language is to translate the training data available for the pivot-language to the target by using an off-the-shelf Machine Translation (MT) system. This is usually referred to as the translate-train setup (Hu et al., 2020; Turc et al., 2021).
Few-shot transfer is another alternative; as shown by Lauscher et al. (2020), a few labelled examples in the target language, that can be obtained cheaply, can lead to substantial improvements over the zero-shot performance.
However, there has not been much work on comparing the performance across these two strategies. In one such study, Hu et al. (2020) compare the performance of translate-train with few-shot transfer on TyDIQA-GoldP (Clark et al., 2020) dataset, but they only evaluate the few-shot case with 1000 examples, which does not provide any insight into how the performance varies with increasing dataset sizes for these two approaches. Additionally, there are trade-offs related to the data acquisition costs as well. The cost per training instance is expected to be much smaller for an MT-based approach than manual translation or labeling of examples. However, depending on the nature of task, language, and quality of the MT output, the amount of data required to achieve the same performance through these two approaches can be drastically different. More importantly, fine-tuning the MMLMs with a combination of the data from the two strategies could be the cheapest alternative for achieving a target accuracy, which, to the best of our knowledge, has not been explored yet.
Inspired by the above observations and gaps, in this paper, we ask the following question: Given a pre-determined budget to fine-tune a multilingual model on a task for which some data is available in a pivot language, what is the best achievable accuracy on a target language by (a) training the model on the pivot-language data, (b) different amounts of machine-translated and (c) manually-collected data in the target language. Solving this requires an understanding of the exact nature of the performance and cost trade-offs between the two kinds of target language datasets and their relative costs of acquisition, apart from factors such as the amount of pivot language data, the task, the MMLM, and
Human Language Technologies, pages 1369 - 1384
the languages concerned.
This problem of modeling and measuring the trade-offs between different input factors and their costs is well-studied in the field of microeconomics. A sophisticated machinery has been developed in the form of Production Functions and allied analytical methods (Miller and Blair, 2009; Cobb and Douglas, 1928), in order to solve the following generic problem: with the best available technology, how are the inputs to a production process (eg. Labor and Capital) related to its output, that is the quantity of goods produced. In this paper, we adapt this framework to address the aforementioned question of MMLM fine-tuning trade-offs.
The key contributions of our work are threefold. 1. We extend the idea of production functions to performance functions that model the relationship between input data sizes and performance of a system; we propose a possible analytical form for this function and derive the performance trends and optimal data collection strategies under fixed costs. 2. We illustrate the usefulness of this framework through a case study on a Q&A task - TyDIQA-GoldP (Clark et al., 2020) and systematically study the various trade-offs for 8 languages. 3. Our study provides several important insights such as (a) if the cost of MT data creation is non-zero, then the optimal performance under a fixed budget is always achieved with either only manually-created data or a combination of the two; (b) the ratio of the two datasets for the least cost combination usually remains constant at different levels of performance.
To the best of our knowledge, this is the first work that applies the idea of production functions to analyze the cost-performance trade-offs of MMLM fine-tuning. The proposed framework can be extended to a multitude of NLP problems where the trade-offs similar to the ones discussed above, are common (e.g., pre-training vs. fine-tuning data). To encourage reproducible research, we have made our code, the performance data, and a detailed list of the results publicly available<sup>1</sup>.
2 Theoretical Foundations
One of the foundational pillars of neoclassical economics is the idea of Production Functions. Simply put, a production function is a mathematical formalization of the relationship between the output of a firm (industry, economy) and the inputs that have been used in obtaining it (Khatskevich and
Pranevich, 2018; Miller and Blair, 2009). A multifactor production function is defined as a map
$$
Q: \mathbf {x} \rightarrow f (\mathbf {x}), \quad \forall \mathbf {x} \in \mathbb {R} ^ {+ n} \tag {1}
$$
where  $Q \in \mathbb{R}^{+}$  is the quantity of output,  $n$  is the number of the inputs, the non-negative function  $f$  is continuously differentiable for all  $\mathbf{x} = (x_{1},\ldots ,x_{n})$  when  $x_{i} \in \mathbb{R}^{+}$ . A sophisticated and extensive set of analytical machinery has been developed over the years in microeconomics theory that allows one to closely model and analyze not only the relationship between the inputs and outputs² of a firm, but also the interdependence between the inputs (i.e.,  $x_{i}$ s). Thus, one can efficiently compute and clearly visualize the various trade-offs and optimal configurations of the production system.
Production functions have been extensively used to model and study systems as diverse as education (Bettinger et al., 2020; Bowles, 1970), environment (Lu et al., 2019; Halicioglu and Ketenci, 2018), sustainability (Yankovyi et al., 2021), cognition (Todd and Wolpin, 2003) and of course, different types of industries (Husain et al., 2016; Batiese, 1992). Along similar lines, in this work we develop the concept of Performance Function that models the performance of an MMLM given the amount of translated and manually labeled data. In this section, we begin by formalizing the notations and defining some key concepts from microeconomics, appropriately adapted to our context. Then we present the functional form of the performance function, and discuss certain practical constraints and assumptions that we will make in our formulation.
Consider a multilingual model  $\mathcal{M}$  pre-trained on a set of languages  $\mathcal{L}$ , which is to be fine-tuned for a task  $\mathfrak{T}$ , for which  $P$  labelled examples are available in a pivot language  $p\in \mathcal{L}$ . Some or all of the  $P$  pivot language examples can be automatically translated to a target language  $l\in \mathcal{L}$  through an MT system to obtain  $T(\leq P)$  examples. Further, let  $M$  be the amount of examples for  $l$  that have been labelled or translated manually.
Definition 1 Performance Function,  $\Pi = \pi (T,M|l,p,P,\mathcal{M},\mathfrak{T})$  , denotes the best possible performance (as per the current state-of-the-art) of
2Production functions can also be defined when there are  $m$  outputs i.e.  $Q\in \mathbb{R}^m$  and  $f:\mathbb{R}^n\to \mathbb{R}^m$
$^{1}$ github.com/kabirahuja2431/PerformanceFunctionAnalysis
a system in language  $l$  for a task  $\mathfrak{T}$ , that has been built on top of a pre-trained MMLM  $\mathcal{M}$ ,  $P$  labelled examples in language  $p \neq l$ ,  $T$  translated examples by an MT system, and  $M$  manually created examples.
Here,  $\Pi \in [0,1]$  is any appropriate and accepted measure of performance, such as accuracy or F1-score. To simplify the notation we will often drop the given conditions from the equation and denote  $\Pi = \pi (T,M)$ . The conditional factors, whenever not obvious from the context, will be explicitly stated. Note that  $T$  and  $M$  are respective equivalents of  $K$  and  $L$  of the neoclassical Capital-Labor production functions. Capital investment in technology or mechanization is similar to machine-translated data, whereas manual dataset creation would require investment on labor.
Definition 2 Total cost of operation (or simply the cost),  $\kappa(T, M) = \kappa_t(T) + \kappa_m(M)$ , is the total cost of procuring translated and manually created datasets for  $l$  for the task  $\mathfrak{T}$ .
We further assume that the translation and manual collection costs are scalar multiples of the unit costs, i.e.  $\kappa_{t}(T) = c_{t}T$  and  $\kappa_{m}(M) = c_{m}M$ , where  $c_{t} > 0$  is the cost of translating a single example from  $P$  into language  $l$  automatically, while  $c_{m} > 0$  is the cost of collecting one training example in  $l$  manually. Therefore,
$$
C = \kappa (T, M) = c _ {t} T + c _ {m} M \tag {2}
$$
Usually,  $c_{m} > c_{t}$ . Also, note that we are ignoring the costs of pivot data collection and computational costs of pre-training and fine-tuning, partly because we are interested in studying the tradeoff between  $T$  and  $M$ . Also,  $P$  is useful for any target language, and therefore, the amortized cost of creating  $P$  tends to zero as the number of target languages increases. Similarly the amortized cost of pre-training tends to zero as the number of tasks grows. The task-specific training cost is proportional to training data-size,  $P + T + M$ , and therefore, can be partially consumed in  $c_{t}$  and  $c_{m}$ .
Definition 3 Isoperf curves are the contours of the performance function that represent the relationship between  $T$  and  $M$  for a fixed performance value  $\Pi_c$ .
Definition 4 Isocost curves are the contours of the cost function that represent the different possible
Figure 1: Hypothetical T-M diagram illustrating isoperfs, isocosts, points of tangency and expansion path.
combinations of  $T$  and  $M$  that result in equal overall costs.
Both isoperfs and isocosts are drawn on a T-M diagram (K-L diagram in micro-economics), which is illustrated in Fig. 1. The x and y axis represent the input factors T and M, respectively. The orange curves are the hypothetical isoperfs, known as isoquants in economics. As the name suggests, each point on these curves represents T-M combinations that result in the same (iso) performance (perf), denoted in the diagram by  $\Pi_1$ ,  $\Pi_2$ , etc. Intuitively, it can be seen that two isoperfs never intersect; as we move towards right and up,  $\Pi$  increases because either  $T$  or  $M$  or both increase. Thus,  $\Pi_1 < \Pi_2 < \Pi_3 < \Pi_4$ . The origin,  $T = 0$ ,  $M = 0$ , represents an isoperf corresponding to the zero-shot performance on  $l$  when  $\mathcal{M}$  is fine-tuned only on  $P$ .
The blue lines represent the isocosts. Considering the nature of the cost function defined, the isocost curves will be straight lines parallel to each other with slope  $-\frac{c_t}{c_m}$ . Like isoperfs, the cost of operation increases for the isocosts as we move towards right and top in the T-M diagram.
Definition 5 Least Cost Operating Point on an isoperf refers to the (possibly multiple) point where the total cost of operation is lowest for a given performance.
Under the assumption of smooth and convex isoperfs,3 the isocost corresponding to the least cost of operation will be a tangent to the isoperf, and the optimal allocation of the  $T$  and  $M$  is given by the point of tangency. The isocosts shown in Fig. 1
<sup>3</sup>Isoperfs are convex for declining marginal rate of technical substitution.
correspond to the least cost curves for respective isoperfs, and the points of tangency are represented by the points E1, E2, etc.
Definition 6 Expansion path is a path connecting the point of tangency of different isoperf and isocost curves, tracing out the cost minimizing combination of the data resources with increasing performance and costs.
Expansion paths are important in determining resource allocations strategies. For instance, when a higher budget is available for dataset expansion in a particular language, should one invest more in translation or in manually collected data? And how does this equation change in the long run, as the system moves towards higher performances?
Thus, isoperfs and isocosts when studied collectively can help determine the allocation of the amount of translation and manual data for a desirable performance value that minimizes the cost of operation.
2.2 Selecting a Functional Form for  $\pi$
In production analysis, one of the difficult problems is to decide on the functional form of the production function that can on one hand accurately represent the input-output relationship, and on the other, is amenable to close-formed analysis (Griffin et al., 1987). Clearly, a linear production function would be an inappropriate choice for  $\pi(T, M)$ , as  $T$  and  $M$  are not perfect substitutes of each other. A popular choice in such case is the Cobb-Douglas performance function (Cobb and Douglas, 1928), which is of the form  $T^{\alpha}M^{\beta}$ . However, the two datasets do not have multiplicative, but rather an additive effect. Therefore, we propose the following performance function:
$$
\Pi = \pi (T, M) = a _ {z s} + a _ {t} T ^ {\alpha_ {t}} + a _ {m} M ^ {\alpha_ {m}} (3)
$$
where  $a_{zs}, a_t, a_m \geq 0$  and  $0 \leq \alpha_t, \alpha_m \leq 1$ . The positive coefficients of the input factors are motivated by assuming that under a reasonable translation and manual annotation quality, the addition of data from these sources should not hurt the zero-shot performance which is given by  $a_{zs}$  (when  $T = M = 0$ ). Bounding the exponents below 1 ensures that the performance is not allowed to increase linearly with increasing data in one of these sources, as we always see diminishing returns with respect to data for any machine learning model.
The commonly used training setups can be obtained as special cases of the above equation. The translate-train setup, can be obtained by setting  $T = P$  and  $M = 0$  in the equation, giving  $\Pi_{TT} = a_{zs} + a_tP^{\alpha_t}$ . Similarly,  $\Pi_{FS} = a_{zs} + a_mk^{\alpha_m}$  gives the few-shot setup with  $k$  examples. We denote this functional form as AMUE (Additive Model with Unequal Elasticities).
The expression for tangency point can be derived by setting  $dM / dT|_{\Pi = \Pi_c}$  to the slope of the isocost,  $-c_t / c_m$ , which gives the following equation for the expansion path.
$$
M = \left(\frac {c _ {t} a _ {m} \alpha_ {m}}{c _ {m} a _ {t} \alpha_ {t}}\right) ^ {\frac {1}{1 - \alpha_ {m}}} T ^ {\frac {1 - \alpha_ {t}}{1 - \alpha_ {m}}} \tag {4}
$$
Thus,  $M / T$  (also called the labor-to-capital ratio) increases with performance if  $\alpha_{m} > \alpha_{t}$ , remains fixed when  $\alpha_{m} = \alpha_{t}$ , and decreases with performance when  $\alpha_{m} < \alpha_{t}$ . Similarly, the ratio of costs of acquiring manually created data to translated data,  $Mc_{m} / Tc_{t}$  is proportional to  $a_{m}M^{\alpha_{m}} / a_{t}T^{\alpha_{t}}$ , which is the ratio of the contributions of the two datasets to the performance  $\Pi$ .
More often than not, actual production systems are too complex to be modeled accurately with simple functional forms. We expect a similar situation, where AMUE might be well suited for modeling and visualizing the trends. However, to obtain the actual operating cost and expansion path that are practically useful, one would need to model the behavior of the performance function more accurately. To this end, we also experiment with Gaussian Process Regression (GPR) for defining the performance function. As we shall see in the next section, GPR is able to fit the data more effectively, though we shall stick to AMUE as the two show identical trends and the latter also allows us to gain deeper insights and richer visualizations.
2.3 Some Practical Considerations
Definition 7 Cost Ratio, defined as  $c_{t / m} = \frac{c_t}{c_m}$ , is the relative cheapness of the translation data, when compared to the cost of obtaining a manually created data point.
We expect the cost ratio to be much smaller than 1. However, both translation and manual annotation costs vary according to the complexity (in case of translation, just the lengths of sentences) of the task at hand.  $c_{m}$  might also vary with the choice of the target language  $l$ , while  $c_{t}$  can be assumed to be uniform across the languages supported by
the commercial MT systems like Google or Bing. In the experiments for our case study, we calculate the expansion paths for different values of  $c_{t / m}$  to systematically study the nature of the trade-offs between the two sources of data.
Realizable region: The forms of the performance function as well as cost function defined above do not place any constraint on the values that the input factors, i.e.  $T$  and  $M$ , can take, which means that the amount of data can be increased indefinitely in order to improve the performance. However, we are aware that the amount of translated data is upper bounded by the amount of pivot data available, i.e.  $T \leq P$ . While this constraint can be explicitly worked out into the equations (by replacing  $T$  with  $\min(T, P)$ ), we stick to the original forms to preserve the smoothness of AMUE. Instead, we define a realizable region  $\mathcal{R}: T \leq P$ , and if a tangency point lies outside  $\mathcal{R}$  we explicitly search for the minimum cost point on the part of the isoperf curve that lies in the realizable region. Note that, in such cases the isocost curves corresponding to the minimum cost point will no longer be tangents to the corresponding isoperfs, and will usually lie at the boundary between the realizable and non-realizable regions.
3 Case-Study on TyDiQA-GoldP
In order to understand the efficacy of the proposed framework, we conduct a case-study on a popular multilingual Question Answering task (cf.  $\mathfrak{T}$ ) using TyDiQA-GoldP (Clark et al., 2020) dataset and consider mBERT as the MMLM  $\mathcal{M}$ . In the following subsections, we provide the details of the task and training setup for generating the performance  $\Pi$  for different combination of the input factors, the procedure for estimating the parameters of the performance functions, and the findings.
We consider the Minimum Answer Span Task from the Typologically Diverse Question Answering benchmark or TyDiQA-GoldP for conducting the experiments. The choice of this particular dataset stems from two main properties of the benchmark. First, question-answering tasks are amenable to translation. Secondly, TyDiQA-GoldP is comprised of manually labelled datasets for nine typologically diverse languages. This enables us to study the effect of different amounts of manually-created data  $M$  on the performance of the MMLM.
The amount of  $M$  varies significantly from language to language with 1.6k examples for Korean to 15k examples in Arabic. 3.7k examples are available for English which we shall consider as the pivot language  $p$  in all the experiments. We use Azure Translator4 to obtain the translated data  $T$  in eight target languages. The answer span alignment between English and the translated languages are obtained based on the technique described in Hu et al. (2020). We measure the performance II as the average F1-score between the predicted and actual answer-spans for the test examples.
3.2 Fine-tuning Setup
We fine-tune mBERT on the TyDiQA-GoldP dataset with different values of the input factors,  $T$  and  $M$ , for each target language, along with the amount of English pivot data,  $P$ . Different values of  $T$  are chosen by translating  $0\%$ ,  $10\%$ ,  $40\%$ ,  $70\%$  or  $100\%$  of the English pivot data. Eleven different values in the range  $[0, |\mathcal{D}_l|]$  ( $\mathcal{D}_l$  is the size of the available training data in  $l$ ) and seven values between 0 and  $3.7k$  are selected for  $M$  and  $P$ , respectively. Considering eight different target languages, this results in 3080 different fine-tuning configurations. In each configuration, we use 3 different random seeds and train for 5 epochs with a learning rate of 2e-5 and a batch size of 32. The models are also jointly trained<sup>5</sup>. We use XTREME repository (Hu et al., 2020) and the Hugging Face Transformer Library (Wolf et al., 2020) to conduct all our experiments.
3.3 Parameter Estimation of the Performance Function
Upon estimating the performance values for the various fine-tuning configurations, we formulate the parameter estimation for the performance functions  $\pi$  as a regression task, with  $T$  and  $M$  as inputs and  $\Pi$  as the output. We use a Non Linear Least Squares algorithm (Levenberg, 1944) to fit the AMUE functional form (cf. Equation (5)), while specifying the bounds on the function parameters. For GPR, we use an RBF Kernel added with a White Kernel to model the noise in the observations, and the kernel parameters are optimized using L-BFGS-B optimization algorithm (Byrd et al., 1995) with 10 restarts. Note that, we fit different
4https://www.microsoft.com/en-us/ translator/business/Translator-api/
<sup>5</sup>We empirically observed that joint training performs better than curriculum learning ( $P \to T \to M$ )
l	at	αt	am	αm
P = 3696
ar	3.7e-01	1.9e-07	2.0e+00	2.2e-01
bn	5.8e-04	6.9e-01	2.3e+00	3.0e-01
fi	7.4e-02	3.9e-01	1.2e+00	3.0e-01
id	2.5e-13	2.5e-01	1.2e+00	2.9e-01
ko	2.6e-15	2.1e-03	1.5e+00	2.6e-01
ru	7.8e-13	5.6e-01	7.1e-01	3.5e-01
sw	5.2e-02	4.2e-01	1.1e+00	3.7e-01
te	5.1e-19	2.5e-01	1.2e+01	1.5e-01
P = 2000
ar	1.7e-01	2.9e-01	2.9e+00	2.1e-01
bn	9.9e-01	1.2e-01	1.9e+00	3.4e-01
fi	9.4e-02	4.6e-01	1.6e+00	3.0e-01
id	4.0e-01	1.2e-01	1.5e+00	3.0e-01
ko	3.0e-13	4.1e-01	1.6e+00	2.8e-01
ru	5.8e-03	6.5e-01	1.1e+00	3.4e-01
sw	9.2e-02	4.3e-01	1.2e+00	3.7e-01
te	1.6e-01	3.0e-01	1.2e+01	1.5e-01
Table 1: Values of AMUE performance function parameters for different languages.
performance functions for each combination of  $l$  and  $P$ . Additionally, we also conducted several experiments with other functional forms including Cobb-Douglas, linear, log-linear and polynomial functions ( $>1$  degree) which either showed higher margins of error or over-fitting.