SlowGuess/NAACL_2022_OCR huggingface dataset:
First, we evaluate how well the two proposed performance functions are able to predict the performance for different fine-tuning configurations. For this, we split the 3080 different training configurations into training  $(80\%)$  and test  $(20\%)$  sets. The test root mean squared error (RMSE) and coefficient of determination  $(r^2)$  values for AMUE and GPR were found to be 5.84, 0.90 and 2.43, 0.98 respectively. Thus, both the models can fit the data reasonably well, though as expected, GPR provides a better fit. Check Appendix for more details.
Expansion Paths: Table 1 shows the estimated values of the AMUE parameters for different languages and pivot sizes. For all the languages,  $a_{m}$  is greater than  $a_{t}$  by at least an order of magnitude, meaning that the manually collected data ends up having a significantly higher contribution towards the model's performance. For  $P = 2000$ , we
see comparatively higher values of  $a_{t}$  (though still  $< a_{m}$ ). This indicates that the machine-translated data might be more beneficial when there is a paucity of training data available in the pivot language, and thus a lower zero-shot performance to begin with.
For  $P = 3696$ , Arabic, Indonesian and Korean has  $\alpha_{m} > \alpha_{t}$  and therefore, the corresponding expansion curves (Eqn 4) will have an increasing  $M / T$  ratio with increasing  $\Pi$ . On the other hand for Swahili, Telugu and Finnish,  $\alpha_{m} < \alpha_{t}$ , and hence the expansion curves will bend towards the x-axis in the T-M diagram, indicating a declining  $M / T$  ratio. In such cases, as we continue to increase the performance at the minimum cost, the optimum strategy would be to collect higher and higher amount of translation data as compared to manually labelled data.
However, notice that the  $\alpha_{m}$  and  $\alpha_{t}$  are close to each other for majority of the cases resulting in nearly linear expansion paths, a situation that is often encountered in economics whenever the production function is homogenous. We did not start with a homogeneity assumption on  $\pi (M,T)$ ; rather, the estimated parameters indicate so. This has two interesting implications: 1)  $M / T$  remains nearly uniform at the different levels of performance; 2) the slope of the expansion path is approximately  $\left(\frac{c_t a_m}{c_m a_t}\right)^{\frac{1}{1 - \alpha_m}}$  (by setting  $\alpha_{m} = \alpha_{t}$  in Eqn 4), meaning if the cost ratio  $\frac{c_t}{c_m}$  is greater than  $\frac{a_t}{a_m}$ , the optimal strategy would be to collect more manually labelled data (since  $\frac{1}{1 - \alpha_m} > 1$  by definition) and vice-versa. Thus, by just looking at the value of these parameters we can gain key insights about the optimal data allocation strategies.
These strategic insights can also be clearly visualized through the isoperf, isocost and expansion path curves on the T-M diagrams, as shown in Fig. 2. Due to paucity of space, we show the diagrams for two languages - Swahili (sw) and Telugu (te) with two different cost ratios for the former (Fig. 2a and 2b), and two different pivot sizes for the latter (Fig. 2c and 2d). Refer appendix (6,7,8,9) for rest.
For  $c_{t / m} = 0.1$ ,  $l = \mathrm{sw}$  (Fig. 2a), the expansion path follows a straight line roughly with a slope  $\left(\frac{c_t a_m}{c_m a_t}\right)^{\frac{1}{1 - \alpha_m}} = 3.2$ . This indicates that even though  $M$  is 10 times more expensive than  $T$ , the optimal allocation policy is to still collect about thrice as much amount of  $M$  as  $T$ . However, for  $c_{t / m} = 0.01$ , which is less than  $\frac{a_t}{a_m}$ , the slope of the expansion path drops to  $\approx 0.08$ , as demonstrated
(b)
(d)
Figure 2: M-T diagrams showing expansion paths obtained through AMUE for Swahili and Telugu for different values of  $P$  and  $c_{t / m}$ . The shaded region represents  $\mathcal{R}$  (cf. Sec.2.3).
by the theoretical expansion path on the right side of the  $M = T$  line in Fig. 2b. This suggests that we can rely on collecting a higher amount of translation data to increase the performance in this case because the manually collected data is much more expensive. As we move to the performance values  $>70$ , we reach at the boundary of the realizable region (marked by translucent gray rectangle), and can no longer keep on collecting more translation data to increase the performance as by definition  $T \leq P$ . Beyond this point, to increase the performance, collecting higher amounts of manual data becomes inevitable.
For Telugu, we study the effect of two different values of  $P$  and keep  $c_{t / m}$  fixed at 0.1. At  $P = 3696$ , the isoperfs are nearly parallel to x-axis with the expansion path lying along the line  $T = 0$  (Fig. 2c), which is expected as  $\frac{a_t}{a_m} \approx 0$  in this case (see Table 1). This particular expansion path indicates that data obtained by translating English examples into Telugu does not have any notable performance improvement, though demands additional cost. The optimal strategy in this case is to only collect manually annotated data. This is not entirely surprising; the translate-train setup
in Hu et al. (2020) also shows low F1-scores for Telugu than the zero-shot setup.6 Interestingly, when  $P = 2000$  (Fig. 2d),  $T$  provides non-trivial performance gains. The expansion curve is bent slightly to the left of the  $M = T$  line, similar to Fig. 2a. This trend of higher  $a_{t} / a_{m}$  for lower  $P$  is observable for all languages (Table 1).
Performance and Cost Trade-off: Fig. 3 plots the cost vs the performance value traced out by the expansion paths for the 8 target languages. To calculate the total cost, we assume  $c_{t} = 0.007$ , which was estimated according to the standard translator Pricing offered by Azure<sup>7</sup>, and consider  $c_{t / m} = 0.01$ . For all the languages, we observe a declining slope as we increase the value of  $C$ . Thus, it becomes increasingly more expensive to improve the performance of the models as we move to the higher values of  $\Pi$  (law of diminishing returns).
Comparing AMUE isoperfs with GPR isoperfs:
<sup>6</sup>Note that this does not invalidate the assumption we made in section 2.2. Hu et al. (2020) fine-tuned their models only on translated data, while we do train them with English Data as well and observe similar performance as zero-shot for Telugu.
$^{7}$ https://azure.microsoft.com/en-us/ pricing/details/cognitive-services/ translator/
Figure 3: Performance vs the minimum costs for different languages. The performance function considered is AMUE. For  $c = 0.1$  case refers to Fig. 10 in appendix.
Figure 4: Comparing the Isoperfs and their corresponding optimum isocosts for AMUE and GPR production functions.
Figure 4 displays the isoperfs and the corresponding optimum isocosts obtained using AMUE and GPR based performance functions. As can be observed, both functions predict similar trends across their isoperfs; however, as expected, the curves are shifted due to different margin of errors for the two models.
In this work, we have proposed a micro-economics inspired framework to study the performance and cost trade-offs between manually annotated and machine-translated data for training multilingual models, and demonstrated its efficacy through a case-study on the TyDiQA-GoldP dataset. The key findings from this case-study are: 1. Some amount of manually collected data in a target language is crucial to attain optimal performance at minimum cost irrespective of how much cheaply MT data can
be procured, as long as the cost is non-zero. 2. The ratio of manually collected and machine-translated data at least cost operating point remains nearly uniform at the different levels of performance 3. The usefulness of translated data is higher when the amount of pivot language data is less. There are several other insights that can be drawn from the T-M diagrams and other plots, which could not be presented here due to the paucity of space.
This work can be expanded in several ways. In the current work we considered a single-pivot and single-target case. Generalizing this to the case where the model is allowed to be trained on multiple pivot languages and then be evaluated on multiple targets is of considerable interest. This implies extension to multiple-output production functions with multiple  $(>2)$  input factors.
Here, we have not considered the effect of multiple technology on the isoperfs. For our problem, multiple technologies may correspond to the different MMLMs such as mBERT, XLMR and mT5, different MT systems, and even different training curricula. Identifying the optimal allocation policy considering the presence of such multiple technological alternatives would be an interesting exercise. In particular, it will be interesting to explore the impact of translation quality on the trade-offs. An important limitation of the current framework is that it presumes availability of certain amounts of M and T datasets such that the performance function can be estimated. However, in practice, one would like to understand the trade-offs before collecting the data. Recently, Srinivasan et al. (2021) showed that it is possible to predict the zero-shot and few-shot performance of MMLMs for different languages using linguistic properties and their representation in the pre-training corpus. Understanding if there exists a similar dependence of the performance trade-offs with the linguistic properties of different languages can help us generalize our framework to the new languages without the need for explicit data collection.
Finally, we believe that performance function-based analysis can be applied to a multitude of three-way trade-offs among technology, cost and data that are commonly encountered in the NLP world. The economics of language data can be a new direction of study with important practical and theoretical applications.
We would like to thank Shanu Kumar and the LIT-MUS team at Microsoft for their valuable inputs and feedback over the course of this work. We are also grateful to the anonymous reviewers for their constructive comments.
George E Batiese. 1992. Frontier production functions and technical efficiency: a survey of empirical applications in agricultural economics. Agricultural economics, 7(3-4):185-208.
Eric Bettinger, Robert W Fairlie, Anastasia Kapuza, Elena Kardanova, Prashant Kumar Loyalka, and Andrey Zakharov. 2020. Does Edtech substitute for traditional learning? experimental estimates of the educational production function. NBER Working Paper, (w26967).
Samuel Bowles. 1970. Towards an educational production function. In *Education, income, and human capital*, pages 11-70. NBER.
Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing, 16(5):1190-1208.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454-470.
Charles W Cobb and Paul H Douglas. 1928. A theory of production. The American Economic Review, 18(1):139-165.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ronald C Griffin, John M Montgomery, and M Edward Rister. 1987. Selecting functional form in production function analysis. Western Journal of Agricultural Economics, pages 216-227.
Ferda Halicioglu and Natalya Ketenci. 2018. Output, renewable and non-renewable energy production, and international trade: Evidence from EU-15 countries. Energy, 159:995-1002.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080.
Shaiara Husain, Md Shahidul Islam, et al. 2016. A test for the Cobb Douglas production function in manufacturing sector: The case of Bangladesh. International Journal of Business and Economics Research, 5(5):149-154.
Guennadi A Khatskevich and Andrei F Pranevich. 2018. Production functions with given elasticities of output and production. Journal of the Belarusian State University. Economics, 2:13-21.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
Kenneth Levenberg. 1944. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 2(2):164-168.
Shibao Lu, Xiao Bai, Wei Li, and Ning Wang. 2019. Impacts of climate change on water resources and grain production. Technological Forecasting and Social Change, 143:76-84.
Ronald E Miller and Peter D Blair. 2009. Input-output analysis: Foundations and extensions. Cambridge university press.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Anirudh Srinivasan, Sunayana Sitaram, Tanuja Ganu, Sandipan Dandapat, Kalika Bali, and Monojit Choudhury. 2021. Predicting the performance of multilingual nlp models. arXiv preprint arXiv:2110.08875.
Petra E Todd and Kenneth I Wolpin. 2003. On the specification and estimation of the production function for cognitive achievement. The Economic Journal, 113(485):F3-F33.
Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. 2021. Revisiting the primacy of english in zero-shot cross-lingual transfer. CoRR, abs/2106.16171.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computational Linguistics.
Oleksandr Yankovyi, Viktor Koval, Larysa Lazorenko, Olga Poberezhets, Marina Novikova, and Viktoriya Gonchar. 2021. Modeling sustainable economic development using production functions. Studies of Applied Economics, 39(5).
Here we derive the expression for the curve traced by the expansion path as given in equation 4. As described in section 2.2 AMUE performance function is given by:
$$
\pi (T, M) = a _ {z s} + a _ {t} T ^ {\alpha_ {t}} + a _ {m} M ^ {\alpha_ {m}}
$$
Setting  $\pi (T,M) = \Pi_c$  i.e. a constant value, we can obtain an analytic expression for the isoperf curves from this functional form, which is given by:
$$
M = \left(\frac {\Pi_ {c} - a _ {z s} - a _ {t} T ^ {\alpha_ {t}}}{a _ {m}}\right) ^ {\frac {1}{\alpha_ {m}}} \tag {5}
$$
Since the expansion path is the locus of the points of tangency between isoperf and isocost curves, we can compute the slope of the isoperf curve and set them equal to each other. The slope for isoperf curve can be computed as:
$$
\begin{array}{l} M ^ {\alpha_ {m}} = \left(\frac {\Pi_ {c} - a _ {z s} - a _ {t} T ^ {\alpha_ {t}}}{a _ {m}}\right) \\ \alpha_ {m} M ^ {\alpha_ {m} - 1} \frac {\mathrm {d} M}{\mathrm {d} T} = - \frac {\alpha_ {t} a _ {t}}{a _ {m}} T ^ {\alpha_ {t} - 1} \\ \frac {\mathrm {d} M}{\mathrm {d} T} = - \frac {\alpha_ {t} a _ {t}}{\alpha_ {m} a _ {m}} \frac {T ^ {\alpha_ {t} - 1}}{M ^ {\alpha_ {m} - 1}} \\ \end{array}
$$
The slope of the isocost curve is simply  $-\frac{c_t}{c_m}$ , equating them we get:
$$
\begin{array}{l} \frac {c _ {t}}{c _ {m}} = \frac {\alpha_ {t} a _ {t}}{\alpha_ {m} a _ {m}} \frac {T ^ {\alpha_ {t} - 1}}{M ^ {\alpha_ {m} - 1}} \\ M ^ {\alpha_ {m} - 1} = \frac {\alpha_ {t} a _ {t} c _ {m}}{\alpha_ {m} a _ {m} c _ {t}} T ^ {\alpha_ {t} - 1} \\ M = \left(\frac {c _ {t} a _ {m} \alpha_ {m}}{c _ {m} a _ {t} \alpha_ {t}}\right) ^ {\frac {1}{1 - \alpha_ {m}}} T ^ {\frac {1 - \alpha_ {t}}{1 - \alpha_ {m}}} \\ \end{array}
$$
A.2 Training Setup
We typically run the fine-tuning experiments on NVIDIA-P100 GPUs with 16 GB of memory. A fine-tuning job with 3 random seeds typically takes 2 hours to run on the specified compute. Having access to 64 of such GPUs we ran multiple jobs in parallel. For fitting performance functions and doing analysis on expansion paths CPU only compute of Intel(R) Xeon(R) CPU E5-2690 was utilized.
We use mBERT configuration bert-base-multilingual-cased for fine-tuning, which supports 104 languages and has around 178 million parameters.
A.3 Goodness of Fit
Table 2 shows the train and test RMSE and  $r^2$  for GPR and AMUE. For training set we also compute the errors corresponding to different fine-tuning setups like translate-train, few-shot etc, which indicates that our models can accurately fit different regions of the performance landscape. The point is again illustrated in Figure5 which compares the predictions of AMUE and GPR with the actual F1-scores for different values of the amount of manual data (i.e.  $M$ ), keeping  $T$ ,  $P$ , and  $p$  as fixed.
Data Split	Fine-tune setup	AMUE	GPR
RMSE↓	r2↑	RMSE↓	r2↑
Train	Zero-Shot	4.19	0.95	4.43	0.95
Translate-Train	5.10	0.93	3.68	0.96
Few-Shot	5.75	0.90	1.63	0.99
Few-Shot + Translate-train	4.71	0.93	1.53	0.99
Overall	5.04	0.93	1.86	0.99
Test	Overall	5.84	0.90	2.43	0.98
Table 2: RMSE and  $r^2$  values for the two performance functions on training and test sets.
Figure 5: Performance function estimated by AMUE and GPR.  $\Pi \equiv \mathrm{F1}$  -score (scaled by 100).
(e)
(g)
Figure 6: M-T diagrams for different languages for  $P = 3696$  and  $c_{t / m} = 0.1$
(f)
(h)
(e)
(g)
Figure 7: M-T diagrams for different languages for  $P = 2000$  and  $c_{t / m} = 0.1$
(e)
(g)
Figure 8: M-T diagrams for different languages for  $P = 3696$  and  $c_{t / m} = 0.01$
(c)
(e)
Figure 9: M-T diagrams for different languages for  $P = 2000$  and  $c_{t / m} = 0.01$
(d)
(f)
Figure 10: Performance vs the minimum costs for different languages for  $c = 0.1$ . As expected the overall costs are now lower than in figure 3, since the manual data is cheaper in this case.