Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency
Year: 2023

Abstract:
The field of Math-NLP has witnessed significant growth in recent years, motivated by the desire to expand LLM performance to the learning of non-linguistic notions (numerals, and subsequently, arithmetic reasoning). However, non-linguistic skill injection typically comes at a cost for LLMs: it leads to catastrophic forgetting of core linguistic skills, a consequence that often remains unaddressed in the literature. As Math-NLP has been able to create LLMs that can closely approximate the mathematical skills of a grade-schooler or the arithmetic reasoning skills of a calculator, the practicality of these models fail if they concomitantly shed their linguistic capabilities. In this work, we take a closer look into the phenomena of catastrophic forgetting as it pertains to LLMs and subsequently offer a novel framework for non-linguistic skill injection for LLMs based on information-theoretic interventions and skill-specific losses that enable the learning of strict arithmetic reasoning. Our model outperforms the state-of-the-art both on injected non-linguistic skills and on linguistic knowledge retention, and does so with a fraction of the non-linguistic training data (1/4) and zero additional synthetic linguistic training data. Our pre-trained models and experimentation codebases are hosted online1.

heading: 1 Introduction
text: Numeracy, involving the comprehension of sizes, magnitudes, and order, is the most prevalent form of non-linguistic information embedded in textual corpora (Joram et al., 1995). Thus, the case for numerically-capable LLMs is rather easy to make: as numerals grant objectivity to language (Porter, 1996), numerically-capable language models are key to optimal performance in a host of downstream tasks such as information extraction (Madaan et al., 2016), inference (Naik et al., 2018),
1https://github.com/Mandar-Sharma/Skill-LM
and data-to-text generation (Sharma et al., 2021, 2022a).

heading: 1.1 Re-thinking the Objective of Math-NLP
text: Progress in Math-NLP: Several notable publications in the Math-NLP space have made rapid strides in numeracy-tinged language-modeling (Thawani et al., 2021) - from investigations of the inherent deficiency of numerical reasoning skills in LLMs induced through unsupervised training, both for numerals that appear in the training corpus (Zhang et al., 2020) and OOD (out-of-domain) numerals (Wallace et al., 2019; Razeghi et al., 2022), to interventions that strengthen the numerical reasoning skills of these models (Spithourakis and Riedel, 2018; Jiang et al., 2020; Geva et al., 2020). Further, advances in chain-of-thought prompting in few-shot learning settings (Li et al., 2022) and
6178
task-specific fine-tuning (Lewkowycz et al., 2022) have shown significant gains in the capacity for quantitative reasoning in LLMs.
Linguistic evaluation remains important: As notable as these accomplishments are, the goal remains not to replicate the reasoning capabilities of a grade-schooler or to proxy a calculator, but rather build LLMs that are empowered with these skills. As such, an area that often goes unaddressed in the Math-NLP space is how these models perform as general language modelers. With the advent and popularity of generative conversational models (OpenAI, 2022), the goal is to have one model capable of a host of skills - not to load separate models for conversation/assistance and reasoning. As depicted in Figure 1, whether a model is designed to perform strict non-linguistic tasks or semi-linguistic tasks, it should never come at the cost of core linguistic competency. After all, language models are intended to model language.

heading: 1.2 Necessitating the Re-thinking
text: LLMs injected with non-linguistic skills forgo their linguistic skills: Consider the task of strict arithmetic reasoning as shown in Figure 1, a subset of possible quantitative reasoning tasks. If a base BERT model (Devlin et al., 2019) is further trained on this non-linguistic task, it suffers significant degradation on 8/9 GLUE tasks (Wang et al., 2018) that evaluate the natural language understanding (NLU) capabilties of the model, as showcased in Table 1. This observation has long been known in the deep learning literature as catastrophic forgetting (Kirkpatrick et al., 2017), wherein when a model pre-trained on task A is further trained on task B, the parameters in the model vital for task A adapt their values to meet the requirements of task B.
LLMs exhibit unconventional forgetting: What is interesting, based on our findings, is that in the case of LLMs, the forgetting of linguistic skills is not evenly spread - the forgetting is rather taskspecific. Akin to other neural network applications,
the forgetting of linguistic skills may likely be grouped as performance loss over a single task A; however, as seen in Table 1, the GLUE tasks suffer various ranges of degradation - the task of finding the referent of a pronoun (WNLI, Levesque et al. (2012)) does not seem to suffer at all, while the grammatical correctness assessment task (CoLA, Warstadt et al. (2019)) suffers severe degradation.
As proponents for skill-empowered LLMs, we thus make a case for disclosing the performance on general NLU tasks when models are trained for superior performance on niche skill-sets such as non-linguistics, an area left wanting in the MathNLP front. Because of this task-specific forgetting, quantitative reasoning models trained in a Q&A fashion may not showcase degradation in similarly modeled downstream tasks such as SQuAD (Rajpurkar et al., 2016) and DROP (Dua et al., 2019) - thus disclosing performance across a range of NLU tasks is crucial.
Substantiating forgetting on the basis of parameter sharing: To establish that observed performance degradation can indeed be accredited to catastrophic forgetting, we take an information theoretic lens to pry into parameter-sharing tendencies across tasks with the aid of Fisher information (Rissanen, 1996). For a single sample y drawn from a distribution with probability desnity f(y; θ), the Fisher information index I(θ) (1) quantifies the sensitivity of the parameter θ to the data instance y. Thus, given a task-specific training corpus (X,Y ) ∈ Dtask, we can estimate the sensitivity of each model parameter θi ∈ θ for the given task.
I(θi) = Ey∈Y ( d log f(y; θi)
dθi )2 (1)
= −Ey∈Y ( d2 log f(y; θi)
dθ2i ) (2)
Using this formulation, we compute the Fisher parameter sensitivities I(θ) for four different models based on continued training of the base BERT model on four datasets:
• Iarith(θ): for BERT trained on an arithmetic reasoning dataset (Geva et al., 2020)
• ICoLA(θ), IMRPC(θ), IRTE(θ): for BERT trained on three GLUE (Wang et al., 2018) tasks CoLA, MRPC, and RTE respectively
To ground our hypothesis of task-specific forgetting as a consequence of parameter-sharing, first, we select n = 800 parameters deemed most sensitive for arithmetic reasoning from Iarith(θ), and compare how important those same parameters are for the three GLUE tasks based on their respective Fisher scores ICoLA(θ), IMRPC(θ), IRTE(θ) (see Appendix §A.1.1 for details on Fisher score computations). As seen in Figure 2, for the first and last self-attention encoder layers, the sensitivities of the parameters across tasks correlate well with the findings of Table 1 - the NLU task that suffers the least performance degradation (WNLI) also has the least sensitivity to these (shared) parameters across the encoder layers, while the NLU tasks that do suffer from performance degradation (MRPC, CoLA, RTE) have varying ranges of shared sensitivities across the encoder self-attention layers. These findings hold consistent across all 12 encoder layers of the BERT model (see Appendix §A.1.2).
Our contributions: In line with the above observations, we offer the following contributions in the form of our proposed model, Skill-LM, for nonlinguistic skill injection in LLMs:
• Novel multi-task skill-injection loss that in-
fuses a sense of numeral structure in the learned representations, leading to better generalization performance than the state-of-theart, all with a significantly lower fraction (n4 ) of training data.
• Weight consolidation schemes for LLMs for better linguistic retention with 0 additional linguistic samples compared to 1 million synthetic textual training samples used by the state-of-the-art.
• Through exhaustive qualitative and quantitative evaluations, we demonstrate the improved generalization performance of Skill-LM over the state-of-the-art. Our experiments also highlight the need for disclosing linguistic performance for models trained on highly-niche non-linguistic tasks.

heading: 2 Designing Skill-LM

heading: 2.1 Non-Linguistic Learning
text: Based on probabilistic modeling, language models are trained to output the next sequential token yt at timestep t based on the n tokens already predicted by the model, formulated as P (yt|yt−1, ..., yt−n) = P (yt|y<t). This probability distribution P is often optimized through measures of uncertainty such as cross-entropy or KL-divergence. The application of these same loss functions used for learning linguistic token distributions may not necessarily translate to the learning
of non-linguistic entities. Unlike linguistic tokens, the magnitude of a numeral is especially tied to its meaning (Dehaene et al., 1998). This magnitude can either be modeled as a continuous linear representation (Dehaene et al., 1990) or a log-compressive representation (Dehaene, 2003). Thus, to inject this numeric-scale representation into a language model, we take a simplistic approach of augmenting the learning of tokens through cross-entropy LCE with a regression loss LREG2. This regression loss is incorporated into the quantitative reasoning loss function LQ as represented in (3, 5).
LQ(θ) = LCE + λ1 .LREG (3) LCE = −log(P (yt|y<t)) (4)
LREG =
√√√√ n∑
i=1
(y2 − ŷ2) (5)
Figure 3 (left) depicts the convergence of LREG for different configurations of λ1. Please see Appendix §A.2.1 for further details on the update schedules for hyperparameter tuning.

heading: 2.2 Linguistic Retention
text: Among prominent strategies for multitask learning, a system-level consolidation scheme consists of stitching-together amalgamated datasets constituting multiple-shared tasks (Kumaran et al., 2016). However, due to the limitless range of possible
2As in the initial phases of model training, incorrect predictions of target numerals can lead to exceedingly large values of LREG, thus our choice of seed values for λ1 were set to {1e−3, 1e−4} as not to exceed the range of LCE {0, 1} by values greater than an order of magnitude.
downstream tasks that LLMs are often employed for, the paradigm consists of building large models that hold linguistic prowess and are intended to be fine-tuned on a single downstream task (Devlin et al., 2019; Brown et al., 2020), thus suited for a continual learning paradigm. As depicted in Figure 1, the high degree of parameterization of these models leads to the belief that there is a solution for {task B, θB}, a non-linguistic skill, that is proximal to the linguistic solution space for the model {task A, θA} (Sharma et al., 2022b). To enable this continual learning, we adapt the elastic weight consolidation (EWC) regularization to LLMs - elastic as it functions as a spring, anchoring the solution space closer to θA (Kirkpatrick et al., 2017). Thus, EWC penalizes changes to specific network weights deemed vital for linguistics while injecting non-linguistic skills into the model.
In line with our task-specific parametric observations from Introduction §1.2, we compute F = IBERT (θ), the Fisher information index for the base BERT model based on a portion of its original pre-training corpus - WikiText (Merity et al., 2016), thus approximating its posterior distribution. Let us assume that θ∗ling represents the set of parameters of a converged base-BERT model pre-trained for linguistics. We now introduce the quadratic penalty LEWC (6, 7) that penalizes changes to any model parameter i crucial to the core linguistic functionality of the pre-trained model.
L(θ) = LQ(θ) + λ2 .LEWC (6)
LEWC = ∑
i
1 2 Fi(θi − θ∗ling,i)2 (7)
In this loss formulation, the hyperparameter λ2
is crucial as it dictates both model convergence and balances the learning of quantitative reasoning skills θQ with linguistic prowess θling. To evaluate the sensitivity of model convergence with respect to λ2, we perform a hyperparameter sweep between {1e−6, 1e−10} - Figure 3 (right) showcases the interplay between LCE and LEWC (color-matched) for the best performing values of λ2 on the validation set. The first sign of model convergence is observed at λ2 = 1e−7, and although slight improvements to model convergence are noted for even smaller values of λ2, the smallest value that allows for convergence, theoretically, allows for balanced learning of θQ with θling.

heading: 3 Experiment Setup and Results

heading: 3.1 Tasks and Datasets
text: The goal of Skill-LM is to empower LLMs with non-linguistic skills in a manner that avoids catastrophic forgetting of linguistic skills without the aid of additional synthetic linguistic training. Thus, we have two categories of tasks that Skill-LM, along with the baselines, should be evaluated on:

heading: 3.1.1 Quantitative Reasoning
text: To hold fair comparisons to GenBERT (Geva et al., 2020), we both train and evaluate all models with the arithmetic reasoning portion of their dataset. The data instances take the form of the sample arithmetic task demonstrated in Figure 1. The corpus consists of Ntrain = 165, 000 training samples and Nval = 1666 validation samples, where the numerals are in the range {1, 203} with numeral ranges stratified between the training and validation sets. For our models, we randomly sample Ntrain
4 instances for training.
OOD Performance: The out-of-domain (OOD) performance of all models are evaluated on data instances generated in the same manner but for numeral ranges {203, 106} that are unseen for all models evaluated.

heading: 3.1.2 Natural Language Understanding
text: Following standard protocols, we employ all 9 tasks in the GLUE benchmark (Wang et al., 2018) as metrics for linguistic prowess of a model. The tasks, as per the benchmark, are categorized into three groups:
• Single Sentence Tasks: CoLA (the Corpus of Linguistic Acceptability) (Warstadt et al.,
2019) for grammatical fidelity (Matthews correlation), SST-2 (the Stanford Sentiment Treebank) (Socher et al., 2013) for sentiment prediction
• Similarity and Paraphrase Tasks: MRPC (the Microsoft Research Paraphrase Corpus) (Dolan and Brockett, 2005), QQP (the Quora Question Pairs), and STS-B (the Semantic Textual Similarity Benchmark) (Cer et al., 2017) for semantic equivalence
• Inference Tasks: MNLI (the Multi-Genre Natural Language Inference Corpus) (Williams et al., 2018) and RTE (Recognizing Textual Entailment) for textual entailment, QNLI (the Stanford Question Answering Dataset) (Rajpurkar et al., 2016) for Q&A, and WNLI (the Winograd Schema Challenge) (Levesque et al., 2012) for pronoun referent selection.
Evaluation Metrics: Besides CoLA (evaluated using the Matthews correlation coefficient) and STS-B (evaluated using a combination of the Spearman’s and Pearson’s correlation coefficients), all result shown represent accuracy for the respective GLUE task.

heading: 3.2 Baselines
text: For assessment of both quantitative reasoning skills and linguistic prowess through natural language understanding, the following three models are used as the baselines for this experimentation. For the training specifics, please see Appendix §A.2.2.
• BERT: In this evaluation, this base pre-trained model establishes the standard for natural language understanding that all BERT-derivatives designed for non-linguistic skills should strive to achieve. Thus, its performance on the set of GLUE tasks are italicized in Table 3.
• BERTArith: This is the model generated from the continued training of the pre-trained BERT model on the quantitative reasoning dataset using the standard cross-entropy LCE loss. This model showcases the current paradigm of skill-injection where the architecture of a model is left unchanged and the training parameters are often adapted to meet performance requirements in the target task.
• GenBERT (Geva et al., 2020): This BERTbased model is trained for numerical reason-
1 4 of the training data) for both the in-domain validation set {1, 20e3} and out-of-domain (OOD) sets for numeral ranges {20e3, 10e4}, {10e4, 10e5}, and {10e5, 10e6}. Using a fraction (n4 ) of the original training data, Skill-LM not only closely matches the in-domain performance of GenBERT, it significantly improves the generalization performance to OOD numerals ranges as well.
Model Training Samples CoLA STS-B MNLI MNLIMM MRPC BERT - 0.59 0.89 83.85 84.05 86.76 BERTArith 0 0.08 0.80 32.73 32.95 70.34 GenBERT 1 Million 0.540.001 0.880.001 83.000.576 83.401.107 85.040.693 Skill-LM 0 0.580.041 0.890.003 84.070.158 84.661.123 86.881.123
Model Training Samples QNLI QQP RTE SST-2 WNLI
BERT - 90.55 90.61 65.34 91.62 56.33 BERTArith 0 50.53 70.49 47.29 88.07 56.33 GenBERT 1 Million 90.830.012 90.780.316 67.862.042 91.510.648 55.630.995 Skill-LM 0 91.540.207 90.960.043 65.701.531 92.370.081 56.180.216

heading: 3.3 Quantitative Results

heading: 3.3.1 Numerical Reasoning
text: From Table 2, we observe that using only 14 th of the training dataset, Skill-LM closely resembles the performance of GenBERT while significantly improving the performance on out-of-domain numeral ranges. This leads to two deductions:
• It is known in the literature that LLMs often struggle to extrapolate numeral ranges that are absent from the training corpus (OOD) (Wallace et al., 2019; Razeghi et al., 2022). The

heading: 3.3.2 Natural Language Understanding
text: Recall that our goal with Skill-LM is to prevent the loss of linguistic prowess as a consequence of nonlinguistic skill injection. The premise therein is that BERT-derivatives, empowered with non-linguistic skills, should at least strive to have linguistic performances of the base model. Thus, the performance of the base BERT model is italicized in Table 3.
In §1.2, we established the degradation of linguistic performance in LLMs as a consequence of non-linguistic skill injection. Thus, the goal of weight consolidation LEWC was to revitalize the linguistic performance of the model back to baseline. However, from Table 3, we observe that employing LEWC that uses 0 additional training data outperforms GenBERT that uses 1 Million additional linguistic training data on 8/9 of the standardized GLUE benchmarks. To further authenticate these findings, the results are presented as µσ (mean and standard deviation) across two runs of training-validation with different seeds for model initialization. Thus Skill-LM showcases improved performance coupled with significant savings in GPU compute costs compared to previous related efforts that train on an additional 1 Million linguistic training samples (Geva et al., 2020).

heading: 3.4 Qualitative Results
text: In §2.1, we theorized that regression loss, in the context of numerical skill injection, would inject a sense of numeric scale and magnitude estimation (Dehaene et al., 1998) to the general learning of numerical representations. From Table 2 we quantified the gains from this skill-specific loss in OOD generalization of numerals, however, in this section we further investigate whether the extrapolation to OOD numerals is indeed due to this learnt sense of numeric scale.
OOD numerals closer to the training range: In Table 2, Skill-LM boosts the predictive performance for OOD numerals in the range [203, 104] from 1.32% to 19.44% - but where does the baseline fail? In Figure 5, as common-case failure scenarios, we showcase 3 sample responses from SkillLM vs baseline to prompts from the OOD range [203, 104]: while the baseline does capture the nuances in difference of the operands (the numerals closer the decimal are correct), it severely fails to extrapolate to the scale of the operands.
OOD numerals further from the training range: In Table 2, the evaluation metric used is accuracy, thus evaluating the capabilities of these models to output the exact token in response to the quantitative reasoning prompts. For larger OOD ranges [104, 105] and [105, 106], all models struggle to predict the exact output - but how close do they get? Figure 4 showcase the distribution of the predicted output based on their powers of 10s - for the OOD range [104, 105], the outputs should mostly center around 105 (left figure) while for the range [105, 106] they should center around 106 (right figure). Although unable to predict the exact tokens, Skill-LM tends to predict tokens closer in magnitude to the ground truth consistently compared to our baseline.

heading: 4 Conclusions
text: Our study shows that LLMs are capable of demonstrating quantiative reasoning without sacrificing the broad palette of linguistic skills that they are traditionally evaluated against. This multi-task framework, together with the weight consolidation strategy, highlights that this framework can be systematized beyond the studies described here. As a result, non-linguistic tasks and linguistic tasks need not be seen as being at odds for LLMs and we can begin thinking about richer integrations of qualitative and quantitative reasoning. Our experimental results also highlight that the improvements showcased here do not require exorbitant training data and in fact require just a fraction of what previous studies have leveraged.
Our future work will be organized in three directions. First, we intend to study at a more finegrained level the dovetailing of different arithmetic reasoning tasks vis-a-vis linguistic counterparts, and any synergies that can be exploited while learning. Second, there are situations where linguistics can help numerical reasoning (math word problems, data-to-text generation) and multi-task formulations that capture the underlying semantics can be developed. Finally, there are other forms of non-linguistic reasoning (diagrammatic reasoning) that can potentially be studied using the multi-task framework that we have described here.
Limitations
In our study, we address the issue of linguistic forgetting via the injection of the strict non-linguistic skill of quantitative reasoning. Although quantitative reasoning with LLMs is an active research area, as discussed above, further fine-grained studies are required to extrapolate this behavior to tasks that leverage synergies between aspects of both linguistics and non-linguistics - such as math word problems or data-to-text generation. Further, investigations into the linguistic forgetting tendencies of different languages would lend an insight into the role of linguistic morphology in this behavior. The restrictions from our in-house GPU resources does not allow scaling this study to more recent models that exceed 100 Billion parameters, although, due to the sharing of similar architectures, we forecast our findings to hold despite of model scaling.
Ethics Statement
Although the ethical waters of the development and deployment of LLMs are difficult to nagivate, we can ascertain that our study does not bring forth further complications. The datasets we use in this study are established benchmark datasets from publicly accessible websites and do not contain any personally identifiable information. Our analyses does not constitute human subjects and thus is not within the purview of the IRB. Further, in the landscape of increasing emission costs from large-scale computation, our study offers avenues for severely restricting the size of the training data - both linguistic and non-linguistic.

heading: A Appendix
text: A.1 Substantial Forgetting on the Basis of Parameter Sharing
A.1.1 Fisher Information Computation The Fisher information score, as depicted in (1) is the expected value of the square of the gradient for a sample y ∈ Y . Thus, to compute the Fisher sensitivity of a model θ to a task A, we compute the sum of the squared gradients averaged by the number of parameters in θ. In our case, where θ is a pretrained transformer-based LLM, the model cross-entropy loss dlogf(y; θ) (4) for each sample y is computed, through which the gradient dlogf(y;θ)dθ can then be computed. The sum of these squared gradients gives us the Fisher information score for each parameter θi in a model θ with respect to a task A.
A.1.2 Parameter Sensitivities for the Self-Attention Encoder Layers
In §1.2, we substantiated the linguistic forgetting of LLMs through parameter sharing tendencies of the model with illustrations of the parameter sensitivities across different tasks for the first (1st) and last (12th) self-attention encoder layer of the transformer. Here, through Figure 6, we show that the findings hold across all self-attention encoder layers of model. Further, it is interesting to observe that the task CoLA shares more parameters with the Arithmetic task in the earlier layers compared to the latter layers.
A.2 Designing Skill-LM A.2.1 Hyperparameterization for LREG The intuition for the selection of hyperparameter λ1 within the range {1e−3, 1e−4} was to scalematch the exceedingly large values of regression loss LREG to the cross-entropy loss LCE during the intial phases of training where incorrect predictions of target numerals are frequent. In addition to evaluating the model convergence with λ1 set to these constants, we also evaluate the following update schedule configurations for λ1:
Algorithm 1 Update Schedule 1
λprev ← 1e−4 for i in epochs do
λcurrent ← LREGLCE+LREG λ1 ← 0.99 ∗ λprev + 0.01 ∗ λcurrent λprev ← λ1
end for
Algorithm 2 Update Schedule 2
λprev ← 1e−4 for i in epochs do
λcurrent ← LREGLCE+LREG λ1 ← 0.01 ∗ λprev + 0.99 ∗ λcurrent λprev ← λ1
end for
A.2.2 Model Training Configurations The models BERTArith, GenBERT, and Skill-LM all share the base BERT architecture. The baseline GenBERT has been employed as-is with the model that the authors provide used for comparative evaluation. For models BERTArith and Skill-LM, these are initialized as pre-trained base BERT models with 160M parameters and further trained on randomly sampled n4 th of the arithmetic portion of GenBERT’s training data. The pre-trained base BERT model is loaded from the HuggingFace library (Wolf et al., 2019).
The scheme for training follows BERT’s standard training protocol of using masked-language modeling. However, instead of randomly masking 15% of the tokens as done in BERT, we mask the result of the each sample quantitative prompt. For instance, from Figure 5, for the sample 61176.23 - 46741.95 = 14434.28, the models BERTArith and Skill-LM are trained to predict 14434.28 for the masked prompt 61176.23 - 46741.95 = [MASK]. With the standard sequence size of 512 for BERT, the models were trained for 60 epochs in a cluster of 4 Tesla P100 GPUs.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Introduction (Section 1)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Introduction (Section 1) , Experiments (Section 3)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Experiments (Section 3) and Appendix
C 3 Did you run computational experiments? Introduction (Section 1), Experiments (Section 3)

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Designing Skill-LM (Section 2)

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Experimentation (Section 3)

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Our future work will be organized in three directions. First, we intend to study at a more finegrained level the dovetailing of different arithmetic reasoning tasks vis-a-vis linguistic counterparts, and any synergies that can be exploited while learning. Second, there are situations where linguistics can help numerical reasoning (math word problems, data-to-text generation) and multi-task formulations that capture the underlying semantics can be developed. Finally, there are other forms of non-linguistic reasoning (diagrammatic reasoning) that can potentially be studied using the multi-task framework that we have described here. Limitations
In our study, we address the issue of linguistic forgetting via the injection of the strict non-linguistic skill of quantitative reasoning. Although quantitative reasoning with LLMs is an active research area, as discussed above, further fine-grained studies are required to extrapolate this behavior to tasks that leverage synergies between aspects of both linguistics and non-linguistics - such as math word problems or data-to-text generation. Further, investigations into the linguistic forgetting tendencies of different languages would lend an insight into the role of linguistic morphology in this behavior. The restrictions from our in-house GPU resources does not allow scaling this study to more recent models that exceed 100 Billion parameters, although, due to the sharing of similar architectures, we forecast our findings to hold despite of model scaling