Task-Incremental Learning on Long Text Sequences
Authors: Natalia Graziuso, Andrea Zugarini, Stefano Melacci (Università di Siena, expert.ai, 2024)
Topic: Language Models

Abstract
The extraordinary results achieved by Large Language Models are paired with issues that are critical in real-world applications. The costs of inference and, in particular, training are extremely large, both in terms of time and computational resources, and they become prohibitive when working in dynamic environments, where data and tasks are progressively provided over time. The model must be able to adapt to new knowledge, new domains, new settings, without forgetting the previously learned skills. Retraining from scratch easily becomes too costly, thus Continual Learning strategies are of crucial importance. This is even more evident when data consist of ``long'' documents, that require several resources to be processed by modern neural models, leading to very long prompts. This paper investigates LLM-based Task-Incremental Learning in the case of tasks exploiting long sequences of text, as it is typical in summarization, question-answering on long documents, reviewing long contracts, and several others. We show how adapting the model by Task Arithmetic with LoRA, which was proposed for visual data, yields promising results also in the case of such ``long'' text data. To our best knowledge, this is the first work along this challenging direction. The outcome of the investigation of this paper is generic enough to represent an important starting point for further research in processing linguistic data in every language.

Introduction
The quality of Language Models (LMs) has been rapidly improving in the last decade, showing outstanding skills when scaled to large data and networks AUTHOR, leading to the nowadays popular Large Language Models (LLMs). Solving more complex tasks with LLMs often requires processing ``long'' documents and articulated long instructions. However, handling lengthy prompts can be a significant obstacle for real-world applications, raising costs and resources required during both inference and, in particular, training. This issue can become critical when the LLM needs to be specialized to many different tasks, domains, and, more generally, when it is applied to dynamic settings that require multiple adaptations. For instance, in real-world applications, models need to be re-trained from time to time, as new data/tasks become available. In such scenarios, the need for Continual Learning (CL) AUTHOR strategies becomes imperative. From a very generic perspective, CL focuses on the development of algorithms capable of sequentially learning from a stream of data, while preserving what was learnt in past experiences, avoiding catastrophic forgetting AUTHOR. %Integrating both the handling of long texts and the development of multi-task models into a CL framework adds another layer of complexity to an already challenging landscape. In this work, motivated by the aforementioned issues, we study the problem of Continual Learning from ``long'' sequences of text, exploiting LLMs. We investigate several strategies based on LoRA AUTHOR to adapt an LLM to multiple tasks that are sequentially proposed over time. In particular, we first follow the route of training a single adapter in a sequential manner, then we explore Task Arithmetic to fuse multiple adapters trained independently AUTHOR. We consider the possibility of assigning different weights to each task, and we shed some light on what are the factors that contribute the most to catastrophic forgetting and to effective task adaptation. The outcomes of such an investigation reveals that: (1) there is limited sensitivity to task-order, i.e., regardless of the sequence in which tasks are presented, the overall average performance remains relatively stable, a property that, to our best knowledge, was never evaluated in the case of tasks composed of long documents; (2) despite its simplicity, Task Arithmetic demonstrates effectiveness in addressing forgetting phenomena when learning from long texts, %, and a very small fine-tuning on few examples from all the tasks, collected in a memory buffer, greatly improves the overall results, strongly reducing the gap from multiple models independently adapted to the task data. Moreover, (3) we are the first to evaluate a recently proposed benchmark (SCROLLS AUTHOR) in a CL setting, offering reference results for further activity in processing long sequences of text. We remark that while our experiments are based on data in English language, the generic issues we explore about handling long sequences of text are intrinsically shared by every language. %Influence of Output Style on Performance: We observed a notable influence of output style on performance outcomes, as illustrated by the comparison between Natural Language Inference and Summarization tasks. The distinct stylistic requirements of these tasks have a discernible effect on performance metrics, highlighting the importance of considering output style in task evaluation and analysis. %may be glued with introduction In the last few years, a variety of approaches were proposed by the scientific community in the context of CL (see AUTHOR and references therein). The main goal is the one of learning from newly provided information, with models that are capable of acquiring new knowledge without forgetting the previously learned one, and, more importantly, without storing the full dataset and retraining from scratch every time AUTHOR. Several efforts are dedicated to the case of lifelong Reinforcement Learning AUTHOR and of Supervised Learning AUTHOR, distinguishing among scenarios and categories of approaches AUTHOR, ranging from parameter isolation, regularization methods, and replays AUTHOR. Unsupervised or Self-Supervised Learning approaches are also becoming popular AUTHOR, and the case of adaptation of pre-trained backbones AUTHOR. Of course, neural models for processing language are a subject of study in the context of CL AUTHOR. We mention the case of language modeling in Lamol AUTHOR, which is trained to concurrently solve a task and mimic training examples, thereby preserving the distribution of previous tasks. Sun et al. AUTHOR introduce Distill and Replay, %which is trained on a sequence of tasks similarly to the approach applied in Lamol. It which learns to solve the task, to generate training examples formatted as context-question-answer, and to distill knowledge from a model trained on the previous task(s). %Before fitting the model to the next task, pseudo samples are generated to represent the data distribution of previous tasks. Differently, Reasoning-augmented Continual Learning AUTHOR focuses on creating reasoning pathways to preserve and improve LLMs’ reasoning abilities and information transfer. %Meaning that the LLM must provide the textual reasoning followed to compute the answer along with the answer itself. Surprisingly, it offers performance comparable to replay-based techniques. Together with works that learn new models from scratch, several approaches devise fine-tuning strategies for pre-trained Transformers in language processing, that turn out to be efficiently adaptable to a downstream task by learning only a small number of task-specific parameters. It is the case of models that tune the input prompt AUTHOR or of generic Adapters AUTHOR, such as the popular LoRA AUTHOR, which introduces new weight matrices, parametrized by the product of low-rank ones. Evaluating these models with long contexts AUTHOR is not frequent in the scientific literature, especially in the case in which multiple fine-tunings are sequentially applied, typical of CL, which is the main focus of this paper. In particular, LoRA and Task Arithmetic AUTHOR has been jointly studied to handle CL problems in vision AUTHOR, that is what this paper extends to the case of language and long sequences. We also mention works that focus instruction-based model for CL, such as ConTinTin AUTHOR, where each task is modelled by a specific instruction that directly defines the target concept along with a few instances that illustrate. %The authors start from NATURAL INSTRUCTIONS benchmark AUTHOR, whose instructions consist of 7 fields: Title, Definition, Caution, Prompt, Things to avoid and Examples, and they restructure the datasets in it in order to process them as a stream of tasks. Briefly, the system is required to perform a new task, generating its expected outputs, from its instruction, transfer the knowledge acquired from old tasks to help solve the new ones and simultaneously improve the performance on the upstream tasks. Scialom et al. AUTHOR and Luo et al. AUTHOR investigate natural language instructions paired with memory buffers and replays. %Remarkably, this approach allows to maintain almost 100% of the initial performance and turns out to be task order invariant, even though storing a buffer of data is still recognized as one of the limitations. Furthermore, their study concludes to exclude that CL adaptability actually emerges from the scale of the model, but rather from an intensive pre-training stage. To state it, they evaluate their model, CT0, not only on new and old tasks, but also on new instructions generated through instruction composition. In this scenario, they evaluate zero-shot performance and CT0 seems to succed in understanding instructions beyond those it was exposed to during training.

Conclusion
We investigated Large Language Models in progressively learning from tasks involving long sequences of text. A pre-trained model was paired with one or more adapters (LoRA), and we analyzed the role of Task Arithmetic, showing that it yields performances that are not far from the ones of multiple models independently trained to solve each task. Our results suggests a viable road to mitigate the need of large computational resources when learning from tasks based on ``long'' documents. While we exploited data in English language, the experiences of this paper can be interpreted as generic attempts to leverage long sequences in Continual Learning, in a sense going beyond the language barrier. Future work will consider schemes to automatically tune the Task Arithmetic AUTHOR. The work was partially funded by: % \item ``ReSpiRA - REplicabilità, SPIegabilità e Ragionamento'', a project financed by FAIR, Affiliated to spoke no. 2, falling within the PNRR MUR programme, Mission 4, Component 2, Investment 1.3, D.D. No. 341 of 03/15/2022, Project PE0000013, CUP B43D22000900004 }; % \item ``enRichMyData - Enabling Data Enrichment Pipelines for AI-driven Business Products and Services'', an Horizon Europe (HE) project, grant agreement ID: 101070284 }.