<figure><img src="image_1.png" /></figure>

HAL
open science

# Transfer Learning Through Knowledge-Infused Representations with Contextual Experts

Daniel Biermann, Morten Goodwin, Ole-Christoffer Granmo

► To cite this version:

Daniel Biermann, Morten Goodwin, Ole-Christoffer Granmo. Transfer Learning Through Knowledge-Infused Representations with Contextual Experts. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.569-578, 10.1007/978-3-031-34107-6_45. hal-05498102

HAL Id: hal-05498102

https://inria.hal.science/hal-05498102v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_4.png" /></figure>

# Transfer Learning Through
Knowledge-Infused Representations
with Contextual Experts

Daniel Biermann(✉), Morten Goodwin, and Ole-Christoffer Granmo

Centre for Artificial Intelligence Research (CAIR), Department of ICT,
University of Agder, Grimstad, Norway
{daniel.biermann,morten.goodwin,ole.granmo}@ua.no

**Abstract.** In recent years, transfer learning in natural language processing has been dominated by incredibly large models following a pretraining-finetuning approach. A problem with these models is that the increasing model size goes hand in hand with increasing training costs. In this work, we instead evaluate the transfer learning capabilities of the recently introduced knowledge-infused representations. Previously, these infused representations have been shown to infuse experts' knowledge into a downstream model when the expert and downstream model operates on the same task domain. We extend this by investigating the effects of different expert task configurations on the performance of the downstream model. Our results show that differing expert and downstream tasks do not affect the downstream model. This indicates a desired robustness of the model towards adding irrelevant information. Simultaneously, the ability to transport important information is retained as we continue to see a significant performance improvement when adding two experts of differing tasks. Overall, this solidifies the potential knowledge-infused representations have regarding the ability to generalize across different tasks and their ability to recycle old computations for smaller new downstream models.

**Keywords:** Machine Learning · Text classification · Transfer learning

## 1 Introduction

Data-driven machine learning suffers from an inherent problem: Data-driven algorithms perform best when trained for narrowly defined tasks on a task-specific dataset, and the disadvantage is that a trained algorithm will fail if the task is changed. Moreover, the information contained in the trained algorithm for one task is, in general, of no use for another task. The trend of increasing performance by increasing the scale of the models makes training the current state-of-the-art increasingly time-consuming and expensive. The ability to effectively use previously trained models could help reduce the temporal, financial, and environmental impact of computations training models for new tasks. This

© IFIP International Federation for Information Processing 2023

Published by Springer Nature Switzerland AG 2023

I. Maglogiannis et al. (Eds.): AIAI 2023, IFIP AICT 676, pp. 569–578, 2023.

https://doi.org/10.1007/978-3-031-34107-6_45
problem lies at the heart of transfer learning, a field that investigates the transfer of knowledge between models across tasks, datasets, and architectures.

Machine learning algorithms have made incredible improvements in performance over the last decade, and transfer learning has been a significant contributor. Many state-of-the-art models use a pretraining-finetuning approach in their methodology. This pretraining-finetuning approach is rooted in transfer learning as the model is first trained on a large general dataset and subsequently finetuned on a smaller task-specific dataset. The general idea is to learn general structures in the data and finetune the learned structure for a downstream task. Examples of this approach find application in all fields of machine learning. Natural language processing has been particularly impacted through the Transformer models pretrained on vast unlabelled datasets [2, 4, 18].

Transfer learning methods are not limited to pretraining-finetuning methods. For example, the text-to-text transformer (T5) architecture [12] performs transfer learning on a model-based level. The idea is to train a single model to solve multiple tasks. Thus, the knowledge gained by training in one task affects all other tasks as the model parameters are shared.

A problem that can arise in both approaches is catastrophic forgetting (or catastrophic inference) [6]. While especially relevant in continual learning, it can occur in all fields of machine learning. This phenomenon occurs when adjusting the weights in a network for a new example can lead to overwriting/forgetting the knowledge that was attained in previous training. In transfer learning, this can be problematic when we forget instead of using the knowledge contained in previously trained models.

In this contribution, we investigate the transfer learning capabilities of a model that combines the output of a task-specific expert with the original input given to the expert. The model thus creates a new representation that is infused with the experts' knowledge and then given to a downstream model. This work expands on Biermann et al. [1], which have shown that knowledge infused representations benefit downstream model performance if expert and downstream models operate in the same task domain. In our investigation, we test the influence of different task domains on the downstream model performance when using knowledge-infused representations. We examine whether the information encoded in the infused representations has any positive or negative effects when considering experts and downstream models of different task domains. We further expand the model by combining the information of two experts and investigating the influence of a task-foreign expert on the performance of downstream models.

## 2 Related Work

This work is based on and a continuation of previous work by the authors [1]. Previously, we established that infusing the information contained in the output of an expert can be transported down to a simple downstream model if the expert and downstream model share the same task domain. Missing from this work is
the important and relevant case where expert task domain and downstream task
domain differ. In this work, we investigate the influence of differing expert and
downstream task domains. Additionally, the previous work considered only cases
with one expert, whereas in this study, we explicitly investigate the influence
of multiple experts on the downstream performance in the context of transfer
learning.

Transfer learning is an important field in machine learning, and with the increase in model size and computation cost over the last decade, its relevance will continue to rise. As discussed in 1, within the NLP field, transfer learning is mainly achieved through pretraining on enormous unlabelled datasets as the various GPT models show [2,8–11]. Another approach is the above-mentioned T5 transformer [12] that trains the same model parameters on subsequent tasks. Overall, the computational cost is again heavy as you have to train transformer models. In our approach, instead of finetuning and retraining transformer models, we aim to recycle and benefit from already existing and finished models. As far as we are aware, there is no other work investigating transfer learning in this direction.

This approach has the added benefit that the models that are actually trained
can be made smaller. This allows for more competition across the machine learn-
ing field as smaller institutions can symbiotically benefit from the enormous
models that can only be trained by a handful of large institutions. Smaller mod-
els might emphasize the problem of catastrophic forgetting again as increasing
model size and dataset size have been shown to reduce the impact of catastrophic
forgetting [13]. With regards to this, the here proposed model remains an elegant
candidate as the nature of the finished and fixed experts prevents any form of
overwriting or forgetting previously trained parameters.

3 Methods

## 3.1 General Architecture

The model follows the general architecture of previous work used in Biermann
et al. [1] and extends it by an additional expert. Adding an additional expert
allows us to test the models’ response to the presence of additional experts that
do not offer direct beneficial information regarding the downstream task. In con-
trast to the previous work, we also test the model for cases in which the expert’s
task domain and downstream task domain differ. The general architecture is
depicted in Fig. 1. An input sequence is separately given to up to two experts
and an embedding layer. The contextual experts are pretrained models that
solve a specific task. The experts are contextual in the sense that they can be
specifically chosen for a downstream task. The outputs of the experts are then
scaled up to the embedding dimension via single linear layers and appended to
the sequence itself. This corresponds to the sequential combination paradigm
in our previous work. We opted to concentrate on the sequential paradigm in
this work as it is suspected that the model scales better with the sequential
paradigm when increasing the number of experts. Additionally, and more impor-
tantly, the sequential paradigm combined with an attention-based combination
offers a natural interpretability lost in the other combination paradigm. The
extended sequence is then given to an attention-based combination mechanism
that reduces the sequence length back to the original input sequence length and
infuses the sequence tokens with the expert's output information, yielding an
information-infused new sequence representation. This infused representation is
then given to a downstream model that is trained to perform a downstream
task. The error is backpropagated from the downstream model to the combi-
nation mechanism when training the model. The experts are seen as finished
products and are not trained or finetuned in any way.

<figure><img src="image_5.png" /><figcaption>**Fig. 1.** Model architecture. The original input is given to contextual experts and to an embedding layer. The experts are pre-trained task-specific models. The outputs of the experts are linearly scaled, via a feed forward layer, to the embedding dimension of the sequence tokens and appended to the sequence. The combination mechanism combines the expert output and original input into a new information-infused representation. Downstream models are arbitrary, to-be-trained models.</figcaption></figure>

3.2 Combination Mechanism

The goal of the combination mechanism is to take in the original input sequence appended by the expert output tokens and reduce it back to the original sequence length. Ideally, this happens in a way that the information contained in the original input tokens and the expert output tokens are fused in the new infused sequence.

We employ the same mechanism used in [1] which is based on Vaswani et al.'s
scaled dot-product and multi-head attention introduced with the Transformer
[16]. To reiterate, the scaled dot-product

$$
\mathcal{A}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \quad (1)
$$

calculates an attention weight map from each word to each word in the sequence
via the query *Q* and key *K* matrices and creates a new representation by multi-
plying and summing with the value matrix *V*. *Q*, *K*, and *V* are created from the
original input, and their respective vectors show dimensionalities of *d*<sub>q</sub> = *d*<sub>k</sub>, *d*<sub>v</sub>.
The new representation then has the dimensionality *d*<sub>v</sub>
In multi-head attention, multiple heads $H_i$ calculate the scaled dot-product in parallel and are concatenated back into a single new representation and scaled to a final new representation via weight matrix $W^O$.

$$ \mathcal{M}(Q, K, V) = \text{Concat}(H_1, \dots, H_h)W^O \qquad (2) $$

$$ H_i = \mathcal{A}(Q_i, K_i, V_i) \qquad (3) $$

This work follows the general practice and chooses $d_k = d_q = d_v = d_{emb}$, where $d_{emb}$ is the dimension of the word embedding. To reduce the sequence length of expert output infused new representation, the sequence is cut off after reaching its original length.

For example, if we set the original sequence length as $N$ and the number of expert outputs to $e$, the attention mechanism calculates $N+e$ new representation tokens. The original sequence length $N$ is subsequently enforced by cutting off the last $e$ tokens in the infused sequence.

## 3.3 Contextual Experts

The general idea is that any arbitrary model or architecture can be used as an expert as long as it is trained to perform a single task. In the sequential combination paradigm we employ in this work, it is also possible for the experts to return a single output token or a sequence of output tokens. This allows us to give the experts a certain contextual component. With contextual experts, we mean that the experts can be handpicked in the context of a downstream task. The idea is to choose experts that might offer beneficial information with regard to the downstream task. For example, in this work, we chose our experts to be a dialogue act classifier and a sentiment classifier. While not directly related to their task, one could reasonably assume that certain dialogue act correlate more with certain sentiments than others. Thus, having the information of the sentiment expert might hold benefit for a downstream model trying to solve dialogue act classification.

For the sake of simplicity, both experts chosen in this work are based on the DistilBERT model [14] implemented via the Huggingface Transformer library [17]. The experts are then fine-tuned in their respective tasks on their respective datasets.

*Dialogue Act classifier (DA):* The dialogue act classifier is fine-tuned on the SwitchBoard Dialogue Act corpus (SwDA) [5, 15]<fnref n="1" />. The SwDA corpus consists of transcribed phone conversations and the cleaned version contains 41 different classes. To counteract the strong class imbalance present in the dataset, the expert fine-tuned with class weights. The expert was finetuned for 5 epochs and reached a test accuracy of 0.781

*Sentiment Analysis classifier (SA):* The sentiment classifier is finetuned on the well-established IMDB movie reviews dataset [7]. The dataset consists of movie

<footnote n="1">¹ This work uses the cleaned and curated train, validation, and test splits given by https://github.com/NathanDuran/Switchboard-Corpus.</footnote>
reviews and categorizes the reviews in 2 possible sentiment classes. The expert was finetuned for 2 epochs following an 80-20 training and validation split and reached a test accuracy of 0.932

## 3.4 Downstream Model

This architecture aims to use the information that is contained and generated by already trained experts. Thus, keeping the downstream model deliberately simple shows that simple models can benefit from this approach. A simple model further emphasizes the idea of combining computationally heavy experts with lightweight downstream models to maximize the amount of computation saved when compared to training or finetuning large models.

To this extent, the downstream model consists of a single GRU [3] layer leading into a softmax classifier consisting of 64 hidden units.

# 4 Experiments

To investigate the transfer learning capabilities of the proposed architecture, we perform several simulations with different expert and downstream task settings. The ‘Expert Task/Downstream Task’ column in Table 1 lists the different expert and downstream task combinations considered.

As a baseline, we train the downstream model in dialogue act classification (DA) or sentiment classification (SA) without adding expert output. This way, we can provide a baseline that includes any benefits that the attention mechanism may provide to the performance. This ensures that any comparisons in performance with the baseline can be more directly attributed to the addition of the expert output.

Subsequently, we first add one expert and run simulations for all possible expert- and downstream task combinations. Adding only one expert allows us to directly investigate the impact that a differing expert and downstream task have on performance. In our previous work, we only considered cases in which the expert and downstream task solve the same task.

Finally, we combine both experts and train the downstream model in each task. Adding both experts allows us to probe the effects that combining multiple expert outputs may have on each other. Mainly, we test whether adding two experts elevates the performance beyond what one expert shows or if the presence of another task-unrelated expert hinders the performance of the downstream model.

The model is trained for the same amount of epochs in all simulations using a learning rate of $2 \times 10^{-4}$. For the input sequence, we train an embedding layer to use 300-dimensional embedding vectors. The experts use their own respective tokenizers and embeddings.
<table><caption>**Table 1.** Baseline and downstream task test accuracies for different expert- and downstream task configurations. DA stands for the dialogue act classification task, and SA stands for the sentiment classification task.</caption><thead><tr><th>Model</th><th>Accuracy<br>Baseline</th></tr></thead><tbody><tr><td>Dialogue Act expert</td><td>0.781</td></tr><tr><td>Sentiment expert</td><td>0.932</td></tr><tr><td>Downstream Model (DA)</td><td>0.748</td></tr><tr><td>Downstream Model (SA)</td><td>0.835</td></tr><tr><th>Expert Task/Downstream Task</th><th>Downstream</th></tr><tr><td>DA/DA</td><td>0.776</td></tr><tr><td>SA/SA</td><td>0.928</td></tr><tr><td>SA/DA</td><td>0.751</td></tr><tr><td>DA/SA</td><td>0.820</td></tr><tr><td>DA + SA/DA</td><td>0.776</td></tr><tr><td>DA + SA/SA</td><td>0.929</td></tr></tbody></table>

## 4.1 Results

Table 1 summarises the baseline and test accuracies for the downstream model for all performed simulations.

Comparing the expert and baseline performances for the different tasks, we see that the expert slightly outperforms the baseline by 3.3% for the DA task and more significantly by 9.4% for the SA task, respectively. Though the difference between baseline and expert performance is small, it is significant enough to see whether information has been transported from the expert to the downstream model.

As expected, in the cases where the expert and downstream model share the same task ('DA/DA' and 'SA/SA'), with 0.776 and 0.928 we see an improvement in the downstream performance close to the performance of the respective expert. This aligns with the results achieved in our previous work and the information of the expert has been transported to the downstream model. This performance improvement can not be observed when the expert and downstream task domains differ from each other. In these cases ('SA/DA' and 'DA/SA'), the downstream performance stays close to their respective baseline performances. While the DA performance of 0.751 lies slightly above the baseline, the SA performance falls slightly below the baseline with 0.820. This shows that the model does not benefit from adding the knowledge of a task-foreign expert but roughly remains around its baseline performance.

When adding task-foreign experts, one could imagine scenarios in which the expert might still hold some information. For example, in dialogue act classification and sentiment classification, it might be reasonable to assume that certain dialogue act classes correlate with certain sentiments, and thus, knowing the
sentiment of a sentence might hold useful information. The simulations show that our model, with the chosen experts, tasks and datasets, could not learn and use this type of cross-benefit. While it would have been nice to observe this type of cross-benefit in our simulations, their absence does not mean that this model cannot achieve them. These cross-benefits are highly dependent on the chosen experts, the form of their output, and the chosen expert and downstream task. Some tasks are naturally more beneficial to each other than more independent tasks.

What is more beneficial is that the performance does not worsen significantly when including an expert in a different task, indicating the model's ability to ignore unhelpful information that might be introduced through the expert output. Notably, this implicates a certain robustness of the model toward adding multiple experts. If the model can ignore irrelevant information, the negative effects of adding irrelevant experts become less problematic.

In fact, we can see this in the simulations in which both experts are added to the model (DA+SA). When both experts are added, the downstream model reaches performances of 0.776 for the DA task and 0.929 for the SA task. Both performances approach the respective expert task performance again. This solidifies the previous observations that the model can pay attention to beneficial information and can, at the same time, ignore seemingly irrelevant information.

# 5 Conclusion

We tested the transfer learning capabilities of the proposed model by testing different expert- and downstream task configurations. When expert and downstream tasks align, we see a clear and significant improvement in downstream model performance. When the expert and downstream tasks differ, we could see no positive effect on the downstream model performance. This behavior is expected as a task-foreign expert does not carry immediate relevant information for the downstream task. Nevertheless, as both chosen tasks are within the NLP domain one could imagine possible indirect cross-benefits the model might gain. The results indicate a limited ability of the model to benefit from possible cross-benefits between tasks. As these cross-benefits are highly dependent on the chosen tasks, it is virtually impossible to make a definite statement on the ability of this model to achieve these types of benefits. Though further investigation is warranted as examples of intuitive possible cross-benefits easily come to mind and cross-benefits may be more apparent for more nuanced tasks.

Additionally, we observe that there are no negative effects when the expert task and downstream task differ. This is a promising result as it indicates robustness regarding irrelevant experts. This is further emphasized by simulations that add two experts of different tasks simultaneously. Here, the model can again reach a downstream performance close to that of its respective expert itself. This shows that the model can pay attention to the beneficial expert information while at the same time ignoring the irrelevant expert information. It also shows that the inclusion of task-foreign experts does not hinder the models' ability to benefit from the task-relevant expert.
Overall, we have shown that this model is able to combine multiple experts without losing its ability to transport relevant and beneficial information to the downstream model. At the same time, we have shown that the model is able to ignore task-foreign information which indicates the models' robustness regarding the negative influences of irrelevant information.

This has nice implications for future applications of this approach. Our results show that adding more experts to the model shows no immediate negative impact on the performance of the model itself. Thus, possible next steps include adding more experts to the model. Adding more experts could lead to a higher probability of encountering positive cross-benefits that we could not observe with only two different tasks. Furthermore, a natural next step is to test the multi-task capabilities of this model and whether we are able better train a downstream model on multiple tasks when infusing it with expert knowledge.

Finally, this approach shows promise in making simpler models competitive again and gives an alternative to the ever-growing, large models as we effectively can 'recycle' old computations.

## References

1. Biermann, D., Goodwin, M., Granmo, O.C.: Knowledge infused representations through combination of expert knowledge and original input. In: Zouganeli, E., Yazidi, A., Mello, G., Lind, P. (eds.) Nordic Artificial Intelligence Research and Development: 4th Symposium of the Norwegian AI Society, NAIS 2022, Oslo, Norway, May 31-June 1, 2022, Revised Selected Papers, vol. 1650, pp. 3–15. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-17030-0_1

2. Brown, T., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

3. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint: arXiv:1406.1078 (2014)

4. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint: arXiv:1810.04805 (2018)

5. Godfrey, J., Holliman, E., McDaniel, J.: Switchboard: telephone speech corpus for research and development. In: [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. 517–520 (1992). https://doi.org/10.1109/ICASSP.1992.225858

6. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. **114**(13), 3521–3526 (2017)

7. Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150. Association for Computational Linguistics, Portland, Oregon, USA, June 2011. http://www.aclweb.org/anthology/P11-1015

8. OpenAI: Gpt-4 technical report (2023)
9. Ouyang, L., et al.: Training language models to follow instructions with human feedback. In: Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744 (2022)

10. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)

11. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)

12. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

13. Ramasesh, V.V., Lewkowycz, A., Dyer, E.: Effect of scale on catastrophic forgetting in neural networks. In: International Conference on Learning Representations (2022)

14. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint: arXiv:1910.01108 (2019)

15. Stolcke, A., et al.: Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput. Linguist. 26(3), 339–373 (2000)

16. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

17. Wolf, T., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)

18. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019). https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf