Plug-and-Play Knowledge Injection for Pre-trained Language Models
Year: 2023

Abstract:
Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. In this work, we are the first to study how to improve the flexibility and efficiency of knowledge injection by reusing existing downstream models. To this end, we explore a new paradigm plug-and-play knowledge injection, where knowledge bases are injected into frozen existing downstream models by a knowledge plugin. Correspondingly, we propose a plug-and-play injection method maptuning, which trains a mapping of knowledge embeddings to enrich model inputs with mapped embeddings while keeping model parameters frozen. Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models. Moreover, we show that a frozen downstream model can be well adapted to different domains with different mapping networks of domain knowledge. Our code and models are available at https://github. com/THUNLP/Knowledge-Plugin.

text: Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. In this work, we are the first to study how to improve the flexibility and efficiency of knowledge injection by reusing existing downstream models. To this end, we explore a new paradigm plug-and-play knowledge injection, where knowledge bases are injected into frozen existing downstream models by a knowledge plugin. Correspondingly, we propose a plug-and-play injection method maptuning, which trains a mapping of knowledge embeddings to enrich model inputs with mapped embeddings while keeping model parameters frozen. Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models. Moreover, we show that a frozen downstream model can be well adapted to different domains with different mapping networks of domain knowledge. Our code and models are available at https://github. com/THUNLP/Knowledge-Plugin.

heading: 1 Introduction
text: Recent years have witnessed rapid development in enhancing pre-trained language models (PLMs) with various external knowledge bases, i.e., knowledge injection for PLMs (Levine et al., 2020; Zhou et al., 2020; Zhang et al., 2019; Peters et al., 2019;
∗ Equal contribution † Corresponding authors
Bosselut et al., 2019; Guan et al., 2020). Knowledge injection improves the performance of PLMs on a wide range of tasks such as information extraction (Liu et al., 2020a; Wang et al., 2021b), question answering (Xiong et al., 2020; Wang et al., 2021a), and text generation (Chen et al., 2020).
Existing injection methods commonly inject knowledge by knowledge-aware pre-training or fine-tuning (Peters et al., 2019; Yamada et al., 2020; Liu et al., 2020a; Wang et al., 2021a). However, rarely studied is how to inject knowledge into a downstream model that is already adapted to a specific task. If we want to apply a new knowledge injection method to enhance models on a specific task, we have to discard task-specific downstream models and retrain them. In addition, one downstream model working with multiple knowledge bases requires retraining itself to inject each knowledge base. Retraining models is time-consuming and resource-intensive, leading to the need for a flexible and efficient injection paradigm.
Toward flexible and efficient injection, we explore a novel paradigm plug-and-play knowledge injection, where knowledge bases are injected into frozen existing downstream models by knowledge modules. The knowledge module bridges the
knowledge base and the model, and we call it a plugin vividly. Under this paradigm, a downstream model would have multiple plugins, each corresponding to a combination of an injection method and a knowledge base, which ensures flexibility. Moreover, knowledge plugins should be small enough to ensure efficiency. Intuitively, as shown in Figure 1, we treat models and knowledge bases as computers and flash disks, respectively.
In this work, we study two settings for the plugand-play knowledge injection paradigm. The first is general plug-and-play knowledge injection, aiming to inject knowledge into all downstream models (trained from a particular PLM) by a general plugin without any task-specific training. In this setting, all downstream models share exactly one plugin for one combination of an injection method and a knowledge base. The second is task-specific plugand-play knowledge injection, where knowledge plugins are trained to better adapt to downstream tasks while keeping downstream models frozen.
By our pilot study, we find that existing methods (Poerner et al., 2020; Ye et al., 2022; Wang et al., 2021a; Lewis et al., 2020) that can be used directly can not be well applied to the plug-andplay injection paradigm. To this end, we propose map-tuning, a preliminary exploration of learning knowledge plugins. Specifically, we train a lightweight mapping network that augments model inputs with mapped knowledge representations, e.g., TransE (Bordes et al., 2013). To meet the general and task-specific injection requirements, we design general map-tuning and task-specific map-tuning, respectively. General map-tuning adopts language modeling as its objective to learn knowledge plugins and seeks better generalizability. Task-specific map-tuning adopts task targets for plugin learning and seeks better task adaptation.
We use three typical knowledge-driven NLP tasks to evaluate our plug-and-play knowledge injection, including relation classification (Han et al., 2018), entity typing (Xin et al., 2018), and question answering (Sciavolino et al., 2021). The experimental results show that: (1) after adapting PLMs to downstream tasks through full-parameter finetuning or parameter-efficient tuning, also known as delta tuning (Liu et al., 2021; Ding et al., 2022), injecting knowledge into these downstream models by general map-tuning leads to performance improvements in almost all cases; (2) using taskspecific map-tuning to inject domain knowledge
further enables a frozen downstream model to work well in different domains. We hope our contribution can draw more attention to the plug-and-play knowledge injection paradigm and inspire more future research.

heading: 2 Plug-and-Play Knowledge Injection
text: Paradigm Description. Given a downstream model D trained on a downstream task with a PLM P as the backbone, we intend to improve its performance on this task by incorporating an extra knowledge base B and freezing D’s parameters, for which we need to train a knowledge pluginM. Note that neither pre-training nor fine-tuning trains the model D to cooperate with B orM.
Two Injection Settings. As shown in Figure 2(a), plug-and-play knowledge injection decouples knowledge injection from model training, which is different from existing paradigms. For general plug-and-play knowledge injection, M is obtained based on only P and B, and then it is directly plugged into all downstream models, D1,D2, . . ., without any additional training. For task-specific plug-and-play knowledge injection, it is allowed to trainM1,M2, . . . for D1,D2, . . . respectively while keeping D1,D2, . . . frozen.
Challenges. The general plug-and-play knowledge injection poses serious challenges to methods designed for it.M is expected to improve the performance of D, yet D has never seenM or been seen byM during training. The only prior condition is that P and B are visible during trainingM. Therefore, the designed methods for general injection need to endowM with enough generalizability such thatM can adapt to unknown D1,D2, . . . well. Even though the knowledge base B may have rich knowledge, without a good adaptation ofM, useful information brought to D will be less than disruptive noise.
The task-specific plug-and-play knowledge injection relaxes the restrictions, where Mi is allowed to be trained with frozen Di. Compared to injection during fine-tuning, the training of Mi should be fast and the parameter number of Mi should be small compared to that of Di. Otherwise, the methods would be meaningless. Hence, it requires simple and efficient architecture designs and informative training objectives forMi.
Potentiality of Using Existing Methods. Few existing knowledge injection methods can be di-
<latexit sha1_base64="XKJizic/5i5YT3Zj/VcmXmFrzf0=">AAACHnicbVDLSsNAFJ3UV31XXboJFsFVSUTRZbEbFy58VYUmlMn0th06MwkzN2oJ+Q239mvciVv9GMFJ7UKtBwYO59zLuXOiRHCDnvfhlGZm5+YXyotLyyura+uVjc0bE6eaQZPFItZ3ETUguIImchRwl2igMhJwGw0ahX97D9rwWF3jMIFQ0p7iXc4oWikIEB4xazXOrsK8Xal6NW8Md5r4E1IlE5y3K59BJ2apBIVMUGNavpdgmFGNnAnIl4LUQELZgPagZamiEkyYjW/O3V2rdNxurO1T6I7VnxsZlcYMZWQnJcW++esV4n9eK8XucZhxlaQIin0HdVPhYuwWBbgdroGhGFpCmeb2Vpf1qaYMbU2/UiJp/6DggcVSUtXJgss8C4rAKMou86Iv/2870+Rmv+Yf1A4vDqr1k0lzZbJNdsge8ckRqZNTck6ahJGEPJFnMnJGzovz6rx9j5acyc4W+QXn/QsC/aQs</latexit>
<latexit sha1_base64="fqWQc6zGpkyDFBQqNdaB73Szm1Y=">AAACHXicbVDLSgNBEJyNrxhfUY9eFoPgKexKRI+iF48ajArZILOTTjJkHstMrxqW/Qyv+jXexKv4MYKzMQdjbGgoqrqp7ooTwS0GwadXmptfWFwqL1dWVtfWN6qbW9dWp4ZBi2mhzW1MLQiuoIUcBdwmBqiMBdzEw7NCv7kHY7lWVzhKoCNpX/EeZxQd1Y4QHjET+h7yu2otqAfj8mdBOAE1MqmLu+pX1NUslaCQCWptOwwS7GTUIGcC8kqUWkgoG9I+tB1UVILtZOOTc3/PMV2/p41rhf6Y/b2RUWntSMZuUlIc2L9aQf6ntVPsHXcyrpIUQbEfo14qfNR+8b/f5QYYipEDlBnubvXZgBrK0KU05RJL94OCB6alpKqbRc08iwrDOM6aeZFX+DedWXB9UA8b9cPLRu3kdJJcmeyQXbJPQnJETsg5uSAtwogmT+SZvHgv3qv35r3/jJa8yc42mSrv4xsJ9qQ+</latexit>
<latexit sha1_base64="HdjqFVT8unCJTgRGjM0IA4G+xaA=">AAACH3icbVDLSsNAFJ34tr6qLt0Ei+CqJKLoUnTjUsU+oAllMr1th85M4syNWkK+w61+jTtx68cITtoubOuBgcM593LunCgR3KDnfTsLi0vLK6tr66WNza3tnfLuXt3EqWZQY7GIdTOiBgRXUEOOApqJBiojAY1ocF34jSfQhsfqAYcJhJL2FO9yRtFKYYDwglkCVKWYt8sVr+qN4M4Tf0IqZILbdvkn6MQslaCQCWpMy/cSDDOqkTMBeSlIDSSUDWgPWpYqKsGE2ejo3D2ySsftxto+he5I/buRUWnMUEZ2UlLsm1mvEP/zWil2L8KMqyRFUGwc1E2Fi7FbNOB2uAaGYmgJZZrbW13Wp5oytD1NpUTS/kHBM4ulpKqTBfd5FhSBUZTd50Vf/mw786R+UvVPq2d3p5XLq0lza+SAHJJj4pNzckluyC2pEUYeySt5I+/Ou/PhfDpf49EFZ7KzT6bgfP8Cx/WlKQ==</latexit>
<latexit sha1_base64="YTurMLA4tl/mqRTdkFPnehqG8wU=">AAACH3icbVBNSwMxFMz6WetX1aOXxSJ4Krui6FH04lGLrUJ3kSR91WCSXZO3aln2d3i1v8abeO2PEcy2Pah1IDDMvMe8DEulsBgEQ29mdm5+YbGyVF1eWV1br21stm2SGQ4tnsjE3DBqQQoNLRQo4SY1QBWTcM0ezkr/+gmMFYm+wn4KsaJ3WvQEp+ikOEJ4wZxliGCK21o9aAQj+NMknJA6meDitvYVdROeKdDIJbW2EwYpxjk1KLiEohplFlLKH+gddBzVVIGN89HRhb/rlK7fS4x7Gv2R+nMjp8ravmJuUlG8t3+9UvzP62TYO45zodMMQfNxUC+TPiZ+2YDfFQY4yr4jlBvhbvX5PTWUuw5+pzDl/qDhmSdKUd3No2aRR2UgY3mzKPsK/7YzTdr7jfCgcXh5UD85nTRXIdtkh+yRkByRE3JOLkiLcPJIXskbGXgD79378D7HozPeZGeL/II3/AbXVKUy</latexit>
<latexit sha1_base64="oR40QGMmh45lOMswonCXgLASu2o=">AAACHHicbVBNS8NAEN3Ur1q/qh69BIvgqSRS0WPRi8da7Ac2pWy203bp7ibsbtQS8i+82l/jTbwK/hjBTZuDbR0YeLw3j5l5fsio0o7zbeXW1jc2t/LbhZ3dvf2D4uFRUwWRJNAgAQtk28cKGBXQ0FQzaIcSMPcZtPzxbaq3nkAqGogHPQmhy/FQ0AElWBvq0dPwYlxxOekVS07ZmZW9CtwMlFBWtV7xx+sHJOIgNGFYqY7rhLobY6kpYZAUvEhBiMkYD6FjoMAcVDeeXZzYZ4bp24NAmhbanrF/HTHmSk24byY51iO1rKXkf1on0oPrbkxFGGkQZL5oEDFbB3b6vt2nEohmEwMwkdTcapMRlphoE9LCFp+bHwQ8k4BzLPqxV09iL13o+3E9SfNyl9NZBc2LslspX95XStWbLLk8OkGn6By56ApV0R2qoQYiSKBX9Iam1tR6tz6sz/lozso8x2ihrK9fwCCjiQ==</latexit>
<latexit sha1_base64="z/vtTQ1yS8oS2ADAkIL9fIaFBFg=">AAACHnicbVBNS8NAFNz4bf2qevQSLIKnkkhFj0URPNZqVWhC2WxfdenuJuy+qCXkb3jVX+NNvOqPEdy0PdjqwMIw8x7zdqJEcIOe9+XMzM7NLywuLZdWVtfWN8qbW9cmTjWDFotFrG8jakBwBS3kKOA20UBlJOAm6p8W/s0DaMNjdYWDBEJJ7xTvcUbRSkGA8IRZ+/KsEeadcsWrekO4f4k/JhUyRqNT/g66MUslKGSCGtP2vQTDjGrkTEBeClIDCWV9egdtSxWVYMJseHPu7lml6/ZibZ9Cd6j+3sioNGYgIzspKd6baa8Q//PaKfaOw4yrJEVQbBTUS4WLsVsU4Ha5BoZiYAllmttbXXZPNWVoa5pIiaT9g4JHFktJVTcLmnkWFIFRlDXzoi9/up2/5Pqg6teqhxe1Sv1k3NwS2SG7ZJ/45IjUyTlpkBZhJCHP5IW8Oq/Om/PufIxGZ5zxzjaZgPP5Aw08pDI=</latexit>
<latexit sha1_base64="muys/2ZQJ40hek8CS3JlhPgiw3k=">AAACGnicbVBNS8NAEN3U7/pV9eglWARPNRFFj6IXjyq2Fpoim+20Xbq7CbsTtYT8Ca/6a7yJVy/+GMFNm4NtHRh4vDfDm3lhLLhBz/t2SnPzC4tLyyvl1bX1jc3K1nbDRIlmUGeRiHQzpAYEV1BHjgKasQYqQwH34eAy1+8fQRseqTscxtCWtKd4lzOKlmoGCM+YHmYPlapX80blzgK/AFVS1PVD5SfoRCyRoJAJakzL92Jsp1QjZwKycpAYiCkb0B60LFRUgmmno3szd98yHbcbadsK3RH7dyOl0pihDO2kpNg301pO/qe1EuyetVOu4gRBsbFRNxEuRm7+vNvhGhiKoQWUaW5vdVmfasrQRjThEkr7g4InFklJVScNbrM0yA3DML3N8rz86XRmQeOo5h/XTm6Oq+cXRXLLZJfskQPik1NyTq7INakTRgR5Ia/kzXlz3p0P53M8WnKKnR0yUc7XL/whopk=</latexit>
<latexit sha1_base64="Dd6HywciKItWZcpdkVWYL5tDQCw=">AAACH3icbVDLSsNAFJ3U97vq0k2wCK5KIoouq24EEeqjKjRBJtPbOnRmEmdu1BLyHW71a9yJ236M4KR2YasHBg7n3Mu5c6JEcIOe13dKE5NT0zOzc/MLi0vLK+XVtWsTp5pBg8Ui1rcRNSC4ggZyFHCbaKAyEnATdY8L/+YRtOGxusJeAqGkHcXbnFG0UhggPGPWPDu8PA3zu3LFq3oDuH+JPyQVMkT9rvwVtGKWSlDIBDWm6XsJhhnVyJmAfD5IDSSUdWkHmpYqKsGE2eDo3N2ySsttx9o+he5A/b2RUWlMT0Z2UlK8N+NeIf7nNVNsH4QZV0mKoNhPUDsVLsZu0YDb4hoYip4llGlub3XZPdWUoe1pJCWS9g8KnlgsJVWtLLjIs6AIjKLsIi/68sfb+Uuud6r+bnXvfLdSOxo2N0s2yCbZJj7ZJzVyQuqkQRh5IC/klbw5b8678+F8/oyWnOHOOhmB0/8GqS2kgA==</latexit>
<latexit sha1_base64="muys/2ZQJ40hek8CS3JlhPgiw3k=">AAACGnicbVBNS8NAEN3U7/pV9eglWARPNRFFj6IXjyq2Fpoim+20Xbq7CbsTtYT8Ca/6a7yJVy/+GMFNm4NtHRh4vDfDm3lhLLhBz/t2SnPzC4tLyyvl1bX1jc3K1nbDRIlmUGeRiHQzpAYEV1BHjgKasQYqQwH34eAy1+8fQRseqTscxtCWtKd4lzOKlmoGCM+YHmYPlapX80blzgK/AFVS1PVD5SfoRCyRoJAJakzL92Jsp1QjZwKycpAYiCkb0B60LFRUgmmno3szd98yHbcbadsK3RH7dyOl0pihDO2kpNg301pO/qe1EuyetVOu4gRBsbFRNxEuRm7+vNvhGhiKoQWUaW5vdVmfasrQRjThEkr7g4InFklJVScNbrM0yA3DML3N8rz86XRmQeOo5h/XTm6Oq+cXRXLLZJfskQPik1NyTq7INakTRgR5Ia/kzXlz3p0P53M8WnKKnR0yUc7XL/whopk=</latexit>
<latexit sha1_base64="gl8lzF6VILwNkPD/JDrZ88PO1Y4=">AAACHHicbVBNS8NAEN34bf2qevQSLIKnkoiiR9GLF6EWW4tNKZvttF3c3YTdiVpC/oVX+2u8iVfBHyO4aXuw1YGBx3szvJkXxoIb9LwvZ25+YXFpeWW1sLa+sblV3N6pmyjRDGosEpFuhNSA4ApqyFFAI9ZAZSjgLny4zPW7R9CGR+oWBzG0JO0p3uWMoqXuA4RnTK9pnLWLJa/sjcr9C/wJKJFJVdrF76ATsUSCQiaoMU3fi7GVUo2cCcgKQWIgpuyB9qBpoaISTCsdXZy5B5bpuN1I21bojtjfGymVxgxkaCclxb6Z1XLyP62ZYPeslXIVJwiKjY26iXAxcvP33Q7XwFAMLKBMc3ury/pUU4Y2pCmXUNofFDyxSEqqOmlQzdIgNwzDtJrlefmz6fwF9aOyf1w+uTkunV9Mklshe2SfHBKfnJJzckUqpEYYUeSFvJKhM3TenHfnYzw650x2dslUOZ8/3/6jnA==</latexit>
<latexit sha1_base64="gl8lzF6VILwNkPD/JDrZ88PO1Y4=">AAACHHicbVBNS8NAEN34bf2qevQSLIKnkoiiR9GLF6EWW4tNKZvttF3c3YTdiVpC/oVX+2u8iVfBHyO4aXuw1YGBx3szvJkXxoIb9LwvZ25+YXFpeWW1sLa+sblV3N6pmyjRDGosEpFuhNSA4ApqyFFAI9ZAZSjgLny4zPW7R9CGR+oWBzG0JO0p3uWMoqXuA4RnTK9pnLWLJa/sjcr9C/wJKJFJVdrF76ATsUSCQiaoMU3fi7GVUo2cCcgKQWIgpuyB9qBpoaISTCsdXZy5B5bpuN1I21bojtjfGymVxgxkaCclxb6Z1XLyP62ZYPeslXIVJwiKjY26iXAxcvP33Q7XwFAMLKBMc3ury/pUU4Y2pCmXUNofFDyxSEqqOmlQzdIgNwzDtJrlefmz6fwF9aOyf1w+uTkunV9Mklshe2SfHBKfnJJzckUqpEYYUeSFvJKhM3TenHfnYzw650x2dslUOZ8/3/6jnA==</latexit>
<latexit sha1_base64="onu66ZApQHer6gmA3Do/knHnV+c=">AAACK3icbVDLSgNBEJz1bXxFPXjwMhgET2FXFD36uOhNxRghG8LsbEcHZ2aXmV41LPs1XvVrPCle/QzB2SQHozY0FFXdFFVRKoVF33/zxsYnJqemZ2Yrc/MLi0vV5ZUrm2SGQ4MnMjHXEbMghYYGCpRwnRpgKpLQjO6OS715D8aKRF9iL4W2YjdadAVn6KhOdS1EeMT8MI5FSTBJT3WaYdGp1vy63x/6FwRDUCPDOetUv8I44ZkCjVwya1uBn2I7ZwYFl1BUwsxCyvgdu4GWg5opsO28H6Cgm46JaTcxbjXSPvvzI2fK2p6K3KVieGt/ayX5n9bKsLvfzkUZCTQfGHUzSTGhZRs0FgY4yp4DjBtXAaf8lhnG0XU24hIpl0HDA0+UYjrOw4siD0vDKMovirKv4Hc7f8HVdj3Yqe+e79QOjobNzZB1skG2SED2yAE5IWekQTgpyBN5Ji/ei/fqvXsfg9Mxb/izSkbG+/wGEqSpWg==</latexit>
<latexit sha1_base64="onu66ZApQHer6gmA3Do/knHnV+c=">AAACK3icbVDLSgNBEJz1bXxFPXjwMhgET2FXFD36uOhNxRghG8LsbEcHZ2aXmV41LPs1XvVrPCle/QzB2SQHozY0FFXdFFVRKoVF33/zxsYnJqemZ2Yrc/MLi0vV5ZUrm2SGQ4MnMjHXEbMghYYGCpRwnRpgKpLQjO6OS715D8aKRF9iL4W2YjdadAVn6KhOdS1EeMT8MI5FSTBJT3WaYdGp1vy63x/6FwRDUCPDOetUv8I44ZkCjVwya1uBn2I7ZwYFl1BUwsxCyvgdu4GWg5opsO28H6Cgm46JaTcxbjXSPvvzI2fK2p6K3KVieGt/ayX5n9bKsLvfzkUZCTQfGHUzSTGhZRs0FgY4yp4DjBtXAaf8lhnG0XU24hIpl0HDA0+UYjrOw4siD0vDKMovirKv4Hc7f8HVdj3Yqe+e79QOjobNzZB1skG2SED2yAE5IWekQTgpyBN5Ji/ei/fqvXsfg9Mxb/izSkbG+/wGEqSpWg==</latexit>
<latexit sha1_base64="XKJizic/5i5YT3Zj/VcmXmFrzf0=">AAACHnicbVDLSsNAFJ3UV31XXboJFsFVSUTRZbEbFy58VYUmlMn0th06MwkzN2oJ+Q239mvciVv9GMFJ7UKtBwYO59zLuXOiRHCDnvfhlGZm5+YXyotLyyura+uVjc0bE6eaQZPFItZ3ETUguIImchRwl2igMhJwGw0ahX97D9rwWF3jMIFQ0p7iXc4oWikIEB4xazXOrsK8Xal6NW8Md5r4E1IlE5y3K59BJ2apBIVMUGNavpdgmFGNnAnIl4LUQELZgPagZamiEkyYjW/O3V2rdNxurO1T6I7VnxsZlcYMZWQnJcW++esV4n9eK8XucZhxlaQIin0HdVPhYuwWBbgdroGhGFpCmeb2Vpf1qaYMbU2/UiJp/6DggcVSUtXJgss8C4rAKMou86Iv/2870+Rmv+Yf1A4vDqr1k0lzZbJNdsge8ckRqZNTck6ahJGEPJFnMnJGzovz6rx9j5acyc4W+QXn/QsC/aQs</latexit>
<latexit sha1_base64="fqWQc6zGpkyDFBQqNdaB73Szm1Y=">AAACHXicbVDLSgNBEJyNrxhfUY9eFoPgKexKRI+iF48ajArZILOTTjJkHstMrxqW/Qyv+jXexKv4MYKzMQdjbGgoqrqp7ooTwS0GwadXmptfWFwqL1dWVtfWN6qbW9dWp4ZBi2mhzW1MLQiuoIUcBdwmBqiMBdzEw7NCv7kHY7lWVzhKoCNpX/EeZxQd1Y4QHjET+h7yu2otqAfj8mdBOAE1MqmLu+pX1NUslaCQCWptOwwS7GTUIGcC8kqUWkgoG9I+tB1UVILtZOOTc3/PMV2/p41rhf6Y/b2RUWntSMZuUlIc2L9aQf6ntVPsHXcyrpIUQbEfo14qfNR+8b/f5QYYipEDlBnubvXZgBrK0KU05RJL94OCB6alpKqbRc08iwrDOM6aeZFX+DedWXB9UA8b9cPLRu3kdJJcmeyQXbJPQnJETsg5uSAtwogmT+SZvHgv3qv35r3/jJa8yc42mSrv4xsJ9qQ+</latexit>
<latexit sha1_base64="HdjqFVT8unCJTgRGjM0IA4G+xaA=">AAACH3icbVDLSsNAFJ34tr6qLt0Ei+CqJKLoUnTjUsU+oAllMr1th85M4syNWkK+w61+jTtx68cITtoubOuBgcM593LunCgR3KDnfTsLi0vLK6tr66WNza3tnfLuXt3EqWZQY7GIdTOiBgRXUEOOApqJBiojAY1ocF34jSfQhsfqAYcJhJL2FO9yRtFKYYDwglkCVKWYt8sVr+qN4M4Tf0IqZILbdvkn6MQslaCQCWpMy/cSDDOqkTMBeSlIDSSUDWgPWpYqKsGE2ejo3D2ySsftxto+he5I/buRUWnMUEZ2UlLsm1mvEP/zWil2L8KMqyRFUGwc1E2Fi7FbNOB2uAaGYmgJZZrbW13Wp5oytD1NpUTS/kHBM4ulpKqTBfd5FhSBUZTd50Vf/mw786R+UvVPq2d3p5XLq0lza+SAHJJj4pNzckluyC2pEUYeySt5I+/Ou/PhfDpf49EFZ7KzT6bgfP8Cx/WlKQ==</latexit>
<latexit sha1_base64="YTurMLA4tl/mqRTdkFPnehqG8wU=">AAACH3icbVBNSwMxFMz6WetX1aOXxSJ4Krui6FH04lGLrUJ3kSR91WCSXZO3aln2d3i1v8abeO2PEcy2Pah1IDDMvMe8DEulsBgEQ29mdm5+YbGyVF1eWV1br21stm2SGQ4tnsjE3DBqQQoNLRQo4SY1QBWTcM0ezkr/+gmMFYm+wn4KsaJ3WvQEp+ikOEJ4wZxliGCK21o9aAQj+NMknJA6meDitvYVdROeKdDIJbW2EwYpxjk1KLiEohplFlLKH+gddBzVVIGN89HRhb/rlK7fS4x7Gv2R+nMjp8ravmJuUlG8t3+9UvzP62TYO45zodMMQfNxUC+TPiZ+2YDfFQY4yr4jlBvhbvX5PTWUuw5+pzDl/qDhmSdKUd3No2aRR2UgY3mzKPsK/7YzTdr7jfCgcXh5UD85nTRXIdtkh+yRkByRE3JOLkiLcPJIXskbGXgD79378D7HozPeZGeL/II3/AbXVKUy</latexit>
<latexit sha1_base64="oR40QGMmh45lOMswonCXgLASu2o=">AAACHHicbVBNS8NAEN3Ur1q/qh69BIvgqSRS0WPRi8da7Ac2pWy203bp7ibsbtQS8i+82l/jTbwK/hjBTZuDbR0YeLw3j5l5fsio0o7zbeXW1jc2t/LbhZ3dvf2D4uFRUwWRJNAgAQtk28cKGBXQ0FQzaIcSMPcZtPzxbaq3nkAqGogHPQmhy/FQ0AElWBvq0dPwYlxxOekVS07ZmZW9CtwMlFBWtV7xx+sHJOIgNGFYqY7rhLobY6kpYZAUvEhBiMkYD6FjoMAcVDeeXZzYZ4bp24NAmhbanrF/HTHmSk24byY51iO1rKXkf1on0oPrbkxFGGkQZL5oEDFbB3b6vt2nEohmEwMwkdTcapMRlphoE9LCFp+bHwQ8k4BzLPqxV09iL13o+3E9SfNyl9NZBc2LslspX95XStWbLLk8OkGn6By56ApV0R2qoQYiSKBX9Iam1tR6tz6sz/lozso8x2ihrK9fwCCjiQ==</latexit>
<latexit sha1_base64="z/vtTQ1yS8oS2ADAkIL9fIaFBFg=">AAACHnicbVBNS8NAFNz4bf2qevQSLIKnkkhFj0URPNZqVWhC2WxfdenuJuy+qCXkb3jVX+NNvOqPEdy0PdjqwMIw8x7zdqJEcIOe9+XMzM7NLywuLZdWVtfWN8qbW9cmTjWDFotFrG8jakBwBS3kKOA20UBlJOAm6p8W/s0DaMNjdYWDBEJJ7xTvcUbRSkGA8IRZ+/KsEeadcsWrekO4f4k/JhUyRqNT/g66MUslKGSCGtP2vQTDjGrkTEBeClIDCWV9egdtSxWVYMJseHPu7lml6/ZibZ9Cd6j+3sioNGYgIzspKd6baa8Q//PaKfaOw4yrJEVQbBTUS4WLsVsU4Ha5BoZiYAllmttbXXZPNWVoa5pIiaT9g4JHFktJVTcLmnkWFIFRlDXzoi9/up2/5Pqg6teqhxe1Sv1k3NwS2SG7ZJ/45IjUyTlpkBZhJCHP5IW8Oq/Om/PufIxGZ5zxzjaZgPP5Aw08pDI=</latexit>
<latexit sha1_base64="muys/2ZQJ40hek8CS3JlhPgiw3k=">AAACGnicbVBNS8NAEN3U7/pV9eglWARPNRFFj6IXjyq2Fpoim+20Xbq7CbsTtYT8Ca/6a7yJVy/+GMFNm4NtHRh4vDfDm3lhLLhBz/t2SnPzC4tLyyvl1bX1jc3K1nbDRIlmUGeRiHQzpAYEV1BHjgKasQYqQwH34eAy1+8fQRseqTscxtCWtKd4lzOKlmoGCM+YHmYPlapX80blzgK/AFVS1PVD5SfoRCyRoJAJakzL92Jsp1QjZwKycpAYiCkb0B60LFRUgmmno3szd98yHbcbadsK3RH7dyOl0pihDO2kpNg301pO/qe1EuyetVOu4gRBsbFRNxEuRm7+vNvhGhiKoQWUaW5vdVmfasrQRjThEkr7g4InFklJVScNbrM0yA3DML3N8rz86XRmQeOo5h/XTm6Oq+cXRXLLZJfskQPik1NyTq7INakTRgR5Ia/kzXlz3p0P53M8WnKKnR0yUc7XL/whopk=</latexit>
<latexit sha1_base64="muys/2ZQJ40hek8CS3JlhPgiw3k=">AAACGnicbVBNS8NAEN3U7/pV9eglWARPNRFFj6IXjyq2Fpoim+20Xbq7CbsTtYT8Ca/6a7yJVy/+GMFNm4NtHRh4vDfDm3lhLLhBz/t2SnPzC4tLyyvl1bX1jc3K1nbDRIlmUGeRiHQzpAYEV1BHjgKasQYqQwH34eAy1+8fQRseqTscxtCWtKd4lzOKlmoGCM+YHmYPlapX80blzgK/AFVS1PVD5SfoRCyRoJAJakzL92Jsp1QjZwKycpAYiCkb0B60LFRUgmmno3szd98yHbcbadsK3RH7dyOl0pihDO2kpNg301pO/qe1EuyetVOu4gRBsbFRNxEuRm7+vNvhGhiKoQWUaW5vdVmfasrQRjThEkr7g4InFklJVScNbrM0yA3DML3N8rz86XRmQeOo5h/XTm6Oq+cXRXLLZJfskQPik1NyTq7INakTRgR5Ia/kzXlz3p0P53M8WnKKnR0yUc7XL/whopk=</latexit>
<latexit sha1_base64="gl8lzF6VILwNkPD/JDrZ88PO1Y4=">AAACHHicbVBNS8NAEN34bf2qevQSLIKnkoiiR9GLF6EWW4tNKZvttF3c3YTdiVpC/oVX+2u8iVfBHyO4aXuw1YGBx3szvJkXxoIb9LwvZ25+YXFpeWW1sLa+sblV3N6pmyjRDGosEpFuhNSA4ApqyFFAI9ZAZSjgLny4zPW7R9CGR+oWBzG0JO0p3uWMoqXuA4RnTK9pnLWLJa/sjcr9C/wJKJFJVdrF76ATsUSCQiaoMU3fi7GVUo2cCcgKQWIgpuyB9qBpoaISTCsdXZy5B5bpuN1I21bojtjfGymVxgxkaCclxb6Z1XLyP62ZYPeslXIVJwiKjY26iXAxcvP33Q7XwFAMLKBMc3ury/pUU4Y2pCmXUNofFDyxSEqqOmlQzdIgNwzDtJrlefmz6fwF9aOyf1w+uTkunV9Mklshe2SfHBKfnJJzckUqpEYYUeSFvJKhM3TenHfnYzw650x2dslUOZ8/3/6jnA==</latexit>
<latexit sha1_base64="gl8lzF6VILwNkPD/JDrZ88PO1Y4=">AAACHHicbVBNS8NAEN34bf2qevQSLIKnkoiiR9GLF6EWW4tNKZvttF3c3YTdiVpC/oVX+2u8iVfBHyO4aXuw1YGBx3szvJkXxoIb9LwvZ25+YXFpeWW1sLa+sblV3N6pmyjRDGosEpFuhNSA4ApqyFFAI9ZAZSjgLny4zPW7R9CGR+oWBzG0JO0p3uWMoqXuA4RnTK9pnLWLJa/sjcr9C/wJKJFJVdrF76ATsUSCQiaoMU3fi7GVUo2cCcgKQWIgpuyB9qBpoaISTCsdXZy5B5bpuN1I21bojtjfGymVxgxkaCclxb6Z1XLyP62ZYPeslXIVJwiKjY26iXAxcvP33Q7XwFAMLKBMc3ury/pUU4Y2pCmXUNofFDyxSEqqOmlQzdIgNwzDtJrlefmz6fwF9aOyf1w+uTkunV9Mklshe2SfHBKfnJJzckUqpEYYUeSFvJKhM3TenHfnYzw650x2dslUOZ8/3/6jnA==</latexit>
<latexit sha1_base64="UqFhaguoowE9uZMWnSSV/GKz9b8=">AAACJHicbVDLTgIxFO3gC/E16tLNRGLiiswYjC6JblwikUfCENIpF2hoO5O2g5LJ/Ilb+Rp3xoUb/8TEDrAQ8CRNTs69J+f2BBGjSrvul5Xb2Nza3snvFvb2Dw6P7OOThgpjSaBOQhbKVoAVMCqgrqlm0IokYB4waAaj+2zeHINUNBRPehJBh+OBoH1KsDZS17Z9DS/GlwwxVxpk2rWLbsmdwVkn3oIU0QLVrv3j90IScxCaMKxU23Mj3Umw1JQwSAt+rCDCZIQH0DZUYA6qk8wuT50Lo/ScfijNE9qZqX8diblKTXhgNjnWQ7U6y8T/Zu1Y9287CRVRrEGQeVA/Zo4OnawGp0clEM0mhmAiqbnVIUMsMTEdLKcE3PxBwDMJOceil/i1NPGzwCBIamnWl7fazjppXJW8cun6sVys3C2ay6MzdI4ukYduUAU9oCqqI4LG6BW9oak1td6tD+tzvpqzFp5TtATr+xfoDKa8</latexit>
<latexit sha1_base64="UqFhaguoowE9uZMWnSSV/GKz9b8=">AAACJHicbVDLTgIxFO3gC/E16tLNRGLiiswYjC6JblwikUfCENIpF2hoO5O2g5LJ/Ilb+Rp3xoUb/8TEDrAQ8CRNTs69J+f2BBGjSrvul5Xb2Nza3snvFvb2Dw6P7OOThgpjSaBOQhbKVoAVMCqgrqlm0IokYB4waAaj+2zeHINUNBRPehJBh+OBoH1KsDZS17Z9DS/GlwwxVxpk2rWLbsmdwVkn3oIU0QLVrv3j90IScxCaMKxU23Mj3Umw1JQwSAt+rCDCZIQH0DZUYA6qk8wuT50Lo/ScfijNE9qZqX8diblKTXhgNjnWQ7U6y8T/Zu1Y9287CRVRrEGQeVA/Zo4OnawGp0clEM0mhmAiqbnVIUMsMTEdLKcE3PxBwDMJOceil/i1NPGzwCBIamnWl7fazjppXJW8cun6sVys3C2ay6MzdI4ukYduUAU9oCqqI4LG6BW9oak1td6tD+tzvpqzFp5TtATr+xfoDKa8</latexit>
<latexit sha1_base64="UqFhaguoowE9uZMWnSSV/GKz9b8=">AAACJHicbVDLTgIxFO3gC/E16tLNRGLiiswYjC6JblwikUfCENIpF2hoO5O2g5LJ/Ilb+Rp3xoUb/8TEDrAQ8CRNTs69J+f2BBGjSrvul5Xb2Nza3snvFvb2Dw6P7OOThgpjSaBOQhbKVoAVMCqgrqlm0IokYB4waAaj+2zeHINUNBRPehJBh+OBoH1KsDZS17Z9DS/GlwwxVxpk2rWLbsmdwVkn3oIU0QLVrv3j90IScxCaMKxU23Mj3Umw1JQwSAt+rCDCZIQH0DZUYA6qk8wuT50Lo/ScfijNE9qZqX8diblKTXhgNjnWQ7U6y8T/Zu1Y9287CRVRrEGQeVA/Zo4OnawGp0clEM0mhmAiqbnVIUMsMTEdLKcE3PxBwDMJOceil/i1NPGzwCBIamnWl7fazjppXJW8cun6sVys3C2ay6MzdI4ukYduUAU9oCqqI4LG6BW9oak1td6tD+tzvpqzFp5TtATr+xfoDKa8</latexit>
<latexit sha1_base64="UqFhaguoowE9uZMWnSSV/GKz9b8=">AAACJHicbVDLTgIxFO3gC/E16tLNRGLiiswYjC6JblwikUfCENIpF2hoO5O2g5LJ/Ilb+Rp3xoUb/8TEDrAQ8CRNTs69J+f2BBGjSrvul5Xb2Nza3snvFvb2Dw6P7OOThgpjSaBOQhbKVoAVMCqgrqlm0IokYB4waAaj+2zeHINUNBRPehJBh+OBoH1KsDZS17Z9DS/GlwwxVxpk2rWLbsmdwVkn3oIU0QLVrv3j90IScxCaMKxU23Mj3Umw1JQwSAt+rCDCZIQH0DZUYA6qk8wuT50Lo/ScfijNE9qZqX8diblKTXhgNjnWQ7U6y8T/Zu1Y9287CRVRrEGQeVA/Zo4OnawGp0clEM0mhmAiqbnVIUMsMTEdLKcE3PxBwDMJOceil/i1NPGzwCBIamnWl7fazjppXJW8cun6sVys3C2ay6MzdI4ukYduUAU9oCqqI4LG6BW9oak1td6tD+tzvpqzFp5TtATr+xfoDKa8</latexit>
<latexit sha1_base64="LlfxlhCsfSHbHEvVS8P9BmuHzRY=">AAACL3icbVDLahsxFNUkaZu6LyddZiNiCt3UzJSUdhmSRboppKF+gMeYK/mOIyJpBulOEzPMIl+TbfM1pZuQbT+iUI3jRV4HBIdz7uXoHlFo5SmO/0Qrq2tPnj5bf9568fLV6zftjc2+z0snsSdznbuhAI9aWeyRIo3DwiEYoXEgTvYbf/ATnVe5/UHzAscGZlZlSgIFadLeSgnPSGTVAVp0oPk3KD5QaZWd1ZN2J+7GC/CHJFmSDlvicNL+l05zWRq0JDV4P0rigsYVOFJSY91KS48FyBOY4ShQCwb9uFocUfN3QZnyLHfhWeIL9fZGBcb7uRFh0gAd+/teIz7mjUrKvowrZYuS0MqboKzUnHLeNMKnyqEkPQ8EpFPhr1wegwNJobc7KcKEGyyeytwYsNMqPaqrtAkUojqqm76S++08JP2P3WSn++n7Tmd3b9ncOtti2+w9S9hntsu+skPWY5Kdswv2i11Gl9Hv6Cq6vhldiZY7b9kdRH//AxhZqt8=</latexit>
<latexit sha1_base64="uJYNi/HxFYwUqicgsQyV5rxJvWw=">AAACNXicbVBNTxsxFPTSL75aQjlycRshcSHaRUHtEcGlFyRABJCyUfTsvA1WbO/Kfts2Wu2ZX8MVfksPvVW99hcg1RtyKNCRLI1m3tP4jSi08hTHP6KFFy9fvX6zuLS8svr23Vpr/f25z0snsSdznbtLAR61stgjRRovC4dghMYLMTls/Iuv6LzK7RlNCxwYGFuVKQkUpGHrQ0r4nURWnYGf7PgCZWPyIyh2qLTKjuthqx134hn4c5LMSZvNcTxs3aejXJYGLUkN3veTuKBBBY6U1Fgvp6XHAuQExtgP1IJBP6hmp9R8KygjnuUuPEt8pv67UYHxfmpEmDRAV/6p14j/8/olZZ8HlbJFSWjlQ1BWak45b3rhI+VQkp4GAtKp8Fcur8CBpNDeoxRhwg0Wv8ncGLCjKj2tq7QJFKI6rZu+kqftPCfnu52k29k76bb3D+bNLbJN9pFts4R9YvvsCztmPSbZNbtht+wuuot+Rr+i3w+jC9F8Z4M9QvTnLxWtrWM=</latexit>
<latexit sha1_base64="onu66ZApQHer6gmA3Do/knHnV+c=">AAACK3icbVDLSgNBEJz1bXxFPXjwMhgET2FXFD36uOhNxRghG8LsbEcHZ2aXmV41LPs1XvVrPCle/QzB2SQHozY0FFXdFFVRKoVF33/zxsYnJqemZ2Yrc/MLi0vV5ZUrm2SGQ4MnMjHXEbMghYYGCpRwnRpgKpLQjO6OS715D8aKRF9iL4W2YjdadAVn6KhOdS1EeMT8MI5FSTBJT3WaYdGp1vy63x/6FwRDUCPDOetUv8I44ZkCjVwya1uBn2I7ZwYFl1BUwsxCyvgdu4GWg5opsO28H6Cgm46JaTcxbjXSPvvzI2fK2p6K3KVieGt/ayX5n9bKsLvfzkUZCTQfGHUzSTGhZRs0FgY4yp4DjBtXAaf8lhnG0XU24hIpl0HDA0+UYjrOw4siD0vDKMovirKv4Hc7f8HVdj3Yqe+e79QOjobNzZB1skG2SED2yAE5IWekQTgpyBN5Ji/ei/fqvXsfg9Mxb/izSkbG+/wGEqSpWg==</latexit>
<latexit sha1_base64="onu66ZApQHer6gmA3Do/knHnV+c=">AAACK3icbVDLSgNBEJz1bXxFPXjwMhgET2FXFD36uOhNxRghG8LsbEcHZ2aXmV41LPs1XvVrPCle/QzB2SQHozY0FFXdFFVRKoVF33/zxsYnJqemZ2Yrc/MLi0vV5ZUrm2SGQ4MnMjHXEbMghYYGCpRwnRpgKpLQjO6OS715D8aKRF9iL4W2YjdadAVn6KhOdS1EeMT8MI5FSTBJT3WaYdGp1vy63x/6FwRDUCPDOetUv8I44ZkCjVwya1uBn2I7ZwYFl1BUwsxCyvgdu4GWg5opsO28H6Cgm46JaTcxbjXSPvvzI2fK2p6K3KVieGt/ayX5n9bKsLvfzkUZCTQfGHUzSTGhZRs0FgY4yp4DjBtXAaf8lhnG0XU24hIpl0HDA0+UYjrOw4siD0vDKMovirKv4Hc7f8HVdj3Yqe+e79QOjobNzZB1skG2SED2yAE5IWekQTgpyBN5Ji/ei/fqvXsfg9Mxb/izSkbG+/wGEqSpWg==</latexit>
<latexit sha1_base64="oAegdQBdBKnynEHkST7p7V2/Osc=">AAACK3icbVBNSwMxFMz6Wb+rHjx4WSyCp7Irih5FLx5rsbXQLSVJXzWYZJfkrVqW/TVe7a/xpHj1Zwhm2x6sOhAYZt5j8oYlUlgMgjdvZnZufmGxtLS8srq2vlHe3GraODUcGjyWsWkxakEKDQ0UKKGVGKCKSbhh9xeFf/MAxopYX+MggY6it1r0BafopG55J0J4cntZAlSnGHVZiggm75YrQTUYwf9LwgmpkAlq3fJX1It5qkAjl9Tadhgk2MmoQcEl5MtRaiGh/J7eQttRTRXYTjY6IPf3ndLz+7FxT6M/Un9uZFRZO1DMTSqKd/a3V4j/ee0U+6edTOgkRdB8HNRPpY+xX7Th94QBjnLgCOVGuL/6/I4ayl0H0ylMuRs0PPJYKap7WVTPs6gIZCyr50Vf4e92/pLmYTU8qh5fHVXOzifNlcgu2SMHJCQn5IxckhppEE5y8kxeyNAbeq/eu/cxHp3xJjvbZAre5zcN+Kns</latexit>
<latexit sha1_base64="oAegdQBdBKnynEHkST7p7V2/Osc=">AAACK3icbVBNSwMxFMz6Wb+rHjx4WSyCp7Irih5FLx5rsbXQLSVJXzWYZJfkrVqW/TVe7a/xpHj1Zwhm2x6sOhAYZt5j8oYlUlgMgjdvZnZufmGxtLS8srq2vlHe3GraODUcGjyWsWkxakEKDQ0UKKGVGKCKSbhh9xeFf/MAxopYX+MggY6it1r0BafopG55J0J4cntZAlSnGHVZiggm75YrQTUYwf9LwgmpkAlq3fJX1It5qkAjl9Tadhgk2MmoQcEl5MtRaiGh/J7eQttRTRXYTjY6IPf3ndLz+7FxT6M/Un9uZFRZO1DMTSqKd/a3V4j/ee0U+6edTOgkRdB8HNRPpY+xX7Th94QBjnLgCOVGuL/6/I4ayl0H0ylMuRs0PPJYKap7WVTPs6gIZCyr50Vf4e92/pLmYTU8qh5fHVXOzifNlcgu2SMHJCQn5IxckhppEE5y8kxeyNAbeq/eu/cxHp3xJjvbZAre5zcN+Kns</latexit>
<latexit sha1_base64="oAegdQBdBKnynEHkST7p7V2/Osc=">AAACK3icbVBNSwMxFMz6Wb+rHjx4WSyCp7Irih5FLx5rsbXQLSVJXzWYZJfkrVqW/TVe7a/xpHj1Zwhm2x6sOhAYZt5j8oYlUlgMgjdvZnZufmGxtLS8srq2vlHe3GraODUcGjyWsWkxakEKDQ0UKKGVGKCKSbhh9xeFf/MAxopYX+MggY6it1r0BafopG55J0J4cntZAlSnGHVZiggm75YrQTUYwf9LwgmpkAlq3fJX1It5qkAjl9Tadhgk2MmoQcEl5MtRaiGh/J7eQttRTRXYTjY6IPf3ndLz+7FxT6M/Un9uZFRZO1DMTSqKd/a3V4j/ee0U+6edTOgkRdB8HNRPpY+xX7Th94QBjnLgCOVGuL/6/I4ayl0H0ylMuRs0PPJYKap7WVTPs6gIZCyr50Vf4e92/pLmYTU8qh5fHVXOzifNlcgu2SMHJCQn5IxckhppEE5y8kxeyNAbeq/eu/cxHp3xJjvbZAre5zcN+Kns</latexit>
<latexit sha1_base64="oAegdQBdBKnynEHkST7p7V2/Osc=">AAACK3icbVBNSwMxFMz6Wb+rHjx4WSyCp7Irih5FLx5rsbXQLSVJXzWYZJfkrVqW/TVe7a/xpHj1Zwhm2x6sOhAYZt5j8oYlUlgMgjdvZnZufmGxtLS8srq2vlHe3GraODUcGjyWsWkxakEKDQ0UKKGVGKCKSbhh9xeFf/MAxopYX+MggY6it1r0BafopG55J0J4cntZAlSnGHVZiggm75YrQTUYwf9LwgmpkAlq3fJX1It5qkAjl9Tadhgk2MmoQcEl5MtRaiGh/J7eQttRTRXYTjY6IPf3ndLz+7FxT6M/Un9uZFRZO1DMTSqKd/a3V4j/ee0U+6edTOgkRdB8HNRPpY+xX7Th94QBjnLgCOVGuL/6/I4ayl0H0ylMuRs0PPJYKap7WVTPs6gIZCyr50Vf4e92/pLmYTU8qh5fHVXOzifNlcgu2SMHJCQn5IxckhppEE5y8kxeyNAbeq/eu/cxHp3xJjvbZAre5zcN+Kns</latexit>
<latexit sha1_base64="ZrgITluuF6vmJwC4sBU06PhfIvY=">AAACHHicbVBNS8NAEN34WetX1aOXYBE8lUQUPRa9eLBQi63FpshmO9XF3U3Ynagl5F94tb/Gm3gV/DGCm7YHqw4MPN6b4c28MBbcoOd9OjOzc/MLi4Wl4vLK6tp6aWOzZaJEM2iySES6HVIDgitoIkcB7VgDlaGAq/D+NNevHkAbHqlLHMTQlfRW8T5nFC11HSA8YVo7r2U3pbJX8Ubl/gX+BJTJpOo3pa+gF7FEgkImqDEd34uxm1KNnAnIikFiIKbsnt5Cx0JFJZhuOro4c3ct03P7kbat0B2xPzdSKo0ZyNBOSop35reWk/9pnQT7x92UqzhBUGxs1E+Ei5Gbv+/2uAaGYmABZZrbW112RzVlaEOacgml/UHBI4ukpKqXBo0sDXLDMEwbWZ6X/zudv6C1X/EPKocXB+XqySS5AtkmO2SP+OSIVMkZqZMmYUSRZ/JChs7QeXXenPfx6Iwz2dkiU+V8fAOBaaNk</latexit>
<latexit sha1_base64="gGEnD0lTvs3n+fgJ2GbS8suyRXk=">AAACHXicbVDLSgNBEJyN7/iKevSyGARPYVciegx68eAhBmOEbJDZSUeHzGOZ6VXDsp/hVb/Gm3gVP0ZwNuagxoaGoqqb6q44EdxiEHx4pZnZufmFxaXy8srq2nplY/PS6tQwaDMttLmKqQXBFbSRo4CrxACVsYBOPDwp9M4dGMu1usBRAj1JbxQfcEbRUd0I4QGzM21tfl2pBrVgXP40CCegSibVvK58Rn3NUgkKmaDWdsMgwV5GDXImIC9HqYWEsiG9ga6DikqwvWx8cu7vOqbvD7RxrdAfsz83MiqtHcnYTUqKt/avVpD/ad0UB0e9jKskRVDs22iQCh+1X/zv97kBhmLkAGWGu1t9dksNZehS+uUSS/eDgnumpaSqn0WtPIsKwzjOWnmRV/g3nWlwuV8L67WD83q1cTxJbpFskx2yR0JySBrklDRJmzCiySN5Is/es/fivXpv36Mlb7KzRX6V9/4F5hSkKQ==</latexit>
<latexit sha1_base64="p8E9BqFW1JMuhZOBcL1dRTZwzb4=">AAACH3icbVDLSsNAFJ34tr6qLt0Ei+CqJFLRpSiISxXbCk2QyfRWB+cRZ27UGvIdbvVr3IlbP0Zw0nah1QMDh3Pu5dw5SSq4xSD49CYmp6ZnZufmKwuLS8sr1dW1ltWZYdBkWmhzmVALgitoIkcBl6kBKhMB7eT2qPTb92As1+oC+ynEkl4r3uOMopPiCOER82Ojn0AVV9VaUA8G8P+ScERqZITTq+pX1NUsk6CQCWptJwxSjHNqkDMBRSXKLKSU3dJr6DiqqAQb54OjC3/LKV2/p417Cv2B+nMjp9LavkzcpKR4Y8e9UvzP62TY249zrtIMQbFhUC8TPmq/bMDvcgMMRd8Rygx3t/rshhrK0PX0KyWR7g8KHpiWkqpuHp0XeVQGJkl+XpR9hePt/CWtnXrYqO+eNWoHh6Pm5sgG2STbJCR75ICckFPSJIzckWfyQl69V+/Ne/c+hqMT3mhnnfyC9/kNnVmlEA==</latexit>
<latexit sha1_base64="uxHmbiLcbFHbY1P8ny/uuNIxXpc=">AAACHHicbVBNS8NAEN34WetX1aOXYBE8lUQqeix68aBQi61iU2SznerS3U3Ynagl5F94tb/Gm3gV/DGCm7YHqw4MPN6b4c28MBbcoOd9OjOzc/MLi4Wl4vLK6tp6aWOzZaJEM2iySET6OqQGBFfQRI4CrmMNVIYCrsL+Sa5fPYA2PFKXOIihI+md4j3OKFrqJkB4wrR+dp7dlspexRuV+xf4E1Amk6rflr6CbsQSCQqZoMa0fS/GTko1ciYgKwaJgZiyPr2DtoWKSjCddHRx5u5apuv2Im1boTtif26kVBozkKGdlBTvzW8tJ//T2gn2jjopV3GCoNjYqJcIFyM3f9/tcg0MxcACyjS3t7rsnmrK0IY05RJK+4OCRxZJSVU3DRpZGuSGYZg2sjwv/3c6f0Frv+JXKwcX1XLteJJcgWyTHbJHfHJIauSU1EmTMKLIM3khQ2fovDpvzvt4dMaZ7GyRqXI+vgGGf6Nn</latexit>
<latexit sha1_base64="400J+XfkSfO33mkbANDTQbOf/Go=">AAACHXicbVBNS8NAEN3U7++qRy/BIngqiVT0WPTiUcWq0BTZbKft0v0IuxO1hPwMr/prvIlX8ccIbmoOtjow8Hhvhjfz4kRwi0Hw6VVmZufmFxaXlldW19Y3qptb11anhkGLaaHNbUwtCK6ghRwF3CYGqIwF3MTD00K/uQdjuVZXOEqgI2lf8R5nFB3VjhAeMbuidpjfVWtBPRiX/xeEJaiRss7vql9RV7NUgkImqLXtMEiwk1GDnAnIl6PUQkLZkPah7aCiEmwnG5+c+3uO6fo9bVwr9Mfs742MSmtHMnaTkuLATmsF+Z/WTrF33Mm4SlIExX6MeqnwUfvF/36XG2AoRg5QZri71WcDaihDl9KESyzdDwoemJaSqm4WXeZZVBjGcXaZF3mF0+n8BdcH9bBRP7xo1JonZXKLZIfskn0SkiPSJGfknLQII5o8kWfy4r14r96b9/4zWvHKnW0yUd7HN85wpBs=</latexit>
<latexit sha1_base64="gGEnD0lTvs3n+fgJ2GbS8suyRXk=">AAACHXicbVDLSgNBEJyN7/iKevSyGARPYVciegx68eAhBmOEbJDZSUeHzGOZ6VXDsp/hVb/Gm3gVP0ZwNuagxoaGoqqb6q44EdxiEHx4pZnZufmFxaXy8srq2nplY/PS6tQwaDMttLmKqQXBFbSRo4CrxACVsYBOPDwp9M4dGMu1usBRAj1JbxQfcEbRUd0I4QGzM21tfl2pBrVgXP40CCegSibVvK58Rn3NUgkKmaDWdsMgwV5GDXImIC9HqYWEsiG9ga6DikqwvWx8cu7vOqbvD7RxrdAfsz83MiqtHcnYTUqKt/avVpD/ad0UB0e9jKskRVDs22iQCh+1X/zv97kBhmLkAGWGu1t9dksNZehS+uUSS/eDgnumpaSqn0WtPIsKwzjOWnmRV/g3nWlwuV8L67WD83q1cTxJbpFskx2yR0JySBrklDRJmzCiySN5Is/es/fivXpv36Mlb7KzRX6V9/4F5hSkKQ==</latexit>
<latexit sha1_base64="p8E9BqFW1JMuhZOBcL1dRTZwzb4=">AAACH3icbVDLSsNAFJ34tr6qLt0Ei+CqJFLRpSiISxXbCk2QyfRWB+cRZ27UGvIdbvVr3IlbP0Zw0nah1QMDh3Pu5dw5SSq4xSD49CYmp6ZnZufmKwuLS8sr1dW1ltWZYdBkWmhzmVALgitoIkcBl6kBKhMB7eT2qPTb92As1+oC+ynEkl4r3uOMopPiCOER82Ojn0AVV9VaUA8G8P+ScERqZITTq+pX1NUsk6CQCWptJwxSjHNqkDMBRSXKLKSU3dJr6DiqqAQb54OjC3/LKV2/p417Cv2B+nMjp9LavkzcpKR4Y8e9UvzP62TY249zrtIMQbFhUC8TPmq/bMDvcgMMRd8Rygx3t/rshhrK0PX0KyWR7g8KHpiWkqpuHp0XeVQGJkl+XpR9hePt/CWtnXrYqO+eNWoHh6Pm5sgG2STbJCR75ICckFPSJIzckWfyQl69V+/Ne/c+hqMT3mhnnfyC9/kNnVmlEA==</latexit>
<latexit sha1_base64="qk2n4EH0S3r+AsR+a1EjWr7xQq4=">AAACG3icbVBNS8NAEN34bf2qevQSLIKnkkhFj0U9eBFqsVVpStlsp7p0dxN2J2oJ+RVe9dd4E68e/DGCm7YHax0YeLw3w5t5YSy4Qc/7cmZm5+YXFpeWCyura+sbxc2tpokSzaDBIhHpm5AaEFxBAzkKuIk1UBkKuA77p7l+/QDa8Ehd4SCGtqR3ivc4o2ip2wDhCdOzi6xTLHllb1juNPDHoETGVesUv4NuxBIJCpmgxrR8L8Z2SjVyJiArBImBmLI+vYOWhYpKMO10eHDm7lmm6/YibVuhO2R/b6RUGjOQoZ2UFO/NXy0n/9NaCfaO2ylXcYKg2MiolwgXIzf/3u1yDQzFwALKNLe3uuyeasrQZjThEkr7g4JHFklJVTcN6lka5IZhmNazPC//bzrToHlQ9ivlw8tKqXoyTm6J7JBdsk98ckSq5JzUSIMwIskzeSGvzqvz5rw7H6PRGWe8s00myvn8AcmNowU=</latexit>
<latexit sha1_base64="uxHmbiLcbFHbY1P8ny/uuNIxXpc=">AAACHHicbVBNS8NAEN34WetX1aOXYBE8lUQqeix68aBQi61iU2SznerS3U3Ynagl5F94tb/Gm3gV/DGCm7YHqw4MPN6b4c28MBbcoOd9OjOzc/MLi4Wl4vLK6tp6aWOzZaJEM2iySET6OqQGBFfQRI4CrmMNVIYCrsL+Sa5fPYA2PFKXOIihI+md4j3OKFrqJkB4wrR+dp7dlspexRuV+xf4E1Amk6rflr6CbsQSCQqZoMa0fS/GTko1ciYgKwaJgZiyPr2DtoWKSjCddHRx5u5apuv2Im1boTtif26kVBozkKGdlBTvzW8tJ//T2gn2jjopV3GCoNjYqJcIFyM3f9/tcg0MxcACyjS3t7rsnmrK0IY05RJK+4OCRxZJSVU3DRpZGuSGYZg2sjwv/3c6f0Frv+JXKwcX1XLteJJcgWyTHbJHfHJIauSU1EmTMKLIM3khQ2fovDpvzvt4dMaZ7GyRqXI+vgGGf6Nn</latexit>
<latexit sha1_base64="TQxrrtf1GGMk+a+tOiQIJsCd1So=">AAACJHicbVDLSsNAFJ34rO+oSzfBIrgqiSi6FN0IblSsCk0pk8ltHZxHmLmplpA/cWu/xp24cOOfCE5qF74ODBzOuZdz5ySZ4BbD8M2bmJyanpmtzc0vLC4tr/ira1dW54ZBk2mhzU1CLQiuoIkcBdxkBqhMBFwnd8eVf90HY7lWlzjIoC1pT/EuZxSd1PH9GOEBi1Ol7wWkPSg7fj1shCMEf0k0JnUyxlnH/4hTzXIJCpmg1raiMMN2QQ1yJqCcj3MLGWV3tActRxWVYNvF6PIy2HJKGnS1cU9hMFK/bxRUWjuQiZuUFG/tb68S//NaOXYP2gVXWY6g2FdQNxcB6qCqIUi5AYZi4AhlhrtbA3ZLDWXoyvqRkkj3BwX3TEtJVVrEF2URV4FJUlyUVV/R73b+kqudRrTb2DvfrR8ejZurkQ2ySbZJRPbJITkhZ6RJGOmTR/JEht7Qe/ZevNev0QlvvLNOfsB7/wSUp6aL</latexit>
<latexit sha1_base64="AOWkeK+AA+CnXE8Spy/xTO+lUjM=">AAACM3icbVBNSwMxEM36/W3VowcXi+DFuiuKHkVBPNZirdAtJZtOazDJLsmsWpY9+mu82h8j3sSrP0EwW3uwrQOBx3sz8zIvjAU36HlvzsTk1PTM7Nz8wuLS8spqYW39xkSJZlBlkYj0bUgNCK6gihwF3MYaqAwF1ML781yvPYA2PFLX2I2hIWlH8TZnFC3VLGwFCE+YljXsoaZccdXZv7C79jDJcdYsFL2S1y93HPgDUCSDKjcL30ErYokEhUxQY+q+F2MjpRo5E5AtBImBmLJ72oG6hYpKMI20f0jm7lim5bYjbZ9Ct8/+nUipNKYrQ9spKd6ZUS0n/9PqCbZPGilXcYKg2K9ROxEuRm6eitviGhiKrgWUaW7/6rI7qilDm92QSyjtDQoeWSQlVa00qGRpkBuGYVrJ8rz80XTGwc1ByT8sHV0dFk/PBsnNkU2yTXaJT47JKbkkZVIljDyTF/JKek7PeXc+nM/f1glnMLNBhsr5+gGS2ayk</latexit>
<latexit sha1_base64="XkSfTJnEe/ceWT2SPqC/cOTtbo0=">AAACJXicbVDLSgNBEJz1bXxFPXpZDIKnsCuKHoN68BjFRCEbwuyko4PzWGZ6jWHZT/FqvsabCJ78EsHZmIOJFgwUVd1UT8WJ4BaD4MObmZ2bX1hcWi6trK6tb5Q3t5pWp4ZBg2mhzW1MLQiuoIEcBdwmBqiMBdzED2eFf/MIxnKtrnGQQFvSO8V7nFF0Uqe8GSE8YXau+8pisZh3ypWgGozg/yXhmFTIGPVO+SvqapZKUMgEtbYVBgm2M2qQMwF5KUotJJQ90DtoOaqoBNvORqfn/p5Tun5PG/cU+iP190ZGpbUDGbtJSfHeTnuF+J/XSrF30s64SlIExX6CeqnwUftFD36XG2AoBo5QZri71Wf31FCGrq2JlFi6PyjoMy0lVd0susqzqAiM4+wqL/oKp9v5S5oH1fCwenR5WKmdjptbIjtkl+yTkByTGrkgddIgjPTJM3khQ2/ovXpv3vvP6Iw33tkmE/A+vwGfSqcZ</latexit>
<latexit sha1_base64="+Cwi+aVNb39//t0rmMsdEPJXGjk=">AAACHnicbVDLSsNAFJ34flt16SZYBFclkYoui27cCLVYKzRFJpPbdnAeYeZGLSG/4Va/xp241Y8RnNQufB0YOJxzL+fOiVPBLQbBuzc1PTM7N7+wuLS8srq2XtnYvLQ6MwzaTAttrmJqQXAFbeQo4Co1QGUsoBPfnJR+5xaM5Vpd4CiFnqQDxfucUXRSFCHcY36mExDFdaUa1IIx/L8knJAqmaB5XfmIEs0yCQqZoNZ2wyDFXk4NciagWIoyCyllN3QAXUcVlWB7+fjmwt91SuL3tXFPoT9Wv2/kVFo7krGblBSH9rdXiv953Qz7R72cqzRDUOwrqJ8JH7VfFuAn3ABDMXKEMsPdrT4bUkMZupp+pMTS/UHBHdNSUpXkUavIozIwjvNWUfYV/m7nL7ncr4X12sF5vdo4njS3QLbJDtkjITkkDXJKmqRNGEnJA3kkT96T9+y9eK9fo1PeZGeL/ID39gmWW6SD</latexit>
<latexit sha1_base64="SsQJTJ3mP3xMsXw0hL4q8C8TXKI=">AAACOXicbVBNTxsxFPQChRQopPTYi0WExCnarUBwjODSikuKCETKRpHtfRtc/LGyvQnRan9Bfw1X8ks49lZx5Vyp3iQHvp5kaTQzT+M3NBPcujB8CJaWVz6srtU+rm9sftrarn/eubQ6Nww6TAttupRYEFxBx3EnoJsZIJIKuKI3p5V+NQJjuVYXbpJBX5Kh4ilnxHlqUN+LHdw6mhZtAyOuc4vPlB4LSIaAf6hfwCpbOag3wmY4G/wWRAvQQItpD+r/4kSzXIJyTBBre1GYuX5BjONMQLke5xYywm7IEHoeKiLB9ovZOSXe80yCU238Uw7P2OcbBZHWTiT1TknctX2tVeR7Wi936XG/4CrLHSg2D0pzgZ3GVTc44cYfLCYeEGa4/ytm18QQ5nyDL1Ko9DcoGDMtJVFJEZ+XRVwFUlqcl1Vf0et23oLLb83ooHn486DROlk0V0Nf0S7aRxE6Qi30HbVRBzH0G92hezQNpsGf4G/wOLcuBYudL+jFBE//AQUcr2U=</latexit>
<latexit sha1_base64="XkSfTJnEe/ceWT2SPqC/cOTtbo0=">AAACJXicbVDLSgNBEJz1bXxFPXpZDIKnsCuKHoN68BjFRCEbwuyko4PzWGZ6jWHZT/FqvsabCJ78EsHZmIOJFgwUVd1UT8WJ4BaD4MObmZ2bX1hcWi6trK6tb5Q3t5pWp4ZBg2mhzW1MLQiuoIEcBdwmBqiMBdzED2eFf/MIxnKtrnGQQFvSO8V7nFF0Uqe8GSE8YXau+8pisZh3ypWgGozg/yXhmFTIGPVO+SvqapZKUMgEtbYVBgm2M2qQMwF5KUotJJQ90DtoOaqoBNvORqfn/p5Tun5PG/cU+iP190ZGpbUDGbtJSfHeTnuF+J/XSrF30s64SlIExX6CeqnwUftFD36XG2AoBo5QZri71Wf31FCGrq2JlFi6PyjoMy0lVd0susqzqAiM4+wqL/oKp9v5S5oH1fCwenR5WKmdjptbIjtkl+yTkByTGrkgddIgjPTJM3khQ2/ovXpv3vvP6Iw33tkmE/A+vwGfSqcZ</latexit>
<latexit sha1_base64="+Cwi+aVNb39//t0rmMsdEPJXGjk=">AAACHnicbVDLSsNAFJ34flt16SZYBFclkYoui27cCLVYKzRFJpPbdnAeYeZGLSG/4Va/xp241Y8RnNQufB0YOJxzL+fOiVPBLQbBuzc1PTM7N7+wuLS8srq2XtnYvLQ6MwzaTAttrmJqQXAFbeQo4Co1QGUsoBPfnJR+5xaM5Vpd4CiFnqQDxfucUXRSFCHcY36mExDFdaUa1IIx/L8knJAqmaB5XfmIEs0yCQqZoNZ2wyDFXk4NciagWIoyCyllN3QAXUcVlWB7+fjmwt91SuL3tXFPoT9Wv2/kVFo7krGblBSH9rdXiv953Qz7R72cqzRDUOwrqJ8JH7VfFuAn3ABDMXKEMsPdrT4bUkMZupp+pMTS/UHBHdNSUpXkUavIozIwjvNWUfYV/m7nL7ncr4X12sF5vdo4njS3QLbJDtkjITkkDXJKmqRNGEnJA3kkT96T9+y9eK9fo1PeZGeL/ID39gmWW6SD</latexit>
<latexit sha1_base64="uxHmbiLcbFHbY1P8ny/uuNIxXpc=">AAACHHicbVBNS8NAEN34WetX1aOXYBE8lUQqeix68aBQi61iU2SznerS3U3Ynagl5F94tb/Gm3gV/DGCm7YHqw4MPN6b4c28MBbcoOd9OjOzc/MLi4Wl4vLK6tp6aWOzZaJEM2iySET6OqQGBFfQRI4CrmMNVIYCrsL+Sa5fPYA2PFKXOIihI+md4j3OKFrqJkB4wrR+dp7dlspexRuV+xf4E1Amk6rflr6CbsQSCQqZoMa0fS/GTko1ciYgKwaJgZiyPr2DtoWKSjCddHRx5u5apuv2Im1boTtif26kVBozkKGdlBTvzW8tJ//T2gn2jjopV3GCoNjYqJcIFyM3f9/tcg0MxcACyjS3t7rsnmrK0IY05RJK+4OCRxZJSVU3DRpZGuSGYZg2sjwv/3c6f0Frv+JXKwcX1XLteJJcgWyTHbJHfHJIauSU1EmTMKLIM3khQ2fovDpvzvt4dMaZ7GyRqXI+vgGGf6Nn</latexit>
<latexit sha1_base64="TQxrrtf1GGMk+a+tOiQIJsCd1So=">AAACJHicbVDLSsNAFJ34rO+oSzfBIrgqiSi6FN0IblSsCk0pk8ltHZxHmLmplpA/cWu/xp24cOOfCE5qF74ODBzOuZdz5ySZ4BbD8M2bmJyanpmtzc0vLC4tr/ira1dW54ZBk2mhzU1CLQiuoIkcBdxkBqhMBFwnd8eVf90HY7lWlzjIoC1pT/EuZxSd1PH9GOEBi1Ol7wWkPSg7fj1shCMEf0k0JnUyxlnH/4hTzXIJCpmg1raiMMN2QQ1yJqCcj3MLGWV3tActRxWVYNvF6PIy2HJKGnS1cU9hMFK/bxRUWjuQiZuUFG/tb68S//NaOXYP2gVXWY6g2FdQNxcB6qCqIUi5AYZi4AhlhrtbA3ZLDWXoyvqRkkj3BwX3TEtJVVrEF2URV4FJUlyUVV/R73b+kqudRrTb2DvfrR8ejZurkQ2ySbZJRPbJITkhZ6RJGOmTR/JEht7Qe/ZevNev0QlvvLNOfsB7/wSUp6aL</latexit>
<latexit sha1_base64="nyzgROdLylaYuaya1zj1yS8EFnA=">AAACKnicbVDLSgMxFM34flsV3LgJFkEXlhlRdCkK4rIWW4VOKZn0tgaTzJDcUcs4P+NWv8aduPU3BDO1C6seCBzOfZybEyVSWPT9N29sfGJyanpmdm5+YXFpubSy2rBxajjUeSxjcx0xC1JoqKNACdeJAaYiCVfR7WlRv7oDY0WsL7GfQEuxnhZdwRk6qV1aDxEeMNsOduiZW7GLqRa6l7dLZb/iD0D/kmBIymSIarv0GXZinirQyCWzthn4CbYyZlBwCflcmFpIGL9lPWg6qpkC28oG9+d0yykd2o2NexrpQP05kTFlbV9FrlMxvLG/a4X4X62ZYveolQmdpAiafxt1U0kxpkUYtCMMcJR9Rxg3wt1K+Q0zjKOLbMQlUu4PGu55rBTTnSys5VlYGEZRVsuLvILf6fwljb1KsF85uNgvH58Mk5shG2STbJOAHJJjck6qpE44eSRP5Jm8eC/eq/fmvX+3jnnDmTUyAu/jC7h2qA0=</latexit>
<latexit sha1_base64="p2a7N1jnb6y1GFf6Cnxuu4MuukI=">AAACKHicbVDLSgNBEJz1bXzFx83LYBD0EnZDRI+iF71pMA/IhjA76SRjZmaXmVk1LvsvXvVrvEmu/ofgbJKDRhsaiqpuqruCiDNtXHfkzM0vLC4tr6zm1tY3Nrfy2zs1HcaKQpWGPFSNgGjgTELVMMOhESkgIuBQDwaXmV5/AKVZKO/MMIKWID3JuowSY6l2fs838GSSo9Ixvpb3QDM2becLbtEdF/4LvCkooGndtPNffieksQBpKCdaNz03Mq2EKMMohzTnxxoiQgekB00LJRGgW8n4+hQfWqaDu6GyLQ0esz83EiK0HorATgpi+npWy8j/tGZsumethMkoNiDpxKgbc2xCnEWBO0zZh/nQAkIVs7di2ieKUGMD++USCPuDhEcaCkFkJ/EraeJnhkGQVNIsL282nb+gVip65eLJbblwfjFNbgXtowN0hDx0is7RFbpBVUTRM3pBr+jNeXPenQ9nNBmdc6Y7u+hXOZ/fUuSnWQ==</latexit>
<latexit sha1_base64="j6OkZfZvhTuSocl1jD5dBiIQ2hQ=">AAACPnicbVDLThsxFPXwKoRHQ1l2YzVCQkhEM1UQXUbthqqbgBpAykSR7bkTTPwY2R5oNJp/4GvYlt/gB7qr2LJBwpNkwetKlo7OOVfH99BMcOvC8C6Ym19YXPqwvFJbXVvf+Fjf/HRidW4YdJkW2pxRYkFwBV3HnYCzzACRVMApHf2o9NNLMJZr9duNM+hLMlQ85Yw4Tw3qu7GDP46mRUfkwz2ikr1MkDH+pfSVgGQI+Ke6AFZ5y0G9ETbDyeC3IJqBBppNZ1B/jBPNcgnKMUGs7UVh5voFMY4zAWUtzi1khI3IEHoeKiLB9ovJTSXe9kyCU238Uw5P2OcbBZHWjiX1TkncuX2tVeR7Wi936bd+wVWWO1BsGpTmAjuNq4Jwwo0/WIw9IMxw/1fMzokhzPkaX6RQ6W9QcMW0lL65Ij4ui7gKpLQ4Lqu+otftvAUnX5tRq7l/1Gq0v8+aW0af0Re0gyJ0gNroEHVQFzF0jW7QX3Qb3Ab/gv/B/dQ6F8x2ttCLCR6eAIVFsRU=</latexit>
<latexit sha1_base64="27LKo39zMNGbwHFFuah0emPaqUg=">AAACI3icbVBNS8NAFNzU789WPXoJFsFTSUTRY9FLj1VsFZpSNttXXbq7Cbsvagn5JV7tr/EmXjz4UwQ3bQ62OrAwzJvHvJ0wFtyg5306pYXFpeWV1bX1jc2t7XJlZ7dtokQzaLFIRPoupAYEV9BCjgLuYg1UhgJuw+FlPr99BG14pG5wFENX0nvFB5xRtFKvUg4QnjFtUGnQ2rJeperVvAncv8QvSJUUaPYq30E/YokEhUxQYzq+F2M3pRo5E5CtB4mBmLIhvYeOpYpKMN10cnjmHlql7w4ibZ9Cd6L+3kjtWWYkQ+uUFB/M/CwX/5t1Ehycd1Ou4gRBsWnQIBEuRm7egtvnGhiKkSWUaW5vddkD1ZTlJcykhNL+QcETi6Skqp8G11ka5IFhmF5neV/+fDt/Sfu45p/UTq9OqvWLorlVsk8OyBHxyRmpkwZpkhZhJCEv5JWMnbHz5rw7H1NrySl29sgMnK8f0vCmKA==</latexit>
<latexit sha1_base64="IiQcAAYHuIDgHl1FpQD455bTZiw=">AAACHHicbVBNS8NAEN34WetX1aOXYBHqpSSi6FH04rEWW8WmlM122i7ubsLuRC0h/8Kr/TXexKvgjxHctD1odWDg8d4Mb+aFseAGPe/TmZtfWFxaLqwUV9fWNzZLW9tNEyWaQYNFItK3ITUguIIGchRwG2ugMhRwE95f5PrNA2jDI3WNwxjakvYV73FG0VJ3AcITphV6kHVKZa/qjcv9C/wpKJNp1Tqlr6AbsUSCQiaoMS3fi7GdUo2cCciKQWIgpuye9qFloaISTDsdX5y5+5bpur1I21bojtmfGymVxgxlaCclxYGZ1XLyP62VYO+0nXIVJwiKTYx6iXAxcvP33S7XwFAMLaBMc3urywZUU4Y2pF8uobQ/KHhkkZRUddOgnqVBbhiGaT3L8/Jn0/kLmodV/6h6fHVUPjufJlcgu2SPVIhPTsgZuSQ10iCMKPJMXsjIGTmvzpvzPhmdc6Y7O+RXOR/fKXSjMA==</latexit>
<latexit sha1_base64="roBzL38q0iLY5OiUBSulvSE7c0s=">AAACHHicbVDLSgNBEJz1bXxFPXpZDIJewq4oehS9eFQxD8wGmZl0dHBmdpnpVcOyf+FVv8abeBX8GMHZJAeT2NBQVHVT3cUSKSwGwbc3NT0zOze/sFhaWl5ZXSuvb9RtnBoONR7L2DQZtSCFhhoKlNBMDFDFJDTYw1mhNx7BWBHra+wl0Fb0Touu4BQddRMhPGO2y/by23IlqAb98idBOAQVMqyL2/JP1Il5qkAjl9TaVhgk2M6oQcEl5KUotZBQ/kDvoOWgpgpsO+tfnPs7jun43di41uj32b8bGVXW9hRzk4rivR3XCvI/rZVi97idCZ2kCJoPjLqp9DH2i/f9jjDAUfYcoNwId6vP76mhHF1IIy5MuR80PPFYKao7WXSVZ1FhyFh2lRd5hePpTIL6fjU8qB5eHlROTofJLZAtsk12SUiOyAk5JxekRjjR5IW8kjfvzXv3PrzPweiUN9zZJCPlff0CKyWjMQ==</latexit>
rectly used for general plug-and-play knowledge injection. We summarize the existing knowledge injection methods1 that have the possibility to be used for general plug-and-play knowledge injection as follows. (1) Embedding-based methods: E-BERT (Poerner et al., 2020) and PELT (Ye et al., 2022) build an entity embedding lookup table in the representation space of token embeddings and combine entity embeddings with token embeddings to construct input embeddings. (2) Retrieval-based methods: RAG (Lewis et al., 2020) retrieves plain text from knowledge bases and augments the original input text with the plain text as injected knowledge. (3) Adapter-based methods: K-Adapter (Wang et al., 2021a) computes knowledgeable representations based on the outputs of the downstream models accompanied by knowledgeable adapters, which are trained with frozen PLMs and plugged into all downstream models. Even though these methods may bring knowledge without training PLMs, it is unclear whether they work well in the plug-and-play knowledge injection paradigm, i.e., whether the knowledge brought by them is utilizable for downstream models that have never learned how to use these methods.

heading: 3 Map-Tuning
text: In this section, we first present the overall framework of map-tuning, which is designed for plugand-play knowledge injection. Then, we show how to use it for general injection and task-specific injection, where the methods are called general maptuning and task-specific map-tuning respectively.
1These methods are originally designed for injecting knowledge during pre-training or fine-tuning.

heading: 3.1 Overall Framework
text: We target knowledge bases consisting of a set of entities and structured or unstructured knowledge about these entities. To utilize such a knowledge base B, we assume a knowledge representation model K to assign each entity e an entity embedding e ∈ RdKE , where dKE is the dimension of entity embeddings. Map-tuning injects knowledge by mapping knowledge representations into the space of token embeddings and using the mapped representations as additional inputs, which is also adopted by Poerner et al. (2020); Ye et al. (2022).
Specifically, given an input text, we first match the entity mentions in the text with the entities in B. The input text is denoted by {w1, w2, . . . , wn}, where wi is the i-th token and n is the number of tokens in the input text. We use a triple (e, l, r) to represent a mention span, where e is the matched entity, l and r are the left and right token indices of the mention span. The corresponding mention span is {wl, wl+1, . . . , wr}. Assume there are m entities in the text, (e1, l1, r1), (e2, l2, r2), . . . , (em, lm, rm), where 1 ≤ l1 ≤ r1 < l2 ≤ r2 < · · · < lm ≤ rm ≤ n. The original sequence of input embeddings are {w1,w2, . . . ,wn}, where wi ∈ RdPLM is the i-th token embedding and dPLM is the dimension of token embeddings. Then, we map each entity embedding ei toM(ei) ∈ RdPLM by a mapping network M. Finally, we replace {wli ,wli+1, . . . ,wri} with {M(ei), /,wli , . . . ,wri} for every (ei, li, ri) to construct a new input sequence. Note that / is the token embedding of “/”.

heading: 3.2 General Map-tuning
text: General map-tuning aims to train a mapping networkM based on P and K. It requiresM to have
enough generalizability to handle different downstream tasks becauseM will be plugged into all downstream models. Hence, we train M with a general pre-training task while plugging it into P , such as language modeling, which has been shown to be an unsupervised multi-task learning (Radford et al., 2019). We freeze the parameters of P and only train the mapping network M to meet the requirement of plug-and-play knowledge injection.
We adopt a variant of Masked Language Model (MLM) (Devlin et al., 2019), named MentionMasked Language Modeling (MMLM), as the task for trainingM. According to our observation in the preliminary experiments, the prediction of most tokens requires only language ability instead of external knowledge, such as that of some stop words, while the prediction of entity mentions relies on external knowledge more often. Hence, as shown in Figure 2(b), we randomly mask only entity mentions2 in the input text to ensure that the mapping network is trained sufficiently and the mapped embeddings are well utilized in the PLM. In this way, the ability of PLMs to predict masked entity mentions is enhanced by the mapped embeddings of both the masked entity and other entities in the context. We mask all tokens of a masked entity mention, and the MMLM loss is the same as the original MLM loss (Devlin et al., 2019).
After general map-tuning,M can be used for the general plug-and-play injection. Although the mapping networkM was not trained with any downstream model D before, we can directly plugM into each D.

heading: 3.3 Task-specific Map-tuning
text: Task-specific map-tuning aims to adapt a mapping networkM for a given downstream model D. We freeze the parameters of D and train the mapping networkM on the downstream task, whose procedure is shown in Figure 2(b). The training objective is identical to the original objective of this task. If the knowledge representations provide useful information for this task, the mapping network will learn to extract this information and to make it recognizable to the downstream model D. Note that the mapping network can not only be trained from scratch, but can also be initialized with a mapping network learned with general map-tuning, which could provide a good starting point.
2If the number of entity mentions is small, we can choose to cover all the masking combinations.

heading: 4 Experiments

heading: 4.1 Experimental Setups
text: Training Methods of Downstream Models. We adopt BERTbase (Devlin et al., 2019) as the backbone PLM in the experiments and consider four training methods for its adaptation to downstream tasks. Besides vanilla full-model fine-tuning, we also consider three parameter-efficient tuning (PET) methods, which have been becoming increasingly important in the era of large-scale PLMs (Liu et al., 2021). As resource-saving are both plug-andplay knowledge injection and PET, it is meaningful to apply this paradigm to downstream models trained by PET methods in the resource-limited scenario. (1) Fine-tuning optimizes all the parameters of a PLM with the task objective following the original BERT. (2) LoRA (Hu et al., 2021) freezes the PLM parameters and injects trainable rank-decomposition matrices as additional parameters. (3) Adapter (Houlsby et al., 2019) injects additional adapter networks with the PLM parameters frozen. (4) BitFit (Zaken et al., 2021) only optimizes the parameters of bias vectors and freezes the rest parameters. The hyper-parameters are reported in Appendix A.
Downstream Tasks. We evaluate methods under the plug-and-play knowledge injection paradigm on three kinds of knowledge-driven NLP tasks including relation classification, entity typing, and question answering. For relation classification, which requires models to classify the relation between two entities given a context, we experiment on both few-shot and full-data settings. In the few-shot setting, we aim to evaluate model performance on long-tail relations whose training instances are not sufficient. Specifically, we use FewRel 1.0 (Han et al., 2018) and FewRel 2.0 (Gao et al., 2019).3 In the full-data setting, we evaluate models on Wiki80 (Han et al., 2019), which contains 80 relation types from Wikidata, and follow the data split of Zhang et al. (2019). For entity typing, which requires models to classify the type of an entity given a context, we evaluate models on Wiki-ET (Xin et al., 2018) containing 68 entity types from Freebase. For question answering, we evaluate models on EntityQuestions (Sciavolino
3We randomly sample 5000 instances from test data of FewRel 1.0 and FewRel 2.0 respectively for fast evaluation. Note that the test data is not publicly released and we get the data from the authors. We experiment with full test data of FewRel 1.0 on the official leaderboard in Section 4.4.
et al., 2021), an open-domain QA dataset consisting of entity-centric questions. We use knowledgeenhanced models to directly answer questions without retrieving related documents. We report accuracy on relation classification and question answering, and F1 score on entity typing.
Knowledge Bases. We use Wikidata5M (Wang et al., 2021b) and UMLS4 as our external knowledge bases for the Wikipedia domain and PubMed5 domain, respectively. To avoid information leakage in the relation classification task, we remove the triples appearing in the datasets from these knowledge bases. We adopt TransE (Bordes et al., 2013) as our knowledge representation model and the dimension of knowledge embeddings is set to 128.
Evaluated Existing Methods. We evaluate existing methods that can be applied to general plugand-play knowledge injection. (1) E-BERT (Poerner et al., 2020) also obtains a mapping network to transform knowledge embeddings. Different from map-tuning, E-BERT builds the connection between the vocabulary and entities by string matching, and then make the mapped knowledge embeddings close to their corresponding token embeddings. In this work, E-BERT uses the same TransE embeddings as map-tuning instead of wikipedia2vec for fair comparisons. (2) PELT (Ye et al., 2022) aggregates the output representations of a specific entity in multiple contexts to build the entity representation. Then, the entity representation can be appended to the model input without any mapping because the input space and output space are the same for most PLMs. The entity-related context can be treated as an external textual knowledge base. (3) Retrieval Augmentation (RA) is to augment input texts with additional retrieved unstructured knowledge, such as RAG (Lewis et al., 2020) and REALM (Guu et al., 2020). In this work, we retrieve the entity descriptions from Wikidata5M and append them to the input texts. (4) K-Adapter (Wang et al., 2021a) implicitly stores knowledge in the parameters of adapter networks. We follow the original procedure of K-Adapter while keeping the parameters of PLMs and adapters frozen.6
Details of Map-tuning. The architecture of the
4UMLS represents the Unified Medical Language System, which is the trademark of U.S. National Library of Medicine.
5https://pubmed.ncbi.nlm.nih.gov/ 6This procedure still requires the training of the final fully connected layer, which does not strictly meet the setting of general plug-and-play Injection.
mapping network is simply an affine transformation We + b, where W ∈ RdPLM×dKE and b ∈ RdPLM . In this work, the parameter amount of the mapping network is 768 × 128 + 768 < 0.1M. For Mention-Masked Language Modeling, we use the raw texts of Wiki20M (Gao et al., 2021), which is sampled from the Wikipedia corpus and provides the annotations of entity linking. The total size is around 300MB, much smaller than common pretraining corpora. Since map-tuning only aims to adapt the mapping network for a PLM, it does not require much training data. We train the mapping network for 5 epochs, which costs only 12 hours on an NVIDIA Tesla V100. General map-tuning essentially builds an entity embedding lookup table. To evaluate its quality, we evaluate it in the traditional injection during fine-tuning paradigm as a preliminary experiment. To be more specific, we fine-tune the PLMs on downstream tasks, during which the mapping network is plugged into them. The details are in Appendix E. We find that maptuning consistently outperforms E-BERT and PELT in the traditional paradigm, which also builds entity embedding lookup tables.

heading: 4.2 General Plug-and-Play Injection
text: In this subsection, we evaluate knowledge injection methods in the setting of general plug-and-play knowledge injection, where we directly plug knowledge modules into downstream models without any training. The results are reported in Table 1.
From this table, we have four observations: (1) All of the four existing methods can not consistently improve the performance of downstream models. In most cases, injecting these knowledge modules degrades the model performance, often to a large degree. It empirically proves that the setting of general plug-and-play injection is challenging and these four methods are not suitable in this setting. The knowledge provided by these methods can not be directly used, so they are basically disruptive noise to the downstream models. (2) Our proposed general map-tuning achieves consistent improvement on almost all downstream models, suggesting that the mapping network effectively transforms knowledge embeddings into the space of token embeddings and the mapped embeddings can be directly used by downstream models. We highlight the importance of Mention-Masked Language Modeling, which provides sufficient training instances for general map-
tuning, while the matched entity-token pairs for E-BERT are insufficient for training the mapping network. (3) Intuitively, general map-tuning may work better with PET methods than with full-model fine-tuning because PET methods change much fewer parameters from the PLM and general maptuning is trained based on the PLM. In fact, the performance improvement brought to models trained by full-model fine-tuning is comparable to that of PET methods. It demonstrates that map-tuning is a promising method regardless of the training methods of downstream models. (4) Remarkably high is the performance improvement brought by RA to fine-tuned BERT on EntityQuestions. We observe that the retrieved entity description contains the exact answer as a substring for 62.19% of instances in the test set, and we remove these instances and report the result in Table 16. We find that RA still gets a slightly higher performance than map-tuning does for fine-tuned BERT, but brings a significant performance drop to other downstream models, while map-tuning brings consistent performance improvement to all downstream models. It suggests that fine-tuned BERT has the surprising generalization ability to extract a substring in the additional context as the answer, and even to reveal the answer hidden in the additional context without string matches. On the contrary, other downstream
models are not able to reveal the hidden answer. Thus, it is worth investigating RA with pluggable knowledge modules to stably provide information for different downstream models, rather than directly appending unstructured text to model inputs.

heading: 4.3 Task-specific Plug-and-Play Injection
text: Since map-tuning achieves the best performance in the general plug-and-play injection setting, we further evaluate it in the setting of task-specific plug-and-play injection, where we train mapping networks based on downstream models with task objectives. If we have already conducted general map-tuning on a PLM, we can initialize the network with the general mapping network. Otherwise, we have to train the network from scratch.
We first evaluate task-specific map-tuning on Wiki80 and Wiki-ET. The results are reported in Table 2. From the table, we have two observa-
tions: (1) Task-specific map-tuning achieves better performance on these two datasets than general map-tuning does. It indicates that the mapping network extracts more informative knowledge for the specific task by task-specific training than the general one does. (2) If the general mapping network is available, it is recommended to use it to initialize the mapping network, which further improves the model performance.
Then, we evaluate task-specific map-tuning in domain adaptation, which is a more challenging setting. In this setting, we aim to plug multiple knowledge bases into a single downstream model. Specifically, a downstream model is trained on a source domain, and then we plug the knowledge modules of the target domain into it for domain adaptation. Here, we use the relation classification datasets on the Wikipedia domain (FewRel 1.0) and the PubMed domain (FewRel 2.0). FewRel 1.0 is the source domain. FewRel 2.0 is the target domain. The knowledge base for FewRel 2.0 is UMLS. Since the original FewRel 2.0 does not provide training instances, we rearrange FewRel 2.0 and have the following data split. As FewRel 2.0 has 25 relations, we separate 15 relations for training and development and the rest 10 relations are used for testing.
From Table 3, we have two observations: (1) For the domain adaptation from Wikipedia to PubMed, map-tuning significantly improves the model performance (e.g., from 76.7 to 81.2 in 5-1) and achieves better performance than the model finetuned on PubMed domain (e.g., from 78.6 to 81.2 in 5-1). It suggests that it is promising to use maptuning to introduce external knowledge for domain adaptation. (2) Multi-domain training degrades the model performance on the Wikipedia domain and maintains its performance on the PubMed domain while map-tuning does not degrade the performance on each domain. It indicates that the pluggable mapping networks are suitable for continual domain adaptation.

heading: 4.4 Computational Efficiency
text: We compare our proposed plug-and-play knowledge injection paradigm with previous knowledge injection paradigms on the time cost. We evaluate the training time on an NVIDIA Tesla V100 and compare the model performance on the 10- way 1-shot setting of FewRel 1.0. ERNIE (Zhang et al., 2019), KEPLER (Wang et al., 2021b), and
LUKE (Yamada et al., 2020) inject knowledge during pre-training. PELT (Ye et al., 2022) injects knowledge during fine-tuning. The results of ERNIE, KEPLER, LUKE, and PELT are taken from Ye et al. (2022). Map-tuning injects knowledge after fine-tuning.
The results are shown in Figure 3. From this figure, we observe that the training time of maptuning is much shorter than those methods under the paradigm of injecting during pre-training, and it is comparable to PELT. Besides, the performance of map-tuning is also competitive compared to previous knowledge injection methods. Moreover, map-tuning only optimizes additional 0.1% of parameters and we report the number of parameters optimized for different knowledge injection methods in Appendix G. Plug-and-play knowledge injection has great potential to be comparable to previous paradigms w.r.t. task performance, while maintaining its innate flexibility and efficiency.

heading: 4.5 Case Study
text: We present a qualitative analysis of map-tuning in Table 4. In the first case, the original downstream model does not understand that “flying officer” is a military rank and wrongly predicts the relation as “occupation”. With the general mapping network, which enriches the meaning of “flying officer”, the model correctly predicts the relation.
The general mapping network, however, may be misleading in some cases. In the second case, it is easy for the original downstream model to recognize “Wasp” as a member of “Avengers” without any external knowledge since this fact could be inferred by the word “other”. Compared to the external knowledge provided by the task-specific mapping network, coarse-grained is that provided by the general mapping network, because there is no additional training before the inference. As a result, the model wrongly recognizes “Avengers” as comic books instead of the fictional superhero
team, and thus changes the correct model prediction. Task-specific map-tuning, which is further adapted to the task, corrects the prediction.

heading: 5 Related Work
text: To enhance PLMs with external knowledge, there are two mainstream paradigms: injection during pre-training and injection during fine-tuning (Yin et al., 2022). For injection during pre-training, researchers usually construct new knowledge-aware objectives, such as entity prediction (Xu et al., 2021), entity discrimination (Xiong et al., 2020), entity and relation discrimination (Qin et al., 2021), and link prediction (Wang et al., 2021b). In this way, knowledge will be implicitly stored in the parameters of PLMs. Injection knowledge during pretraining can simultaneously improve performance on a range of downstream knowledge-driven tasks. However, the training cost of this paradigm is expensive. Taking the typical knowledge-enhanced PLMs LUKE (Yamada et al., 2020) and KEPLER (Wang et al., 2021b) as an example, it takes more than 3,000 GPU hours to train them.
Injection knowledge during fine-tuning is a relatively lightweight paradigm, where external knowledge is often used to augment model inputs for specific tasks (Zhou et al., 2019; Lin et al., 2019; Liu et al., 2020b; Cheng et al., 2021; Kang et al.,
2022). When injecting unstructured textual knowledge, some methods retrieve task-related information from external corpora to augment the original input text (Karpukhin et al., 2020; Liu et al., 2020a). When using structured knowledge, such as knowledge graphs, existing methods usually apply knowledge representation learning methods (Bordes et al., 2013; Lin et al., 2015) to encode structured knowledge into embeddings, and then fuse these knowledge embeddings with input token embeddings using knowledge injection methods (Sun et al., 2020; Su et al., 2021; Yasunaga et al., 2021).
In general, existing knowledge injection methods mainly target PLMs and adopt paradigms where knowledge and models are highly coupled. Toward flexible and efficient injection, we study a new paradigm, plug-and-play knowledge injection, where we decouple models and knowledge sources, and then inject knowledge into downstream models without retraining the models. This work is also related to parameter-efficient tuning (Liu et al., 2021; Ding et al., 2022) and plugins for large language models (Xiao et al., 2023; Dathathri et al., 2020; Lauscher et al., 2021; Chronopoulou et al., 2022; Yu et al., 2023; Xu et al., 2023; Alayrac et al., 2022) while we are the first to study knowledge injection in a parameter-efficient and plug-and-play way.

heading: 6 Conclusion
text: In this work, we propose a new paradigm of injection toward flexible and efficient knowledge injection. In this paradigm, downstream models can be enhanced with little computational cost, which benefits large amounts of models. We first systematically evaluate existing knowledge injection methods and find that they are not suitable for plugand-play injection. Then, we propose map-tuning for this paradigm, which effectively injects knowledge into downstream models to enhance them.
There are four promising directions for future investigation into plug-and-play knowledge injection. (1) How can we reduce the performance gap between methods for this novel paradigm and those for the previous injection paradigms, while maintaining superior flexibility and efficiency? (2) Besides factual knowledge, how can we effectively plug diverse knowledge bases, such as text corpora, voice, images, and even other PLMs? (3) After injecting the knowledge in a plug-and-play way, how can the PLMs do various types of complex reasoning based on the injected knowledge (Onoe et al., 2023)? (4) Can the plug-and-play knowledge injection methods for these sources be unified, so we can plug a combination of multiple sources? We hope this work can attract attention to and inspire research on these problems.
Limitations
In this paper, we present a novel knowledge injection paradigm plug-and-play knowledge injection for PLMs. We show existing methods can not be well applied to the new paradigm and propose maptuning as a preliminary exploration of methods.
The paradigm plug-and-play knowledge injection has a limitation in terms of its assumption. It assumes that a PLM should be fine-tuned for downstream tasks. However, very large-scale PLMs can perform zero-shot learning or in-context learning on downstream tasks without being fine-tuned. Future work may extend the definition of the proposed paradigm to make it meaningful in these scenes.
The method map-tuning has three limitations in terms of its applicability. Firstly, we did not evaluate map-tuning for PLMs pre-trained by other language modeling objectives (e.g., casual language modeling) besides MLM. As its spirit can be easily generalized to various language modeling objectives, we leave this evaluation as future work. Secondly, we did not evaluate whether the PLM can
do complex reasoning (e.g., multi-hop reasoning) based on the knowledge injected by map-tuning. Thirdly, map-tuning is designed to plug structural fact knowledge. It is also meaningful to plug other diverse knowledge bases, including text corpora, voice, images, and even other PLMs, which are not covered by our work.

heading: Acknowledgments
text: This work is supported by the National Key R&D Program of China (No.2022ZD0116312), National Natural Science Foundation of China (No. 62236004).
Author Contributions Zhengyan Zhang, Zhiyuan Zeng, Huadong Wang, and Deming Ye wrote the code and conducted the experiments. Zhengyan Zhang constructed the basic experimental framework including codes and datasets. Zhiyuan Zeng was in charge of plug-and-play and fine-tuning experiments. Huadong Wang and Deming Ye provided TransE and PELT embeddings respectively. Zhengyan Zhang and Zhiyuan Zeng contributed to the analysis experiments. Zhengyan Zhang and Zhiyuan Zeng wrote the initial draft. Yankai Lin, Huadong Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu significantly edited and improved the paper. Peng Li, Maosong Sun, and Jie Zhou provided valuable advice to the research.

heading: A Hyper-parameters
text: A.1 Fine-tuning downstream PLMs
We experiment with four training methods for the adaptation of PLMs on downstream tasks, which are Full-model fine-tuning, LoRA, Adapter, and BitFit. The embedding layer is frozen during training. We train all the models using AdamW with 10% warming-up steps. We list our hyperparameters in Table 5.
A.2 General Map-tuning
For general map-tuning, we search the dropout rate in {0.15, 0.25, 0.35, 0.45}. We train all the mapping networks using Adam (Kingma and Ba, 2015). The learning rate is 3E-5 and the batch size is 64. We train the mapping network on the Wikipedia corpus for 5 epochs. The hyper-parameters of the best mapping network in all cases are listed in Table 6. When we evaluate RA on these datasets, we set the sequence length to 512.
A.3 Task-specifc Map-tuning
We report hyper-parameters for task-specific maptuning in Table 7. We train all mapping networks using Adam with 10% warming-up steps
Regarding the results reported in Table 2, during task-specific map-tuning, we use dropout in the attention probabilities and all fully connected layers
of the PLM. The dropout rate is 0.30, 0.20, and 0.00 for Wiki80, Wiki-ET, and EntityQuestions, respectively. Regarding the results reported in Table 3, when using training data from the source domain for task-specific map-tuning, the dropout rate is 0.35. In these cases, the training data for task-specific map-tuning are identical to those for fine-tuning the downstream models. We search the dropout rate in {0.00, 0.15, 0.20, 0.25, 0.30, 0.35}. When using training data from the target domain for task-specific map-tuning, we do not use dropout.
The hyper-parameters for experiments with RoBERTa are identical to those with BERT.
A.4 Fine-tuning with the Mapping Network
Regarding the results reported in Table 14, the hyper-parameters for fine-tuning BERT are identical to those in Table 5. We train all mapping networks using Adam without dropout, and the batch size is 64. For map-tuning on the Wikipedia corpus, the learning rate is 1E-5. We report other hyper-parameters for map-tuning on the Wikipedia corpus in Table 8, and those for map-tuning on downstream data in Table 9.
A.5 Details of K-Adapter
We use the open-source implementation of KAdapter7, and we only consider facAdapter (Factual Adapter). The BERTbase layers where adapter layers plug in are {5, 10}. The hyper-parameters for pre-training facAdapter are identical to those reported in Wang et al. (2021a).
In order to plug K-Adapter into frozen downstream models in the setting of general plug-andplay injection, we tune the final fully connected layer on downstream data. We use Adam with 10%
7https://github.com/microsoft/k-adapter
warming-up steps, and other hyper-parameters are listed in Table 10.
A.6 Details of Data Preprocessing
For FewRel and Wiki80, we mark the subject and object spans by # and $ tokens respectively. For WikiET and EntityQuestions, we mark the entity span by $ token.
To evaluate encoder PLMs on EntityQuestions, we append the [MASK] token to the question, and only keep the instances whose answers are in the PLM token vocabulary. We train the model to fill in the [MASK] token. It is a classification task, where all tokens in the vocabulary are choices. Only when the answer token is ranked as the top 1 result is the model considered to give a correct prediction. We further remove the instances whose entity is not in the database. Finally, we have 37800 training instances, 4693 validation instances, and 4731 test instances.
FewRel, Wiki80, and WikiET provide the annotation of entity linking, and for EntityQuestions we do entity liking by string matching.

heading: B Stability of Map-tuning
text: We evaluate the stability of map-tuning in general plug-and-play knowledge injection. Training the PLMs on downstream tasks with three different seeds (one of which is used in all main experiments), for each task, we have three different downstream models, into which we plug the mapping network. The mean and standard deviation of performance improvement brought by map-tuning is shown in Table 11. From this table, we observe that map-tuning is not sensitive to downstream models overall, showing its decent stability.

heading: C How Map-tuning Works with Other PLMs?
text: In this section, we experiment map-tuning with RoBERTa (Liu et al., 2019), another representative PLM, on the domain transfer setting using taskspecific map-tuning. The setting is identical to that in Section 4.3. The results are shown in Table 12. From this table, we observe that task-specific maptuning significantly improves the performance of the model trained on the source domain by introducing the knowledge of the target domain. Moreover, the model plugged with map-tuning is even much better than the model trained on multiple domains. It indicates that map-tuning is a universal knowledge injection method for different PLMs.

heading: D Empirical Analysis of MMLM
text: We conduct an empirical analysis of what MMLM trains the mapping network to learn. Concretely, we split the general map-tuning corpus into a training set and a test set. During training on the training set, we plugM(e1) andM(e2) before two entity mentions e1 and e2 for each instance, and mask only the mention span of e1. During inference on the test set, we evaluate the MMLM loss in four settings. (1) No-Perturbation plugs the M(e1) and M(e2), which is identical to the setting of training. (2) Self-Perturbation replaces M(e1) with M(ei), where ei is a random entity. (3) Other-Perturbation replacesM(e2) withM(ei). (4) All-Perturbation replaces both M(e1) and M(e2) with random ones. We also evaluate these
settings with a randomly-initialized mapping network without map-tuning. For analysis, we report the result in the setting No-Plug where there is no plugged embedding.
The result is shown in Table 13. From this table, we have three observations. (1) With maptuning, the loss in Self-Perturbation is significantly larger than that in No-Perturbation, even close to that in All-Perturbation. It proves that MMLM trains the mapping network to extract the entity information stored in the knowledge embedding so that PLMs can utilize the information. (2) The loss in Other-Perturbation is also larger than that in No-Perturbation, which indicates that the mapping network learns to extract the connections between different entities and to feed such information into PLMs. (3) Interestingly, the loss in AllPerturbation with map-tuning is smaller than that in No-Plug, and the loss in settings without maptuning is close to the latter. The trained mapping network may be able to convert an arbitrary knowledge embedding to an embedding that can activate the PLM’s own memory of some factual knowledge. In conclusion, the three mentioned abilities of mapping networks trained by MMLM enable PLMs to know new knowledge or better recall their own knowledge. Future work may improve MMLM to get stronger mapping networks.
E Is Map-tuning Competitive in the Traditional Paradigm?
It is natural to use the general mapping network in the traditional injection during fine-tuning paradigm, as the general network essentially builds an entity embedding lookup table. We freeze the parameters of the mapping network and fine-tune the PLM on downstream tasks, during which we augment model inputs with mapped knowledge representations. Intuitively, the models learn to effectively extract information from mapped knowledge representations during fine-tuning. Inspired by ULMFiT (Howard and Ruder, 2018), we also experiment on the setting where we use the task’s training data as the corpus for general map-tuning. Our results are shown in Table 14.
From this table, we have two observations: (1) map-tuning consistently outperforms E-BERT and PELT in the traditional paradigm. Considering that E-BERT and map-tuning use the same knowledge embedding, we suggest that map-tuning provides more useful knowledge representations for BERT than E-BERT. (2) General map-tuning on downstream data achieves comparable performance to that on the large-scale unsupervised corpus. It indicates that general map-tuning does not necessitate a large amount of training data for a specific task.
F How do We Ensure the Generality of Map-tuning?
In the setting of general plug-and-play injection, we train a general mapping network based on a PLM and directly plug it into various downstream models during inference. There exists a gap between the general map-tuning procedure and the inference on downstream tasks, i.e., the PLM used for maptuning is different from downstream models. To reduce this gap, we use dropout (Hinton et al., 2012) in the attention probabilities and all fully connected layers of the PLM during general map-tuning. In-
tuitively, dropout simulates different variants of the PLM and makes the mapping network have better generality for different downstream models trained from the PLM. We explore five different dropout rates. The results on the 5-way 1-shot of FewRel 1.0 are chosen as the representative and shown in Figure 4.
From this figure, we have two observations: (1) Training without dropout leads to the worst performance, which indicates that the generality of the mapping network is not good enough and downstream models can not utilize the knowledge. (2) Large dropout rates are also not optimal. Empirically, the dropout rate of 0.25 is a good choice.

heading: G Numbers of Optimized Parameters
text: Compared to previous knowledge injection methods, map-tuning is a parameter-efficient method. The numbers of optimized parameters for different knowledge injection methods are shown in Table 15. In order to introduce external knowl-
edge, previous methods usually optimize all parameters during pre-training and fine-tuning while map-tuning only optimizes additional 0.1% of parameters and freezes the original model, which makes it flexible to use mapping networks for different inputs with the same models.

heading: H Performance on EntityQuestions
text: We report the performance on filtered EntityQuestions in Table 16.

heading: Future Work
text: Future work may improve MMLM to get stronger mapping networks. E Is Map-tuning Competitive in the Traditional Paradigm?
It is natural to use the general mapping network in the traditional injection during fine-tuning paradigm, as the general network essentially builds an entity embedding lookup table. We freeze the parameters of the mapping network and fine-tune the PLM on downstream tasks, during which we augment model inputs with mapped knowledge representations. Intuitively, the models learn to effectively extract information from mapped knowledge representations during fine-tuning. Inspired by ULMFiT (Howard and Ruder, 2018), we also experiment on the setting where we use the task’s training data as the corpus for general map-tuning. Our results are shown in Table 14. From this table, we have two observations: (1) map-tuning consistently outperforms E-BERT and PELT in the traditional paradigm. Considering that E-BERT and map-tuning use the same knowledge embedding, we suggest that map-tuning provides more useful knowledge representations for BERT than E-BERT. (2) General map-tuning on downstream data achieves comparable performance to that on the large-scale unsupervised corpus. It indicates that general map-tuning does not necessitate a large amount of training data for a specific task. F How do We Ensure the Generality of Map-tuning?
In the setting of general plug-and-play injection, we train a general mapping network based on a PLM and directly plug it into various downstream models during inference. There exists a gap between the general map-tuning procedure and the inference on downstream tasks, i. e. , the PLM used for maptuning is different from downstream models. To reduce this gap, we use dropout (Hinton et al. , 2012) in the attention probabilities and all fully connected layers of the PLM during general map-tuning. In-
tuitively, dropout simulates different variants of the PLM and makes the mapping network have better generality for different downstream models trained from the PLM. We explore five different dropout rates. The results on the 5-way 1-shot of FewRel 1. 0 are chosen as the representative and shown in Figure 4. From this figure, we have two observations: (1) Training without dropout leads to the worst performance, which indicates that the generality of the mapping network is not good enough and downstream models can not utilize the knowledge. (2) Large dropout rates are also not optimal. Empirically, the dropout rate of 0. 25 is a good choice