Codebert with injection of vocabulary
CodeBert with injected tools Bioinformatics vocabulary.
The vocabulary added in the pretrained language model is a list of bioinformatics tool names and binary commands extracted from different Bioinformatics tools Knowledges Bases :
- Bio.tools
J. Ison, K. Rapacki, H. Ménager, M. Kalas, E. Rydza, P. Chmura, C. Anthon, N. Beard, K. Berka, D. Bolser, T. Booth, A. Bretaudeau, J. Brezovsky, R. Casadio, G. Cesareni, F. Coppens, M. Cornell, G. Cuccuru, and et al. Tools and data services registry: a community effort to document bioinformatics resources. Nucleic Acids Research,44(D1):D38–D47, Jan. 2016. ISSN 0305-1048. doi: 10.1093/nar/gkv1116. URL https://doi.org/10.1093/nar/gkv1116.
- Bioconda
B. Grüning, R. Dale, A. Sjödin, B. A. Chapman, J. Rowe, C. H. Tomkins-Tinch, R. Valieris, and J. Köster. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods, 15 (7):475–476, July 2018. ISSN 1548-7105. doi: 10.1038/s41592-018-0046-7. URL https://www.nature.com/articles/s41592-018-0046-7.
- Biocontainers
F. da Veiga Leprevost, B. A. Grüning, S. Alves Aflitos, H. L. Röst, J. Uszkoreit, H. Barsnes, M. Vaudel, and et al. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics, 33 (16):2580–2582, Aug. 2017. ISSN 1367-4803. doi: 10.1093/bioinformatics/btx192. URL https://doi.org/10.1093/bioinformatics/btx192.
- and Bioweb (Knowledge base manually curated at Institut Pasteur, France)
Vocabulary Injection Strategy
Tokenizer Update: new words are added to the existing tokenizer of CodeBert
Embedding Initialization: embeddings for newly introduced words are initialized as the average of the embeddings of their constituent existing words
Model Adaptation
After extending the vocabulary, two main strategies can be applied when performing NER with this model:
Option 1: Fine-Tuning (Recommended): the language model is fine-tuned on domain-specific corpora to better integrate the new vocabulary into its internal representations
Option 2: No Fine-Tuning: the extended model is used directly without further training
Use in
This model is used to do NER experiments on CPL-Article and CPL-Code.
This model can be used with :
- https://github.com/ClemenceS/nlstruct
- Other experiments are presented in https://gitlab.liris.cnrs.fr/sharefair/copalink-experiments
- Downloads last month
- 216