INDUS: Effective and Efficient Language Models for Scientific
  Applications
Authors: name: Bishwaranjan Bhattacharjee, name: Aashka Trivedi, name: Masayasu Muraoka, name: Muthukumaran Ramasubramanian, name: Takuma Udagawa, name: Iksha Gurung, name: Rong Zhang, name: Bharath Dandala, name: Rahul Ramachandran, name: Manil Maskey, name: Kayleen Bugbee, name: Mike Little, name: Elizabeth Fancher, name: Lauren Sanders, name: Sylvain Costes, name: Sergi Blanco-Cuaresma, name: Kelly Lockhart, name: Thomas Allen, name: Felix Grazes, name: Megan Ansdel, name: Alberto Accomazzi, name: Yousef El-Kurdi, name: Davis Wertheimer, name: Birgit Pfitzmann, name: Cesar Berrospi Ramis, name: Michele Dolfi, name: Rafael Teixeira de Lima, name: Panos Vegenas, name: S. Karthik Mukkavilli, name: Peter Staar, name: Sanaz Vahidinia, name: Ryan McGranaghan, name: Armin Mehrabian, name: Tsendgar Lee

Large language models (LLMs) trained on general domain corpora showed
remarkable results on natural language processing (NLP) tasks. However,
previous research demonstrated LLMs trained using domain-focused corpora
perform better on specialized tasks. Inspired by this pivotal insight, we
developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
biology, physics, heliophysics, planetary sciences and astrophysics domains and
trained using curated scientific corpora drawn from diverse data sources. The
suite of models include: (1) an encoder model trained using domain-specific
vocabulary and corpora to address natural language understanding tasks, (2) a
contrastive-learning-based general text embedding model trained using a diverse
set of datasets drawn from multiple sources to address information retrieval
tasks and (3) smaller versions of these models created using knowledge
distillation techniques to address applications which have latency or resource
constraints. We also created three new scientific benchmark datasets namely,
CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
(IR) to accelerate research in these multi-disciplinary fields. Finally, we
show that our models outperform both general-purpose encoders (RoBERTa) and
existing domain-specific encoders (SciBERT) on these new tasks as well as
existing benchmark tasks in the domains of interest.