ChipNeMo: Domain-Adapted LLMs for Chip Design
Authors: name: Mingjie Liu, name: Teo Ene, name: Robert Kirby, name: Chris Cheng, name: Nathaniel Pinckney, name: Rongjian Liang, name: Jonah Alben, name: Himyanshu Anand, name: Sanmitra Banerjee, name: Ismet Bayraktaroglu, name: Bonita Bhaskaran, name: Bryan Catanzaro, name: Arjun Chaudhuri, name: Sharon Clay, name: Bill Dally, name: Laura Dang, name: Parikshit Deshpande, name: Siddhanth Dhodhi, name: Sameer Halepete, name: Eric Hill, name: Jiashang Hu, name: Sumit Jain, name: Brucek Khailany, name: Kishor Kunal, name: Xiaowei Li, name: Hao Liu, name: Stuart Oberman, name: Sujeet Omar, name: Sreedhar Pratty, name: Ambar Sarkar, name: Zhengjiang Shao, name: Hanfei Sun, name: Pratik P Suthar, name: Varun Tej, name: Kaizhe Xu, name: Haoxing Ren

ChipNeMo aims to explore the applications of large language models (LLMs) for
industrial chip design. Instead of directly deploying off-the-shelf commercial
or open-source LLMs, we instead adopt the following domain adaptation
techniques: custom tokenizers, domain-adaptive continued pretraining,
supervised fine-tuning (SFT) with domain-specific instructions, and
domain-adapted retrieval models. We evaluate these methods on three selected
LLM applications for chip design: an engineering assistant chatbot, EDA script
generation, and bug summarization and analysis. Our results show that these
domain adaptation techniques enable significant LLM performance improvements
over general-purpose base models across the three evaluated applications,
enabling up to 5x model size reduction with similar or better performance on a
range of design tasks. Our findings also indicate that there's still room for
improvement between our current results and ideal outcomes. We believe that
further investigation of domain-adapted LLM approaches will help close this gap
in the future.