ChipNeMo: Domain-Adapted LLMs for Chip Design
Authors: name: Mingjie Liu; name: Teo Ene; name: Robert Kirby; name: Chris Cheng; name: Nathaniel Pinckney; name: Rongjian Liang; name: Jonah Alben; name: Himyanshu Anand; name: Sanmitra Banerjee; name: Ismet Bayraktaroglu; name: Bonita Bhaskaran; name: Bryan Catanzaro; name: Arjun Chaudhuri; name: Sharon Clay; name: Bill Dally; name: Laura Dang; name: Parikshit Deshpande; name: Siddhanth Dhodhi; name: Sameer Halepete; name: Eric Hill; name: Jiashang Hu; name: Sumit Jain; name: Brucek Khailany; name: Kishor Kunal; name: Xiaowei Li; name: Hao Liu; name: Stuart Oberman; name: Sujeet Omar; name: Sreedhar Pratty; name: Ambar Sarkar; name: Zhengjiang Shao; name: Hanfei Sun; name: Pratik P Suthar; name: Varun Tej; name: Kaizhe Xu; name: Haoxing Ren
Published: 2023-10-31 22:35:58

Abstract: ChipNeMo aims to explore the applications of large language models (LLMs) for
industrial chip design. Instead of directly deploying off-the-shelf commercial
or open-source LLMs, we instead adopt the following domain adaptation
techniques: custom tokenizers, domain-adaptive continued pretraining,
supervised fine-tuning (SFT) with domain-specific instructions, and
domain-adapted retrieval models. We evaluate these methods on three selected
LLM applications for chip design: an engineering assistant chatbot, EDA script
generation, and bug summarization and analysis. Our results show that these
domain adaptation techniques enable significant LLM performance improvements
over general-purpose base models across the three evaluated applications,
enabling up to 5x model size reduction with similar or better performance on a
range of design tasks. Our findings also indicate that there's still room for
improvement between our current results and ideal outcomes. We believe that
further investigation of domain-adapted LLM approaches will help close this gap
in the future.