Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Authors: name: NVIDIA, name: Aaron Blakeman, name: Aaron Thomas, name: Aastha Jhunjhunwala, name: Abhibha Gupta, name: Abhinav Khattar, name: Adam Rajfer, name: Adi Renduchintala, name: Adil Asif, name: Aditya Vavre, name: Adriana Flores Miranda, name: Ahmad Bilal, name: Aileen Zaman, name: Ajay Hotchandani, name: Akanksha Shukla, name: Akhiad Bercovich, name: Aleksander Ficek, name: Alex Gronskiy, name: Alex Kondratenko, name: Alex Steiner, name: Alex Ye, name: Alexander Bukharin, name: Alexandre Milesi, name: Ali Taghibakhshi, name: Alice Gatti, name: Alisa Liu, name: Alok Kumar, name: Amar Phanishayee, name: Ameya Sunil Mahabaleshwarkar, name: Amir Klein, name: Amit Zuker, name: Amnon Geifman, name: Anahita Bhiwandiwalla, name: Ananth Subramaniam, name: Andrea Santilli, name: Andrew Fulks, name: Andrew McHarg, name: Andrew Tao, name: Andrii Skliar, name: Anjulie Agrusa, name: Ankur Srivastava, name: Ankur Verma, name: Anna Shors, name: Anna Warno, name: Antoni-Joan Solergibert I Llaquet, name: Arham Mehta, name: Arkadiusz Nowaczynski, name: Arti Jain, name: Ashwath Aithal, name: Ashwin Poojary, name: Asif Ahamed, name: Asit Mishra, name: Asma Kuriparambil Thekkumpate, name: Atefeh Sohrabizadeh, name: Avinash Kaur, name: Avinash Vem, name: Ayush Dattagupta, name: Barath Subramaniam Anandan, name: Bardiya Sadeghi, name: Ben Lanir, name: Benedikt Schifferer, name: Besmira Nushi, name: Bilal Kartal, name: Bill Thiede, name: Bita Darvish Rouhani, name: Bo Deng, name: Bob Schatz, name: Boris Ginsburg, name: Boxin Wang, name: Brad Nemire, name: Brandon Norick, name: Brian Dang, name: Brian Westphal, name: Brian Yu, name: Brucek Khailany, name: Bryan Catanzaro, name: Carlo del Mundo, name: Caryln Aarish, name: Chankyu Lee, name: Chantal Hwang, name: Charbel Sakr, name: Charles Wang, name: Charlie Truong, name: Chen Cui, name: Cheng Cheng, name: Cheng-Ping Hsieh, name: Chenghao Zhang, name: Chenhui Deng, name: Chintan Patel, name: Chris Alexiuk, name: Christian Cosgrove, name: Christian Munley, name: Christine Harvey, name: Christopher Parisien, name: Chunyang Shen, name: Coco Li, name: Collin Neale, name: Cynthia Gao, name: Cyril Meurillon, name: Dan Gil, name: Dan Su, name: Dan Zhao, name: Dane Corneil, name: Daniel Afrimi, name: Daniel Egert, name: Daniel Korzekwa, name: Daniel Lo, name: Daniel Machlab, name: Daniel Serebrenik, name: Daniil Sorokin, name: Daria Gitman, name: Daria Levy, name: Darko Stosic, name: David Mosallanezhad, name: David Yu, name: Davit Karamyan, name: Deena Donia, name: Deep Debroy, name: Deepak Narayanan, name: Devin O'Kelly, name: Dheeraj Peri, name: Dhruv Nathawani, name: Di, name: Wu, name: Dima Rekesh, name: Divyanshu Kakwani, name: Donald Plummer, name: Dong Anh, name: Dongfeng Yu, name: Dongfu Jiang, name: Donnie Kim, name: Dorrin Poorkay, name: Duncan Riach, name: Dusan Stosic, name: Dustin VanStee, name: Eavan Meng, name: Edgar Minasyan, name: Edward Lin, name: Eileen Margaret Peters Long, name: Elad Sarafin, name: Elad Segal, name: Elena Lantz, name: Ellie Evans, name: Elliott Ning, name: Eric Chung, name: Eric Harper, name: Eric Pham-Hung, name: Eric Tramel, name: Eric Yang, name: Erick Galinkin, name: Erik Pounds, name: Erika Goncalves Goncalves, name: Evan Briones, name: Evan Wu, name: Evelina Bakhturina, name: Evgeny Tsykunov, name: Ewa Dobrowolska, name: Faisal Ladhak, name: Farzan Memarian, name: Fay Wang, name: Fei Jia, name: Felipe Soares, name: Felipe Vieira Frujeri, name: Feng Chen, name: Fengguang Lin, name: Ferenc Galko, name: Frank Sun, name: Frankie Siino, name: Frida Hou, name: Gal Hubara Agam, name: Gal Kaplun, name: Gantavya Bhatt, name: Gargi Prasad, name: Garvit Kulshreshtha, name: George Armstrong, name: Gerald Shen, name: Giulio Borghesi, name: Gordana Neskovic, name: Gorkem Batmaz, name: Grace Lam, name: Greg Mason, name: Greg Pauloski, name: Grigor Nalbandyan, name: Grzegorz Chlebus, name: Grzegorz Karch, name: Guan-Ting Liu, name: Guoming Zhang, name: Guyue Huang, name: Haggai Maron, name: Haifeng Qian, name: Haim Elisha, name: Haoxing Ren, name: Haran Kumar Shiv Kumar, name: Haribhau Hud, name: Harris Nover, name: Harrison Saturley Hall, name: Hayate Iso, name: Helen Ngo, name: Herbert Hum, name: Herman Sahota, name: Hexin Wang, name: Himanshu Soni, name: Hovhannes Tamoyan, name: Hua Li, name: Huanhuan Chen, name: Hui Li, name: Hui Wang, name: Huy Nguyen, name: Ian Chiles, name: Ido Galil, name: Ido Shahaf, name: Igor Gitman, name: Igor Shovkun, name: Ilya Loshchilov, name: Ingo Guehring, name: Itamar Schen, name: Itay Levy, name: Itay Neeman, name: Ivan Moshkov, name: Izik Golan, name: Izzy Putterman, name: Jaemin Choi, name: Jakub Slowikowski, name: Jan Kautz, name: Jane Polak Scowcroft, name: Jared Casper, name: Jatin Mitra, name: Jeffrey Glick, name: Jenny Chen, name: Jesse Oliver, name: Jiacheng Xu, name: Jiafan Zhu, name: Jialin Song, name: Jian Zhang, name: Jiantao Jiao, name: Jiaqi Zeng, name: Jie Lou, name: Jim King, name: Jimmy Zhang, name: Jingquan Wang, name: Jinhang Choi, name: Jinju Chu, name: Joey Conway, name: Joey Guman, name: Johan Jatko, name: Johannes Rausch, name: John Kamalu, name: John Roberts, name: Johnny Greco, name: Johnny Mensel, name: Jonah Alben, name: Jonas Yang, name: Jonathan Cohen, name: Jonathan Raiman, name: Joseph Jennings, name: Joshua Mabry, name: Joshua Pierce, name: Joyjit Daw, name: Julien Veron Vialard, name: Junkeun Yi, name: Jupinder Parmar, name: Kajal Jain, name: Kan Zhu, name: Kari Briski, name: Katherine Cheung, name: Katherine Luna, name: Keith Willowhawk, name: Keith Wyss, name: Keshav Santhanam, name: Kevin Shih, name: Kezhi Kong, name: Khanh Nguyen, name: Khushi Bhardwaj, name: Kirthi Shankar Sivamani, name: Konstantinos Krommydas, name: Krishna C. Puvvada, name: Krzysztof Pawelec, name: Kumar Anik, name: Kyle Keprios, name: Kylie Day, name: Lawrence McAfee, name: Leo Du, name: Leon Derczynski, name: Li Ding, name: Linda Liu, name: Lingjie Wu, name: Lior Kadoch, name: Lizzie Wei, name: Luis Vega, name: Luke Robison, name: Lun Su, name: Maarten Van Segbroeck, name: Maciej Jakub Mikulski, name: Maer Rodrigues de Melo, name: Magda Sypula, name: Mahan Fathi, name: Makesh Narsimhan Sreedhar, name: Makesh Tarun Chandran, name: Manoj Kilaru, name: Maor Ashkenazi, name: Marc Cuevas, name: Marc Romeijn, name: Marcin Chochowski, name: Mark Cai, name: Mark Mozolewski, name: Markus Kliegl, name: Marta Stepniewska-Dziubinska, name: Martyna Patelka, name: Mattei Machczynski, name: Matvei Novikov, name: Mauricio Ferrato, name: Maximilian Golub, name: Mehrzad Samadi, name: Melissa Corpuz, name: Mengru Wang, name: Mengxi Wu, name: Meredith Price, name: Meriem Boubdir, name: Micah Schaffer, name: Michael Andersch, name: Michael Boone, name: Michael Gschwind, name: Michael Lightstone, name: Michael Loh, name: Michal Bien, name: Michal Zawalski, name: Michelle Gill, name: Miguel Martinez, name: Mikail Khona, name: Mike Chrzanowski, name: Mike Houston, name: Mingyuan Ma, name: Minseok Lee, name: Mohamed Fawzy, name: Mohammad Dabbah, name: Mohammad Shoeybi, name: Mostofa Patwary, name: Nabin Mulepati, name: Najeeb Nabwani, name: Namit Dhameja, name: Narimane Hennouni, name: Natalie Hereth, name: Nathaniel Pinckney, name: Nave Algarici, name: Nave Assaf, name: Netanel Haber, name: Nicholas Knight, name: Nick Reamaroon, name: Nickson Quak, name: Nidhi Bhatia, name: Nikhil Desai, name: Nikolai Ludwig, name: Nima Tajbakhsh, name: Ning Xu, name: Nir Ailon, name: Nirmal Juluru, name: Nitin Nitin, name: Ofri Masad, name: Oleg Rybakov, name: Oleksii Hrinchuk, name: Oleksii Kuchaiev, name: Olivia Viessmann, name: Olivier Delalleau, name: Oluwatobi Olabiyi, name: Omer Ullman Argov, name: Omri Puny, name: Oren Tropp, name: Pablo Ribalta, name: Pallab Bhattacharya, name: Panos Lampropoulos, name: Parth Mannan, name: Pasha Shamis, name: Patrick Legresley, name: Paul Gibbons, name: Pavlo Molchanov, name: Pawel Morkisz, name: Peter Dykas, name: Peter Jin, name: Pierre-Yves Aquilanti, name: Pinky Xu, name: Piotr Januszewski, name: Piotr Laskiewicz, name: Pooya Jannaty, name: Prakash Gurumurthy, name: Pranav Prashant Thombre, name: Prasoon Varshney, name: Pritam Gundecha, name: Przemek Tredak, name: Puhui Meng, name: Qiyu Wan, name: Rabeeh Karimi Mahabadi, name: Rachel Oberman, name: Rachit Garg, name: Radha Sri-Tharan, name: Rahul Kandu, name: Rakshit Sanadhya, name: Ran El-Yaniv, name: Ran Zilberstein, name: Rasoul Shafipour, name: Ray Macalisang, name: Rayen Tian, name: Reka Kovacs, name: Renjie Pi, name: Rick Izzo, name: Rima Shahbazyan, name: Rishabh Garg, name: Rishi Puri, name: Rita Fernandes Neves, name: Ritchie Zhao, name: Ritika Borkar, name: Ritu Gala, name: Riyad Islam, name: Robert Clark, name: Robert Hesse, name: Robert Kirby, name: Roger Waleffe, name: Rohit Watve, name: Roi Koren, name: Ron Banner, name: Ruoxi Zhang, name: Russell J. Hewett, name: Ryan Prenger, name: Ryan Stewart, name: Ryota Egashira, name: Sadegh Mahdavi, name: Saee Paliwal, name: Sagar Singh, name: Sahil Modi, name: Salika Dave, name: Samantha Shinagawa, name: Samuel Kriman, name: Sandip Bhaskar, name: Sangkug Lym, name: Sanjay Kariyappa, name: Sanjeev Satheesh, name: Saran Vikas Murari, name: Satish Pasumarthi, name: Saurabh Mishra, name: Saurav Muralidharan, name: Scott Hara, name: Sean Narentharen, name: Selvaraj Anandaraj, name: Seonjin Na, name: Seonmeyong Bak, name: Seonmyeong Bak, name: Sepehr Sameni, name: Seph Mard, name: Serge Panev, name: Seth Henneman, name: Seth Poulos, name: Shahar Mor, name: Shantanu Acharya, name: Shaona Ghosh, name: Sharath Turuvekere Sreenivas, name: Sharon Mendelson, name: Shaun Kotek, name: Shawn Wang, name: Shay Aharon, name: Shaya Gharghabi, name: Sheng-Chieh Lin, name: Shi Chen, name: Shiqing Fan, name: Shirish Baskaran, name: Shreya Gopa, name: Shrimai Prabhumoye, name: Shubham Pachori, name: Shubham Toshniwal, name: Shuoyang Ding, name: Shwetha Krishnamurthy, name: Siddharth Singh, name: Simeng Sun, name: Sirshak Das, name: Sivakumar Arayandi Thottakara, name: Smita Ithape, name: Somshubra Majumdar, name: Soumye Singhal, name: Sri Harsha Singudasu, name: Sridhar Bhuvanapalli, name: Srimukh Veccham, name: Stas Sergienko, name: Stefania Alborghetti, name: Stephen Ge, name: Su Rong, name: Sugam Dipak Devare, name: Sukrit Rao, name: Sumeet Kumar Barua, name: Sungsoo Ha, name: Sunny Gai, name: Suriya Gunasekar, name: Suseella Panguluri, name: Suyog Gupta, name: Sviataslau Hinzburh, name: Sweta Priyadarshi, name: Syeda Nahida Akter, name: Talor Abramovich, name: Tan Bui, name: Tanay Varshney, name: Tatevik Ter-Hovhannisyan, name: Teodor-Dumitru Ene, name: Terry Kong, name: Thanh Do, name: Tianhe Zhang, name: Tiffany Moore, name: Tijmen Blankevoort, name: Tim Moon, name: Tiyasa Mitra, name: Tom Balough, name: Tomasz Grzegorzek, name: Tomasz Hliwiak, name: Tomer Asida, name: Tomer Bar Natan, name: Tomer Keren, name: Tomer Ronen, name: Tony Salim, name: Tony Wang, name: Traian Rebedea, name: Tugrul Konuk, name: Twinkle Vashishth, name: Udi Karpas, name: Ushnish De, name: Vahid Noorozi, name: Venkat Srinivasan, name: Venmugil Elango, name: Vibhor Agrawal, name: Victor Cui, name: Vijay Korthikanti, name: Vikas Mehta, name: Vinay Rao, name: Virginia Wu, name: Vitaly Kurin, name: Vitaly Lavrukhin, name: Vladimir Anisimov, name: Vu Pham, name: Wanli Jiang, name: Wasi Uddin Ahmad, name: Wataru Ishihara, name: Wei Du, name: Wei Ping, name: Weiheng Chai, name: Wenliang Dai, name: Wesley Helmholz, name: Will Jennings, name: Will Zhu, name: Wojciech Prazuch, name: Xiaowei Ren, name: Xiwen Yu, name: Yan Breek, name: Yang Chen, name: Yang Yu, name: Yangyi Chen, name: Yaniv Galron, name: Yashaswi Karnati, name: Yejin Choi, name: Yev Meyer, name: Yi-Fu Wu, name: Yian Zhang, name: Ying Lin, name: Yonatan Geifman, name: Yonggan Fu, name: Youngeun Kwon, name: Yu Yao, name: Yugi Guvvla, name: Yuki Huang, name: Yunsheng Liu, name: Zach Moshe, name: Zachary Newell, name: Zhilin Wang, name: Zhiyu Li, name: Zhongbo Zhu, name: Zhuolin Yang, name: Zihan Liu, name: Zijie Yan, name: Zsolt-Alon Wertheimer
Organization: NVIDIA

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.