Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
Authors: name: Wanghan Xu, name: Yuhao Zhou, name: Yifan Zhou, name: Qinglong Cao, name: Shuo Li, name: Jia Bu, name: Bo Liu, name: Yixin Chen, name: Xuming He, name: Xiangyu Zhao, name: Xiang Zhuang, name: Fengxiang Wang, name: Zhiwang Zhou, name: Qiantai Feng, name: Wenxuan Huang, name: Jiaqi Wei, name: Hao Wu, name: Yuejin Yang, name: Guangshuai Wang, name: Sheng Xu, name: Ziyan Huang, name: Xinyao Liu, name: Jiyao Liu, name: Cheng Tang, name: Wei Li, name: Ying Chen, name: Junzhi Ning, name: Pengfei Jiang, name: Chenglong Ma, name: Ye Du, name: Changkai Ji, name: Huihui Xu, name: Ming Hu, name: Jiangbin Zheng, name: Xin Chen, name: Yucheng Wu, name: Feifei Jiang, name: Xi Chen, name: Xiangru Tang, name: Yuchen Fu, name: Yingzhou Lu, name: Yuanyuan Zhang, name: Lihao Sun, name: Chengbo Li, name: Jinzhe Ma, name: Wanhao Liu, name: Yating Liu, name: Kuo-Cheng Wu, name: Shengdu Chai, name: Yizhou Wang, name: Ouwen Zhangjin, name: Chen Tang, name: Shufei Zhang, name: Wenbo Cao, name: Junjie Ren, name: Taoyong Cui, name: Zhouheng Yao, name: Juntao Deng, name: Yijie Sun, name: Feng Liu, name: Wangxu Wei, name: Jingyi Xu, name: Zhangrui Li, name: Junchao Gong, name: Zijie Guo, name: Zhiyu Yao, name: Zaoyu Chen, name: Tianhao Peng, name: Fangchen Yu, name: Bo Zhang, name: Dongzhan Zhou, name: Shixiang Tang, name: Jiaheng Liu, name: Fenghua Ling, name: Yan Lu, name: Yuchen Ren, name: Ben Fei, name: Zhen Zhao, name: Xinyu Gu, name: Rui Su, name: Xiao-Ming Wu, name: Weikang Si, name: Yang Liu, name: Hao Chen, name: Xiangchao Yan, name: Xue Yang, name: Junchi Yan, name: Jiamin Wu, name: Qihao Zheng, name: Chenhui Li, name: Zhiqiang Gao, name: Hao Kong, name: Junjun He, name: Mao Su, name: Tianfan Fu, name: Peng Ye, name: Chunfeng Song, name: Nanqing Dong, name: Yuqiang Li, name: Huazhu Fu, name: Siqi Sun, name: Lijing Cheng, name: Jintai Lin, name: Wanli Ouyang, name: Bowen Zhou, name: Wenlong Zhang, name: Lei Bai
Published: 2025-12-18 12:44:36

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.