Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
Authors: name: Wanghan Xu; name: Yuhao Zhou; name: Yifan Zhou; name: Qinglong Cao; name: Shuo Li; name: Jia Bu; name: Bo Liu; name: Yixin Chen; name: Xuming He; name: Xiangyu Zhao; name: Xiang Zhuang; name: Fengxiang Wang; name: Zhiwang Zhou; name: Qiantai Feng; name: Wenxuan Huang; name: Jiaqi Wei; name: Hao Wu; name: Yuejin Yang; name: Guangshuai Wang; name: Sheng Xu; name: Ziyan Huang; name: Xinyao Liu; name: Jiyao Liu; name: Cheng Tang; name: Wei Li; name: Ying Chen; name: Junzhi Ning; name: Pengfei Jiang; name: Chenglong Ma; name: Ye Du; name: Changkai Ji; name: Huihui Xu; name: Ming Hu; name: Jiangbin Zheng; name: Xin Chen; name: Yucheng Wu; name: Feifei Jiang; name: Xi Chen; name: Xiangru Tang; name: Yuchen Fu; name: Yingzhou Lu; name: Yuanyuan Zhang; name: Lihao Sun; name: Chengbo Li; name: Jinzhe Ma; name: Wanhao Liu; name: Yating Liu; name: Kuo-Cheng Wu; name: Shengdu Chai; name: Yizhou Wang; name: Ouwen Zhangjin; name: Chen Tang; name: Shufei Zhang; name: Wenbo Cao; name: Junjie Ren; name: Taoyong Cui; name: Zhouheng Yao; name: Juntao Deng; name: Yijie Sun; name: Feng Liu; name: Wangxu Wei; name: Jingyi Xu; name: Zhangrui Li; name: Junchao Gong; name: Zijie Guo; name: Zhiyu Yao; name: Zaoyu Chen; name: Tianhao Peng; name: Fangchen Yu; name: Bo Zhang; name: Dongzhan Zhou; name: Shixiang Tang; name: Jiaheng Liu; name: Fenghua Ling; name: Yan Lu; name: Yuchen Ren; name: Ben Fei; name: Zhen Zhao; name: Xinyu Gu; name: Rui Su; name: Xiao-Ming Wu; name: Weikang Si; name: Yang Liu; name: Hao Chen; name: Xiangchao Yan; name: Xue Yang; name: Junchi Yan; name: Jiamin Wu; name: Qihao Zheng; name: Chenhui Li; name: Zhiqiang Gao; name: Hao Kong; name: Junjun He; name: Mao Su; name: Tianfan Fu; name: Peng Ye; name: Chunfeng Song; name: Nanqing Dong; name: Yuqiang Li; name: Huazhu Fu; name: Siqi Sun; name: Lijing Cheng; name: Jintai Lin; name: Wanli Ouyang; name: Bowen Zhou; name: Wenlong Zhang; name: Lei Bai
Published: 2025-12-18 12:44:36

Abstract: Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.