Case2Code: Learning Inductive Reasoning with Synthetic Data
Authors: name: Yunfan Shao; name: Linyang Li; name: Yichuan Ma; name: Peiji Li; name: Demin Song; name: Qinyuan Cheng; name: Shimin Li; name: Xiaonan Li; name: Pengyu Wang; name: Qipeng Guo; name: Hang Yan; name: Xipeng Qiu; name: Xuanjing Huang; name: Dahua Lin
Published: 2024-07-17 11:35:00

Abstract: Complex reasoning is an impressive ability shown by large language models
(LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought
prompting or iterative tool-using to solve challenging tasks step-by-step. In
this paper, we hope to focus on evaluating and teaching LLMs to conduct
inductive reasoning, that is, LLMs are supposed to infer underlying rules by
observing examples or sequential transformations. However, collecting
large-scale and diverse human-generated inductive data is challenging. We focus
on data synthesis in the code domain and propose a Case2Code task by
exploiting the expressiveness and correctness of programs. Specifically, we
collect a diverse set of executable programs, synthesize input-output
transformations for each program, and force LLMs to infer the underlying code
implementations based on the synthetic I/O cases. We first evaluate
representative LLMs on the synthesized Case2Code task and demonstrate that the
Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale
Case2Code training samples to train LLMs to perform inductive reasoning.
Experimental results show that such induction training benefits not only in
distribution Case2Code performance but also enhances various coding abilities
of trained LLMs, demonstrating the great potential of learning inductive
reasoning via synthetic data.