Enhancement of In-Context Reasoning in LLMs through Inductive Rule Learning

Authors: Tien-Dat Nguyen, Hai-Toan Nguyen, Nguyen Viet Ha
Primary area: generative models
Keywords: In-Context Learning, Inductive Reasoning

Abstract:
Currently, Large language models (LLMs) have achieved remarkable performance across various language tasks, largely due to their training on extensive datasets and their considerable model size. These models exhibit in-context learning abilities, which is to learn through few-shot learning. However, the underlying reasoning process remains ambiguous, it is unclear whether the model simply retrieves relevant information and instructions from its training data to generate similar responses, or whether it generalizes examples to form overarching rules, which are then applied to produce accurate answers. Another method for improving few-shot learning is Chain-of-Thought prompting that complement steps by steps instruction for LLMs, so they can follow this instruction to solve many reasoning tasks. Several approaches for evaluating the reasoning abilities of LLMs typically involve task-solving through code generation, which enables models to formalize problems and leverage a code compiler to solve them precisely. However, these methods are constrained to specific task types and are insufficient for a comprehensive assessment of the model's broader reasoning capabilities. Therefore, this paper proposes a method to enhance in-context learning capabilities through two main stages: generating general rules from the provided examples and utilizing LLMs to verify these general rules, thereby aiming to improve reliability and accuracy. At the same time, this approach seeks to investigate the inductive and deductive reasoning abilities, and can improve our understanding of the model’s reasoning by generating and applying general rules to provide transparent, clearly explained responses. The proposed method demonstrates competitive performance on the 1D-ARC benchmark and several traditional language tasks, suggesting its potential for more robust evaluation of LLM reasoning abilities.

Decision: desk reject

Reviews and discussion:

Metareview: