ItD: Large Language Models Can Teach Themselves Induction through Deduction (2024)

Abstract: Although Large Language Models (LLMs) are showing impressive performance on a wide range of Natural Language Processing tasks, researchers have found that they still have limited ability to conduct induction. Recent works mainly adopt “post processes” paradigms to improve the performance of LLMs on induction (e.g., the hypothesis search & refinement methods), but their performance is still constrained by the inherent inductive capability of the LLMs. In this paper, we propose a novel framework, Induction through Deduction (ItD), to enable the LLMs to teach themselves induction through deduction. The ItD framework is composed of two main components: a Deductive Data Generation module to generate induction data and a Naive Bayesian Induction module to optimize the fine-tuning and decoding of LLMs. Our empirical results showcase the effectiveness of ItD on two induction benchmarks, achieving relative performance improvement of 36% and 10% compared with previous state-of-the-art, respectively. Our ablation study verifies the effectiveness of two key modules of ItD. We also verify the effectiveness of ItD across different LLMs and deductors. The data and code of this paper can be found at https://github.com/forangel2014/ItD.

heading: 1 Introduction
text: Induction can take we humans from the observed to the unobserved (Sloman and Lagnado, 2005). The task of Induction aims to discover consistent transformations from a set of input-output pairs, where the transformations map the inputs to the outputs well (Wang et al., 2023). As shown in Figure 1, given the input-output pairs {xi, yi}ni=1, the model
1* Corresponding author: Kang Liu (kliu@nlpr.ia.ac.cn)
needs to predict the latent transformation f . For a detailed example, given the input [1,2] with the output [1] and other input-output pairs, the tested model is supposed to figure out the transformation output the first element of the input list. The Induction task is an important task in Natural Language Processing (NLP) and the mastery of the induction ability is an important sign of intelligence (Peirce, 1868; Lake et al., 2017; Chollet, 2019).
Currently, humans have already mastered the capability of induction and have found thousands of laws from the physical world and human society. However, machine intelligence still struggles
2719
to induce basic logic rules in structure data like knowledge graphs (Zhang et al., 2021; GrzymalaBusse, 2023). Recently, with the rapid development of Large Language Models (LLMs), many works have begun to adopt the LLMs to induce the transformations given the input-output observations of various tasks and express the induced transformations as rules (Yang et al., 2023; Sun et al., 2023; Zhu et al., 2023; Zhao et al., 2023), guidelines (Pang et al., 2023), instructions (Honovich et al., 2022), and codes (Alet et al., 2021; Wang et al., 2023). These methods take advantage of the interpretability and generalization ability of LLMs in solving the Induction task.
However, recent research (Bang et al., 2023; Mitchell et al., 2023; Mirchandani et al., 2023; Gendron et al., 2023) have revealed that LLMs have inherently limited ability in induction. To tackle such a limitation, work like Hypothesis Search (Wang et al., 2023) proposes to select the generated hypotheses from LLMs by evaluating them on the observations, while another following work Iterative Hypothesis Refinement (Qiu et al., 2023) proposes to further refine them through LLMs based on the evaluating results on the observations. Nevertheless, as shown in Figure 2(a), these hypothesis
search & refinement methods are essentially “post processes” to the directly induced hypotheses of LLMs. They still heavily rely on the inherent induction ability of LLMs which are Weak Inductors.
Even though LLMs are limited in induction, recent work finds out that they possess much better capability in deduction (Bang et al., 2023; Tang et al., 2023). Different from induction, deduction aims to infer the correct output given the transformation and the input. Despite the distinction that induction associates multiple (x, y) pairs with the latent transformation f , whereas deduction links x and f to the resultant y, both approaches fundamentally share the commonality of reasoning within the framework of input, output, and transformation (x, y, f). Therefore, it motivates us to propose a novel framework ItD (Induction through Deduction), to enable the LLMs to teach themselves induction through deduction. Different from previous methods, ItD fine-tunes the LLMs on their deduced data to make them Strong Inductors, as shown in Figure 2(b). For a given induction task, ItD first proposes Deductive Data Generation to leverage the deductive capability of the LLMs to generate a set of task data (x, y, f), which is simple yet effective and does not rely on human annotations or any larger LLMs’ assistance. The data will then be used to fine-tune the LLMs to obtain better inductive capability.
However, it is non-trivial to utilize the deduced data. We find out that directly fine-tuning the LLMs with the IO prompt (§2.2) used in the previous methods (Honovich et al., 2022; Wang et al., 2023; Qiu et al., 2023) cannot effectively leverage the observed samples (as shown in Figure 5). Thus, ItD further proposes Naive Bayesian Induction as a strategy to optimize the use of each sample. Moreover, we also observe performance gains with the increase in the number of samples using our approach. Specifically, this novel technique fine-tunes the LLM to predict f conditioned on single pair x, y (p(f |x, y)) instead of n pairs (p(f |{xi, yi}ni=1)). During the decoding phase, it utilizes the Naive Bayesian approach to equivalently infer the probability distribution of f under all n (x, y) conditions (p(f |{xi, yi}ni=1)) with the probability distribution of f under a single (x, y) condition (p(f |x, y)).
We conduct experiments on two different types of induction tasks for evaluation: Instruction Induction and List Function. Compared with previous methods, The experiment results show that ItD is
superior to the existing methods in assisting LLMs in induction, and both the Deductive Data Generation and the Naive Bayesian Induction components effectively contribute to ItD. We also make discussions to show that ItD can be effectively applied to different LLMs, and a more powerful deductor, e.g. ChatGPT, will further improve the performances of ItD. In summary, the major contributions of this paper are as follows: • We propose a novel framework ItD to enable
the LLMs to teach themselves induction through deduction.
• We propose Deductive Data Generation to effectively leverage the deductive capability of LLMs to generate task data. which is fully selfsupervised and needs no human annotations or any larger LLMs to assist.
• We propose Naive Bayesian Induction to allow LLMs to optimize the use of each observed sample and be able to take advantage of the increase in the number of observed samples.

heading: 2 Preliminary

heading: 2.1 Induction Task
text: As shown in Figure 1, induction aims to infer the latent transformation, f , from a few of observed samples, {xi, yi}ni=1, where yi = f(xi).
An induction task T will include multiple inputoutput data pairs D = {xi, yi}mi=1, and all (xi, yi) share the same latent ground truth transformation f . The complete task data D of task T is then split into an induction set Din and a deduction set Dde.
The testing model is asked to first run the induction process on Din. Din is split into multiple batches, with each batch containing n samples {xi, yi}ni=1. The batches will be fed into the model sequentially. The testing model observes the input batches and induces the predicted transformation f∗. All f∗ induced from Din will be collected for deduction.
In the deduction process, a shared Reasoner R is used to execute all induced f∗ from different methods on Dde for fairness. For all test samples (xtest, ytest) from Dde, the candidate f∗ and test input xtest will be fed into R and then R generates the prediction y∗. Finally, we evaluate the metric between ytest and y∗ and average it over f∗.

heading: 2.2 IO Prompt
text: As the induction task offers the model n observed samples at a time, it is natural to organize the sam-
ples into the IO (Input-Output) prompt as follows: x1, y1;x2, y2; ...;xn, yn, which is also widely used by previous works (Honovich et al., 2022; Wang et al., 2023; Qiu et al., 2023). Note that we omit the instructions and other connection words in the prompt above. For example, for the Input2 in Figure 1, the IO prompt can be: Please figure out the transformation that transforms the following input lists to the output lists: Input:[1,2], Output:[1], ......, Input:[5,1], Output:[5]. So the transformation is:.

heading: 3 Framework
text: In this section, we introduce ItD, a framework for empowering the induction capability of LLMs through their own deduction capability. As shown in Figure 3, ItD is composed of two modules: Deductive Data Generation, and Naive Bayesian Induction. For a given induction task, Deductive Data Generation will first leverage the deductive capability of the LLMs to generate the task data. Then we propose Naive Bayesian Induction to allow LLMs to optimize the use of each observed sample, while taking advantage of the increase in the number of observed samples.

heading: 3.1 Deductive Data Generation
text: To empower the induction ability of LLMs, a set of training data (x, y, f) is needed. Here we consider sampling from their joint distribution p(x, y, f). As we introduced in §1, compared with induction p(f |x, y), LLMs are better at deduction p(y|f, x). Thus we propose the following derivation to leverage the LLMs to generate the task data in a deductive behavior.
p(x, y, f) = p(x, y|f)p(f) = p(y|x, f)p(x|f)p(f) (1)
As shown in Eq (1), to generate data (x, y, f), we propose to sample p(f), p(x|f), and p(y|x, f) sequentially. The pipeline of Deductive Data Generation is shown in the upper half of Figure 3.
3.1.1 Sampling p(f) through Initial Induction To ensure that the generated data (x, y, f) approximates the real task data distribution, we first need to sample the transformation f that approximates the ground truth transformation of the task. Thus, we let LLMs induce f on the induction set Din in the sampling decoding mode with the IO prompt, and we consider the induced f as samples from the prior distribution p(f).

heading: 3.1.2 Deduction with In-Context Learning
text: For the p(y|x, f)p(x|f) part in Eq (1), this paper leverages the deductive capability with In-Context Learning (ICL) of LLMs to generate samples (x, y). We first manually create several cases of deduction as the few-shot demonstrations and then ask LLMs to generate samples (x, y) for each f (Figure 4).
As shown in Figure 4, the upper half is the fixed prompt and the content of the last instruction is replaced by each f from §3.1.1. In the lower half, the LLMs will follow the demonstrations to continuously first generate an input xi according to the instruction f , and then generate yi based on their deductive capability. We then parse the output text of LLMs to obtain the samples {xi, yi}ni=1. The deductive capabilities of LLMs will determine the extent to which (x, y) satisfies the given f . For each f , we generate n corresponding (x, y) pairs for later tuning.

heading: 3.2 Naive Bayesian Induction
text: Having obtained the generated task data, we propose Naive Bayesian Induction which incorporates tuning and decoding to empower the inductive capability of LLMs. The pipeline of Naive Bayesian Induction is shown in the lower half of Figure 3. Instead of the plain IO prompt (§2.2), Naive Bayesian Induction proposes the Group Decoding (GD) prompt template as follows: x, y. Compared with the IO prompt, the GD prompt contains only one input-output pair (x, y).
By using the GD prompt in Naive Bayesian Induction, we allow LLMs to optimize the use of each observed sample (p(f |x, y)) and can take advantage of the increase in the number of observed samples. Naive Bayesian Induction further proposes Naive Bayesian Group Decoding, which enables us to equivalently infer the probability distribution of f under all n (x, y) conditions (p(f |{xi, yi}ni=1)) with the probability distribution of f under a single (x, y) condition (p(f |x, y)).
Specifically, the IO prompt and GD prompt finetune the LLM and decode with the LLMs according to the following distribution respectively.
• IO prompt: pLLM (f |{xi, yi}ni=1) • GD prompt: pLLM (f |x, y)

heading: 3.2.1 Fine-tuning on the Deduced Data
text: For the shared fine-tuning data collected in §3.1, we organize them into the training data with IO prompt and GD prompt, respectively. Then, we adapt LoRA (Hu et al., 2021) and QLoRA (Dettmers et al., 2023) to fine-tune the original LLMs to gain a better capability of induction.

heading: 3.2.2 Naive Bayesian Group Decoding
text: For the model trained with IO prompt, in the induction stage, we directly convert the n observed sample from Din into the IO prompt, feed it into the model, and use beam search to decode the f . This method is denoted as ItD-IO.
For the model trained with GD prompt, ItD proposes the following Naive Bayesian Group Decod-
ing (NBGD) algorithm. NBGD allows us to equivalently infer the probability distribution of f under all n (x, y) conditions (p(f |{xi, yi}ni=1)) with the probability distribution of f under a single (x, y) condition (p(f |x, y)).
p(f |{xi, yi}ni=1) = p({xi, yi}ni=1|f)p(f)
p({xi, yi}ni=1) ∝ p({xi, yi}ni=1|f)p(f) = p(f) n∏
i=1
p(xi, yi|f)
= p(f) n∏
i=1
p(f |xi, yi)p(xi, yi) p(f)
∝ p(f)−(n−1) n∏
i=1
p(f |xi, yi)
(2) Here we assume that given the transformation f , the input-output pairs (x, y) are independent to each other, i.e. p({xi, yi}ni=1|f) =
∏n i=1 p(xi, yi|f). This assumption is quite natural in the scene of induction, where each yi is only determined by f and the corresponding xi.
As shown in Eq (2), we derive the probability p(f |{xi, yi}ni=1) into two parts, i.e. the prior term p(f)−(n−1) and the product term ∏n i=1 p(f |xi, yi) respectively. Suppose the text of f is a sentence t = [t1, t2, ..., tm]. For the ∏n i=1 p(f |xi, yi), we modify the ordinary beam search decoding process as follows:
n∑
i=1
log p(t|xi, yi) = n∑
i=1
m∑
j=1
log p(tj |xi, yi, t<j)
=
m∑
j=1
n∑
i=1
log p(tj |xi, yi, t<j)
(3) As shown in Eq (3) and Figure 3, in the induction stage, NBGD will first convert all samples (xi, yi) into GD prompt, tokenize them and feed them into the LLMs in a batch. Then in each step of decoding (j), the LLMs receive the already decoded part of transformation t<j , and every sample (xi, yi) and generate the next-token scores (log-probabilities) for tj . Then we will add up the next-token scores from all the samples (i). Like ordinary beam search, in each step j, we will maintain the top-k beams with the largest beam scores.
After NBGD decodes the top-k f , we finally rerank them through the prior term p(f)−(n−1). In the log scale, we only need to calculate the log probabilities log p(f) with the same LLMs and add −(n − 1) log p(f) to their beam scores. We consider this training & decoding method as the complete method of our framework, denoted as ItD.

heading: 4 Experiments

heading: 4.1 Dataset and Setups
text: We use two datasets to test the inductive capability of LLMs on two types of induction tasks: commonsense inductive reasoning and symbolic inductive reasoning.
For commonsense inductive reasoning, we adapt the task Instruction Induction (Honovich et al., 2022). The input x and output y are two short sentences while the transformation f is an instruction. This dataset contains 24 sub-tasks. For symbolic inductive reasoning, we adapt the task List Function (Rule, 2020). The input x and output y are two
integer lists while the transformation f is a natural language description of the latent list transformation. This dataset contains 250 sub-tasks.
We adopt ChatGPT as the Reasoner R for both datasets and all tested methods (Note that the Reasoner R will not participate in Deductive Data Generation but only execute the induced f for evaluation). The reported results are average execution scores (Honovich et al., 2022) over all sub-tasks. The detailed setups of the experiments can be found in Appendix A and the detailed results of each subtask of all methods can be found in Appendix B.

heading: 4.2 Baselines
text: We adopt the following baselines to compare with our proposed ItD: • IO (input-output, Honovich et al. 2022). This
baseline is the plain prompt, i.e. directly splice the observations x1, y1, x2, y2, ..., xn, yn as the IO prompt, and feed this prompt for LLMs to conduct induction. • SC (self-consistency, Wang et al. 2022). Based on the IO prompt, the SC method will sample k hypotheses and select the most consistent one by taking a majority vote. • HS (hypothesis search, Wang et al. 2023). Based on the IO prompt, the HS method will evaluate the generated hypotheses by applying the hypotheses to the observed samples. The deductive reasoning results will be used to filter out the most qualified hypothesis. • HS&R (hypothesis search & refinement, Qiu et al. 2023). After selecting the best hypothesis, this baseline allows LLMs to refine the hypothe-
sis to a better one based on the execution results.

heading: 4.3 Main Results
text: We first compare ItD with previous methods to see whether ItD trains the LLMs to become better inductors.
For Instruction Induction, we adopt Llama-2-7bchat as the LLM for all methods. For List Function, as List Function is a task of symbolic reasoning and LLMs are found poor at symbolic deduction than semantic deduction (Tang et al., 2023), we adopt a larger and more powerful LLM, Mixtral-8x7B, for all methods.
As shown in Table 1, ItD is significantly superior to all existing methods on both datasets, bringing relative performance improvement of 193% and 16% compared with the base model (IO), while bringing relative performance improvement of 35% and 10% compared with the previous SOTA (HS&R). These results verify that ItD is better than previous methods in empowering the inductive capability of LLMs.

heading: 4.4 The Effectiveness of Deductive Data Generation
text: To verify the effectiveness of Deductive Data Generation, we here compare the ItD-IO version with the base model (IO). The only difference between these two models is that ItD-IO is fine-tuned with the same data generated by Deductive Data Generation but with the IO prompt.
As shown in Table 1, ItD-IO is superior to the base model on both datasets, bringing the relative performance improvement of 146% and 8%. These results indicate that Deductive Data Generation can produce effective fine-tuning data for the LLMs.

heading: 4.5 The Effectiveness of Naive Bayesian Induction
text: The Naive Bayesian Induction allows us to optimize the use of each observed sample and to take advantage of the increase in the number of observed samples. To verify the effectiveness of Naive Bayesian Induction, we first compare the complete ItD with ItD-IO. As shown the Table 1, the model trained with complete ItD significantly outperforms ItD-IO on both datasets, indicating the effectiveness of Naive Bayesian Induction.
Moreover, we conduct experiments to verify that Naive Bayesian Induction can benefit from the increase in the number of observed samples. While ItD-IO is tuned with 5 pairs of (x, y) per batch, we
test both ItD-IO and ItD with 2, 5, 8, and 20 pairs of (x, y) per batch. As shown the Figure 5, the performance of ItD-IO remains almost unchanged with the increase in the number of samples, with ItD-IO-20 only outperforming ItD-IO-2 by 1.15% and 0.77%. In contrast, the performance of ItD enjoys a natural improvement as the number of samples grows, with ItD-20 outperforming ItD-2 by 3.86% and 2.77%. These results verify the effectiveness of Naive Bayesian Induction in both
directly improving the induction performance and making LLMs capable of taking advantage of the increase in the number of observed samples.

heading: 4.6 Discussion

heading: 4.6.1 The Effectiveness of ItD on Different Sizes of LLMs
text: To verify whether ItD is effective with different sizes of LLMs, we adopt extra LLMs of different sizes for each task: For Instruction Induction, besides Llama-2-7b-chat, we adopt Llama-2-13bchat, and ChatGPT for experiments. For the List Function, besides Mixtral-8x7B, we adopt ChatGPT for the experiments of this task. Note that for ChatGPT, as we are not able to modify the decoding algorithm during its inference time, we only apply the ItD-IO version for it. We use the official API for the fine-tuning and inference of the ChatGPT.
As shown in Table 2, for LLMs of different sizes, ItD can effectively enhance the performance of the model, with the relative performance improvement ranging from 9% to 193% across different models. These results support that ItD can effectively em-
power the inductive capability of LLMs of different sizes.

heading: 4.6.2 A More Powerful Deductor Can Bring
text: Further Improvements for ItD
Both HS and ItD need a deductor to improve the induction process. For HS, the deductor is used to search for the best-proposed hypothesis by evaluating them on the observed samples. For ItD, the deductor is used to deduce data for fine-tuning. In the experiments above, the deductor used in these two methods is both the tested model itself. However, here we would like to discuss whether a more powerful deductor will further improve these methods. So we adopt the Reasoner R of the tasks, i.e. ChatGPT, as a more powerful deductor for these methods as the comparison.
As shown in Table 3, After being equipped with a more powerful deductor (denoted as +D), both HS and ItD gain performance improvements on both datasets, while ItD still consistently outperforms HS whatever the deductor is the base model or ChatGPT. These results further inform us that the more powerful the Deductor, the better it helps in training the Inductor.

heading: 4.6.3 Held-out Task Generalization
text: To investigate the inductive capability of LLMs under the Instruction-tuning (ItD) framework on outof-distribution (OOD) tasks, we selected 5 held-out tasks from the 24 sub-tasks in the Instruction Induction dataset: sum, translation_en-de, antonyms, first_word_letter, and sentiment. These tasks were drawn from the Spelling, Lexical Semantics, Numerical, Multilingual, and GLUE categories (categorized by Honovich et al., 2022), respectively.
During the Deductive Data Generation stage, we masked the function f and its corresponding x, y pairs for these 5 held-out tasks, making them OOD tasks. We then fine-tuned the LLM using the f, x, y pairs generated from the remaining sub-tasks. This
setting is referred to as ItD-OOD. The results, as shown in Table 4, indicate that compared to the full ItD framework, ItD-OOD generally exhibits a significant performance drop on the held-out tasks. However, compared with the naive IO baseline, ItD-OOD shows improvements on most tasks, and even surpasses ItD on the first_word_letter sub-task. This suggests that the ItD framework has a certain degree of cross-task generalization capability, but the effectiveness of this generalization depends on the similarity between the transformations f of different sub-tasks.

heading: 5 Related Work

heading: 5.1 Capability of Induction of LLMs
text: Although LLMs have shown great power in a large number of fields of NLP (Chen et al., 2024b,a; Li et al., 2024; Ling et al., 2023; Xu et al., 2024), it is shown by previous research that they are poor on induction. Mirchandani et al. 2023 and Gendron et al. 2023 found that LLMs are poor on abstract induction tasks like Abstraction and Reasoning Corpus (Chollet, 2019). Another research (Mitchell et al., 2023) found that even GPT-4 and GPT-4V are still not able to robustly form abstractions and reason in contexts not previously seen in their training data. However, Bang et al. 2023 and Tang et al. 2023 have made quantitative evaluations on LLMs and found that they are much better at deduction than induction. Inspired by the findings of these works, we propose a novel framework, ItD, to leverage the powerful deductive capability of LLMs to enhance their inductive capability.

heading: 5.2 Memory-Oriented Induction
text: LLMs have shown strong ability in reasoning in various down-steam tasks. However, they still struggle when it comes to an unfamiliar task. Thus, many previous works have designed a working memory to help LLMs store and use task-specific knowledge (Yang et al., 2023; Sun et al., 2023; Zhu et al., 2023; Zhao et al., 2023). The LLMs are prompted to induce task-specific knowledge in the form of facts or rules and store them in the memory during the induction stage. In the deductive reasoning stage, a retriever will be called to retrieve relevant knowledge about the current question from the memory and prompt it to the LLMs. For these applications, ItD is supposed to be a powerful framework for these methods to tune the LLMs to gain better inductive capability to further improve their
performances.

heading: 5.3 Hypothesis Search and Refinement
text: Some previous works have proposed methods to improve the induced hypotheses of LLMs by conducting Hypothesis Search and Refinement. Hypothesis Search (Wang et al., 2023) proposes to implement the natural language hypothesis to the Python program and then execute them on the observed samples, the execution results are then used to filter out the better hypotheses. Based on Hypothesis Search, Iterative Hypothesis Refinement (Qiu et al., 2023) proposes to iteratively refine the hypothesis through LLMs based on the feedback of execution results. Compared with these methods, ItD improves the inherent inductive capability of LLMs by fine-tuning them with high-quality deduced data and producing a better induction algorithm.

heading: 5.4 Naive Bayes-based Context Extension
text: NBCE (Su, 2023) is recently proposed as an effective method to extend the context for LLMs. It is proposed for the scenes of conducting QA with a batch of documents. However, the documents are likely to be coupled with others and thus cause NBCE poorly infer the answers. Compared with NBCE, Naive Bayesian Induction applies this derivation to the problem of induction, where the samples are conditionally independent of each other given f in nature. Moreover, we involve the tuning process with GD prompt in ItD, which not only optimize the use of each observed sample but also take advantage of the increase in the number of samples.

heading: 6 Conclusion
text: In this paper, we propose a novel framework, ItD, to enable LLMs to teach themselves induction through deduction. We conduct a series of experiments on two types of induction datasets and verify that ItD is superior to existing methods in empowering the inductive capability of LLMs. Moreover, we verify the effectiveness of Deductive Data Generation and Naive Bayesian Induction. More experiment results support that ItD can be effectively applied to LLMs of different sizes, and a more powerful deductor can further improve the performance of ItD.
Limitations
With our ItD framework, we can improve both the symbolic deductive reasoning and semantic deductive reasoning tasks. However, constrained by the limited capability of LLMs in symbolic reasoning, the performance of ItD on List Function (a symbolic deductive task) is not as satisfying as it is on Instruction Induction (a semantic deductive task). Besides, our proposed Naive Bayesian Group Decoding is still categorized to greedy algorithms. It does not involve planning and may likely fall into local optima. We leave further exploration of these directions as future work.
Ethics Statement
This paper proposes a method for LLMs to teach themselves induction through deduction. All experiments are conducted on publicly available datasets. Thus there is no data privacy concern. Meanwhile, this paper does not involve human annotations, and there are no related ethical concerns.

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China (No.2022ZD0160503) and the National Natural Science Foundation of China (No.62376270) and OPPO Research Fund.

heading: A Setups
text: For Instruction Induction, it contains 24 sub-tasks, with the induction set Din of each sub-task including 100 batches, each batch includes n = 5 pairs of (x, y). The deduction set Dde of each sub-task including 100 pairs of (x, y) for testing. For List Function, it contains 250 sub-tasks, with the induction set Din of each sub-task including 3 batches, each batch includes n = 5 pairs of (x, y). The deduction set Dde of each sub-task including 17 pairs of (x, y) for testing.
Induction on both tasks is conducted in a zeroshot manner by LLMs. For both ordinary beam search and Naive Bayesian Group Decoding used during the induction phase, we adopt the beam size of 5.
For Deductive Data Generation, we adopt top-p = 0.95 and temperature = 0.3 to sample 5 transformations f from each batch of Din. And then we generate 5 pairs of (x, y) for each f . For both ItD and ItD-IO, we fine-tune them using the same data above, with a learning rate of 1e-4 and for 3 epochs. For Naive Bayesian Group Decoding, we create a patch for the utils.py in the transformer library, it can be easily installed and uninstalled using our scripts.
The prompts used in Induction (§3.2.2) and Deduction with In-Context Learning (§3.1.2) for Instruction Induction and List Function are shown in Table 5 and Table 6, respectively. Note that the text in the Induction part is shared by both the IO prompt and the GD prompt (for the IO prompt, n > 1, and for the GD prompt, n = 1).

heading: B Detailed Results
text: The detailed results of Instruction Induction are shown in Table 7. As the List Function contains 250 sub-tasks and we have 19 methods in all, the table of its detailed results will be too large for the paper. Instead, you can find it at https://github.com/forangel2014/ItD.

heading: Future Work
text: We leave further exploration of these directions as future work