Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models (2024)

Abstract: Although Large Language Models (LLMs) demonstrate remarkable ability in processing and generating human-like text, they do have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language(e.g., chemical molecular formula). Injecting a collection of symbolic data directly into the training of LLMs can be problematic, as it disregards the synergies among different symbolic families and overlooks the need for a balanced mixture of natural and symbolic data. In this work, we tackle these challenges from both a data and framework perspective and introduce Symbol-LLM series models. First, we curated a data collection consisting of 34 tasks and incorporating 20 distinct symbolic families, intending to capture the interrelations and foster synergies between symbols. Then, a two-stage tuning framework succeeds in injecting symbolic knowledge without loss of the generality ability. Extensive experiments on both symboland NL-centric tasks demonstrate the balanced and superior performances of Symbol-LLM series models. The project page is https:// xufangzhi.github.io/symbol-llm-page/.

heading: 1 Introduction
text: Large Language Models (LLMs), such as GPTseries (Radford et al., 2019; Brown et al., 2020; OpenAI, 2023) and LLaMA-series (Touvron et al., 2023a,b), boosted the performance in various Natural Language Processing (NLP) tasks (Zhao et al., 2023; Wei et al., 2022b; Zhou et al., 2023; Yao et al., 2023). The success of these models heavily relies on natural language (NL) as the primary interface1 for interaction and reasoning. However, the
∗Corresponding Author. 1Interface in this paper refers to the communication be-
tween LLM and environment (i.e., external tools).
NL-centric interface confines the inputs and outputs to an NL form, which can only address certain aspects of world knowledge, such as fact (Bordes et al., 2015), commonsense (Talmor et al., 2019).
Nevertheless, a substantial amount of abstract knowledge, notably in areas like molecular formula (e.g., C6H12O6) and first-order logic (e.g., IsTriangle(X) → SumOfAngles(X, 180◦)), is more effectively represented in symbolic forms rather than in NL.2
Compared to the NL form, the symbolic form covers a wide spectrum of scenarios and tends to be more concise and clear, enhancing its communication effectiveness (Gao et al., 2023; Qin et al., 2023). In particular, when interacting with robots, symbolic command sequences (such as PICKUP, WALK) are more accurate and efficient than NL. Similarly, when using programming languages (like SQL and Python) to call external tools (Gao et al., 2023), expressing this structured information in NL form can be difficult.
Despite the symbolic form offering a wealth of information, deploying LLMs directly via a symbolic-centric interface poses a significant challenge. This is largely attributed to the fact that LLMs are trained via large-scale unsupervised pre-training on extensive general text datasets, which inherently lack a symbolic foundation. The most straightforward approach to incorporating symbolic knowledge into LLMs is through finetuning (Yang et al., 2023; Xu et al., 2023b). However, the format of symbolic data significantly diverges from that used during pre-training. Consequently, merely fine-tuning with large heterogeneous data can lead to catastrophic forgetting (Kirkpatrick et al., 2017).
2For clarity, the world knowledge covered in this paper refers to the symbolic form of knowledge, rather than the broad concept of ’knowledge’ stored in LLMs.
13091
Meanwhile, existing injection methods primarily stress on specific symbols, it is important to note that symbolic forms can be quite complex and vary across tasks. Training LLM for a specific symbolic form is both time-consuming and labor-intensive. Furthermore, treating each symbol independently often overlooks the interconnections between different symbols, e.g., atom unit (e.g., BornIn(Obama,USA)) in first-order logic (FOL) is similar in form to function (e.g., query(Paris, nwr(hotel))) in API calls .
Upon this observation, we conduct a comprehensive collection of 34 text-to-symbol generation tasks with 20 standard symbolic forms introduced with instruction tuning format. The symbolic data comes from three sources: (1) 88.3% of the data was collected from existing benchmarks. (2) 5.8% of the data was prompted by LLMs. Compensating for the natural absence of symbolic representations in some NL-centric tasks, prompting powerful LLMs can generate more novel text-to-symbol pairs. (3) 5.9% of data was generated by introducing the Symbol-evol strategy, with replaced symbolic definitions to prevent the model from memorizing specific symbols. The above sources are uniformly leveraged to capture the underlying connections between symbols from the data view.
For the framework aspect, we apply a two-stage continual tuning framework including the Injection Stage and the Infusion Stage. The Injection Stage prioritizes the exploitation of the inherent connections between different symbols, thereby enabling the model to thoroughly learn a wide range of symbolic knowledge. After tuning LLaMA-2Chat models with all collected symbolic data, we obtain Symbol-LLMBase variants. The Infusion Stage focuses on balancing the model’s dual capabilities by utilizing both symbolic data and general instruction tuning. After combining the general instruction-tuning data with the sampled symbolic data and tuning based on Symbol-LLMBase, we can obtain Symbol-LLMInstruct. Finally, Symbol-LLM series models are widely tested on both symbolcentric and NL-centric tasks, which are verified to exhibit substantial superiority.
Our contributions are as follows, with key values and research insights attached in Appendix A, B:
• A comprehensive collection of text-to-symbol generation tasks is the first collection to treat symbolic data in a unified view and explore the underlying connections among symbols.
• The open-sourced Symbol-LLM series models build a new foundation LLM with balanced symbolic and NL abilities.
• Extensive experiments on both symbol- and NLcentric tasks are conducted to prove the superiority of Symbol-LLM.

heading: 2 Approach
text: In this section, we first introduce the overall symbolic data collection procedure in Section 2.1 and then describe the two-stage tuning framework and the comprehensive test settings in Section 2.2.

heading: 2.1 Data Collection
text: Conducting comprehensive symbolic knowledge injection and exploiting their interrelations requires a large collection of symbolic data. However, achieving diverse knowledge coverage continues to be a significant hurdle in language modeling. Therefore, we curate an extensive collection of symbolic tasks, which is under-explored in NLP.
The overview of the symbolic data collection procedure is shown in Figure 1. The ultimate symbolic dataset is Ds = Ds1 ∪ Ds2 ∪ Ds3 . Here, Ds1 represents the existing benchmarks. The dataset Ds2 is a novel dataset, resulting from prompting GPT-4. Ds3 is another new dataset, generated by introducing the Symbol-evol strategy. Generally, we compile a set of 34 text-to-symbol generation tasks, covering ∼20 different standard symbolic forms. To maintain the general capability in NL-centric tasks, this work also includes general instruction data Dg. Details of each dataset are attached in Appendix C.

heading: NL2Bash NL2SQL

heading: Test Tasks (In-domain + OOD)
text: Ds1: the existing symbolic datasets and benchmarks Previous efforts have been dedicated to specific symbolic forms, offering a natural and strong foundation for Symbol-LLM. We include plenty of text-to-symbol tasks from various data sources such as Spider (Yu et al., 2018), MTOP (Li et al., 2021), SCAN (Lake and Baroni, 2018), and further shape them in the defined formats. Such collection is named as Ds1 .
Ds2: novel text-to-symbol pairs by prompting GPT-4 While Ds1 has broad coverage, it lacks certain text-to-symbol pairs in some crucial scenarios. For example, some mathematical problems can be better handled when converted to programming language, but labeled samples are limited. To address this, we prompt GPT-4 to generate the corresponding symbolic outputs given the NL instructions, following Gao et al. (2023). Correct outputs judged by executing solvers (e.g., code interpreter) are retained to form new text-to-symbol pairs, constructing the collection Ds2 .
Ds3: new samples generated by applying Symbol-evol strategy The above collection can cover a vast range of standard definitions of symbolic forms. However one concern is that large tuning data with the same symbolic definitions magnify LLM’s propensity to memorize the patterns instead of truly learning to follow instructions. Thus, we introduce the Symbol-evol strategy, expecting
to enhance the diversity of symbolic systems. The strategy of Symbol-evol, as depicted in Figure 1(3), is exemplified using SCAN dataset (Lake and Baroni, 2018). In the original data collection, some action commands (in red background) are defined to control robots. Randomly generated strings (in green background) are leveraged to replace the original symbolic definitions. For example, the originally defined command I_TURN_RIGHT is replaced by shY2sW. In this way, diverse symbol instruction samples can be derived based on some original tasks in Ds1 , forming the collection Ds3 . Dg: general data These collected data are from three sources: (i) sampled flan collection data (Wei et al., 2022a; Longpre et al., 2023); (ii) Code Alpaca instruction tuning data (Chaudhary, 2023); (iii) sampled Evol-data from WizardLM (Xu et al., 2023a). Full details are given in Appendix C.2.

heading: 2.2 Symbol-LLM
text: The overview of Symbol-LLM is shown in Figure 2, comprised of both the tuning and testing phases.
The tuning framework, as illustrated in Fig.2a, encompasses two stages: the Injection stage and Infusion stage. After the Injection stage, we can obtain the Symbol-LLMBase model, which is expected to address various symbol-related scenarios. However, Injection stage focuses on injecting symbolic knowledge into LLMs regardless of the general capability. But we also expect Symbol-LLM to
maintain the necessary proficiency in general tasks, to achieve balanced symbol and NL interfaces for interaction and reasoning. Thus, we introduce the Infusion stage to obtain the Symbol-LLMInstruct.
The test phase, represented in Fig.2b, covers comprehensive settings on the symbolic and NL scenarios.
Tuning Phase 1: Injection Stage In this stage, we purely focus on injecting various symbolic knowledge into LLMs by conducting supervised fine-tuning (SFT) on the Ds collection. The training loss of Injection stage is the maximum likelihood estimation (MLE):
LMLE(Ds) = − ∑
i
log pθ(yi|si ⊕ xi), (1)
where pθ is the tunable LLM with parameters θ, which is initialized from LLaMA-2-Chat models. si⊕xi refers to the input format: the instruction (si) covering the task definition concatenates (⊕) with the natural language query (xi). And yi is the symbolic output.
Tuning Phase 2: Infusion Stage In this stage, we randomly sample Ds to obtain a subset Ds′ ⊂ Ds, the data are proportioned to ensure a fair distribution. They are combined with general instruction tuning data Dg to form the training set in this stage. The loss function to be minimized is based on MLE:
LMLE(Ds′ ∪ Dg) = − ∑
j
log pθ1(yj |sj ⊕ xj), (2)
where the tunable parameters θ1 are initialized from Symbol-LLMBase. sj , xj , and yj are the instruction, input, and output for a single sample, respectively.
Testing Phase This work presents comprehensive testing settings for border applications. For detailed task descriptions refer to Appendix E. • Symbolic Tasks: Extensive symbolic generation tasks stress the unique advantages of addressing symbolic language beyond NL.
• General Tasks: Classical benchmarks of general tasks are leveraged to verify the balanced capabilities in symbol- and NL-centric scenarios.
• Symbol+Delegation Tasks: Verifying the effectiveness of LLM with symbolic-centric interface. We refer to this promising setting as Symbol+Delegation, where the model first generates the symbolic representation of the question and then relies on the external solvers for solution (e.g., Python interpreter, SQL execution).

heading: 3 Experiments
text: In this section, we fully evaluate Symbol-LLM3 on three parts of experiments: the symbolic tasks in Sec. 3.1, the general tasks in Sec. 3.2, and the Symbol+Delegation tasks in Sec. 3.3. The implementation details refer to Appendix E and Appendix F. The overall performances of Symbol-LLM are concluded in Appendix G.

heading: 3.1 Symbolic Tasks
text: Table 1 presents the results of 34 symbolic generation tasks. For model comparison, we include GPT3.5, Claude-1, LLaMA-2-Chat, and the optimized model after single-domain SFT on LLaMA-2-Chat. Limited by space, we compare other baseline models (e.g., CodeLLaMA-Instruct) in Appendix G. For extra OOD symbolic tasks, please refer to Appendix H. The main findings are as follows:
Symbol-LLM largely enhances the symbolrelated capabilities of LLM. In comparison with the LLaMA-2-Chat model, Symbol-LLM presents overwhelming advantages in symbolic tasks. It improves the baseline performances of 7B and 13B by 49.29% and 55.88%, respectively. Also, cutting-edge close-source LLMs like GPT3.5 and Claude-1, are far behind Symbol-LLM, with the minimum gaps of 39.61% (GPT-3.5 v.s. Symbol-LLM-7B). In short, Symbol-LLM brings huge advantages in symbolic scenarios.
The unified modeling helps Symbol-LLM successfully capture the intrinsic relationships between different symbols. Fine-tuning LLaMA2-Chat on single-domain tasks fully overfits domain-specific symbolic forms, as shown in Single SFT of Table 1. Compared with it, SymbolLLM shows better performances, with averaged 0.42% and 2.02% gains in 7B and 13B. It verifies that the unified modeling of various symbolic forms is beneficial to capturing symbolic interrelations.

heading: 3.2 General Tasks
text: To verify Symbol-LLM’s power in tackling NLcentric tasks, we conduct the experiments on two widely-used benchmarks, MMLU and BIG-BenchHard (BBH). Results are shown in Table 2.
Competitive performances in general tasks are maintained in Symbol-LLM. Overall, Symbol-
3Unless otherwise specified, Symbol-LLM represents the final model after two stages (i.e., Instruct version).
LLM is well optimized with the two-stage framework in keeping general abilities. For 7B models, Symbol-LLMInstruct shows consistent superiority on MMLU and BBH benchmarks, with ∼4% gains compared with LLaMA-2-Chat. For 13B models, although Symbol-LLMInstruct slightly falls behind its LLaMA counterpart, it achieves 7.20% performance advantages in BBH. The superiority on average is obvious. While Symbol-LLM may not yet match the performance of closed-source LLMs, its well-rounded general capability is notable. These remarkable performances can make Symbol-LLM stand out as a foundational LLM, which combines the basic NL ability and proficiency in handling symbolic language.
To verify the consistent superiority of Symbol-
LLM in general tasks, we include a broader scope of evaluation in Appendix I.

heading: 3.3 Symbol+Delegation Tasks
text: A wide range of experiments are done under the Symbol+Delegation paradigm, covering the fields of math reasoning, symbolic reasoning, logical reasoning, robotic planning, visual reasoning as well as table question answering. For detailed settings, please refer to Appendix E.3. Limited by space, we only present the results of the math reasoning in the main paper. The remaining parts are attached in Appendix J.
We select 9 commonly used math datasets for testing, including both in-domain and OOD tasks. To demonstrate the surprising performances
of Symbol-LLM, we also include several mathdomain LLMs (e.g., WizardMath (Luo et al., 2023), MAmmoTH (Yue et al., 2023)) as strong baselines. Comparison results are presented in Table 3.
Advanced abilities in math reasoning are possessed by Symbol-LLM. GSM8K and MATH are widely used to evaluate the math reasoning capabilities of LLMs. Compared with recent mathdomain LLMs, Symbol-LLM presents great competitive results on them. Especially on GSM8K, Symbol-LLM consistently wins all strong baselines with great margins with all the model variants. On the MATH dataset, Symbol-LLM merely falls behind MAmmoTH, which is a strong LLM specially designed for math reasoning tasks. Notably, MAmmoTH includes GSM8K and MATH in the tuning stage and it also uses delegation (i.e., Python In-
terpreter) for inference, thus our comparisons are fair. Similar superiority is also observed under the OOD tasks (e.g., SVAMP).
Symbol-LLM exhibits competitive performances in extrapolating to OOD tasks. More surprisingly, Symbol-LLM consistently presents its significant superiority among all 7 OOD math tasks. Even compared with GPT-3.5 under the three-shot setting, our Symbol-LLM-7B series won 4 (out of 7) OOD tasks under the zero-shot setting. As we scale the model size to 13B, obvious performance improvements are observed in most of the tasks. These findings verify the prospects of Symbol-LLM under the Symbol+Delegation paradigm.

heading: 4 Analysis
text: In this section, we include the ablation studies (Sec.4.1) and the analysis on Alignment and Uniformity (Sec.4.2). Notably, additional supplementary experiments are attached in Appendix K, including tuning design choices K.1, comparison with single SFT K.2, and new symbol extrapolation K.3.

heading: 4.1 Ablation Studies
text: Here we present two ablation experiments from both the framework and data views: (1) tuning only in one stage, and (2) tuning only on general data collection. For a fair comparison, we introduce two settings for one-stage tuning. The first setting (named One-stage 1.46M) simply mixes Ds, Ds′ and Dg, regardless of sample overlap. The second setting (named One-stage 1.20M) mixes Ds and Dg, which ensures consistency in diversity and avoids duplication. The model exclusively finetuned on general task Dg is referred to as Generalonly. Comparison results are shown in Table 4.
Two-stage tuning framework shows superiority over one-stage, especially for 13B. Simply mixing the training data in one stage is prone to affecting the symbol-related tasks. Especially under the Symbol+Delegation setting, the two-stage framework witnesses 3∼6% advantages over the one-stage models. In the 13B model comparison, our two-stage framework consistently demonstrates superiority across symbolic tasks, general tasks, and Symbol+Delegation tasks.
The incorporation of symbolic data yields a modest impact on the performances of general tasks. Compared with General-only, Symbol-LLMInstruct is optimized to largely enhance the symbol-centric capabilities. Meanwhile, it maintains the capability to address general NL-centric tasks without significant sacrifices (< 2%).

heading: 4.2 Alignment and Uniformity
text: Motivated by (Wang and Isola, 2020; Gao et al., 2021b), we include Alignment and Uniformity metrics to delve into the factors contributing to the superiority of Symbol-LLM.
Alignment measures the representation similarity within each symbolic form, while Uniformity quantifies the uniformity of all the symbolic representations. They are based on Eq. 3 and Eq. 4 respectively in Appendix L. The calculation results
are visualized in Figure 3. Further, we extend the definition to measure the interrelations between any two symbolic forms, based on Eq. 5. Limited by space, we only include a part of the symbolic forms for illustration and present the results of 13B models in Figure 4 in Appendix L.
The item-wise conclusions are listed as follows: Symbol-LLM optimizes symbol distinctiveness and overall expressiveness in the embedding space. From Fig. 3, compared with the LLaMA2-Chat models, Symbol-LLM series is optimized towards superior Alignment and Uniformity. It ensures the discernment of shared features within each symbolic form, simultaneously enhancing the overall information entropy. Specifically for the 7B model, the two-stage framework effectively maintains a balance of uniformity, preventing the collapse of the embedding space. Symbol-LLM excels at capturing symbolic interrelations. From Fig. 4, the LLaMA-2-Chat model exhibits significant representation sparsity between symbolic forms. Even under the same form (e.g., Bash, FOL), the features are scattered. On the contrary, Symbol-LLM largely enhances the perception of symbolic interrelations by (1) achieving better alignments between symbols (e.g., PythonAMR and CheBi-RX) and (2) pulling closer sample features within each symbolic form (e.g., FOL).

heading: 5 Related Works
text: Large Language Models Plenty of recent efforts have been made to develop foundation language models (Zhao et al., 2023), which are expected to promote the subsequent applications, such as
AI agents (Wang et al., 2023a; Sun et al., 2023; Wu et al., 2024; Cheng et al., 2024). These works on LLMs are universally categorized into closedsource and open-source models. Close-source LLMs, represented by GPT-4 (OpenAI, 2023), Claude, PaLM (Chowdhery et al., 2023), have greatly shaped our daily life through NL-centric interactions. However, their closed-source and black-box properties limit further optimization. Under such circumstances, open-source LLMs (Zeng et al., 2023; Jiang et al., 2023; Touvron et al., 2023b) receive significant attention because of their tunable and small-scale properties. However, current attempts on these LLMs mainly explore NLcentric abilities, which treats NL as the interface to express knowledge and achieve interactive reasoning. In contrast, our work focuses on improving the symbol-centric capabilities of open-source LLM, which leads to a balanced symbol-centric and NLcentric foundational LLM.
Instruction Tuning To make LLMs capable of following human instructions, instruction finetuning (Zhang et al., 2023) is widely adopted. Meanwhile, self-instruct methods (Wang et al.,
2023c; Xu et al., 2023a; Ouyang et al., 2022) have been proposed to generate diverse and abundant instruction data, based on a small collection of seed instructions. In our work, we follow previous instruction tuning strategies in both tuning stages. For symbolic tasks, we construct instructions, covering the task and symbolic descriptions. For general tasks, we sample the instruction-tuning datasets (e.g., Flan (Longpre et al., 2023)).
Symbol-centric Scenarios LLMs have dominated plenty of NL-centric tasks (Rajpurkar et al., 2016; Talmor et al., 2019; Nallapati et al., 2016), where NL is leveraged as the core interface for interaction, planning, and reasoning. But world knowledge is not purely represented by NL. In fact, symbolic language is also of great significance in expressing abstract world knowledge (Edwards et al., 2022; Bevilacqua et al., 2021; Li and Srikumar, 2019) and leveraging external tools (Gao et al., 2023; Liu et al., 2023; Pan et al., 2023). Some concurrent works (Xu et al., 2023b; Yang et al., 2023) shift focus to the specific forms of symbols (e.g., code), either through prompting off-the-shelf LLMs or tuning on open-source LLMs. These efforts fail to lay a solid symbolic foundation, which is expected to grasp the interrelations among various symbolic forms. In our work, we explore the possibility of treating symbols in a unified manner and lay foundations to build balanced symbol and NL interfaces.

heading: 6 Conclusion and Broader Impact
text: This work pioneeringly proposes to enhance the LLM capability in symbol-centric tasks while preserving the performances on general tasks, leading to balanced symbol and NL interfaces. To address the challenges of capturing symbol interrelations and maintaining a balance in general abilities, we tackle the problem from both data and framework perspectives. Data-wise, we include a collection of
34 text-to-symbol tasks to systematically explore underlying symbol relations. Framework-wise, we implement SFT in a novel two-stage manner to mitigate catastrophic forgetting. Extensive experiments across three task settings (i.e., symbolic tasks, general tasks, and symbol+delegation tasks) demonstrate Symbol-LLM’s superiority in harmonizing symbol- and NL-centric capabilities. Moreover, all models and resources will be made public to facilitate a broader range of research. Notably, the exploration of the role of symbolic language is not limited to a single modality or the mentioned scenario (e.g., reasoning). It is worthy of extending the neural-symbolic paradigms to other multi-modal applications, such as visual web navigation (Cheng et al., 2024), 3D understanding (Zhang et al., 2024), and embodied agents (Bigazzi et al., 2023).

heading: Acknowledgement
text: This work is supported by Shanghai Artificial Intelligence Laboratory. And it is also supported by National Key Research and Development Program of China (2022YFC3303600), National Natural Science Foundation of China (62137002, 62293550, 62293553, 62293554, 61937001, 62250066, 62176209, 62176207, and 62192781), "LENOVO-XJTU" Intelligent Industry Joint Laboratory Project, Natural Science Basic Research Program of Shaanxi (2023-JC-YB-593), the Youth Innovation Team of Shaanxi Universities, XJTU Teaching Reform Research Project "Acquisition Learning Based on Knowledge Forest".

heading: Limitations
text: The insight of Symbol-LLM is to build a balanced symbol- and NL-centric interface for interaction and reasoning. We achieve it from both data (comprehensive symbolic collection to open-source) and framework (two-stage tuning to reduce forgetting) perspectives. It is expected to expand the scope of cutting-edge open-source LLMs largely and lay a new foundation for future work. Though plenty of experiments covering three settings are conducted, there still exist the following two directions for exploration: (1) The model’s ability to self-correct or interact with environmental feedback in symbolic scenarios. It is also key to building language agents from language models. (2) Model size scaling to 70B or larger. As widely recognized, 7B or 13B LLMs are still not sufficient to build excellent language agents, especially when complex interaction
is involved. Thus, it needs further exploration for the size scaling to the larger ones.

heading: A Key Contributions and Values
text: Beyond the proposed data collection strategy and the two-stage tuning strategy, we would also like to emphasize the distinctive value of our work: (1) Insights on synergies between symbolic and natural language for LLMs. Our motivation is sourced from the lack of symbolic foundation in off-the-shelf LLMs, which heavily focus on the NL capabilities. Equipping LLMs with a symbolic foundation brings (i) diverse expression of knowledge; (ii) great power in controlling robots and leveraging tools. Also, the synergies between symbolic and natural language can offer potential insights into transforming language models into language agents, which cover extensive code interaction and tool-using scenarios. (2) Thorough evaluations. Quite different from previous works which merely focus on specific symbols and specific tasks, our work builds comprehensive evaluation studies under nearly 200 tasks. The evaluations are mainly divided into three folds: (i) symbolic tasks. (ii) general NL tasks. (iii) symbol+delegation tasks. (3) Open-sourcing effort for the LLM community. Large numbers of recent works are based on closed-source LLMs (e.g., GPT-4). However, such methods are high-cost and inefficient, limiting custom training. Therefore, our efforts to open-source Symbol-LLM series as well as the comprehensive data collection are valuable to the community.
B Insights behind Symbol-LLM
This part provides key insights on (1) two-stage tuning strategy; (2) data scaling; and (3) intrinsic superiority.
Insight 1: Two-stage strategy. Despite their strong natural language processing abilities, opensource LLMs lack symbol-centric capabilities. Naturally, we think about how LMs grasp the foundation in addressing NL? The rough pipeline of training LMs can be in two stages. The first stage focuses on pretraining with large-scale corpus and language modeling objectives. This stage disregards high-level skills (e.g., instruction-following and complex reasoning) and solely focuses on establishing the natural language foundation. In the second stage, the potential for instructionfollowing, and aligning human preference is enhanced through finetuning (or preference optimization).
Motivated by it, we adopt the two-stage training strategy. Initially, we lay a symbolic foundation, known as the Injection stage, where we solely inject symbolic knowledge into LLMs. Then, in the Infusion stage, we balance the NL-centric abilities and excite the symbolic ability through further finetuning on the mixture of datasets.
In fact, several concurrent works share similar paradigms with ours. For example, CodeLLaMAInstruct is based on LLaMA-2-Chat, utilizing largescale code data for pretraining and NL data for optimizing NL capability. CodeGeeX2 and Lemur are also similar works, primarily effective in coderelated scenarios. Symbol-LLM extends the scope to symbolic language (as we stated, code is one of the specific symbols).
Insight 2: Data scaling. One of the interesting insights behind Symbol-LLM is to explore how much data is sufficient to establish a symbolic foundation? Based on the previous works (Zheng et al., 2022), mastering new tasks in large models primarily entails modifications to high-layer parameters. Therefore, we employ the KolmogorovSmirnov Test to reflect changes in LLM parameters, with the training data scaling.
In Figure 5, we present the average performances on 34 symbolic tasks with data scaling. Specifically, we sample Ds in the Injection stage at a ratio of 10%, 40%, 70%, and 100%. As observed, when symbolic training data surpasses 70% (>620K), model performance breaks through the bottleneck of saturation.
Accordingly, the parameter changes layerwise are presented in Figure 6. When training data scales
from 70% to 100%, significant parameter changes emerge in higher layers. It is expected that scaling the symbolic training data can lay a new foundation of symbolic knowledge.
Insight 3: Intrinsic superiority. It is insightful to reveal the intrinsic superiority of Symbol-LLM over symbolic tasks. That is to say, What is behind the superior performances of Symbol-LLM ? We study it from Alignment and Uniformity views. Details are discussed in Section 4.2 and Appendix L.

heading: C Details of Data Collection
text: In this section, detailed information on the data collection is attached, including both text-to-symbol task collection, and general task collection.

heading: C.1 Text-to-symbol Task Collection
text: We provide a detailed illustration of the symbolic task collection, which consists of 34 different textto-symbol generation tasks. They are categorized into 12 domains in Table 5.
Note that the symbolic task collection includes but is not limited to the listed 34 tasks. To expand the diversity, we also consider some similar tasks. For example, we include some domain-specific NLto-SQL tasks to provide diverse schema. The data is only used at the tuning stage but is not for a test. Thus, the whole collection (only count training samples) reaches ∼880K samples. All of them are leveraged in the first SFT stage.
Also, it is mentioned above that we sample parts of symbolic task collection in the second stage to reduce forgetting. For it, we uniformly sample each task domain with a ratio of 0.3, leading to a sampled collection of ∼260K.

heading: C.2 General Task Collection
text: In the second tuning stage, we include a collection of general instruction-tuning data to keep the LLM capability in some NL-centric settings and further improve the instruction-following capability of Symbol-LLM.
The general data collection contains ∼ 570K samples, which are sourced from the following three parts:
(1) Sampled Flan collection (Longpre et al., 2023) of 150K samples. We obtain the collection directly following Tulu (Wang et al., 2023b).
(2) Code Alpaca collection (Chaudhary, 2023) of 20K samples. In fact, this collection is not in an NL-to-NL form as we expected. However, it stresses much on the instruction-following capabilities, which may help enhance the general ability of LLMs. Also, it is expected to act as the bridge between NL data and symbolic form (i.e., code in this case).
(3) Sampled WizardLM collection (Xu et al., 2023a) of 143K samples. To further expand the diversity of our instruction-tuning collection, we leverage the evol-data from WizardLM.

heading: D Data Format
text: To support the instruction tuning, each piece of data i in the training collection contains three parts, i.e., instruction si, input xi, and output yi. During the training process, instruction si and input text xi are concatenated as the whole input sequence. The model is optimized to generate output yi. One example in the FOLIO dataset is as follows: [Instruction] Transform the natural language sentence into first-order logic forms. [Input] All people who regularly drink coffee are dependent on caffeine. [Output] ∀x (Drinks(x) → Dependent(x))
In the implementation, we rewrite the instruction for each sample by prompting GPT-4, keeping the diversity of the instruction.

heading: E Test Datasets and Benchmarks
text: Our main experiments are conducted on both text-to-symbol tasks and general NL-centric tasks. Then this work also extends the scope to Symbol+Delegation setting, which uses LLM to generate symbolic representation and delegate the reasoning process to the external solver. Such a setting satisfies our expectation to build a better symbol interface.

heading: E.1 Tests in Text-to-Symbol Generation Tasks
text: Planning These tasks involve controlling the robot to finish some tasks in the defined environments. The input is the natural language description of the initial states and the final goals, while the symbolic output in the Planning Domain Definition Language (PDDL) form can be executed by the symbolic planner. For the four settings, Blocksworld involves stacking blocks in order. Termes involves moving blocks to a specific position in the grid. Floortile is to color the floors with the instructions. Grippers is to gripper and move balls from room to room. We use the BLEU metric to measure the correctness of generated forms.
SQL They cover three representative Text-toSQL datasets, Spider, Sparc and Cosql. Given the schema and the natural language query, the output is the corresponding SQL. We use the exact match as the metric.
KG / DB It is similar to the Text-to-SQL tasks, which require generating the symbolic form of the query given the natural language question and schema. But WebQSP and GrailQA leverage the sExpression form while CompWebQ uses SPARQL format. We use the F1 metric for WebQSP and the exact match metric for GrailQA and CompWebQ, following previous work (Xie et al., 2022).
AMR They are classical semantic parsing tasks, where the input sentence is parsed into an abstract syntax graph. We use the Smatch metric to measure the generated form on AMR3.0, AMR2.0, and BioAMR datasets.
Ontology It focuses on the domain of knowledge graph construction. Given the ontology (i.e, predefined relations or entities) and natural language sentence, it is required to output the triples. We employ F1 scores introduced in (Mihindukulasooriya
et al., 2023) to measure the performances on Tekgen and WebNLG.
API These tasks require the output of the API calling form based on the natural language query. MTOP and TOPv2 cover various domains like controlling the music player, and setting alarms. NLMAPS focuses on calling the maps.
Command SCAN involves outputting action sequences based on the commands to control robots. The exact match metric is used to measure the generation accuracy.
Code It involves five representative programming languages, including Bash, Regular Expression, Python, Java and GO. They are tested with the BLEU metric.
FOL It covers three datasets in NL-to-FOL domain, that is FOLIO, MALLS and LogicNLI. Logic Equivalence (LE) is leveraged as the metric, following (Yang et al., 2023).
Visual Three multi-modal question answering datasets GQA, Clevr and Geometry3K are included for test. In these scenarios, we only focus on the natural language parts and transform the natural language query into function symbol forms. The exact match metric is used to measure the performances.
Math As we discussed, transforming the natural language question into Python code is one of the faithful ways to solve math problems. Hence, we measure the accuracy of the generated Python code with the BLEU metric. The ground-truth code is derived by prompting GPT-4, where the ones that can execute the correct answer are preserved.
AI4Science In CheBi dataset, the model is required to generate the correct molecular formula given the natural language descriptions. Exact match metric is used for measure.

heading: E.2 Tests in General Tasks
text: MMLU It covers 57 tasks including different subjects STEM, humanities, social sciences, and others. Our evaluations are based on (Hendrycks et al., 2021a).
Big Bench Hard The benchmark is designed for testing LLM capability in challenging reasoning
tasks. We select 21 tasks in BBH for the test, based on Open-LLM-Leaderboard4.

heading: E.3 Tests in Symbol+Delegation Setting
text: Math Reasoning We generate Python code with Symbol-LLM and use Python interpreter as the delegation. The datasets include GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GSM-Hard (Gao et al., 2023), SVAMP (Patel et al., 2021), Asdiv (Miao et al., 2020), AddSub (Hosseini et al., 2014), SingleEQ (Roy et al., 2015), SingleOP (Roy et al., 2015) and MultiArith (Roy and Roth, 2015). The former two datasets are indomain, while the latter seven datasets are under OOD settings.
Note that MATH dataset includes various ground-truth answer formats (e.g., with diverse units), thus it is difficult to parse the correct values to evaluate the LLMs. Hence, we use manuallycrafted templates to derive the ground-truth values, leading to around 4,000 samples for test.
Symbolic Reasoning Same as math reasoning, we use Python code + Python interpreter to solve the problems. Two OOD tasks are used for test, i.e., Colored Objects (Suzgun et al., 2023) and Last Letter Concatenation 5.
Logical Reasoning We take three representative datasets into consideration, i.e., FOLIO (Han et al., 2022), ProofWriter (Tafjord et al., 2021) and ProntoQA (Saparov and He, 2022). We follow the strategy proposed in (Pan et al., 2023) to conduct the reasoning. Detailedly, for FOLIO, we generate FOL representations first and delegate the solution to the FOL solver. For ProofWriter and ProntoQA tasks, we generate logic programming language and delegate the reasoning to Pyke expert system.
Robotic Planning For robotic planning tasks, we transform the natural language description into PDDL and use fastdownward (Helmert, 2006) as the symbolic solver. Besides the four datasets mentioned in text-to-symbol generation tasks, we also employ two OOD datasets into account, i.e. Barman and Tyreworld.
Visual Question Answering We further extend the application scope of Symbol-LLM to the multimodal domain and test on Geometry3K dataset (Lu
4https://huggingface.co/spaces/HuggingFaceH4/ open_llm_leaderboard
5Test data is based on: https://huggingface.co/ datasets/ChilleD/LastLetterConcat
et al., 2021) for illustration. But we only concentrate on the processing of the NL part. Detailed, we parse the natural language sentence into logic forms and rely on the baseline method (Lu et al., 2021) to conduct the multi-modal reasoning.

heading: F Experimental Settings
text: In the implementation, this work fully finetunes the models with 8 * A100 (80GB) with maximum sequence length set to 4,096. The tuning process is optimized and accelerated by deepspeed zero3 and FlashAttention2. We leverage the AdamW optimizer with a learning rate of 2e-5 for both Injection and Infusion stages. The learning rate schedular is set to Linear. The epoch number is set to 1 for both stages. In the Injection stage, the model weights are initialized from LLaMA-2-Chat and the tuned model is named Symbol-LLMBase. In the Infusion stage, we initialize the model from Symbol-LLMBase and obtain Symbol-LLMInstruct at last. These settings are consistent for both 7B and 13B variants.
The inference process is conducted under a single GPU of A100 (80GB) with greedy search (beam size=1).
For a comprehensive evaluation, we include the following strong baselines. They are categorized into Close-source and Open-source ones:

heading: Close-source Baselines
text: • GPT-3.5 We access OpenAI API to call the model. Specifically, GPT-3.5-turbo version is employed for evaluation across a wide range of tasks.
• Claude-1 We access Anthropic API to call the model. We select Claude-instant-1.2 version for evaluation.

heading: Open-source Baselines
text: • LLaMA-2-Chat Since Symbol-LLM is initialized from LLaMA-2-Chat, we include it as the baseline. In general, LLaMA-2-Chat series is regarded as an excellent NL-centric interface for interaction and reasoning, which exhibits great performance on vast NL tasks.
• Single SFT We conduct SFT on LLaMA-2Chat models for tasks in one specific domain. The obtained models can fully overfit the single domain, thus serving as a strong baseline for comparison.
• CodeLLaMA-Instruct Based on the origin LLaMA-2 models, the CodeLLaMA series is continually pretrained and finetuned with code data. Considering code is one of the specific symbols in our work, we include it as one of the strong baselines. For balanced capabilities in general tasks, we leverage CodeLLaMAInstruct for evaluations.

heading: G Overall Performances
text: Figure 7 presents the overall performance comparison among baseline models. It intuitively demonstrates the obvious advantages of Symbol-LLM on the wide range of tasks. Also, it supports our claim to make Symbol-LLM a balanced foundation LLM on symbols and NL.

heading: H Results on OOD Symbolic Tasks
text: We supplement the experiments and compare the performances on 12 new OOD symbolic tasks. These OOD tasks can be divided into two folds: (i) novel tasks with seen symbolic forms (i.e., outof-distribution); (ii) novel symbolic forms (i.e., outof-domain). The results are presented in Table 6.
From the results, Symbol-LLM shows consistent and significant superiority, compared with LLaMA2-Chat.

heading: I Results on Extensive General Tasks
text: In Table 7, we select extensive general tasks for comparison. According to OpenCompass (Contributors, 2023), these tasks are divided into several categories, covering Examinations, Knowledge, Understanding and Reasoning. Considering the computation cost, we only report the performances of 7B models. The major takeaways are as follows:
Symbol-LLM demonstrates better overall performances compared with LLaMA-2-Chat. Generally speaking, Symbol-LLM wins more of the tasks than LLaMA-2-Chat. Such superiority is consistent with the findings in MMLU and BBH benchmarks in Table 2. It illustrates that SymbolLLM can serve as a solid foundational model, significantly enhancing its symbolic capabilities while maintaining its generality.
Optimization empowers Symbol-LLM with improved understanding and reasoning abilities. Among the four task categories, Symbol-LLM is
particularly better at understanding and reasoning, beating LLaMA-2-Chat on almost all tasks. Such findings are intuitive because text-to-symbol can be regarded as an abstract form of NL, which enriches the understanding abilities of the model. Meanwhile, the generation of some symbolic forms (e.g., code) involves the implicit reasoning process, which is actually similar to the chain-of-thought strategy. To this end, the superior reasoning capability is within our expectations.

heading: J Results on Symbol+Delegation Setting
text: In the main paper, we present the results of math reasoning under the Symbol+Delegation paradigm. Next, we will provide the remaining 5 scenarios, i.e., symbolic reasoning (J.1), logical reasoning (J.2), robotic planning (J.3), visual question answering (J.4) and table question answering (J.5).

heading: J.1 Symbolic Reasoning
text: In symbolic tasks, we also adopt the Python code as the generated symbolic representations, and lever-
age a Python interpreter to conduct the reasoning. Two representative tasks, Colored Objects and Last Letter Concatenation are selected for testing under the zero-shot setting.
From the results in Table 8, the Symbol-LLM series are competitive in both tasks. Notably, even Symbol-LLM-7B shows over 10% superiority over GPT-3.5-turbo in Colored Object task. It is worth noticing that LLaMA-2-Chat models underperform consistently in Last Letter task. Since samples in this dataset share similar forms, the model tends to fail if the model does not master the techniques required for solving it.

heading: J.2 Logical Reasoning
text: In logical reasoning tasks, we take three tasks into consideration, i.e., FOLIO, ProofWriter and ProntoQA. For the FOLIO task, Symbol-LLM first transforms the natural language into FOL forms and delegates the solution to the FOL solver. ProofWriter and ProntoQA are represented in logic programming language and integrate Pyke expert
system for deductive reasoning. Results are listed in Table 9. Symbol-LLM-7B series performs relatively better than 13B counterparts. Among all three tasks, the Symbol-LLM-7B series outperforms GPT-3.5-turbo with large advantages. In comparison with the SOTA model Logic-LM, which is based on off-the-shelf LLMs, Symbol-LLM also wins the ProofWriter tasks, with 5%-6% improvements.

heading: J.3 Robotic Planning
text: In the field of robotic planning, Symbol-LLM first transforms the natural language description into PDDL forms and relies on the fast downward solver to give the faithful action sequence.
In total, we select 6 different robotic settings to verify the proposed method. Results are presented in Table 10. Among four in-domain tasks, SymbolLLM performs pretty well compared with strong baselines, achieving the best results in most cases. Even with GPT-3.5-turbo and Claude-1, both our 7B and 13B series win 3 (out of 4) tasks. However, it struggles a lot in OOD tasks. Only in Tyreworld scenario, Symbol-LLMInstruct-13B achieves the best result, beating all close-source and opensource baselines. It is required to state that these selected robotic planning tasks are very challeng-

heading: Models Del. ColoredObject LastLetter

heading: Models Del. FOLIO ProofWriter PrOntoQA
text: ing, given the length and rigor requirements of the generated programming language. Even closesource LLMs fail in some scenarios. Therefore, we argue it is still an open question for future studies.
J.4 Visual Question Answering
We also explore our potential in the multi-modal scenario. Geometry question answering is selected as the task for the test. Note that the understanding of image is not within our scope, we only focus on the text part and transform the natural language query into logical forms. Then the solution is delegated to the off-the-shelf baseline methods. Comparison results are shown in Figure 8.
The top red bar means the performances using the annotated logic forms from the text. Note that since the utilized delegation method is a neuralbased baseline just for a simple evaluation, the upper boundary does not represent the boundary of this task. From the figure, Symbol-LLM variants are approaching it and significantly outperform all the other baselines.

heading: J.5 Table Question Answering
text: Table (or database) question answering is also a hot topic in recent years. Thus, we select two OOD tasks WikiSQL and WikiTQ for evaluations. The natural language query is first transformed into an SQL query and it is executed by an SQL solver over the given tables or databases under the zero-shot setting. We report experimental results in Table 11.
Symbol-LLM series are consistently superior to all open-source and close-source baselines, with over 40% margins in WikiSQL and 3%∼14% advantages in WikiTQ.

heading: K Supplementary Experiments

heading: K.1 Tuning Design Choices
text: Since our motivation is to build a symbolic foundation for current LLMs, we select the full fine-tuning strategy in the implementation. This part provides a comparison with different tuning design choices: (1) LoRA fine-tuning and (2) curriculum learning. In Figure 9, the performances on 34 symbolic tasks are presented.

heading: Models Del. WikiSQL WikiTQ
text: LoRA tuning To inject new symbolic knowledge and lay a symbolic foundation for LLM, adapter tuning (e.g., LoRA) is not sufficient compared with the full fine-tuning design.
Curriculum learning Based on the performances of LLaMA-2-Chat on these 34 tasks, we rank the difficulty level of tasks, forming the training order of curriculum learning (from easy to hard tasks). From the results, curriculum learning performs 4%-5% worse than our design choice.

heading: K.2 Comparison with Single Domain SFT
text: As discussed above, one of our hypotheses is that various symbols share underlying interrelations, though they are in quite different forms. Thus, we expect that the learning of symbolic knowledge will mutually benefit each other if they are treated in a unified manner.
We present the comparison results between single SFT and unified SFT in Figure 10. The light
blue bar denotes the single domain SFT while the dark blue one is the unified SFT. We categorize the text-to-symbol generation tasks into 12 task domains, according to their similarity in symbolic forms. Within one domain, all tasks are tuned together and the performances on test splits are averaged as the single SFT results. To reduce the effect of the tuning strategy, we utilize the SymbolLLMBase model to measure the results on the unified SFT setting. Each sub-figure in Figure 10 corresponds to one specific domain.
In most domains, unified SFT is superior to single-domain SFT. Larger gains are observed in some uncommon symbolic forms, such as PDDL for planning tasks and molecular formulas in AI for Science scenarios. It presents the possibility that unified SFT on various symbols may help extend the model coverage to low-resource cases. It is also worth noting that in some cases, single-domain SFT performs a little better than Symbol-LLMBase. This is because purely overfitting on specific symbolic forms with powerful LLMs is usually easy to get promising results.

heading: K.3 Extrapolating to New Symbols
text: In the above section, we introduce symbol-evol strategy to expand sample diversity and facilitate the training of instruction-following ability. Following this strategy, we can also automatically generate abundant novel instructions to extrapolate to new symbols. To this end, we further evaluate Symbol-LLM by following novel instructions.
The experiments are based on Clevr and SCAN tasks. Applying symbol-evol strategy, we obtain Clevr-evol and SCAN-evol datasets. Evaluation
results are presented in Figure 11. From the results, the more complex setting (i.e., green bar) does not induce a significant decrease in model performance. Especially, in the Clevr task, Symbol-LLM even does better given the novel instructions. It uncovers that Symbol-LLM follows the instructions during the reasoning process, instead of merely memorizing the specific symbolic forms.

heading: L Analysis: Alignment and Uniformity
text: Beyond performances on symbolic tasks, it is also required to reveal what leads to superiority. Inspired by (Wang and Isola, 2020; Yuan et al., 2023), we extend the ideas of Alignment and Uniformity to evaluate the model perception of symbolic knowledge. Alignment 6 is utilized to measure the interrelations between symbolic forms. Uniformity quantifies the degree of evenness or uniformity in the distribution of symbolic representations. The concept of a uniform feature distribution is valuable as it encourages a higher information entropy, representing more information retention.
Alignment Different from the original implementation (Wang and Isola, 2020) which considers positive pairs in the contrastive learning, this work takes the symbolic sequences under the same symbolic form as the positive pairs. For any symbolic form X, their data distributions are referred to as PX . The alignment within X can be measured with the following formula:
Lalign(X) = E x1,x2 i.i.d∼ PX ∥f(x1)− f(x2)∥2 , (3)
where x1 and x2 are samples from the specific symbolic form X. f(·) returns the LLM embeddings of the symbolic sequences. ∥·∥ returns the norm of the vector.
In the implementation, we select 16 main symbolic forms and sample 100 symbolic sequences for each form to measure the alignment. f(·) leverages the mean pooling representation of the last hidden states of the LLM. We average all the alignment scores from the 16 symbols to obtain the final one. Notably, we employ logarithmic operations on the Alignment loss to reduce scale, without impacting their relative comparison.
6Here, Alignment refers to the concept in contrastive learning, but is not related to the alignment technique in LLMs.
Uniformity Apart from alignment, we also calculate the uniformity of the LLMs on symbolic sequences. The evaluation of the uniformity Luniform is implemented by the following formula:
Luniform = log E x,y i.i.d∼ Pdata e−2∥f(x)−f(y)∥ 2 , (4)
where the data distribution Pdata covers all the symbolic sequences. f(·) also utilizes the mean pooling representation of the last hidden states of the LLM.
Leveraging the above definitions, further analysis and comparison on Symbol-LLM are conducted. The item-wise conclusions are listed as follows: (1) Symbol-LLM optimizes symbol distinctiveness and overall expressiveness in the embedding space (superior Alignment and Uniformity).
Based on the equation 3 and 4, we can assess the proficiency of LLMs in handling symbols. Figure 12 presents the visualization of AlignmentUniformity. The x-axis stands for uniformity while the y-axis is the alignment. Both of these metrics are better when kept as small as possible.
From the figure, Symbol-LLMInstruct models perform consistently better than the original LLaMA2-Chat models, with obvious merit in Alignment and Uniformity. It can be regarded as an in-depth explanation for the superior performances on symbolic generation tasks. Further, it witnesses that the two-stage tuning framework actually corrects the weakness of Uniformity under 7B settings (SymbolLLMInstruct v.s. Symbol-LLMBase).
Both metrics are well optimized with the proposed two-stage tuning framework as well as the symbolic data collection. For Symbol-LLMBase models, though the 7B version witnesses some loss in Uniformity, they consistently achieve superior alignment. (2) Symbol-LLM excels at capturing symbolic interrelations.
The above calculation of Alignment roughly depicts the similarity among samples under the same symbolic form. To this end, we extend the idea of Alignment to measure the interrelation between any two symbolic forms X and Y. The score S(X,Y ) is calculated based on the following formula:
S(X,Y ) = E x∼PX ,y∼PY ∥f(x)− f(y)∥2 , (5)
where x is one symbolic sample in the form of X, while y is one sample in the symbolic form Y. We binarize the scores with the manually defined
threshold for a more intuitive illustration. We ensure the same threshold under a fair comparison of the same model size. And the set of thresholds will not affect the overall conclusion.
The visualization is presented in Figure 13, where the dark blue denotes the closer relation in the representation while the light one is the opposite. We make comparisons between the original LLaMA-2-Chat models and Symbol-LLMInstruct models, separately for the size of 7B and 13B.
For the original LLaMA model (Figure 13a and 13c), the representations between different symbols exhibit significant sparsity. There are only three pairs of symbolic forms that effectively demonstrate the interrelations in the embedding space, i.e., AMR-PDDL, AMR-SPARQL and CLEVR-NLMaps. Also, under several symbol systems (e.g., Bash, FOL), the representation space of samples is also very scattered. The above observations demonstrate that previous foundational LLMs (i.e., LLaMA-2-Chat) lack the ability to capture the interrelations among symbolic systems.
In comparison, Symbol-LLMInstruct series models excel at reflecting the interrelations between symbols. As presented in Figure 13b and 13d: 1) Symbols exhibiting potential connections are effectively aligned within the representation space, i.e., Python-AMR and CheBi-RX. 2) Samples within each symbol are pulled closer together.
Combining the above two observations and analysis, the superior performances of Symbol-LLM on the symbolic generation tasks are sourced from better alignment among symbols in the embedding
space as well as the optimized uniformity.

heading: M Prompting Details
text: This section provides the zero-shot prompting details during the Symbol-LLM test. Totally, 34 symbolic tasks are covered.
Planning prompts for planning domain cover 6 tasks: Blocksworld, Termes, Floortile, Grippers, Barman, and Tyreworld.
Given the description of a scene, please transform the part of the scene from natural language into PDDL file. The defined atom is: [Atom Description] The natural language description is: [NL Description] The PDDL file to this problem is:
SQL prompts for SQL domain cover 3 tasks: Spider, Sparc, and Cosql.
Using valid SQLite, transform the question into SQL form for the tables provided above. The database schema is: [Scheme] The question is: [NL Query] The corresponding SQL for the question is:
KG/DB prompts for KG domain cover 3 tasks: WebQSP, GrailQA (s-expression), and CompWebQ (SPARQL).
Given the knowledge graph, give me the sExpression/SPARQL for request. The given knowledge graph is: [Serialized KG] The request is: [NL Query] The corresponding s-expression/SPARQL for the request is:
AMR prompts for AMR domain cover 3 tasks: AMR 3.0, AMR 2.0, and BioAMR.
Transform the natural language sentence into AMR form. The input sentence is: [NL sentence] The AMR to this sentence:
Ontology prompts for ontology domain cover 2 tasks: TekGen and WebNLG.
Given the following ontology, extract the triples from the sentence. In the output, split each triple with ’|’. The ontology concepts are: [NL Concepts] The ontology relations are: [NL Relations] The sentence is: [NL sentence] The output triples are:
API prompts for API domain cover 3 tasks: MTOP, TOPv2 and NLMaps.
Given api calls, output the TOP Representation for the request. The candidate api calls: [candidate API calls] The request is: [NL Query] The corresponding TOP representation for the query is:
Transform the natural language question into the formal language to query the map apis. The natural language question is: [NL Query] The corresponding formal language to query the map apis is:
Command prompts for Command domain cover SCAN task.
Transform the natural language sentence into action sequence from the candidate actions. The candidate actions are: [Candidate Actions] The NL sentence is: [NL sentence] The corresponding action sequence for the query is:
Code prompts for Code domain cover 5 tasks: NL2BASH, NL2RX, NL2Python, NL2Java, and NL2GO.
Generate the [Programming Language] code given the natural language instruction. The natural language question is: [NL Query] The corresponding code to this question:
FOL prompts for FOL domain cover 3 tasks: FOLIO, MALLS, and LogicNLI.
Transform the natural language sentence into first-order logic forms. The input sentence is: [NL sentence] The corresponding first-order logic for the request is:
Visual prompts for Visual domain cover 3 tasks: GQA, CLEVR, and Geometry3K.
Transform the natural language question into logical functions. Each part of the logical functions starts with the specific operation and they are connected with –>. The candidate operations include (but not limited to): [candidate operations] The natural language question is: [NL Query] The corresponding logical function for the question is:
Transform the natural language question into logical functions. The logical functions are organized in the compositional forms of operations. The candidate operations include: [candidate operations] The natural language question is: [NL Query] The corresponding logical function for the question is:
Transform the description of geometric questions into the formal language in the logic form. We define the following valid predicates. [candidate predicates] The natural language question is: [NL Query] The corresponding logical function for the question is:
Math prompts for Math domain cover 3 tasks: GSM8K, MATH and AQUA.
Write Python code to solve the question. The question is: [NL Query] The solution code is:
AI4Science prompt for AI4Science domain cover CheBi task.
Generate the SMILES (Simplified Molecular Input Line Entry System) representation from the natural language description of the molecular. The molecular is: [NL Description] The corresponding SMILES representation is:

heading: Future Work
text: Therefore, we argue it is still an open question for future studies. J. 4 Visual Question Answering
We also explore our potential in the multi-modal scenario. Geometry question answering is selected as the task for the test. Note that the understanding of image is not within our scope, we only focus on the text part and transform the natural language query into logical forms. Then the solution is delegated to the off-the-shelf baseline methods. Comparison results are shown in Figure 8. The top red bar means the performances using the annotated logic forms from the text. Note that since the utilized delegation method is a neuralbased baseline just for a simple evaluation, the upper boundary does not represent the boundary of this task. From the figure, Symbol-LLM variants are approaching it and significantly outperform all the other baselines