RuAG: Learned-rule-augmented Generation for Large Language Models
Authors: Yudi Zhang
Pei Xiao
Lu Wang
Chaoyun Zhang
Meng Fang
Yali Du
Yevgeniy Puzyrev
Randolph Yao
Si Qin
Qingwei Lin
Mykola Pechenizkiy
Dongmei Zhang
Saravan Rajmohan
Qi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
Logic Rule Learning
Monte Carlo Tree Search
Primary area: interpretability and explainable AI

Abstract: In-context learning (ICL) and Retrieval-Augmented Generation (RAG) have gained attention for their ability to enhance LLMs' reasoning by incorporating external knowledge but suffer from limited contextual window size, leading to insufficient information injection. To this end, we propose a novel framework to automatically distill large volumes of offline data into interpretable first-order logic rules, which are injected into LLMs to boost their reasoning capabilities. Our method begins by formulating the search process relying on LLMs' commonsense, where LLMs automatically define head and body predicates. Then, we apply Monte Carlo Tree Search (MCTS)  to address the combinational searching space and efficiently discover logic rules from data. The resulting logic rules are translated into natural language, allowing targeted knowledge injection and seamless integration into LLM prompts for LLM's downstream task reasoning. We evaluate our framework on public and private industrial tasks, including Natural Language Processing (NLP), time-series, decision-making, and industrial tasks, demonstrating its effectiveness in enhancing LLM's capability over diverse tasks.

Published as a conference paper at ICLR 2025
RuAG: LEARND--RULE-AUGMENTDD GENERATION FOR LARGE LANGUAGE MODELS
Yudi Zhang', Pei Xiao", Lu Wang', Chaoyun Zhang', Meng Fang", Yali Du Yevgeniy Puzyrev', Randolph Yao", Si Qin', Qingwei Lin', Mykola Pechenizkiy! Dongmei Zhang', Saravan Rajmohan', Qi Zhang'
'Eindhoven University of Technology "Peking University 'University of Liverpool
*Microsoft
'King's College London
ABSTRACT
In-context learning (ICL) and Augmenall-ugmentted Generation (RAG) have gained attention for their ability to enhance LLMs' reasoning by incorporating external knowledge but suffer from limited contextual window size, leading to in- sufficient information injection. To this end, we propose novel framework RuAG to automatically distill large volumes of offline data into interpretable first-order logic rules, which are injected into LLMs to boost their reasoning capabilities. Our method begins by formulating the search process relying on LLMs' com- monsense, where LLMs automatically define head and body predicates. Then, RuAG applies Monte Carlo Tree Search (MCTS) to address the combinational searching space and efficiently discover logic rules from data. The resulting logic rules are translated into natural language, allowing targeted knowledge injection and scamless integration into LLM prompts for LLM's downstream task reason- ing. We evaluate our framework public and private industrial tasks, including natural language processing, time-series, decision-making and industrial tasks, demonstrating its effectiveness in enhancing LLM's capability over diverse tasks. Project link: https::/ /tt///ggttttttt www.mooocciiooooooo.eoo wtt.:/ccooooot///mi....
INTRODUCTION
How learn from external data?
(a) Supervised Fine-Turme Fine-Tuning: Parameter Tuning
Inserted knowledge edde
Computational Cost
Rule
Temperature
Sunny
Lee ming
Wind speed 20 OR Rainy
External Data
(b) Ours: Leernabie-rull--agmeettt generation
(c) In-Context Learning Lemminng/PPerriresssuggnnnne genersiion
Figure I: Comparison of supervised fine-tuning, in-context leaming/retriival--aggge generation, and our proposed learned- weaeed-rul-uuggneee augmented generation (RuAG), which injects logic knowledge to boost generation while reducing computational cost.
Leveraging external datasets enhance the performance of pretrained Large Language Models (LLMs) on downstream tasks has become a significant focus in recent research (Brown et al. 2020a Hu af., Fan al. 2024 Dong et al. (002). Methods such as supervised fine-tuning (SFTT (Hu
et al.) 2021: Li & Liang 2021). in-context learning (ICL) (Dong et al. 2022; Wang et al.) 2020
Ravi & Larochelle 2016; Chan et al.) 2022 Fang et al. 2024), retrieval-augmente generation
"Equal contribution. Work done during the internship of Yudi and Pei in Microsoft. 1 "Correspondence Lu Wang <whn@microsoft.com Meng Fang <Meng. liverpool.ac.uk>.

---

Published as a conference paper at ICLR 2025
(RAG) ( Izacard et al., 2023; Fan et al., 2024), and the utilization of knowledge graphs (KGs) (Pan
et al. 2024; Shu et al., |024 Wang et al. al.] 2024) have been explored to incorporate external knowl-
edge into LLMs (Digg et al. 2023 Zhang et al. 2024), enhancing their reasoning and decision-
making capabilities.
Despite these advancements, these methods face notable challenges. Fine-tuning large LLMs on extensive datasets is computationally intensive and time-consuming, often leading to overfitting and catastrophic forgetting (McCloskey & Cohen (989). ICL relies on handcrafted demonstrations and templates that may not effectively summarize large volumes of data, leading to inefficiencies and the "needle in a haystack" problem when processing long contexts (Li al. 2024)) and the ex- tremely long context window significantly increases computational costs (Peng et al. 2024 Naveed et al., 2023). RAG depends heavily on the quality and relevance of retrieved documents and faces computational hurdles when integrating large-scale retrieval into prompts (Fan et al. 2024). Thus, RAG is not able to use the whole of vast knowledge base. KG-based methods incorporate structured representations of knowledge to improve LLMs' understanding and reasoning (Pan et al. 20224] Shu
ct al. 2024 Wang et al. 2024). While KGs can enhance decision-making by providing explicit relational data, constructing and maintaining them requires significant manual effort and domain expertise, making scalability challenging.
These challenges underscore the urgent need for efficient knowledge transformation to en- hance LLMs' understanding. Logic rules, with their high information density, act as a promising bridge between vast, diverse data types (including numerical, textual, and vi- sual data) and LLMs' understanding. Previ- ous work has demonstrated their learnability from external data and their efficiency in providing explanations to enable transparent Al processes (Qu & Tang, 2019, Qu et al.). logic rule, as shown in Figure 2 typically expressed as -> h, indicates that if a set of events a (referred to as body predicates) occurs, then the event (called the target predicate) will also occur. As an example, the logic rule "Temperature > 30 AND Humidity S 50 + Sunny Day" represents knowl- edge in symbolic structures, suitable for learning from data. Additionally, this rule can be easily translated into natural language: "If the temperature is 30 degrees or higher and the humidity is 50 percent or lower, it will be a sunny day." Logic rules are understandable to both humans and LLMs as they encapsulate complex relationships in concise, structured form. Unlike lengthy text passages or extensive datasets in ICL. and RAG, logic rules distill essential information into clear, interpretable statements. Compared the complex node-and-edge structure of KGs, logic rules re- duce cognitive load and align better with LLMs' natural language training. Their direct translation into natural language further improves alignment with LLMs, facilitating more efficient processing and understanding.
Body Predicates
Targe Predicates
Temperature
AND
Humldity 50
Sunny
Wind_speed
Cloud_ coverage
Expected
Logical connective
Figure Illustration of logic rules, which are uti- lized to guide LLMMs generation in this paper.
Inspired by this, we propose a novel framework, raareed-rule-augmmentedd generation (RuAG), to automatically compress large external data into logic rules through LLM-aided Monte Carlo Tree Search (MCTS) (Swiechowski et al. 2023) and then inform L.MMs domain expertise by applying translated logic rules into prompts. Our framework consists of the following three phases. LLM- based Logic Rule Search Formulation: Learning logic rules is expensive due to the involved hu- man effort in formulating the domain-specific search process. Therefore, we automate this process by relying on LLMs' commonsense to define the target and body predicates in logic rules. First, the target predicate is defined to be relevant,,, like a class label in classification task or a game state labeled as "'win", while the body predicates are initialized Then, given the task and dataset descriptions, I.LM generates new target predicates and eliminates most irrelevant data attributions from the body predicates. For example, in navigation, L.MSs may infer some special place as the key steps to the destination and suggest to search the rules for agents reaching the places individually. Also, LLMS may regard some data attributes as irrelevant to the target predicate, thus excluding them from the candidates. Consequently, the logic rule search space can be significantly reduced, and a domain-specific search process can be automatically established. Logic Rule Search with MCTS: Searching rules requires to discover the relationship among the predicates, suffering from the compositional search space (Qu Tang 2019; Zhang et al. [020) Evans & Grefenstette 2018). To this end, RuAG exploits MCTS, which works well in large search spaces, to generate structured and understandable first-order logic rules, which are applied in the
all the data attributions in the dataset.
2

---

Published as a conference paper at ICLR 2025
rule-based generation phase. Soreeed-Rule-Augmeee
Generation: RuAG translates the abstract logic rules into natural language and injects them into LLMs' prompts. By addressing the limitations of SFT, ICL, RAG, and KG-based methods, RuAG offers scalable and computationally efficient solution for integrating extensive domain knowledge into LLMs, improving LLM's reasoning, com- prehension, and task performance with minimal manual intervention.
Our contributions are fourfold. First, we introduce a novel rummeedd-ul-uugmeeette generation framework as a potential alternative SFT, ICL, RAG, and KG-based methods. This framework systematically and nearly automatically compresses external knowledge into compact, interpretable logic rules that prioritize enhancing LLM generation. Second, we propose an automated formulation for MCTS, eliminating the need for manual, domain-specific rule search and enabling a generaliz- able approach applicable across a wide range of tasks. Third, we apply MCTS to efficiently handle the large compositional search space of logic rule discovery. Fourth, we evaluate our framework across diverse scenarios, including public tasks in NLP (relation extraction on DWIE), time-series (log anomaly detection on HDFS), decision-making (the cooperative game Alice and Bob), and an industrial task in abuse detection, demonstrating the effectiveness of our approach in both academic and real- weoldworld settings.
2 RELATED WORK
In this section, we review the most relevant topics related our work, including the techniques to exploit external data in LLMs and logic rule learning.
External data usage in LLMs. There are several ways to inject external knowledge into large lan- guage models. The most common way is distilling knowledge from external data, but it suffers from high computational costs (Xu et al. 2024 Gu et al.] ZZhagg et al,, 2025). In-context learning (Brown al. 2020a prompts LLMs with few handcrafted demonstrations which are understandable for the LLMs. More fancy, Retrieval-Aul Augenal---uueeeeddeee Generation (Chn) et al. 2024a) complements
LLMs by retrieved relevant knowledge from external databases (Li et al.) [023] Shen et al.
2023
Chen et al. 2024c) or constructing demonstrations for in-context learning (ICL) (Poesia et al. 2022
Agrawal et al.) 2023), showing promise in tasks like OpenQA (Borgeaud et al. [022]] Guu et al.
2020) and games (Zhu et al. 2023a Hu et al. (024). Knowledge graphs are welcome in external
knowledge formats well, especially in structured tasks like relation extraction and entity recogni- tion (Shu et al.) 2024, Wang et al. 2024). improving task-specific decisions. Recent research also investigates how LLMs can summarize logic rules from large datasets serve as a knowledge stor- age (Zhu et al. 2023b Luo et al.) 2023). but shows high computational costs due to frequent calls
to commercial LLMS (Brown et al. 2020b OpenAl
2023).
Logic rule learning. Logic rules are increasingly employed to enhance the interpretability and ac- curacy of matisiiinmmaiigg// AI systems (Chiu al.] [223] An et al., (024)). Manually defined logic rules have been used to describe how certain events or outcomes are triggered by predefined condi- tions. However, this process labor-intensive and highly domain-dependent (Evans & Grefenstette 2018 Li Lietal al. 2020). Researchers have explored automatic methods for extracting logic rules, such as statistical approaches and likelihood estimation (Cheng et al. 2022 Qu et al. Ru et al.) [021). Despite these advances, the process still involves extensive domain knowledge and commonsense reasoning, requiring expert intervention to identify the candidate target and body predicates.
3 ENHANCE LLMS' REASONING THROUGH APPLYING LOGIC RULES
In this section, we introduce RuAG, our novel approach to augment I Large I Language Models (LLMs) with logic rules learned from pre-collected training data. Instead of directly fine-tuning the LLM- which can be costly and prone to overfitting or using retrieval-augmenned generation limited by input length, we transform the data into concise logic rules. These rules encapsulate essential pat- terns and guide the L.M during generation, enhancing performance and interpretability.
As shown in Figure 3 RuAG comprises three key steps: 1) LLM-Based Logic Rule Search For- mulation: leverage the LLM to automatically formulate the logic rule learning problem, defining predicates, actions, states, and rewards. (Section 3.I) 2) Logic Rule Search with Monte Carlo Tree Search (MCTS): employ MCTS to efficiently search for effective logic rules based the
3

---

Published as a conference paper at ICLR 2025
LLM-based Logic Rule Search Formulation
Learned-Rus
Target predicate
Augumented Generation
resk-eelated: Obtain
Body predicate
Input:
LLM suggests more:
Remove Impossible ones:
You A, your task collsborate with 2 to find in the game.
Predicate V:Visit/B V
Stand M) Stand
Stand ). Stand
Predicate 2: Visiitt.. M
Here are some logic rulas you may find helpful:
Pre rre-collected data
Stand/28. Reward 10.0
Predicate loBoy(
Body Predicates Target Predicate
Rule 1:1f A visited and her horizonal distance to is then stands on
Logic Rule Search with MCTS
Given target predicate: Stend(o4t,
Rule 2: visited and verticel.
State: rute/ = some body predicates: [Visitie4 DisX pm G..
distance to is -2, then stends
Action: add new predicate into &: [Visit, Disk ...J- DisY/B
Reward: F1-score of applying the rule in the pre-collected data
Rule 3:
Set ( & At. )
Searched
[Current Observation]
Visitwa
1. Visit(at.
@ & DisXLut,
Please response with
Visit/B, m) DisY(B, mm.28 Stand M)
DisX
3.
Response: turn left,
A: Alice
:B: Bob
Treasure
Blocks with different colors
Stand/,4. //Visitt M): A stands on/visited
DisK
F- it's's horizonal distance to @ is DisY(B,
( tB's vertical distancce -2.
generation (RuAG). RuAG automati- cally compresses large external knowledge into compact logic rules using LLM-aided Monte Carlo Tree Search (MCTS), through three phases: LLM-based Logic Rule Search Formulation, Logic Rule Search with MCTS. and the MCTS search by defining the target and body predicates. Then we apply MCTS to generate structured first-order logic rules, which are applied guide generation. Our framework provides an efficient alternative to RAG.
Figure 3: The framework of our novel saanee-rull--uggeeeee
Rule-Augmented Generation. First, the LLM formulates
Table I: Task-relevant target predicates, body candidates and their examples in relation extraction, log-based anomaly detection and cooperative game.
Targef
Task
Training data
Body predicate candidates
predicate
Relation
relations between entities, c.g., A specific All relations except the target predicate, c.g., for target pred- icate inc (A, C), all other relations excepting in0 (A, C).
Extraction
in0 (A, C) (A is in country C) relation.
Anomaly
Pairs of log sequences and Anomaly
Log events, e.g., BS (receiving a bluck).
Detection
anomaly labels (Anomaly)
Observations and actions, c.g.. IeYellow (Alice, Ri Right): Alice's right block is yellow, Move (Bob, Right): Bob moves right.
Cooperative
Triplets of observations, actions GameWin and game win label (SameWin)
Game
3) Learned-Rull-Auggeenn Generation: integrate the learned logic rules into the LLM's generation process, improving its generation. (Section
formulatel problem. (Section 3.2
3.1 FROM DATA RULE SEARCH: LLM-BASED LOGIC RULE SEARCH FORMULATION
Search for logical rules traditionally requires significant human effort, particularly in defining domain-specific head predicates and selecting relevant features that characterize data samples. This process demands domain knowledge and impacts both the quality of the derived logic rules and the computational cost of search. To address this challenge, our method begin with LLM-based Logic Rule Search Formulation, where we leverage the capabilities of LLMs to automatically formulate the logic rule learning problem through defining the predicates.
Initial Predicates. Given a dataset D {(x, y)), where each data sample x [1, .2.... ZN X' is N-dimensional and E {0, }} is the label, we initial the label as the target predicate and the features as the body predicates, as shown in Table [ We can directly translate discrete variables into Boolean values through one-hot vectors. For continuous variables, we can translate them into Boolcan-valuce attributes through Gini-indes (Strobl et al. 2007). Furthermore, we suggest prompt- ing LLMs to remove impossible body predicates to reduce logic rules search space suggest new target predicates to search more logic rules for a better understanding of the task.
Removing Impossible Body Predicates. Given the definition of logic rules and the candidate pred- icates with their descriptions, LLM aids in filtering out impossible or irrelevant body predicates, reducing the computational burden. By utilizing commonsense reasoning, the LIM can identify
4

---

Published as a conference paper at ICLR 2025
predicates that are unlikely to contribute effective logic rules. For instance, in system log anal- ysis, the LLM might determine that certain attributes like user IDs are less relevant for anomaly detection compared to enror codes access patterns. We provide the prompt in Figure A3
Suggesting New Target Predicates. In addition to the primary target predicate (c.g., achieving a specific classification label), the LLM can suggest additional head predicates to explore, given log- ical rule definition, task description, and data schema. This is particularly useful in tasks requiring long-horizon planning, where intermediate goals can guide the search for effective logic rules. By generating these new head predicates, the LLM enables a more comprehensive exploration of the logic rule space. We provide the prompt in Figure A4
Our LLM-based logic rule search formulation enjoys the following advantages:
Automation and Scalability: The LLM automates the setup of the logic rule learning problem,
i.e., defining the target and body predicates, avoiding human experts, and making it scalable to large and complex datasets.
Enriched rule generation: By generating relevant target predicates, our method can extract
more meaningful rules.
Reduced Computational Burden: By eliminating irrelevant predicates, the LLM narrows
down the search space, improving efficiency.
3.2 LOGIC RULE SEARCH WITH MCTS
Following the definition of predicates in logic rule searching, we apply Monte Carlo Tree Searching (MCTS) to perform logic rule learning, inspired by its effectiveness in searching optimal policy in large state spaces.
States, Actions, and Rewards in MCTS. With the predicates defined, the state, action, and reward in MCTS for logic rule searching can be defined as:
States (S): Each state represents a partial logic rule, consisting of a set of predicates. The initial state is the empty set, So = 0. Subsequent states are defined as: S, Sn-1 {at), where a; is the predicate added by an action.
Actions (A)): Actions involve adding a new predicate to the current state. The action space is defined as: A = {Add a: a; is candidate predicate generated by the LLM).
Rewards (R): The reward function evaluates the quality logic rule. For example, the reward for state Sn can be defined as the precision of the rule evaluating on the dataset D.
Typically, MCTS involves building a search tree and simulating outcomes to estimate the value of actions. It consists of four key phases: selection, expansion, simulation, and backpropagation. Selection and expansion. The process begins at the root node, where the algorithm selects the most promising child nodes based on the Upper Confidence Bound applied to Trees (UCT). This continues until a leaf node is reached. If the leaf node is not terminal, new child nodes are created to explore potential moves. As example, (income 2 $50, 000). if we select a new candidate predicate (income 2 $50, 000) according to its UCT value, then we add it into the rule: [(age 2 30), + (income 2 $50, 000) and enter the new state of [[age 2 30), (education = bachelor's) ')] + income 2 $50, 000. $50, Simulation: For the newly expanded nodes, random simulations (also known as rollouts) are performed calculate the reward of the state. Backpropagation: The calculated reward is then propagated back up the tree, updating the nodes' statistical information. The UCT algorithm plays crucial role in MCTS, balancing
expand a new node at the state of [[age 2 30), =
exploration and exploitation by selecting actions that maximize: UCT, X3 CV No. Nc. where /2in
X, is the average reward of action j. .N is the total number of visits to the parent node, is the trade-off.
number of visits to node j. C' is constant that adjusts the exploration-exploiaaie
Finally, we collect all the rules constructed the terminal nodes when 1) the constructed rule reaches a predefined maximum length (i.e., the number of body predicates exceeds threshold).2 2 If the reward of the final node (i.e., the precision of the rule) exceeds predefined threshold, indicating that the rule is sufficiently accurate.
3.3 LEARNED-RUI. AUGMEETTE GENERATION
After the logic rule search, we gather a set of logic rules and follow the following steps to perform leamed-rule-augmented generation. 1) Clean Searched Rules: The collected nules may contain
5

---

Published as a conference paper at ICLR 2025
duplicates, exhibit low quality, cover only limited subset of the data. We first eliminate those with low rewards or minimal data coverage. Then, we compare each pair of rules and retain the one with the higher reward if its body predicates are subset of the other's. s. 2) Translate Rules into Natural Language: To enhance the I.MS'' comprehension, we translate these symbolic rules into natural language, resulting in group of sentences. These sentences LLM prompts to guide generation more effectively. 3) Retrieve Relevant Rules: It is optional to retrieve only the most relevant rules inject all the rules, depending on the contextual window size and the long-text understanding capability of the LLM. 4) Generation: The generator component can be modeled using any LLM. We use GPT-4 (OpenAl (202)) no specific model is clarified. To combine the input with the rules during generation, we simply apply the rules in a prompt template.
then be injected into the
EXPERIMENTS
Most decision-making and prediction tasks be abstracted into state chains achieve their ulti- mate goals, which allows our method to adapt to a wide variety of tasks. In this section, we evaluate our method over diverse domains, including NLP (relationship extraction in Section 4.I), time- series predication (log-based anomaly detection in Section 4.2). decision-makingg task (cooperative game (Chen al., (2024b) in Section
(.3) and a private industrial task (unauthorized party abuse detection in Appendix (A). We compare our method with the domain-specific baselines for each task and LLM--bssed methods, including vanilla, ICL, RAG, and HtT (Zhu et al.) 2023b). HtT shares similar goal of using constructed rule libraries with us to enhance LLM generation.. but relies on LLMs to generate and verify rules, Icading to high computational cost of LLMs. The specific implementation details of the experimental setup can be found in Appendix C
4.1 RELATION EXTRACTION
Decument-level relation extraction is a critical task in natural language processing (NLP), where the goal is to identify and classify relationships between entities across entire documents rather than isolated sentences. This task becomes more complex at the document level due to the larger text and the need to resolve long-range dependencies and co-references between entities scattered throughout the document. However, using only LLMs for this task is often limited by their inabil- ity to consistently capture complex document-wide relationships, especially when reasoning across multiple entities and contexts.
Setup. We conduct experiments on the DWIE dataset (Zaporojets et al. 2021), which contains 802 documents and 130 entities. After excluding irrelevant articles, 700 documents are used for training and 97 for testing. we utilized the LLM to identify and eliminate 15% of the relationships (i.e. appears_in, and player_off that were unlikely to function as valid body predicates, based on their descriptions and the given rule descriptions. We evaluate the performance of our method us- ing standard relation extraction metrics, including Precision, Recall, and Fl-score. For comparison, evaluate our method against several state-of-the-as models for document-lewe relation extrac- tion, including CNN, BiLSTM (Yoo et al. 2211)). Context-Awaid (Sorokin & Gurevych| (2017), and BERT-based models & Lin 2019), which are widely used in document-level relation extraction tasks. Additionally, we compare with the LLM--ased methods: Vanilla, ICL. RAG and HtT
al. 2023b) which employs predefined logical rules to extract relations.
Main Results. As shown in Fig- our method outperforms both deep learning-based and LI LLM-based baselines in document-level relation extraction. Deep learning methods, while achieving decent performance in document-level relation ex- traction, struggle to capture long-range se- mantic dependencies. LLMs, such as GPT- 4, demonstrate superior performance due to their strong natural language understand- ing. For example, ICL achieves an Fl score of 50.26%, outperforming bL-based meth- ods. RAG further improves results (F1:
Table 2: Results on Relation Extraction.
Model
F1
Precision
Recall
CNN 43.78%
47.13%
45.03%
BiL.TTM
48.17%
44.32%
41.53%
DL-based
Bert
49.84%
49.35%
54.13%
Context- Aware
45.37%
49.87%
38.76%
Vanilla
46.94%
69.61%
35.41%
ICL
50.26%
74.09%
38.02%
LLM-based
RAG
52.30%
78.64%
39.17%
H[T
52.59%
68.20%
42.80%
Ours
60.42%
69.44%
53.48%

---

Published as a conference paper at ICLR 2025
52.30%) by retrieving similar cases from training- knowledge base, while HtT extracts rules document by document (F1: 52.59%), which limits its global perspective. Our method addresses these limitations by using MCTS to search for rules globally, mining potential rules efficiently from the entire training data. This ensures reliability during the search process and combines the learned rules with LLM reasoning capabilities, achieving F1 score of 60.42%. These results highlight the effectiveness of our approach in delivering more accurate and comprehensive relation identification compared to both traditional DL-based and other LLM-based methods.
4.2 LOG-BASED ANOMALY DETECTION
Log-based anomaly detection is fundamentally a time-series prediction task, where the goal is to predict whether sequence of log events indicates abnormal system behavior. This task is crucial for maintaining system reliability and security by identifying patterns that signal potential failures or attacks. Given the temporal nature of log data, both sequential patterns and the semantic content of the logs must be analyzed to accurately detect anomalies. Effective anomaly detection in time- series log data is essential for preventing system downtime and ensuring the smooth functioning of distributed infrastructures.
Setup. We evaluate our method on the HDFS dataset (Xu al. 2009) for the log-based anomaly detection task. This dataset consists of over 11 million log entries generated from Hadoop-based map-reduce jobs on more than 200 Amazon EC2 nodes. In practice, we sampled 20,000 blocks of log sequences from the HDFS dataset, consisting of approximately 486,060 log entries. The dataset split chronologically into training, validation, and test sets with a ratio of 8:1:1. We evaluate our method using F1 score, Precision, and Recall to compare it against several baselines, including tradi- tional methods like LogCluster (Lin et al, 2016), DeepL.gg (Du al.) 2017), and LogRobust (Zhang) al. 2019), well as LLM-based models like Vanilla, HtT, LogGPT (Qi et al.) (202)) ICL and
RAG, providing a comprehensive assessment of performance across various approaches.
Main Results. Table compares our method with traditional baselines and LLM- based models the log- log-based anomaly de- tection task. Traditional deep learning meth- ods like LogCluster and DeepLog rely heav- ily on training data, making it hard for them detect new anomalies. While LogRo- bust improves performance (FI: 87.31% its ability to generalize remains limited. LLMs like GPT-4 perform well using simple prompts. For example, LogGPT achieves an Fl score of 72.56%, showing strong seman- tie reasoning- However, low precision (56.82%) leads to misclassification of mi- nor issues as anomalies. HtT learns patterns from training data but struggles with efficiency and global pattern recognition, resulting in an F1 score of 58.73%. RAG improves LLM performance by retrieving similar cases, but the limited information in a single case restricts its effectiveness. Our method addresses these issues by using MCTS to extract reliable rules from the entire dataset, providing clear guidance to the LLM. This approach eliminates miselassifications achieving an Fl score of 92.59%, outperforming all baselines and effectively balancing accuracy and generalization.
Comparison under different methods on Log-based anomaly detection.
Table 3:
Mehtod F1
Precision
Recall
LogCluster
70.97%
96.70%
56.05%
DL--hased
Deepl.og
79.64%
84.81%
75.08%
LogRobust
87.31 %
89.12 89.12%
85.54%
Vanilla
60.10%
47.05%
83. 83.16%
ICL
69.77%
78.95%
62.50%
RAG
84.32%
98.97%
73.46%
LLM--ased
LogGPT 72.56%
56.82%
100%
HiT 58.73%
45.46%
82.31%
Ours 92.59
100%
86.21 16.21%
4.3 MULNTI-GGENT GAME: ALICF& BOB
In the real world, plenty of scenarios involve decision-making, planning, and collaborating, espe- cially in partially observable environments. Moreover, often the optimal strategy often contradicts human intuition. You can not walk towards the treasure directly as there may be walls blocking the path. In such tasks, it is crucial inject domain knowledge to make informed decisions, only by integrating specific domain expertise can the model accurately identify the optimal strategy and make sound judgments.
Setup. We choose the cooperative multi-agent game Alice&Bob, which requires both planning and collaboration. In the game Alice and Bob work together to find the treasure (Chen al. 2024b).
7

---

Published as a conference paper at ICLR 2025
and the optimal paths for both agents often go against intuition. They required to sequentially experience key blocks, with one agent needing to remain on a block to enable the other to obtain the treasure. Episodes last up to 50 steps. Metric We evaluate the method by reporting the average win rate (WR), the accumulative rewards (AR), and the average episode length (AL across 30 episodes. Baselines. We compare our method with RL baselines (hehavior cloning; offline tabular Q), rule generated method (PLLB (Srivastava et al.) (024) and HcT (Zhu et al. (002b)), ICL-Good (ICL with 3 good demonstrations) and "CL-Contrastive (ICL with good and 2 bad demonstrations) and RAG, RAG retrieves timesteps with similar observations and informs LLMs both observations and actions. We provide the results of random policy and LL--based grounded policy (with handeraft rules) as well. Data Collection We collect 1000 episodes of trajectories by applying handcraft policy where the agent has the probability of p to follow the optimal policy and 1 - -p to follow a random policy. We set p 0.7 in the default setting. Generated target predicates by I.I.s.. We search the logic rules from different aspects following the LLMs' suggestion: 1) team reward = -00 2) Alice or Bob stand on yellow, purple, skyblue blocks; 3) Game Win. During the evaluation, LLM serves as Alice and Bob, providing them with the observations, historical information, and the action space and prompting them to respond with the chosen action.
make different
Main Results. In Table [ we com-
Table 4: Experimental results over the decision-making task, Alice&Bob. The standard error is provided in the bracket.
pare the performance of various RL- based and LLM--based methods on the Alice & Bob task. Overall, our method achieves the sota perfor- mance. RL-based methods perform relatively well and surpass most LLM-based methods, as they can accumulate knowledge during train- ing. contrast, LLM-based meth- ods face significant challenges in this task. Methods like Vanilla, ICL- Good, and ICL-Contrastive show negative accumulative rewards (- 0.08, -0.71, and -0.83, respectively) with a win rate of O, indicating a clear lack of strategy reasoning and task optimization. Vanilla performs badly due to the absence of domain knowledge. However, once domain knowledge is correctly incorporated, performance im- proves significantly, as seen with methods like Ours (win rate of 0.7) and Grounded Policy (win rate of 0.9). Among those in-context-based LLM methods, ICL. and RAG insert relevant demonstrations. However, they perform bad as L.LMs may suffer from long-text understanding. HtT, and PLIB rely on LLM to summarize rules, which not only need to understand long text but also require more domain knowledge than our method to summarizing rules, therefore the summarized rules may not provide enough domain knowledge for LLMs.
Method
AR
AL
WR (%)
Behavior Cloning 54.67( (.)))
32,46
0.56
RL-based
Offline Tabular Q 59.51(#:2.71)
32.60
0.63
Vanilla
-0.08(+0.11)
50.0
0.0
ICL-Good
-0.71(10.55)
50.0
0.0
ICL-Contrastive
-0.83(10066)
50.0
0.0
LLM-based
RAG
-0.14(11:22)
50.0
0.0
HrT
-0.26 (+=0.22)
50.0
0.0
PLLB (Offline)
-0.15(++026)
50.0
0.0
Ours
69.45(+t:461)
33.23
0.7
Random
-2.((+0.52)
50.0
0.0
Grounded
89.87(#30.06)
32.1
0.9
4.4 ABLATION STUDY
In this section, we present an ablation study to evaluate the robustness and effectiveness of our method across several dimensions. First, we analyze the performance of our method when using different LLM backbones, examining whether the choice of LLM impacts overall task performance. Second, we explore the contribution of different components in our method, including the use of ofain-of-thought (CoT) reasoning and rule-based guidance, to assess how each component improves the task. Lastly, we investigate the effectiveness of the MCTS rule extraction process by varying the number of search episodes.
Ablation on Different LLM backbones. Table presents the results of our ablation study on different LLM backbones across relation extraction, log anomaly detection and cooperative games. compares baseline models (Vanilla), chain-of-thought (CoT), and our RuAG for GPT-3.5 and GPT-4. While COT improves performance by promoting step-by-step reasoning, it falls short in tasks requiring domain knowledge. In contrast, RuAG learned rules from external data, provides the required context, and consistently enhances performance across different backbones.
Ablation on searching episodes in MCTS. Table [6 shows the impact of MCTS search episodes for three tasks. In relation extraction and cooperative games, report the number and accuracy of
8

---

Published as a conference paper at ICLR 2025
Table 5; Ablation on LLM backbones across different tasks.
Relation Extraction
Log Anomaly Detection
Cooperative Game
Backbone Method
F1
Precision
Recall
Precision
Recall
AR
AL
WR
Vanilla
18.94%
31.06%
13.62%
48, 48.42%
62.71%
39.43%
-0.58(+::477)
50.0
0.0
GPT3.5
+CoT
19.85%
28.19% 19%
15.32%
73.19%
75.42%
71.08%
-0.38(#0.26)
50,0
0,0
+rule
26.63%
39.82%
20.00%
91.39%
100.00%
84.16%
45.2(++49.11)
42.73
0.45
Vanilla 46,94%
69.61%
%
60.10%
47 47.05%
83.16%
-008(+#0111)
50.0
0,0
GPT4
+CoT 10%
63.13%
37.39%
76.11%
63.62%
94.69%
-0.83(+0.66)
50.0
0.0
+rule
60.42%
69.44%
53.48%
92.59%
100.00%
86.21%
69.45(+:46.1)
33.23
0.7
Table 6: Ablation on searching episodes MCTS. Num. denotes the number of searched rules.
Relation Extraction
Anomaly Detection
Cooperative Gamc
Times
Num.
Precision
F1
Precision
Recall
Num.
Precision
50
13
100%
65.75%
100.00%
48.89%
14
100%
200
100%
86.86%
98.7%
77.55%
100%
500
21
100%
91.30%
100%
84%
21
100%
1000
23
95.65%
91.30%
100%
84%
23
91.30%
extracted rules evaluated, while log anomaly detection is assessed based on the final task per- formance. According to the results, fewer search episodes still yield high-quality rules. Increasing episodes expands the search space, leading to more rules, but with diminishing returns as excessive episodes introduce ineffective searches and, in some cases, incorrect rules (e.g., relation extraction).
Ablation on hyperparameter for data collection in decision-makinn task. We adjust the prob- ability of performing optimal policy and report the searched rule numbers and their precision in Table to investigate the impact of data collection policies the searched rules.
4.5
CASE STUDY
In this section, we present a case study to demonstrate how the extracted rules help LLMs perform tasks more effectively across different do- mains. The extracted rules serve guiding mechanism, assisting the LLM making more accurate predictions and improving task perfor- mance by providing structured logic and patterns that the LLM can fol- low. Figure illustrates the most representative cases where extracted
Table Ablation on hy- perparameter p.
Num
Precision
0.2
25
80%
0.5
35
88%
rules helped LLMs improve performance across three tasks: relation extraction, log-based anomaly detection, and multi-agent gaming.
0.7
21
100%
In the relation extraction task, without the aid of extracted rules, LLMs typically rely solely on the literal content of the document, extracting only obvious relational triples while missing more implicit ones. As shown in Figure 4 [)), the LLM can infer the relationship ("Ariel Sharon", "head_of_gon "Israel") based on the document's 's semantics. However, it misses the implicit relationship ("Ariel Sharon". "citizen_off "Isracl"). By providing the LLM with the rule "head_of-gor - citizen_of", our method helps the LLM extract this additional, less obvious relation. This demonstrates how our rule-based approach enables LLMs more comprehensively complete the relation extraction task by accounting for logical patterns that might otherwise be overlooked.
In the log-based anomaly detection task, LLMs can struggle due insufficient domain knowledge, leading to hallucination issues. In Figure bb), the log sequence lacks clear semantic indicators of an anomaly, making difficult for the L.M to detect. Our method uses MCTS to extract rules from historical logs that indicate abnormal patterns. When processing sample, the log sequence is matched with the rule base, and the corresponding rule, along with its confidence score, is provided to the LLM. This enables the LLM combine semantic information with historical patterns and rule reliability to make accurate anomaly detections. In this case, Rule 1 triggered by "E11, E28" indicates high probability of anomaly, allowing the LLM to correctly assess the system state.
In the decision- becision-making task (Figure 4 (c)), the vanilla LLM only takes as input Bob's observation, therefore has straightforward policy to walk towards the treasure directly. However, RuAG awares

---

Published as a conference paper at ICLR 2025
Table 8: Searched rule examples across different tasks.
Task
Rule
Description
If a person holds the position of head of government, they ure also citizen of that country.
Relation Extraction
head.of gow citizen.of
It a person holds the position of head government a nal variation of country. they are also citizen of that nominal variation of the country.
head of gov-x -> citizen.of-x
Anomaly Detection
[E11 & E28] abnormal,
If events El and E28 occur sequentially, it indicates high probability of anomaly with precision of 0.96.
precision 0.96
[E1] & E26 & F22] - ahnormal,
II evenis ETT, E26, and E20 occur sequentially, indicales very high probability of anomaly with precision of 0.99.
precision 0.99
[IsGreení Left) & Move(Alice, right) dz (Alice, treasure)==0 & (Alice, trea-
When Alice's 's center left block is green. if Alice moves right, then the team will receive Reward = 100.0. all these cases, Alice locates at blocks down, block to the left of the trea- sure, Bob stands on skyhlue block, Bob visited skyblue block, Alice visited yellow block, Bob visited purple block.
Cooperative sure) & Stand(Bob, skyblue) Visiled(Bob, Game
skyblue) & Visited(Bob, purple) & Vis- ited(Alice, yellow)) ] - Reward = 100.0
Inpute
Vnndiiee:
Varilla
SeqgSeggeeee (155) lizz tesy ES, E11:" Edr E2s, E1T, EE) EE1;; les: Eze, E2s Es, 'Ed: E23, Eaa, 126, 1921)
BLog Seq
Vanilla
Ariel Sharontameel Mahrnoud Abbas #Document in storic mosting the Indderian rosecal rown of Agytt Nen- mescay, lereed Prime
(Aris) SunttBaanci Staroit Toad of aun. famwll i (GGange W. Buee! lagenn, of, Liited Bustes", (Molnoou Abbos, _ppess
A..ttohhh there are scree dihlock mometerr, indicmme db normal behievio
Shi-
con his Pulcatinion counterpatt Man-
ESs Faarslaas jiccd"oo
System Statn:Poormal)
X
inputa
Cura
oputs
Retreival
of /aoaagg W. Busli Vagor of." United (Matroout Abtous i rreed of goee, Matesssore Artet heardst gtr, (Maumoa Paloodiiaane
Retraival
Rule insiceting chight probedt)
and F-1, ragge
snomaly
Lof gooo citber.. .st
IT: KTI, 1500 1,000M2.00
ind caliuaoou probabiltyooa remely BE
head
rioss
82: EL1.E36
Loonf
Rue Base
Rue Bese
Systom State (Abnormaa)
(a) Relation Extraction
(b) Log bagg- Anomaly Detection
Inputs
You are Bob, currently collaborating with Alica in gid world to obtain the trensure ogeen block).
Vieiilis
You are ourrantly located at blocks down and 5 blocks to the left of tressuse. Your toammate, Alios, is unently located E Blocka down and o blocks to the left of tressure. The blacks surrounding and colors lower left block white (machable)...
Move Right oo Droosure loloctied right
dift
Pless choose from
night stand.
Team recalues Rewardd e 100.0 [Game Whnt:
center right grors Alio moveoriggtt the
inpuss
at trumurs, stands skyebus block, Alcs viattd yello wiblock, Bob rished purple dkyboue blocks.
Ours
Move Left Insed stse oi the skyliun bookk olt Alice realht the tredisa
Bob stands ongurpte block:
When Tosh ocatne blecks drwn ooke leti of soun, #llch thein then ites will ittnniin purple Mloce.
(c) Cooperative Game
Figure 4: Case studies on relation extraction, log-based anomaly detection, and cooperative game.
Bob the domain-specific knowledge: stand on the skyblue block is a significant step for your team's 's success. Therefore, in RuAG, Bob chooses to walk to the skyblue block first. This cooper- ative game highlights the significance of domain-specifff knowledge in socision-making tasks, and demonstrates the effectiveness of our RuAG to integrate domain-specine knowledge by logic rules.
5 CONCLUSION
In this paper, we introduce a novel framework RuAG that automatically distills large volumes of of- fline data into understandable first-order logic rules, which are then injected into LLMSs to enhance their generation capabilities. By leveraging L.MMs commonsense, we first automatically formulate the searching process through detining the target predicate and body predicates. Then, we apply Monte Carlo Tree Search (MCTS) to efficiently address the combinatorial search space. As a con- sequence, our method discovers logic rules that can be seamlessly integrated into LLM prompts for downstream task reasoning. Empirical evaluations across variety of tasks, including NLP, time- series, decision-makine and industrial applications, demonstrate the effectiveness of our approach in improving L.LM performance over diverse domains.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In this paper, we strictly obey the principles outlined the ICLR Code of Ethics, including careful consideration of potential ethical concerns, including the impact human subjects, data privacy, and fairness in algorithmic decisions. Specifically, the three public datasets do not have potential risk. As for the private industrial dataset, promise that any data used in this study released in compliance with legal and ethical standards, and proper security measures were implemented to safeguard personal information.
REPRODUCIBILITY STATEMENT
We provide all the details of our method in the paper and appendix, including evaluation prompts, detailed experimental setup, implementation, and hyperparameters for both LLM reasoning and MCTS. The code will be available upon the paper's publication. The above ensures that others can reproduce our method.
REFERENCES
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In- context examples selection for machine translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisties: ACL 2023, pp. 8857-8877 Toronto. Canada, July 2023. Association for Computational Linguis- tics. doi: 10. 10.18653/v//202 findings-acl.566 ac1.564. URL http://////aaaattllooooooooooo org/2023. Findings-on acl. 564
Ziyan An, Hendrik Baier, Abhishek Dubey, Ayan Mukhopadhyay, and Meiyi Ma. Enabling mets explainability for sequential planning through computation tree logic. In Proceedings of the 27th European Conference on Artificial Intelligence (ECAI), 2024. URL https::/ httss///axxv..oo
abs 0063/2407.22 10820
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang. Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irv- ing, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rac, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Pm- ceedings of the 39th International Conference Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pp. 2206-2240. PMLR, 17-23 Jul 2022. URL https: //proceedings.o mlr wrw.s / vv2 http://////ooeuu22...aaaa
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, 1877-1901. Curran Associates, Inc., 2020a. URL httpps://proceeedngggg neurips wttpsspr.pppppppeeeeeeeeeeee http://r_fiileeaapee///00000 E2ee2299cccerree9111100aa2222222
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learn- ers. ArXiv, abs/2005.14165, 2020b. URL https: //api. seciissene semant eesaniicchooass orrg
CorpusID:21897115
11

---

Published as a conference paper at ICLR 2025
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh. Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Pro- cessing Systems, volume 35, pp. 18878-18891. Curran Associates, Inc., 2022. URL
https://prcceeinnss.eeiiioooooo: newr cc/ (ttpeeerrfilessppper/222/2 77occaaff992222377fc66666reeee
Paper- Cage Confereenn
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38 (16):17754-17762, Mar. 2024a. doi: 10. 10.1669/aami..v. www.w..2811.222222 URL URLpttps:://jss
Sirui Chen, Zhaowei Zhang. Yaodong Yang, and Yali Du. Stas: Spatial-temporee return decompo- sition for multi-agent reinforcement learning. In The 38th Amnual AAAI Conference on Artificial Intelligence, 2024b.
Zhongwu Chen, Chengjin Xu, Dingmin Wang. Zhen Huang, Yong Dou, and Jian Guo. Rulerag: Rule-guided retrieval-augnee augmented generation with language models for question answering. arXiv preprint arXiv:2410.22333 2024c.
Kewei Cheng, Jiahao Liu, Wei Wang, and Yizhou Sun. Rlogic: Recursive logical rule learning from knowledge graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining, KDD '22. pp. 179-189, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 450393850. doi: 10.1 145/35344678.33334211 URL https:
7/doi. otg/10..rg1...0 15/ 3784678 3539421
Tzu-Yi Chiu, Jerome Le Ny, and Jean-Pierre David. Temporal logic explanations for dynamic decision systems using anchors and monte carlo tree search. Artificial Intelligence. 318: 103897,,, 2023. ISSN 0004-3702. doi: http:///...eeerrraaae//////////0 www. sci encedi rect. com sci
103897. URL https::/
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang. Si Qin, Saravan Ra- jmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint urXiv:2311.04220 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234 2022.
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 1285-1298, 2017.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61: 1-64, 2018.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang. Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmenttd large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 6491-6501, New York, NY, USA, 2024. Association for Com- doi: 10. 10.1145/6335528.66744777
puting Machinery. ISBN 9798400704901. //doi. 10. 1 11411113335555 .3671470777
URL https:
Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, and Jun Wang- Large language models are neurosymbolic reasoners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17985-17993, 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. In The Twelfth International Conference Learning Representations.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In Hal Daumé III and Aari Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 1 119 of Proceedings of Machine Learning
12

---

Published as a conference paper at ICLR 2025
Research, pp. p
PMLR, 13-18 Jul 2020. URL https::/ppcceeciggss..
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learming Representations.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 2021.
Sihao Hu, Tiansheng Huang, and Ling Liu. Pok) eellmon: A human-parie agent for pok) 'emon battles with large language models. arXiv preprint 2xiv:2442.001118. 2024.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251): 1-43, 2023.
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can Ilms do retrieval and reasoning in million context window? arXiv preprint arXiv:2407.11963 2024.
Shuang Li, Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi. and Le Song- Temporal logic point processes. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5990-6000. PMLR, 13-18 Jul 2020. URL https: httt:::://proccednngs... mlr. alr.preess/v119/1122 html
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 arXiv:2101.00199 2021.
Xianzhi Li, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, and Samcena Shah. Are chatgpt and gpt- 4 general-purpose solvers for financial text analytics? examination several typical tasks. arXiv preprint arXiv:2305.05862. 2023.
Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. Log clustering based problem identification for online service systems. In Proceedings of the 38th International Con- ference Software Engineering Companion, pp. 102-111, 2016
Linhao Luo, Jiaxin Ju, Bo Xiong, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Chatrule: Mining logical rules with large language models for knowledge graph reasoning. arXiv preprint arXiv:2309.01538 2023.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pp. 109-165. Academic Press, 1989. doi: http://dw.aar//00000000000771111
URL https:
/twp://wcieen..ccc www.:/ccienceii.....mmm rect. cww http://www................ooaaaa
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprini urXiv:2307.00 06435, 2023.
OpenAl. Gpt-4: Openai's generative pre-trained transformer 4 model, 2023.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. JEEE Transactions on Knowledge and Data Engineering, 36(7):3580 3599, July 2024. ISSN 2326-3865. doi: 10.1199/kdee2222433211000 1119/TKDDE..T. 2021
URL httpt::pp:/...o..ootoo
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URI.htpps httptttsss.peeeeeiiiiieeeeee net
13

---

Published as a conference paper at ICLR 2025
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In Interna- tional Conference Learning Representations, 2022. URL https:://ppereeaatt net/
forum? d=Kmt VD 97]43e
Jiaxing Qi, Shaohan Huang. Zhongzhi Luan, Shu Yang, Carol Fung, Hailong Yang, Depei Qian, Jing Shang. Zhiwen Xiao, and Zhihui Wu. Loggpt: Exploring chatgpt for log-based anomaly detection. In 2023 IEEE International Conference High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCCISSS/SarrCi///Dba)s
pp. 273 280. IEEE, 2023.
Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gamett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://pproceeeengggssoo wew.eeedddd.nn..ne...e.e........ www.ipe..ip..... Eile/13e htte://e5aabbaaaaeeeeeaaa111122/ wttpr//ss.copppppeeeeee/////e/ps
pdf
Meng Qu, Junkun Chen, Louis-Pascal Xhonneux, Yoshua Bengio, and Jian Tang. Rnnlogic: Learn- ing logic rules for reasoning on knowledge graphs. In International Conference on Learning Representations.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations, 2016.
Dongyu Ru, Changzhi Sun, Jiangtao Feng. Lin Qiu, Hao Zhou, Weinan Zhang, Yong Yu, and Lei Li. Learning logic rules for document-level relation extraction. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1239-1255 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational I Linguistics, doi: 10. 18653/ v1/2021 112201.emmnlpmminn
URL https: //acl www.llathhhhhoggoo hww.hhoolloooooooo.oooooo 2021 emn p-ma-------- n. 95
Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In chatgpt we trust? measuring and characterizing the reliability of chatgpt. arXiv preprint ar arXiv:2304.08979, 2023.
Peng Shi and Jimmy Lin. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXhv:1904.05255 2019.
Dong Shu, Tianle Chen, Mingyu Jin, Chong Zhang, Mengnan Du, and Yongfeng Zhang, Knowledge graph large language model (kg-Ilm) for link prediction, 2024. URL httpts://aaxi......
abs/2405 07311
Daniil Sorokin and Iryna Gurevych. Context-aware representations for knowledge base relation extraction. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 1784-1789, 2017.
Megha Srivastava, Cedric Colas, Dorsa Sadigh, and Jacob Andreas. Policy learning with language bottleneck. arXiv preprint arXiv:240S.04/11 2024.
Carolin Strobl, Anne-Laure Boulesteix, and Thomas Augustin. Unbiased split selection for classi- fication trees based on the gini index. Computational Statistics & Data Analysis, 52(1):483-501 2007.
Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Maádziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3): 2497-2562, 2023.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Yuqi Wang, Boran Jiang, Yi Luo, Dawei He, Peng Cheng, and Liangeai Gao. Reasoning efficient knowledge kaths:knowledge graph guides large language model for domain question answering, 2024. htt.tttsss:/// . orgoor/ggoooooo abs 2404. 10384
14

---

Published as a conference paper at ICLR 2025
Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I Jordan. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd symposium Operating systems principles, pp. 117-132, 2009.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint 1.X.::2402.131166 2024.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang. Jie Zhou, and Maosong Sun. Docred: A large-scale document-level relation extraction dataset. arXiv preprint arXiv: 1906 06127, 2019.
Klim Zaporojets, Johannes Deleu, Chris Develder, and Thomas Demeester. Dwie: An entity-centric dataset for multi-task doeument-level information extraction. Information Processing & Manage- ment, 58(4): 102563, 2021.
Chaoyun Zhang. Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. UFO: Ul-Focused Agent for Windows OS Interaction. arXiv preprint arXiv:2402.07999, 2024.
Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xic, Xin- sheng Yang. Qian Cheng, Ze Li, et al. Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM joint meeting European software engineering conference and symposium on the foundations of software engineering, pp. 807-817, 2019.
Yudi Zhang, Lu Wang, Meng Fang. Yali Du, Chenghua Huang, Jun Wang. Qingwei Lin, Mykola Pechenizkiy, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Distill not only data but also rewards: Can smaller language models surpass larger ones?, 2025. URL https::/arxiv
org/ (0e//bb/2500 25022 19557
Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Efficient probabilistic logic reasoning with graph neural networks. In International Conference Learn- ing Representationss 2020. URL
net/ fo 6kSttH
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao. Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.177 170.11444, 2023a.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules, arXiv preprint arXiv:2310.07064 2023b.
15

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL RESULTS ON PRIVATE INDUSTRAIL DATASET: UNAUTHORIZED PARTY ABUSE DETECTION
The Unauthorized Party Abuse (UPA) detection task is a binary classification problem, where the goal is to predict whether incident is a case of UPA (ISUPA) based on series of features. These features include both time-dependent data, such as resource acquisition velocities and user activity history, as well as static features, like resource descriptions and types of compromised subscriptions. The task is to accurately classify each event as either UPA or not, while maintaining high precision and recall to avoid misclassifying legitimate customer activities.
Setup The dataset used for this task comes from private industrial source, consisting of histor- ical incidents of Unauthorized Party Abuse (UPA). It includes both time-dependent features, such as resource acquisition velocities and user activity history, as well as static features, like resource descriptions and types of compromised subscriptions. The dataset is imbalanced, with significantly fewer UPA cases compared to legitimate ones, and the overall data volume is large. To address this, we sampled a balanced dataset and tested the algorithm on smaller batches. For evaluation, we used common fraud detection metrics, including -score, Recall, Precision, and Accuracy. We compared our method against several baselines, including XGBoost, Decision Tree, and Rule Grounding. In Rule Grounding, the extracted rules were directly used for prediction to evaluate the effectiveness of rule extraction.
Implement Details In our task, most features in the dataset are continuous. To adapt to the re- quirement of Monte Carlo Tree Search (MCTS) for discrete state mining, we used the Gini index to discretize these continuous features. Specifically, for each continuous feature, we divided it into 10 discrete states. The discretization process involved calculating the Gini index to determine the opti- mal split points, ensuring that each resulting interval maintains a high degree of data purity, Thus, each data sample was converted into a sequence of discrete states.
We used Monte Carlo Tree Search (MCTS) to extract rules from the training set. MCTS was initial- ized with a root node representing the initial state. Child nodes were created and expanded using the Upper Confidence Bound (UCB) formula. Simulations were performed to explore different paths, and optimal rules were generated for both IsUPA=1 and IsUPA=0 targets. The rollout was set to 500, and the reward was based the precision derived from the rule. The maximum rule length was set to 5. Additionally, if a node's precision exceeded 0.85, we considered it a terminal node, as further expansion was deemed unnecessary. This allowed to collect all reasonable rules with lengths ranging from 1 to 5.
Main result Table Al shows the results different methods on the small batch dataset for abuse detection. We observe that the rules extracted using MCTS achieve high precision, similar tradi- tional machine learning methods, but also exhibit a higher recall. This because MCTS explores a broader search space, allowing it to capture a more comprehensive set of abuse patterns. On the other hand, directly using the LLM for this task yields poor performance, with an FI score of only 22.64%. The lack of domain-specific knowledge and the difficulty in processing purely numerical features hinder the LLM's effectiveness in this scenario.
However, our method, which provides the MCTS-extractes rules historical guidance to the LLM, enables the LLM to make better decisions by combining the extracted rules with feature information from specific scenarios. The results indicate that our approach significantly improves the L.MM's performance on this type of numerical task. With the help of rules, the LLM's F1 score increases to 96%, demonstrating the effectiveness of our method in guiding the LLM to handle such tasks better. The table shows several representative rules extracted using MCTS, along with their pre- cision, recall, and F1-score if used directly for detection. As can be seen, just using the first rule alone yields Fl score of 0.6623. Additionally, precision is crucial for rules in this task, high precision means that the rule for predicting IsUPA=1 is highly reliable and unlikely to make false positive errors.
16

---

Published as a conference paper at ICLR 2025
Table Al: Comparison under different methods on Fraud detection
F1
Precision
Recall
Decision tree
83.72%
100%
72%
XGBoost
88.89%
100%
80%
Rule grounding
93.62%
100%
88%
Vanilla Ours
22.64%
21.43% 96%
24%
96%
96%
Table A2: Representative rule, precision, and description of unauthorized party abuse detection.
Conditions
Target
Precision
Recall
F1
Featurel & 0.030 and Feature2 and 0.003 < Feature3 < 0.547
1
0.8632
0.5372
0.6623
0.348 < Feature4 & 0.712
I
0.8229
0,4202
0.5563
Featurel & 0.030 and Feature2 and 0.258 < Feature4 0.348
1
0.9630
0.1383
0.2419
B MORE EXAMPLES OF SEARCHED RULES
We provide the searched rules in Table A3 (Relation Extraction), Table A4 Log-based anomaly
detection), Listing [ Cooperative game) and Table A2
(Abuse detection).
Table A3: Representative rule, precision, and description of relation extraction
Rule player.of-mmembrr.of
Precision
Description
1.0
If someone is player of certain team, then they are also member of that team. For example, "John is player_of _f TeamA" be deduced "John is member of TeamA".
minister.offragent..oo 0.9928
If someone is a minister of a certain organization or country, then they are also an agent of that organization or country. For example, "Alice is minister_of Country X" can he deduced as "Alice agent of Country X":
st_ttee__ gpe0 + head_of_star state
0.7472
someone is head state a nominal variation a coun- try, and that nominal variation corresponds to an official coun- try name, then they also the head of state of that country. For example, "PersonA is the head of state-x of German" " and "German is gpe0 Germany" can be deduced as "PersonA is the dead_offssate of Germany".
head of gov, in0-x +
0.8235
If someone the head of govermment of country, and geo- graphic location in that country has nominal variation, then the head of govermment can be considered a cilizen the nominal variation. For example, "PersonB is the head_of of gov of Israel" and "Tel Aviv is in0-x of Isracli" can be deduced as "PersonB is citizen_of-x of Isracli".
ditizen_of-
head.of, agency.of + citizen.off
0.6364
someone the head an organization, and that organization is an agency of a country, then the head of the organization can be considered citizen of that country. For example, "PersonC is head_of Organization. Y" and "Organization_Y is agency.oo oof Country 7." be deduced "PersonC. is citizen_of Coun- T!".
17

---

Published as a conference paper at ICLR 2025
Burnari and exper iences zelated Lo eBBob atands yellow blockr.
Conditions Alfce visited ye! block, visited purple oookk, visited skyblue block,
Mhen Bob louates at blucks down and block to the left of the Lreasure, Bub noves down, then Beb will stand on yellow block.
Sumnari red
lences related to satto stands purple blockks blacks down will stand an purple block.
Mhen Bob louates
blooks Lo the left of the Limasuse, LE Bob
right, Uhen Bob
When Rob locates at 1 black down and B blocks to the left of the treasure, if Bob noves down, then Rob
will stand puzple block.
Mhen Bob locates
blocks doen and 8 blooka to the left of the treasure, if Bob keep standing
current block, then Rob will stand on purple af lock, In all these cases, Rob vislted purple block.
Nhwn Bob louales at 2 blucks duwn and 8 blooks Lo the lett of the Lieasuee, of Bob muves right, then. Bob
will stand on purple block, Ie all these cases, Bob visited
block.
Rob
blocks
left the treasure, Reb chase cases, Bub visited purple block.
8 blooks
down then Bob
will stand on purple block,
3) Summari red experiences related to *vAlice stands or skyblue blookes
Condit ionas Allce visited lox block, and Bob visised purgle block.
Alice locales
block
and blocka
the left of the Lenasure, if Alice noves left, Bob
did not visit ekyblue block, then Alice will stand on skyblue block.
Sumnar and exper lences sel seld bo AxAlice stands or green bluck
Conditions: Bob stand ckyblue block, Bob visited styo_ue block, Alice visites yellov alock, Bob visited purple block
blook down and black to the left of the treaxure, Al ilice noves up, then Alice sill stard on green plock.
Mhen ilice locates
When ine loostes
block down and 1. block to the left of the tressure, if Ali noves right, then green block.
Alice will
Sumnari zed experiences related to **Alice stande or yellow ploske* Condit innitionns fons: Rob vigited purple block
it Alice's action Alice's action not lest, ther Alice will stand on yellow block, In all these cases, Alice visited ye ylolow bloock
Mhon Niles lecates
blocks
and blocs to the left of the
Whon Alice Lecates at blocks sown and blocs to the left of the treasure, it Alice noves rigat, ther
A1ice will stard yollow black.
and block the of treasure, if Alice noves down, Micce will stard en yollow block.
ine locates
When Alice looates at blocks cown and blocs to the left of the treasure, it Alice keep standing on Alice visited yellow black. or the treasure, it Alice moves dom, shen
current block,
Alice will stand yellow block.
blocks cown and bloss to the Alice will stard en yollow block. In all these cases, Alice visited yellow block.
Mhon Alice Locates
When Alies lecates at blacks down and block to the left of the treasure, if Alice noves rigat, then Alice visites yellow block.
Miie will stard yollow block.
these
Sumnari experi lences related to
stands styblue block.
Conditions: Alice visited yellow block, and Bob visited purpic block. Mhen Boa locates at 0 block cown and 5 blocse to the lett et tae treasure, it Bob noves lett, Alice does stand skyblue block, then Rob will stand skyblue black.
Mhon Boo locates at block down and 5 blocke to the left of the treasurs, Alice's action not left Bob noves lett, then Bob will stand on skyblue block.
7) Sunnarized experiences related to whe tean receive a Fenalty of -10.0 zovard..
Conditions: Bob
vicited skyolus blocs, Aiies visited yellow block, Bob block, stand.
visited purple bleck, Bob's actior
Whon Aiies's apper right block is green, Alics's acticn is not dewn, 12 Beb noves right, then the zean Mill receive Penalty ot -10.0 reward. In all these casce, Alice locates at block dowr. and block to the lett the treasure.
black down and block to the left of the treasure, Alics's acticn down, Bob nover right, then tae tean will receive Penalty of -10.0 reward. Sunnarized experienesa zelazed to e*tho tean receive a Baward 100.0 (Game Kini ..
Mhon Alics iccates
Conditians: Bob stands skybive block,
visited skyalus block, Alies visited yellow block, Bob
visited purple bleck
Whon Alics's center zicht bleck is areen, i2 Alice noves right, then the zean will zecsive Reward
1.0. In all these cases, Alies locates at black down and 1 block to the left of the treasure.
block to the at the treasure, Mide right, then the tean. will zeceiva Reward 103.0.
Mhen Aiids Lecates
block
Listing I: Searched rules in Bob Scenario
18

---

Published as a conference paper at ICLR 2025
Table A4: Representative rule of Log-based anomaly detection
Rule
Precision Description
E7,E15 -
abnormal
1.0
If events E11 and E28 occur sequentially, it indicates high probability of anomaly with confidence of 100%.
E11,E28 - abnormal
0,9553
If events EI1 and E28 occur sequentially, it indicates a high probability of anomaly with a confidence of 95.53%
E2, abnormal
0.99
If events ETT and E28 occur sequentally. it indicates high probability anomaly with a confidence 99%
C IMPLEMENTATION DETAILS
We provide detailed implementation for the three public tasks and the hyperparamter in Table AS
C.I RELATION EXTRACTION
We employed Monte Carlo Tree Search (MCTS) for relation extraction across all relation triplets in the training set. The rules corresponding terminal nodes were saved, and only those with precision greater than 0.5 were retained, resulting in final set of 20 rules. During decision-making the LLMSs select the most relevant rule based on similarity for each input. We experimented with both GPT-3.5 (gpt-35-turbo 1ourbo-1kk-20 1ono-66k-222200 (6k-20230)13) and GPT-4 (gpt-4-20230613) For more arper-parameter,
please refer to Table A5
C.2 LOG-BASED ANOMALY DETECTION
For experiments, sampled 20,000 blocks of log sequences from the large HDFS dataset, which contained nearly 486,060 log entries. We split the dataset in a time-ordered fashion into training, validation, and test with a ratio of 8:1:1. Both the sequential and semantic information of log events were used for anomaly detection. In this task, we defined rules such that if a subset of events (e.g.. Em: En, E - abnormal) appears in order in a sequence, indicates an abnormal log sequence. For example, the rule Em, E., Ej + abnormal indicates that if Em, En, Et appear in order within sequence, the sequence identified as having abnormal characteristics. We employed MCTS to search for rules in the log event sequences of the training set, with the rule's accuracy serving as the reward. During anomaly detection, both event sequence and semantic information are input into the LLM, and matching rules retrieved from the rule library. If no matching rule is found, the LLM is notified that the log sequence does not reflect any known abnormal patterns from historical data.
C.3 ALICE&BOR SCENARIO
We choose the cooperative puzzle-solving game Alice& Bob (shown in Figure A6), as it is both challenging requiring planning and collaboration, where two agents, Alice and Bob, navigate 13x9 grid to find a treasure (Chen et al., (024b) and the optimal path for them are both against to the intuition. Each agent starts different positions and can move up, down, left, right, or keep stand, constrained by walls and map boundaries. Keys open corresponding doors, and lever removes walls, unlocking new areas. The agents only receive rewards upon reaching the treasure (+100), with penalties for hitting walls (-0.1 for general walls, -10 for removable ones). Each agent has limited visibility (a 3x3 area), and they must cooperate, using their abilities to overcome obstacles. Episodes last up 50 steps.
The observation of the agents includes their surrounding 8 blocks, their relative distance to the treasure, their teammate's 's relative distance to the treasure, as well as the special blocks they visited. The candidate body predicates, including the agents' observations and their actions. We search the logic rules from different aspects following the LLMSs suggestion: 1) team reward -10; 2) Alice or Bob stand on yellow, purple, skyblue blocks; 3) Game Win.
19

---

Published as a conference paper at ICLR 2025
You relation extraction assistant, and your task is extract specific relationships between given entities from a document. The format relationship triple should (entity I, relation, entity2), for example, ("University of Cologne', 'based in', 'Germany"). will supply you with document, 20 relationships with their descriptions, and the entities whose relationships need to be uncovered. Your mission is to sift through the document and extract all potential relationships between the given entities, based on the content of the document.
#### Task ####
You need extract the relationships mentioned below. Here are the descriptions and explanations of these relationships:
(relationships)
To improve Recall and precision in relationship extraction, we apply set of logic rules to deduce additional relationships based the ones already identified. You can follow these logic rules to find more relationships between entities: {rules)
Remember, the goal is to use these rules to fill in missing information and enhance the accuracy of relationship extraction. Apply these rules systematically every piece of information you process. Please use logical rules to derive more comprehensive relation triples same time, the relation triples inferred using Logic rule should identified and distinguished from the original triples.
possible. the
have given you the following relationship triples. Based on these and the provided logical rules, derive additional relationship triples. 2. Explain your derivation process and the logical rules you applied.
###
## Entities: {Entities] }
## Document: { (Document)
Now, based on the relationships, Document, and specified Entities provided, extract the triples from the Document that include these Entities and relationships, and briefly state the for each extraction.
Let's think step by step.
#### Output ####
## result:
//Please return the relationship triples in the following JSON format, and after each relation you
attach reason:
{ {'' ("entityl", 'rllation'', "entity")/RRcason //Rcason After cach relation triple you can attach reason.
('entityl', 'relation2', 'entity3' 3//ily/33///aa:
To summarize, your task is to extract relation triples from the given document and follow logical rules to get more comprehensive relation triple. focusing only on the entities and relationships mentioned. Please ensure that you do not extract any duplicate triples, and you should only extract triples that involve the entities and relationships provided me. Output the triples in the strict format (entityl1 relation, entity2), such as (University of Cologne, based_in0, Germany).
Figure A1: Instruction prompt template for generating relation extraction triples.
20

---

Published as a conference paper at ICLR 2025
I'm fetching logical rules between relationships. I need you to help me complete the relationship pre-
processing of rule generation.
The logical rule of the form:
relationl + relation3, which means that if relation exists for entity and entity B then relation3 exists for entity A and entity B.
relationl, relation2 + relation3, which means that if relation1 exists for entity A and entity B and relation2 exists for entity B and entity C, then relation3 exists for entity A and entity C. Given the following 20 relations and their interpretation, I need you tell me which relations can- didates for which it completely impossible to derive relation 3 when each a target relation. relation- ships
In fact. I need you to eliminate some options to help me reduce the amount of calculation when extract rules. It's important make the decision carefully possible that you don't miss rule that will lead to target relationship.
You need to find as much as possible and not miss any potential relationships. Return a json format with a reason after cach item, like this:
relation3: [relation2, relationl], //reason: Explain why relationl, relation2 does not derive relation3
}
Figure A2: Instruction prompt template I for removing impossible body predicates in relation ex- traction.
I need your assistance in completing the preprocessing for generating logical rules between relation- ships.
The logical rules follow format (where predicates before the arrow are considered Body predicates and the ones after the arrow are Head predicates):
relationn relation3: This means if relation1 exists between entity A and entity B, then relation3 also exists between entity A and entity relationl, relation2 relation3: This means if relation exists between entity and entity B, and relation2 exists between entity and entity C, then relation3 exists between entity A and entity C.
Given the following twenty relations their descriptions, I need you to identify which relations are suitable for being Body predicates. Please remove the ones that are not appropriate for Body predicates. (relationships)
Please return the results dictionary where the key represents the relations suitable Body predicates, the value explains why.
Figure A3: Instruction prompt template2 for removing impossible body predicates in relation ex- traction.
21

---

Published as a conference paper at ICLR 2025
You are a helpful assistant. Your task is to build body predicate and the head predicate for searching logic rules of a game through the Monte Carlo Tree Search.
Game Description:
Two player, Alice and Bob are collaborating to obtain a treasure in the grid world. At each timestep. the agent can observe their own paritial observation of the grid world, consisting of the following information:
the color
blocks surrounding agent
the color of the block where the agent is located
3) the relative position of the agent's 's teammate to the treasure the relative position of the agent to the treasure
Each agent can choose to move up, move down, move right, move left and no action. move towards an unmovable block, the agent gets a penalty of reward -0.1. The two agents share team reward whose values fails in [-0.2, -0.1, 0, -10, 10 99.9, 100].
There blocks with five colors: movable white blocks, unmovable blocks, movable yellow blocks, movable purple blocks, movable skyblue blocks and the green blocks representing treasure. Yellow, purple, skyblue blocks are with different functions but we do not know what will happen if any of the agent stand on any blocks. At timestep, the agents can not stand on same block.
Definitions of Logic Rule:
The logic rules are defined as:
[body predicates: (feature satisfies condition 1) & (feature satisties condition 2) ...] [head predicate: special game state].
As example. (relative x-position of agent equals value 1) (relative y-position of agent equals value Alice obtains green block.
Your Task: Define Head Predicates:
To help the agents know the environment better, please suggest all events that may significant for the game win. Below
examples,
if alice obtain treasure if bob obtain the green block the agent get reward of -10,
Please think step by step finish your task.
Figure A4: Instruction prompt template for suggesting new target predicates in cooperative game.
Relationship Extraction
Anomaly Detection
Abuse Detection
Phase
Parameter
Ali
Total rollouts
500
500
500
500
Rule
Reward metric
Precision
FI-score
FI-score
Precision Recall
Generation
Maximum
2
10
body predicates
Terminal
Precision >> 0.9
Precision > 0.9
Precision 0.85
Precision
condition
Maximum
1000
1000
1000
1000
tokens
LLM
Temperature
0
0
Reasoning
Top-p
Frequency
penalty
Presence
0
penalty
Table A5: Summary of MCTS Parameters and LLM Configuration Across Tasks
22

---

Published as a conference paper at ICLR 2025
You will see complete log event sequence from Block in HDFS file system. will also provide you with the content of each log event sequence. Based on the current log sequence, you need predict whether the system is in a [Normal] or [Abnormal] state, along with written description of your reasoning.
## Input
The log sequence window requiring anomaly detection is:
[logs}
The content of cach log event in this sequence is as follows:
evvent.contet)
## The Guidelines for anomaly detection is
(guidelines)
The provided guidelines are very reliable. You need to trust the guidelines provide to you first, unless there is more obvious and direct evidence to the contrary. If there obvious unusual messages in your logs like "error," "failure," "exception." and The provided guidelines are very reliable.
you judge for yourself
need to trust guidelines provide to you first, unless there is more obvious and direct evidence to the contrary. If there are obvious unusual messages in your logs like "error," "failure." "exception," and so you can judge for yourself.
## And you should 'System State:[Normal] or "System State: Abboomallla
You should first provide brief explanation of your evaluation, and then always end your response with either 'System [Nommal) 'System State:[Abnormal] verbatim.
Figure A5: Instruction prompt template for Log-based anomaly detection
Door can be opened after A stand on yellow block.
Door can be opened after B stand purple block.
Walls that removed
agent keep standing on the skyblue blocks
A
Figure A6: Illustration of Alice& Bob.
23

---

Published as a conference paper at ICLR 2025
You are (agent's i name), currently collaborating with your teammate, (teammate's s name), ) in grid world to obtain the treasure (green block). The final goal of your cooperation. Your team's performance will evaluated based on the game and the number of steps taken to find the treasure. Due to the impossible communication with your teammate, please monitor the state of your teammate and adjust your plan in time. ## Game Win: You [teammate's name) reaches the treasure. Please actively collaborate with your teammate achieve the goal.
secure the treasure through rewards you collect during
## Candidate actions: 'up': move to stand on your "upper center** block if not black; 'down': move to stand on your **lower center## block if not blackk; 'left': move to stand on your **center left** block if not blackk; 'right": move to stand on your **center right*** block if not blackk; 'stand': keep standing the current block. Be careful to stand the same block for long time. Explanation
your surrounding blocks: Center left, center right, upper center, lower center blocks: you can only move to any of them as long as they are non-black blocks; otherwise, you will receive a penalty and stay on original block. Upper left, Upper right, Lower left, lower right: You need move twice to reach those blocks. So if you want to move to those blocks, please be careful to plan the path and make right, please make sure both center right and upper center blocks are reachable.
all the blocks in the path movable. As an example: if you want move up then
## Some examples to avoid obstacles: If you want to move to the lower right block and your center right block is black. you can move down first then right if your lower center blobk is white. If moving right would bring you closer your destination but the 'center right block' is unmovable and 'lower center block' movable, try moving down first, then moving left twice and finally up if applicable. Mention this in your plan if you want to do so.
[Searched Logic Rules)
Please response with your thoughts, plan, and chosen action in the following format: Describe your initial thoughts, like analysising the key steps towards game win, identifying your subgoals, comparing your candidate actions, analysising the progress your teammate, assessing your previous plan and making future plan. "Thoughts": "Let's 's think step by step! [your analysis here]".
// Make your future plan after you take action at this timestep. The plan will be the reference of your future decision making. // Do not include the current chosen action the plan. "Plan": "[fill your future plan herel".
Your action, make sure to choose from 'up', 'down', 'left', 'right', 'stand'. "Chosen Action": "[fill your final action choice here]"
## Your last action: (previous action) }
## Your plan at last timestep: (previous plan]
Please reaccess your situation and make decisions based on your current observations and the previous plan. If necessary, you can choose to act without considering your plan.
## Your current observation: couurent_obsrvaatee
Figure A7: Instruction prompt template for generating Alice's action in Alice&Bob.
24


Reviews:
[{"review_id": "a2afUTIsMw", "reviewer": "Reviewer_fTJ7", "summary": "The paper presents a rule-augmented generation approach, where rules are learned from the training dataset using Monte Carlo Tree Search (MCTS).\nIt shows that leveraging rules can outperform retrieved passages or even other supervised trained models.", "questions": "1. Did you use only the \"train\" split to construct the rules, testing them exclusively on the \"test\" split?\n2. How did you partition the data for the cooperative game into train, validation, and test sets?\n3. Given the reliance on deterministic predicates, how do you anticipate this approach adapting to real-world RAG scenarios requiring dynamic, knowledge-based decision-making?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper contributes by integrating rule-based augmentation with generation models, leveraging rules learned from data.", "weaknesses": "There are several areas of concern:\n\n1. Clarity in Section 3.1 (LLM-based Logic Rule Search): This section is difficult to understand. Here are some follow-up questions for clarification:\n\n    1.1. What do the initial predicates look like across the three different datasets?\n\n    1.2. How does the LLM eliminate impossible predicates? Could you provide prompt examples?\n\n    1.3. How does the LLM propose new target predicates? Any prompt examples for this?\n\n2. Performance of Rules Alone: It appears that in cases where the rules generalize well to the test set, predictions might be straightforward using only the rules. However, this may not extend to more complex or varied test cases.\n\n3. Applicability of Rules: The rules generated in this paper may not directly translate to real-world retrieval-augmented generation (RAG) settings, which often require \"external knowledge\" represented in both body and target predicates. For instance, the example in Figure 2 does not reflect how real-world decisions about weather predictions are made. The rules discussed here seem applicable primarily to scenarios with deterministic target predicates (e.g., classification tasks). How might this approach be extended to real-world scenarios, as suggested in the conclusion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9M3FZBQEb", "reviewer": "Reviewer_ME5G", "summary": "This paper proposes to use LLM to create abstract rules that can be provided in context for better decision making when using LLM. The method contains a few steps where first the LM is used to define the features and labels for which we would like to forumlate rules over. Then MCTS is used to find the rules that explain the data best, and last the rules are provided in-context for prediction. The idea of using LLM to create features to search over given data is very nice. The method is evaluated on tasks of relation extraction, anomaly detection and a multi-agent game. Using GPT-4 is much better than weaker models (unsurprisingly) and using the rules is better than some prior work termed HtT (which needs more explanation). In the game experiment there is advantage over ICL and RAG.", "questions": "** Table 4 -- where do confidences come from?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The idea of using large language models for \"feature extraction\" is very interesting. It is related to https://arxiv.org/abs/2409.08466 (which I don't expect to be in the paper since it's very recent)\n\n* The empirical results show that for some use cases abstract rules can be leveraged to improve performance.", "weaknesses": "* The paper needs better **scoping** -- when is this method likely to be useful and when not? For example, the point of machine learning is that some things are hard to express by rules -- for example, what makes a face a face or a cat a cat? We learn machine learning models for cases where rules are hard to formulate. Is this method restricted to things that can be defined by rules or not? I think it's important to discuss this. Second - the method assumes an input of N features with some feature description. Often nowadays we work with more raw data like a sequence of words etc. Is this restricted to such cases? The tasks chosen are diverse but rare and it is not clear how general the method is and when we should expect it to work? Overall the generality of the method remains unclear and needs further discussion.\n\n* Line 188: it is not clear if the LM looks at the actual data. The authors talk about using the LLM to look at the data and find patterns, but it seems maybe the LLM is only used to look at the schema, or the featuer descriptions to define what are the body and target predicates and it does not use the actual values of these features in the data at all? Choosing the rules is done with MCTS where it is not clear if the LM is used - it seems like rules are applied on the data to see if they work well. So if the LLM is not used to find patterns in the data this might be a bit limited.\n\n* The paper has many clarity issues:\n** Clarity - I don't understand figure 3 well enough - it seems important but is a mix of using text and emoji without proper explanation I can kind of squint at it and guess but it seems really difficult to understand what are the details of each step.\n\n** Clarity: line 190: “initial the features as the body predicates” - what are the features exactly? can you give some examples and intuition at this point already?\n\n** The paper talks about \"impossible body predicates\". What are those? Why are they impossible? What is exactly the input provided to the LLM to perform this task (please don't send me to the appendix in author response). Similarly \"suggesting new target predicates\" how? What is the task given to the LLM to do that? All those seem like crucial aspects that are not explained. I can imagine the LM doing an OK job in these things with some prompt but they don't seem necessarily like something that is well defined that even humans can do with reasonable agreement.\n\n** HtT -- this seems like a key baseline that is not explained properly.\n\n** Line 285: During the rule extraction process, we leveraged the LLM to filter out 15% of the relationships that were unlikely to serve as valid predicates. Unclear;\n\n** Experimental details: There is very little detail in the paper on what are the featuers/labels/target and body predicates in each of the experiments, this makes it hard to understand the task. There are a few examples in a table but this is insufficient for understanding.\n\n* Experimental evaluation: a lot of the comparisons in the experiment are between models that have very different abilities. Comparing GPT-4 to some CNN or BiLSTM or BERT is not a fair comparison at all. Baselines should be between the same model that uses different methods for providing in-context information and this is missing.\n\n* Experimental evaluation: The main claim in the abstract and introduction is that using rules in context is a better alternative to in-context learning and RAG - but I don't see any RAG or ICL baselines in section 4.1 or section 4.2 so how can this claim be made? Did the author try ICL and RAG in 4.1 and 4.2? or only in 4.3? It would also be good to have much more detail on what the ICL and RAG baselines look like in Section 4.3.\n\n* Minor: lines 189 and onwards have the same few sentences twice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOl05sNZ41", "reviewer": "Reviewer_31bM", "summary": "This work introduces the RuAG framework that automatically distills large volumes of offline data into logical rules , which are then included in LLM prompts to enhance their reasoning capabilities.", "questions": "1. L190-195 probably has some copy-pasting errors?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. RuAG offers a scalable solution for integrating extensive domain knowledge into LLMs that improves upon RAG or SFT.\n2. Model performance is tested on a wide array of tasks and show improvement on strong baselines\n3. RuAG is more computationally efficient than other methods that summarize external dataset as knowledge storage, as the calls to API models only happen once during logic rule constructions.", "weaknesses": "1. Ablation studies in Table 5 could include RAG or SFT on open-source LLMs, as the current baselines only include COT which does not include external knowledge.\n2. How do LLMs suggest new rules to explore and detect impossible body predicates? These parts seem unclear to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)