COOL: Efficient and Reliable Chain-Oriented Objective Logic with Neural Networks Feedback Control for Program Synthesis

## Abstract

Abstract Program synthesis methods, whether formal or neural-based, lack fine-grained control and flexible modularity, which limits their adaptation to complex software development. These limitations stem from rigid Domain-Specific Language (DSL) frameworks and neural network incorrect predictions. To this end, we propose the Chain of Logic (CoL) , which organizes synthesis stages into a chain and provides precise heuristic control to guide the synthesis process. Furthermore, by integrating neural networks with libraries and introducing a Neural Network Feedback Control (NNFC) mechanism, our approach modularizes synthesis and mitigates the impact of neural network mispredictions. Experiments on relational and symbolic synthesis tasks show that CoL significantly enhances the efficiency and reliability of DSL program synthesis across multiple metrics. Specifically, CoL improves accuracy by 70% while reducing tree operations by 91% and time by 95%. Additionally, NNFC further boosts accuracy by 6%, with a 64% reduction in tree operations under challenging conditions such as insufficient training
data, increased difficulty, and multidomain synthesis. These improvements confirm COOL as a highly efficient and reliable program synthesis framework.

## 1 Introduction

Figure: Figure 2: Performance Enhancements with CoL and NNFC. The CoL DSL surpasses non-CoL DSL in all metrics. While NNFC increases computation time due to neural network calls, it significantly boosts accuracy in dynamic experiments, enhancing reliability.
Refer to caption: x2.png

Program synthesis is becoming increasingly important in computer science for enhancing development efficiency Gulwani et al. (); Jin et al. (). Despite the effectiveness of current state-of-the-art methods in dealing with simple tasks, the complexity of modern software demands more advanced and sophisticated approaches Sobania et al. ().

To address these challenges, an effective solution must offer programmers fine-grained control and flexible modularity in the synthesis process Groner et al. (); Sullivan et al. (). First, fine-grained control tailors the synthesis path to specific tasks, ensuring the interpretability of the synthesis process. Secondly, flexible modularity enhances reusability and guarantees the quality of the entire program by ensuring the correctness of the modules Le et al. ().

However, these principles are often overlooked in current state-of-the-art program synthesis methods. For example, symbolic approaches such as SyGus Alur et al. (), Escher Albarghouthi et al. (), and FlashFill++ Cambronero et al. () struggle to scale to complex tasks because their traversal-based Domain-Specific Language (DSL) framework lacks fine-grained control. A compensatory strategy involves using neural networks for guidance or search space pruning, as seen in projects such as Neo Feng et al. (), LambdaBeam Shi et al. (), Bustle Odena et al. (), DreamCoder Ellis et al. (), and Algo Zhang et al. (), but the control logic remains disconnected from the programmer. On the other hand, LLM-based projects like CodeGen Nijkamp et al. (), CodeX Finnie-Ansley et al. (), and Code Llama Roziere et al. () allow programmers to control synthesis through prompt interactions. However, they lack modularity, as all tasks rely on the same LLM, making the logic vulnerable to biases in training data and leading to subtle errors that require manual verification. In summary, there is an urgent need for fine-grained control and flexible modularity to ensure the efficiency and reliability of these methods when tackling complex synthesis tasks.

In this paper, following the principles of fine-grained control and flexible modularity, we present COOL (Chain-Oriented Objective Logic), a neural-symbolic framework for complex program synthesis. At the core of our approach, we introduce the Chain-of-Logic (CoL), which integrates the functions of the activity diagram to enable fine-grained control Gomaa (). As illustrated in Figure 1, programmers can precisely organize rules into multiple stages and manage control flow using heuristics and keywords. Additionally, we leverage neural networks on top of CoL to dynamically fine-tune the synthesis process. For this purpose, we introduce Neural Network Feedback Control (NNFC) Turan & Jäschke (), which enhances future synthesis by learning from data generated during synthesis and suppresses neural network incorrect predictions through filtering. To ensure modularity, each neural network is bound with a specific CoL DSL, stored in separate library files for clear isolation and easy reuse. Thus, through the combination of CoL and NNFC, COOL achieves high efficiency and reliability when tackling complex synthesis tasks.

We conduct static experiments (constant domain and difficulty tasks, using pre-trained neural networks without further training) and dynamic experiments (mutative domain and difficulty tasks, where neural networks are created and continuously trained during the experiment) to evaluate the impact of CoL and NNFC on program synthesis. Figure 2 illustrates the significant improvements achieved by CoL and NNFC: In static experiments, CoL improves accuracy by 70%, while reducing tree operations by 91% and time by 95%. In dynamic experiments, NNFC further increases the accuracy by 6%, with a 64% reduction in tree operations. The results underscore that achieving fine-grained control and flexible modularity can greatly improve efficiency and reliability in DSL program synthesis.

The contributions of our work are as follows:

- 1.
We propose the Chain-of-Logic (CoL), which enables fine-grained control in complex program synthesis by structuring rule applications into distinct and manageable stages.
- 2.
We further introduce Neural Network Feedback Control (NNFC), a dynamic correction mechanism for CoL that continuously learns from the synthesis process, ensuring modularity by pairing neural networks with specific CoL DSLs.
- 3.
We present COOL, an efficient and reliable neural-symbolic framework for complex program synthesis, combining the strengths of CoL and NNFC to achieve fine-grained control and flexible modularity in DSL-based synthesis.

Figure: Figure 3: Chain-of-Logic. In this illustrative CoL DSL, each node represents a stage or activity where a set of rules can be applied to generate partial programs. The flow between stages is managed by keywords return, logicjump(n), and abort, allowing for the implementation of complex control flow in program synthesis.
Refer to caption: x3.png

## 2 Method

In this section, we detail the implementation of CoL and NNFC, outlining the principles that ensure high efficiency and reliability for complex program synthesis tasks.

### 2.1 Chain-of-Logic (CoL)

Activity diagrams, widely used in software engineering, effectively describe how an initial state transitions to a final state through multiple stages. This feature aligns with the DSL-based program synthesis process. A DSL, defined as a context-free grammar, converts partial programs with nonterminal symbols into complete programs by applying given rules. However, as the rule set grows, DSL becomes inefficient in exploring partial programs. To enhance the efficiency of DSL, the Chain-of-Logic, drawing inspiration from activity diagrams, organizes rule applications during synthesis into a sequence of manageable stages, as illustrated in 3.

CoL improves the control flow of the DSL with two key features: heuristic vectors and keywords. Heuristic vectors specify the stages where rules apply and their corresponding values. For example, in Figure 1, a rule with the heuristic vector (0,7,3) is applicable in stages 2 and 3 with heuristic values of 7 and 3, respectively. These vectors form the core of CoL’s control flow.

Second, CoL introduces three keywords—return, logicjump(n), and abort—to dynamically choose the next stage during synthesis:

- 1.
return: Ends the current rule, staying within current stage or advancing to following stages.
- 2.
logicjump(n): Jumps directly to the stage $n$, enabling branching and loops within CoL.
- 3.
abort: Terminates the current synthesis branch, pruning the search space.

In summary, CoL provides fine-grained control through heuristic vectors and keywords. This structured and detailed approach enhances the efficiency of DSL synthesis.

Figure: Figure 4: Neural Network Feedback Control. The left side illustrates the complete control loop of NNFC. In the forward flow (green path), heuristic values $u$ guide the synthesis process as control signals. In the feedback loop (red path), the DSNN (Domain-Specific Neural Network, the neural network paired with a DSL) generates initial error signals $e_{0}$ from partial programs $y$. These singals are then filtered to produce high-quality error signals $e_{1}$, which adjust the initial heuristic values $u_{0}$. In multidomain synthesis, the CoL DSL and DSNN from the self-domain use partner domain information (dashed path) to clarify tasks and avoid competition, ensuring modularity. The right side details the feedback loop: The DSNN comprises multiple neural networks coupled in series via noise signals, with each network generating its own error signal $e_{0}$, then these signals with large discrepancies are filtered, retaining the final high-quality error signals $e_{1}$.
Refer to caption: x4.png

### 2.2 Neural Network Feedback Control (NNFC)

While CoL enables programmers to fine-tune the synthesis process, the control flow may lack detail or vary by task. To this end, Neural Network Feedback Control (NNFC) dynamically refines control flow through feedback from neural networks, improving precision and adaptability. However, neural networks present the risk of generating incorrect predictions, threatening reliability.

Therefore, a robust control flow in NNFC is crucial to ensuring overall performance. As illustrated in Figure 4, NNFC enhances the CoL DSL in the following ways: In the forward flow, the Clipper prioritizes control signals aligned with DSNN guidance by capping any inconsistent signals, while the CoL DSL applies rules based on the adjusted heuristic values. Meanwhile, in the feedback loop, the DSNN generates error signals from partial programs across domains. To suppress the impact of mispredictions, the Filter refines these signals before they influence the forward flow.

The quality of the signals generated in the feedback loop directly determines the effectiveness of NNFC. If the error signals are of poor quality, NNFC may not only fail to provide additional improvements but also degrade CoL DSL performance. We ensure the error signal quality through an inner coupling structure within DSNN. As shown in Figure 4 (right), during synthesis tasks, DSNN processes partial programs using a series of sequentially connected neural networks. Each neural network takes both the partial programs and intermediate results from the preceding neural network as input, generating its own predictions. When errors occur in earlier networks, they propagate downstream as noise signals, amplifying at each stage. The difference in the outputs between these neural networks is positively correlated with the accumulated error. To mitigate this, we set a threshold to filter out signals with a significant difference in outputs. Finally, DSNN uses passed signals to generate multi-head outputs to fine-tune the forward flow:

- 1.
Task Detection Head (TDH): Improves modularity by determining whether the partial program contains components that the CoL DSL can process.
- 2.
Search Space Prune Head (SSPH): (Active when TDH is true) Evaluate the feasibility of synthesizing the final complete program from the current partial program, and CoL DSL will avoid exploring infeasible spaces.
- 3.
Search Guidance Head (SGH): (Active when both TDH and SSPH are true) Guides the CoL DSL in applying the most promising rules to the partial program.

By adopting filtering and multi-head outputs, the feedback loop delivers high-quality error signals to the forward path, ensuring that NNFC enhances the synthesis process on top of CoL.

## 3 Experiments

We conduct the experiments in two stages to evaluate the improvements introduced by CoL to DSL and to assess how NNFC further enhances performance. First, we carry out static experiments under fixed conditions, including task domain, difficulty level, and neural network. These controlled conditions allow us to accurately measure CoL’s impact on performance. Next, we proceed with dynamic experiments, where conditions vary throughout. This dynamic setup evaluates NNFC’s ability to improve reliability under changing situations.

### 3.1 Experimental Setup

Improvements of DSL by CoL and NNFC is evaluated across benchmarks using various metrics.

**Table 1: Benchmark configurations. Relational benchmarks are divided into easy and difficult groups based on the number of relationship edges, while symbolic benchmarks are based on the number of nodes in the tree.**
| Benchmark Type | Difficulty Level A | Difficulty Level B |
| --- | --- | --- |
| relational | 300 tasks with 3 edges | 200 tasks with 4 edges |
| symbolic | 300 tasks with around 5 nodes | 200 tasks with around 9 nodes |

Benchmarks. We evaluate CoL and NNFC using relational and symbolic tasks with varying difficulty levels, as detailed in Table 1. Specifically, the relational tasks are drawn from the CLUTRR Sinha et al. () dataset, where the goal is to synthesize programs that capture specific target relationships based on human common-sense reasoning. In contrast, the symbolic tasks are generated by GPT Achiam et al. (). They involve synthesizing standard quadratic equation programs from non-standard quadratic forms by performing manual calculation steps. Although these tasks are simple for humans, they serve as a straightforward demonstration of how fine-grained control, derived from programmer expertise, can significantly improve program synthesis efficiency.

Metrics. Besides accuracy, we also focus on the following points: (1) CPU Overhead is assessed by the number of tree operations required for synthesis. (2) Memory overhead is assessed by the number of transformation pairs (a partial program paired with the rule to be applied)(^1^11Each partial program must be completed with at most 1000 transformation pairs, though this may exceed 1000 if additional tasks are generated during synthesis.). (3) GPU Overhead is measured by the number of neural network invocations. (4) Time overhead is referenced by the actual time spent on program synthesis tasks. (5) Filtering Performance is evaluated by the attenuation ratio of invalid to passed neural network predictions.

Chain-of-Logic. We utilize the CoL approach to enhance DSL by making the synthesis process more in line with human problem-solving strategies. For relational tasks, by mirroring the way humans typically reason about family relationships, CoL organizes the synthesis process into stages illustrated in Figure 3. For symbolic tasks, CoL structures the DSL to follow the manual quadratic equation simplification strategy, with stages such as expanding terms, extracting coefficients, permuting terms, and converting equations to standard form. The specific CoL DSL configurations are shown in Table 2, where the significant differences in DSLs highlights the generality of CoL.

**Table 2: CoL DSL configurations. The DSL for relational benchmarks has a limited search space and shorter CoL, facing challenges from numerous production rules leading to larger trees. Conversely, the DSL for symbolic benchmarks offers an unlimited search space with a longer CoL, but the many permutation rules increase the risk of cyclic rule applications.**
| Benchmark | Rules | Length of CoL |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Total | Production Rules | Reduction Rules | Recursive Rules | Permutation Rules |  |  |
| relational | 40 | 36 | 2 | 16 | 0 | 4 |
| symbolic | 55 | 17 | 26 | 3 | 11 | 7 |

Groups. We use multiple groups to comprehensively evaluate CoL and NNFC (as shown in Table 3). First, in static experiments, we evaluate CoL by comparing DSL groups with and without CoL enhancements. Second, to isolate the impact of heuristic vectors—both as guides and as structuring tools for rule application—we create groups enhanced only by heuristic values. Third, we introduce groups enhanced by neural networks to assess whether combining CoL with neural networks yields better results and to explore the filtering effect of the inner coupling structure. In dynamic

**Table 3: Group configurations. Groups marked with ★ are the main experiments, those with ✩ are for ablation and extended experiments, and the unmarked group is the baseline.**
| Group | Experiment | Pretrained DSNN | NNFC | Inner Coupling Structure |
| --- | --- | --- | --- | --- |
| DSL | static |  |  |  |
| ✩DSL (Heuristic) | static |  |  |  |
| ★CoL DSL | static, dynamic |  |  |  |
| ✩DSL+NN | static | ✓ |  |  |
| ✩DSL (Heuristic)+NN | static | ✓ |  |  |
| ✩ CoL DSL+NN | static | ✓ |  |  |
| ✩CoL DSL+NNFC | dynamic |  | ✓ |  |
| ✩DSL+NN (Cp) | static | ✓ |  | ✓ |
| ✩DSL(Heuristic)+NN (Cp) | static | ✓ |  | ✓ |
| ✩CoL DSL+NN (Cp) | static | ✓ |  | ✓ |
| ✩CoL DSL+NN (Cp) | static | ✓ |  | ✓ |
| ★CoL DSL+NNFC (Cp) | dynamic |  | ✓ | ✓ |

experiments, we design control groups with and without NNFC to evaluate its impact. Additionally, we include a group without the inner coupling structure to confirm its necessity.

Environment. Experiments are carried out on a computer equipped with an Intel i7-14700 processor, a GTX 4070 GPU, and 48GB RAM.

### 3.2 Static Experiments

We start with static experiments. With the task domain, difficulty level, and neural network conditions unchanged in each group, a series of controlled experiments confirm that CoL has remarkably boosted DSL program synthesis in all metrics.

The results in Table 4 clearly demonstrate that CoL significantly improves accuracy while minimizing overhead. Most notably, CoL improves the accuracy of the DSL from less than 50% to 100% across both relational and symbolic benchmarks. Additionally, CoL achieves remarkable reductions in relational tasks, cutting tree operations by 90%, transformation pairs by 88%, and time by 95%. Similarly, in symbolic tasks, CoL reduces tree operations by 92%, transformation pairs by 96%, and time by 97%. These findings showcase CoL’s substantial impact on improving performance across all key metrics.

**Table 4: Static performance of DSL and CoL DSL for relational and symbolic tasks. CoL DSL significantly outperforms DSL in all metrics.**
| Benchmark | Group | Accuracy (%) | Avg. Tree Operation | Avg. Transformation Pair | Avg. Time Spent (s) |
| --- | --- | --- | --- | --- | --- |
| relational | DSL | 11.3 | 463.9 | 1432.2 | 9.43 |
| CoL DSL | 100.0 | 46.6 | 177.8 | 0.48 |  |
| symbolic | DSL | 48.3 | 411.2 | 2285.3 | 3.31 |
| CoL DSL | 100.0 | 33.8 | 92.7 | 0.11 |  |

Further ablation and extension experiments clarify the sources of CoL’s enhancement, confirm CoL’s effective integration with neural networks, and explore when filtering via inner coupling structures is most beneficial. Our findings are as follows:

First, CoL’s enhancement stems from both heuristics and structured rule application stages. As illustrated in Figure 5, the DSL (Heuristic) group outperforms the DSL group in most metrics, and the CoL DSL group significantly surpasses DSL (Heuristic) in all metrics. Such results indicate that CoL positively impacts synthesis by guiding and structuring rule application. Moreover, on top of guidance, the structured rule application stages achieve greater improvement.

Second, integrating CoL with neural networks further improves the search efficiency. As shown in Figure 5, despite additional GPU and time overhead, the top-performing CoL DSL + NN group reduces tree operations by 43% and transformation pairs by 19% in relational tasks compared to the CoL DSL group. In symbolic tasks, the CoL DSL + NN (Cp) group reduces tree operations by 64% and transformation pairs by 46%. The results showcase that neural networks can further narrow the search space for program synthesis beyond CoL. Importantly, the group with the inner coupling structure outperforms non-neural groups in both tasks. In contrast, the group without it presents an accuracy decline in symbolic tasks, validating the structure’s role in improving reliability.

Third, the inner coupling structure is more effective when error tolerance is low. As indicated in Figure 5, for symbolic tasks, CoL DSL-based groups with the inner coupling structure significantly outperform those without it. However, for relational tasks and DSL-based groups (without CoL or heuristic), those without such structure perform better. This difference indicates that the filtering effect of the inner coupling structure comes at a cost: it filters out both incorrect and correct predictions. So, its effectiveness depends on the positive impact of eliminating incorrect predictions outweighing the loss of correct ones. Therefore, for relational tasks with a limited search space and DSL-based groups with higher error tolerance, the cost of filtering outweighs the benefit. However, in symbolic tasks, where avoiding errors is more critical, CoL DSL-based groups benefit significantly from the inner coupling structure.

Figure: Figure 5: Static performance on relational and symbolic tasks at difficulty level A. CoL DSL-based groups outperform DSL (Heuristic) and DSL groups. Performance varies for DSNN-enhanced groups with the inner coupling structure. Error bars show 95% confidence intervals across 6 batches.
Refer to caption: x5.png

### 3.3 Dynamic Experiments

Static experiments confirm CoL’s improvements on DSL and its enhancement with neural networks. However, real-world program synthesis involves varying task domains and difficulty, facing the risk of neural network mispredictions due to underperformance. Therefore, we introduce these factors in dynamic experiments to evaluate how NNFC further improves the performance of CoL DSL.

**Table 5: Dynamic performance of CoL DSL and CoL DSL+NNFC(Cp). NNFC significantly improves the dynamic performance of CoL DSL in accuracy, tree operations, and transformation pairs.**
| Bench-mark | Group | Accuracy (%) | Avg. Tree Operation | Avg. Transformation Pair | Avg. Neural Network Invocation | Avg. Time Spent (s) |
| --- | --- | --- | --- | --- | --- | --- |
| relational | CoL DSL | 100.0 | 70.0 | 259.8 | 0 | 1.05 |
| CoL DSL+NNFC (Cp) | 100.0 | 54.6 | 224.5 | 21.7 | 2.08 |  |
| symbolic | CoL DSL | 82.6 | 233.5 | 977.1 | 0 | 1.42 |
| CoL DSL+NNFC (Cp) | 99.4 | 50.3 | 222.2 | 21.6 | 1.12 |  |
| multi-<br>domain | CoL DSL | 97.5 | 115.2 | 367.6 | 0 | 0.99 |
| CoL DSL+NNFC (Cp) | 99.0 | 45.6 | 250.5 | 72.84 | 3.91 |  |

The results in Table 5 confirm that NNFC significantly enhances the reliability of CoL DSL in challenging conditions. As task difficulty increases and multidomain scenarios emerge, the accuracy of the CoL DSL group declines compared to its performance in static experiments. However, the NNFC-enhanced group maintains an accuracy of at least 99%, demonstrating its strong reliability in challenging situations. Additionally, compared with the original CoL DSL group, it reduces tree operations by 22% and transformation pairs by 14%. For symbolic tasks, despite the added time for neural network invocations, the NNFC-enhanced group still shortens the time spent by 21%.

Figure: Figure 6: Dynamic performance differential to CoL DSL in singledomain tasks. The NNFC group without the inner coupling structure shows 12 accuracy declines across 20 batches, while the group with the structure shows none. Each batch consists of 50 tasks, and NNFC continuously trains DSNNs using generated data after each batch, starting from scratch.
Refer to caption: x6.png

Figure: Figure 7: Dynamic performance differential to CoL DSL in multidomain tasks. The NNFC group without an inner coupling structure degrades across all 4 batches, while the group with the structure experiences degradation only in the first batch. Each batch includes 50 relational and 50 symbolic tasks, and DSNNs are continuously trained from those for tasks at difficulty level A in Figure 6.
Refer to caption: x7.png

Further ablation experiments confirm that reliability provided by NNFC primarily stems from the filtering effect of the inner coupling structure. As shown in Figures 6 and 7, the inner coupling structure reduces the occurrence of accuracy declines due to DSNN mispredictions by 94%. Additionally, the dynamic performance reveals how the inner coupling structure enhances NNFC:

In the scenarios where a DSNN underperforms due to issues such as insufficient training data Mikołajczyk & Grochowski () (as seen in Figure 6, tasks 51-100), inadequate generalization to more challenging tasks Yosinski et al. (); Wei et al. () (Figure 6, tasks 301-350), and catastrophic forgetting when tasks from a new domain are learned Kirkpatrick et al. (); Van de Ven & Tolias () (Figure 7, tasks 1-100), incorrect predictions lead the actual synthesis path to deviate from the CoL, which in turn causes inefficiency and reduced accuracy. During these phases, for NNFC with the inner coupling structure, the attenuation ratio spikes, indicating that a large percentage of neural network predictions are filtered out. Consequently, the inner coupling structure ensures that the synthesis process adheres to the Course of Learning (CoL), effectively mitigating the negative impact of DSNN mispredictions and enhancing reliability.

As the DSNN improves and reaches a relatively stable state (as seen in Figure 6, tasks 101-300, 351-500, and Figure 7, tasks 101-400), the attenuation ratio shows a decreasing trend accordingly. This adaptive adjustment demonstrates how the inner coupling structure dynamically regulates the DSNN’s impact, leveraging neural network contributions while mitigating risks to ensure both efficiency and reliability in program synthesis.

## 4 Related Work

Neural Search Optimization: Neural networks are key for optimizing search in program synthesis. Projects like Kalyan et al. (); Zhang et al. () and Li et al. () use neural networks to provide oracle-like guidance, while Neo Feng et al. (), Flashmeta Polozov & Gulwani (), and Concord Chen et al. () prune search spaces with infeasible partial programs. COOL employs both strategies to enhance efficiency.

Multi-step Program Synthesis: Chain-of-Thought (CoT) Wei et al. () enhances LLMs by breaking tasks into subtasks. Projects like Zhou et al. (); Shi et al. () and Zheng et al. () use this in program synthesis. Compared to CoT, which directly decomposes tasks, CoL does so indirectly by constraining rule applications.

Reinforcement Learning: Reinforcement learning improves neural agents in program synthesis through feedback, as seen in Eberhardinger et al. (); Liu et al. (); Bunel et al. (), Concord Chen et al. (), and Quiet-STaR Zelikman et al. (). NNFC similarly refines control flow but serves an auxiliary role for programmer strategies in synthesis rather than dominating it.

## 5 Conclusion

We explored fine-grained control and flexible modularity for complex program synthesis through the Chain-Oriented Objective Logic (COOL) framework. Inspired by activity charts and control theory, we developed Chain-of-Logic (CoL) and Neural Network Feedback Control (NNFC) to achieve these goals. Static and dynamic experiments across relational, symbolic, and multidomain tasks demonstrated that COOL offers strong efficiency and reliability. We believe that continued research and refinement of CoL and NNFC will inspire advancements not only in program synthesis but also in broader areas of neural network reasoning.

## Appendix A Rule in CoL DSL

In addition to the heuristic vector and keywords, COOL extends the flexibility of the synthesis process by enhancing DSL rules. These enhancements are exemplified in Figure 8, which clarifies the rule introduced in Figure 1.

Figure: Figure 8: DSL rules in COOL. The framework allows for defining rule heads using expressions or terminals, which are enhanced with modifiers for additional attributes. Rule bodies can incorporate any valid expression or function name. Besides, COOL provides built-in operations for accessing program fragment information and facilitates dynamic rule head generation.
Refer to caption: x8.png

## Appendix B Stage Progression Driven by Heuristic Vectors

Let $s$ denote the CoL stage, $h$ donate the heuristic value, and $n$ donate the length of CoL. A rule’s heuristic vector can be mathematically represented as:

$$ $\displaystyle\mathbf{H}=$ $\displaystyle\{(s_{0},h_{0}),(s_{1},h_{1}),\ldots,(s_{n},h_{n})\},\quad n\in \mathbb{N}^{+}$ (1) $$

Upon applying a rule with heuristic vector $\mathbf{H}$, the subsequent stage, $s_{\text{next}}$, can only advance or remain the same, and the next stage should be as close to the current stage as possible:

$$ $\displaystyle\min{s_{\text{next}}}\quad\text{such that}\quad\exists$ $\displaystyle(s_{\text{next}},h_{\text{next}})\in\mathbf{H}\quad\text{and} \quad s_{\text{next}}\geq s_{\text{current}}$ (2) $$

## Appendix C Neural Networks in DSNN

COOL performs synthesis tasks using Three-Address Code (TAC), also utilized as input by DSNN. TAC serves as an intermediate representation (IR), allowing program synthesis to be conducted without the constraints of specific DSL syntax or the machine code format of the execution platform Sujeeth et al. (). As TAC embodies both the graphical properties of a syntax tree and the sequential properties of execution, the design of the neural network must be capable of capturing these dual characteristics.

Figure: Figure 9: Layer architecture of neural networks in DSNN. Each neural network consists of embedding layers for domains, types, identifiers, strings, and operators, followed by GAT layers for tree feature extraction. LSTM layers provide sequential modeling for programs, with fully connected layers combining the outputs. Various output layers handle domain identification for task detection, feasibility judgment for search space pruning, tree jumps, stage prediction, heuristic constraint (sign and value), and constraint on the type of rule’s head (expression or terminal) for search guidance.
Refer to caption: x9.png

The detailed layer architecture of neural networks in DSNN is illustrated in Figure 9. The processing flow consists of the following steps:

- 1.
Embedding Node Features: We start by employing embedding layers with learning capabilities. These layers convert categorical inputs into dense, continuous vectors, which enhances the stability and efficiency of subsequent processing layers Hrinchuk et al. ().
- 2.
Graph Feature Extraction: Next, we use a Graph Neural Network (GNN) to extract graph features from each line of TAC code Drori et al. (); Wu et al. (). To adaptively extract intricate details such as node types, graph attention (GAT) layers are applied after the embedding layers Velickovic et al. ().
- 3.
Sequential Feature Processing: We adopt Long Short-Term Memory (LSTM) networks to capture the sequential features inherent in TAC Chen et al. (); Nye et al. (). Recognizing the equal importance of each TAC line, bidirectional LSTM layers are employed following the GAT layers to enrich the contextual understanding Huang et al. ().
- 4.
Multi-Head Output: Finally, the processed data is channeled through multiple output layers to prevent task interference and ensure clarity in results.

Figure 4 (right) illustrates using three neural network units arranged in series to construct the internal coupling structure of DSNN. Labeling these neural networks with A, B, and C in order of their sequence, Table 6 details the specific input features for each network: Neural network B receives its input feature ”applied” from network A’s output feature ”jumps,” while network C’s input features ”applied” and ”next stage” are derived from the output features ”jumps” and ”next stage” of network B.
The output features of three neural network units are consistent and comparable, Table 7 presents the output features of the neural networks.

It is necessary to note that the DSNNs without internal coupling structures in Table 3 contain only neural network A.

**Table 6: Input features of neural networks in DSNN. Each entry specifies the feature, its size, and the neural networks it pertains to, along with a description of its role. These features contribute to the neural network’s understanding of the syntax tree’s structure and semantics, aiding in the accurate synthesis of programs.**
| Feature | Feature Size | Neural Network | Signification |
| --- | --- | --- | --- |
| grounded | 2 | A, B, C | The node is in a fully specified expression. |
| domain | 1 | A, B, C | Domain of the subtask represented by the subtree where the node is located. |
| root | 2 | A, B, C | The tree representing the subtask is rooted at this node. |
| non-terminal | 2 | A, B, C | The node is a non-terminal. |
| type | 1 | A, B, C | Type of the node. |
| identifier | 1 | A, B, C | Identifier of the node. |
| string | 1 | A, B, C | The node contains a string as the immediate value. |
| number | 1 | A, B, C | The node contains a number as the immediate value. |
| operator | 1 | A, B, C | The node is an operator. |
| current stage | 1 | A, B, C | Current CoL stage (valid when this node is grounded). |
| operand position | 3 | A, B, C | Placement of nodes in a binary operation tree (left operand node, right operand node, operation node). |
| applied | 1 | B, C | A rule is applied to the subtree rooted at this node (derived from the output feature “jumps” of the previous neural network). |
| next stage | 1 | C | The CoL stage to advance to after applying the rule (derived from the output feature “next stage” of the previous neural network). |

**Table 7: Output features of neural networks in DSNN. These features provide comprehensive optimizations for CoL DSL during program synthesis, including task detection, search space pruning, and search guidance.**
| Feature | Feature Size | Neural Network | Signification |
| --- | --- | --- | --- |
| domain | 2 | A, B, C | Relevance of task domains to DSNN. |
| feasibility | 2 | A, B, C | Feasibility of synthesizing the complete program. |
| jumps | max_tree_depth*3 | A, B, C | The path from the tree’s root to the subtree’s root where the rule is applied (jump left, right, or stop in each step). |
| next stage | 1 | A, B, C | The CoL stage to advance to after applying the rule. |
| heuristic sign | 2 | A, B, C | Sign of the rule’s heuristic value. |
| heuristic value | 1 | A, B, C | Rule’s heuristic value. |
| expression | 2 | A, B, C | Type of rule’s head (expression or terminal). |

## Appendix D Signal Clipper

The Clipper, as illustrated in Figure 4 (left), caps signals that do not align with the DSNN guidance to zero:

$$ $u_{2}=\begin{cases}0&\text{if }u_{1}>0\text{ and current rule doesn't align with}\\ &\text{ the guidance and there exists another rule in }\\ &\text{ the search space that aligns with the guidance}\\ u_{1}&\text{otherwise}\end{cases}$ (3) $$

### D.1 A* Search in Program Synthesis

During the exploration phase of program synthesis, we leverage the A* algorithm to perform the heuristic search. This algorithm is renowned for its efficacy in discrete optimization tasks, utilizing heuristic guidance to navigate the search space effectively Hart et al. (). Each action or decision is associated with a specific cost in this context. By evaluating the cumulative cost of actions taken so far and the estimated costs of future actions, A* seeks to determine the path with the least overall cost. In our approach, heuristic values promoting forward progression are considered rewards. Therefore, we treat them as negative costs in calculations. Algorithm 1 illustrates the implementation details.

Figure: Algorithm 1 Search Algorithm for DSL Program Synthesis

## Appendix E Implementation Toolchain

To fully implement the CoL DSL and adapt it to NNFC, we choose to build COOL from the ground up rather than extending existing DSL frameworks such as Xtext Bettini () or Groovy King (). We use C++ as the primary language to meet the execution efficiency requirements for the numerous tree operations inherent in the DSL program synthesis process. For development efficiency, we utilize Lex Lesk & Schmidt () and YACC Johnson et al. () for syntax and semantic parsing, respectively. The neural network components are implemented in Python, leveraging the PyTorch library Imambi et al. () to support machine learning tasks effectively. Table 8 shows the detailed code effort involved in developing the different components of COOL across various programming languages.

**Table 8: Code Effort in COOL. Components of COOL are developed across different programming languages.**
| Language | Lines | Components |
| --- | --- | --- |
| C++ | 60k | framework and CoL DSL solver |
| Python | 3k | DSNN |
| Lex | 1k | syntax parser |
| YACC | 2k | semantic parser |

## Appendix F Optimization strategy

In practice, we observe that as the CoL length increases, the frequency of skipping stages rises. While skipping can lead to shorter synthesis paths and improved efficiency, it may cause task failures by omitting necessary stages. To manage this, we propose two strategies:

- 1.
Gradient-Based Regulation: We employ gradient-based regulation, a widely used strategy in program synthesis Cui & Zhu (); Liang et al. (); Chaudhuri et al. (). By evaluating the slope or rate of change between consecutive stages, gradients help us make dynamic adjustments to synthesis paths. In our approach, we regulate skipping by applying a gradient to the heuristic values at each stage in the CoL. We encourage skipping when the heuristic gradient from one stage to the next is positive. Conversely, if the gradient is negative, we suppress skipping.
- 2.
NNFC Regulation: Once we establish a feasible synthesis path, we can treat partial programs derived through skipping as infeasible. Then, we will utilize the feedback loop to suppress unwarranted skipping actions. However, since these partial programs might still contain feasible solutions, we need further investigation to understand and fully leverage the potential impact of this data.

In our experiments, we prioritize accuracy by suppressing skipping behavior, ensuring essential stages are included in synthesis paths.

## Appendix G Future work

In future work, we aim to enhance the capability of the COOL framework by exploring the implementation of CoL and NNFC in more complex scenarios, such as managing dependencies among DSL libraries and object-oriented development. We plan to facilitate community collaboration by developing more DSL libraries to expand COOL’s applications. Additionally, we are interested in integrating COOL with language models. As these models evolve, ensuring ethical and accurate reasoning becomes increasingly crucial Jacovi & Goldberg (); Chen et al. (); Li et al. (). The COOL framework, including CoL’s constraints on rule application and NNFC’s structured agent interactions, helps to enhance reasoning faithfulness, preventing harmful reasoning logic. We hope our work will serve as a reliable bridge for interaction and understanding between human cognitive processes and language model reasoning.

## Appendix H CoL DSL for Relational Tasks

We present only the specific code for the CoL DSL group, while the code for the DSL and DSL (Heuristic) groups, referenced in Table 3, is not displayed. This omission is because their differences from the CoL DSL group are confined to their heuristic vectors. In both the DSL and DSL (Heuristic) groups, the heuristic vectors have a dimension of 1. However, the DSL group employs a fixed heuristic value of -1, whereas the DSL (Heuristic) group utilizes variable values. The experimental codes are presented concisely, showcasing only the framework. Please refer to the attached [supplementary materials](https://github.com/default01234/COOL-DSL) for the complete content.

//1 Separate Relations and Gendersexpr:@(9){(a) is (b)s grandson}{    return:(a) is male &  (a) is (b)s grandchild & (b) is (a)s grandparent;}...//2 Reason Inverse Relationsexpr:@(0,7,3){(a) is (b)s grandchild}{    if(this expr.exist subexpr{(b) is (a)s grandparent} == false){        return: (a) is (b)s grandchild & (b) is (a)s grandparent;    }    abort;}...//3 Reason Indirect Relationsexpr:@(0,0,5){(a) is (b)s sibling}{    placeholder:p1;    while(this expr.find subexpr{(p1) is (a)s sibling}){        if(this expr.exist subexpr{(p1) is (b)s sibling} == false && p1 != b){return: (a) is (b)s sibling & (p1) is (b)s sibling;        }        p1.reset();    }    p1.reset();    while(this expr.find subexpr{(p1) is (a)s parent}){        if(this expr.exist subexpr{(p1) is (b)s parent} == false){return: (a) is (b)s sibling & (p1) is (b)s parent;        }        p1.reset();    }    p1.reset();    while(this expr.find subexpr{(p1) is (a)s pibling}){        if(this expr.exist subexpr{(p1) is (b)s pibling} == false){return: (a) is (b)s sibling & (p1) is (b)s pibling;        }        p1.reset();    }    p1.reset();    while(this expr.find subexpr{(p1) is (a)s grandparent}){        if(this expr.exist subexpr{(p1) is (b)s grandparent} == false){return: (a) is (b)s sibling & (p1) is (b)s grandparent;        }        p1.reset();    }    p1.reset();    abort;}...//4 Recombine Relations and Genders, Eliminate Irrelevant Relationsexpr:@(0,0,0,8){(a) is (b)s ($relation)}{    //immediate family    placeholder:p1;    while(this expr.find subexpr{(a) is (b)s grandchild}){        if(this expr. exist subexpr{(a) is male}){return: $relation == "grandson";        }        if(this expr.exist subexpr{(a) is female}){return:$relation == "granddaughter";        }        p1.reset();    }    p1.reset();    while(this expr.find subexpr{(a) is (b)s child}){        if(this expr. exist subexpr{(a) is male}){return: $relation == "son";        }        if(this expr.exist subexpr{(a) is female}){return:$relation == "daughter";        }        p1.reset();    }    ...    abort;}...expr:@(0,0,0,10){a & ($b == c)}{    return:b == c;}...

## Appendix I CoL DSL for Symbolic Tasks

// Common Transformationsexpr:@(2,2,2,2,2){0+#a}{    return:a;}expr:@(2,2,2,2,2){#a+0}{    return:a;}...// 1 Expand Square Termsexpr:@(5,0,0,0){(#?a + #?b)^2}{    return:a^2+2*a*b+b^2;}expr:@(5,0,0,0){(#?a - #?b)^2}{    return:a^2+(-2)*a*b+b^2;}expr:@(6,0,0,0){(#a*#b)^2}{    return:a^2*b^2;}...// 2 Expand Bracketed Termsexpr:@(0,4,0,0,0){#?a-(#?b+#?c)}{    return:a-b-c;}expr:@(0,3.8,0,0,0){(#?b+#?c)*#?a}{    return:b*a+c*a;}...// 3 Extract Coefficientsexpr:@(0,0,5,0){$x*a}{    return:a*x;}expr:@(0,0,4.8,0){(immediate:a*$x)*(immediate:b*$x)}{    new:tmp = a*b;    return:tmp*x^2;}expr:@(0,0,4.6,0){$x*(a*$x)}{    return:a*x^2;}...// 4 Re-Express Negative Coefficientsexpr:@(0,0,0,3.5,0){#a-$x}{    placeholder:p1;    placeholder:p2;    if(x.exist subexpr{p1*p2}){        abort;    }    return:a+(-1)*x;}expr:@(0,0,0,3.7,0){#a-immediate:b*$x}{    new:tmp = 0 - b;    return:a+tmp*x;}...//5 Arrange Terms in Descending Order, Combine Like Termsexpr:@(0,0,0,0,3){immediate:a*$x+immediate:b*$x}{    new:tmp = a+b;    return:tmp*x;}expr:@(0,0,0,0,2.8){a1*$x+a2*$x^2}{    return:a2*x^2+a1*x;}...//6 Convert to Standard Formexpr:@(0,0,0,0,0,2.5){a*$x^2+b*x == #d}{    return: a*$x^2+b*x + 0 == d;}expr:@(0,0,0,0,0,2.5){b*$x == $d}{    if(d.exist subexpr{x^2}){        return: 0*x^2 + b*x + 0 == d;    }else {        abort;    }}expr:@(0,0,0,0,0,-4){$a==$b}{    return:b==a;}...//7 Apply Solution Formula@(0,0,0,0,0,0,0,10){a*$x^2+b*x+c==0}{    if(b^2-4*a*c<0){        x="null";    }    else {        new:x1=(-b+(b^2-4*a*c)^0.5)/(2*a);        new:x2=(-b-(b^2-4*a*c)^0.5)/(2*a);        x={x1,x2};    }};

## Appendix J Relational Tasks at Difficulty Level A

#load(family) // Load the CoL DSL library for Relational Tasksnew:relation = "";// [Francisco]’s brother, [Wesley], recently got elected as a senator. [Lena] was unhappy with her son, [Charles], and his grades. She enlisted a tutor to help him. [Wesley] decided to give his son [Charles], for his birthday, the latest version of Apple watch.// Ans: (Francisco) is (Lena)s brothernew:Lena = "Lena";new:Charles = "Charles";new:Wesley = "Wesley";new:Francisco = "Francisco";(Charles) is (Lena)s son & (Wesley) is (Charles)s father & (Francisco) is (Wesley)s brother & (Francisco) is (Lena)s ($relation);relation-->"#FILE(SCREEN)";// [Clarence] woke up and said hello to his wife, [Juanita]. [Lynn] went shopping with her daughter [Felicia]. [Felicia]’s sister [Juanita] was too busy to join them.// Ans: (Lynn) is (Clarence)s mother-in-lawnew:Clarence = "Clarence";new:Juanita = "Juanita";new:Felicia = "Felicia";new:Lynn = "Lynn";(Juanita) is (Clarence)s wife & (Felicia) is (Juanita)s sister & (Lynn) is (Felicia)s mother & (Lynn) is (Clarence)s ($relation);relation-->"#FILE(SCREEN)";...

## Appendix K Relational Tasks at Difficulty Level B

#load(family) // Load the CoL DSL library for Relational Tasksnew:relation = "";// [Antonio] was happy that his son [Bernardo] was doing well in college. [Dorothy] is a woman with a sister named [Tracy]. [Dorothy] and her son [Roberto] went to the zoo and then out to dinner yesterday. [Tracy] and her son [Bernardo] had lunch together at a local Chinese restaurant.// Ans: (Roberto) is (Antonio)s nephewnew:Antonio = "Antonio";new:Bernardo = "Bernardo";new:Tracy = "Tracy";new:Dorothy = "Dorothy";new:Roberto = "Roberto";(Bernardo) is (Antonio)s son & (Tracy) is (Bernardo)s mother & (Dorothy) is (Tracy)s sister & (Roberto) is (Dorothy)s son & (Roberto) is (Antonio)s ($relation);relation-->"#FILE(SCREEN)";// [Bernardo] and his brother [Bobby] were rough-housing. [Tracy], [Bobby]’s mother, called from the other room and told them to play nice. [Aaron] took his brother [Bernardo] out to get drinks after a long work week. [Tracy] has a son called [Bobby]. Each day they go to the park after school. ans: (Bobby) is (Aaron)s brothernew:Aaron = "Aaron";new:Bernardo = "Bernardo";new:Bobby = "Bobby";new:Tracy = "Tracy";(Bernardo) is (Aaron)s brother & (Bobby) is (Bernardo)s brother & (Tracy) is (Bobby)s mother & (Bobby) is (Tracy)s son & (Bobby) is (Aaron)s ($relation);relation-->"#FILE(SCREEN)";...

## Appendix L Symbolic Tasks at Difficulty Level A

#load(quadratic) // Load the CoL DSL library for Symbolic Tasksnew:x = 1;6*$x^2 == 3*x - 7;x-->"#FILE(SCREEN)";($x - 6)*(x + 3) == x;x-->"#FILE(SCREEN)";...

## Appendix M Symbolic Tasks at Difficulty Level B

#load(quadratic) // Load the CoL DSL library for Symbolic Tasksnew:x = 1;$x*($x + 11) == 16*($x + 22);x-->"#FILE(SCREEN)";$x*(36*$x + 50) - 11*(19 - 30*$x) == $x^2;x-->"#FILE(SCREEN)";...

## Appendix N Multidomain Tasks

#load(quadratic) // Load the CoL DSL library for Symbolic Tasks#load(family) // Load the CoL DSL library for Relational Tasksnew:x = 1;$x^2 - 4*$x == 6;x --> "#FILE(SCREEN)";...new:relation = "";// [Dolores] and her husband [Don] went on a trip to the Netherlands last year. [Joshua] has been a lovely father of [Don] and has a wife named [Lynn] who is always there for him.// Ans: (Dolores) is (Lynn)s daughter-in-lawnew:Lynn = "Lynn";new:Joshua = "Joshua";new:Don = "Don";new:Dolores = "Dolores";(Joshua) is (Lynn)s husband & (Don) is (Joshua)s son & (Dolores) is (Don)s wife & (Dolores) is (Lynn)s ($relation);relation-->"#FILE(SCREEN)";...

## Appendix O Partial Program as Neural Network Input

"codeTable": [    {        "boundtfdomain": "",        "grounded": false,        "operand1": {            "argName": "x",            "argType": "identifier",            "changeable": 1,            "className": "",            "isClass": 0        },        "operand2": {            "argName": "2",            "argType": "number",            "changeable": 0,            "className": "",            "isClass": 0        },        "operator": {            "argName": "^",            "argType": "other"        },        "result": {            "argName": "1418.4",            "argType": "identifier",            "changeable": 1,            "className": "",            "isClass": 0        },        "root": false    },    ...]

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: n/a
soundness: 1
strengths: n/a
summary: The submission discusses a technique for synthesis of logical rules from data.
weaknesses: The paper fails at presenting the contribution, and I am not able to provide a review of the actual contribution due to the presentation.

After reading the paper, I do not know
* what the form of the actual inputs to the system are, and how a user would interact with it
* how the components of the system are constructed (i.e., what is "DSNN" - is it a feed-forward network? An LLM? What shape does its input take? What is its actual output? Similarly, THD, SSPH, SGH are only explained at the highest possible level)
* what ideas can transfer to other contexts

The only way forward I see is to rewrite the entire submission. I would recommend to ask colleagues who are not closely familiar with this project to review draft submissions, and provide feedback on whether they understand what you are trying to present, before submitting the paper again.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 1
questions: See weaknesses section for questions
soundness: 1
strengths: The paper is very thorough and complex. There is a lot of detail and technical content. I have never seen a control system like that used for the neural network feedback control and it seems like a very novel way of synthesizing programs. While I don't fully understand it, the authors analysis of inner coupling as a key factor to the improved synthesis quality of COOL seems interesting.
summary: This paper describes a form of program synthesis called COOL and applies it to the CLUTTR dataset (logical family relation problems embedded in natural language) and a toy quadratic equation dataset. COOL is a combination of a "Chain of Logic" (CoL) DSL and a neural network feedback control system (NFCC) for guiding search. CoL is a framework for sketching templates of useful DSL rule applications to guide the problem solving process. For example, for the CLUTTR dataset, the CoL DSL contains rules "Separate relations and genders", "Reason inverse relations", "Reason Indirect Relations", "Recombine relations and genders, eliminate irrelevant info", while the CoL DSL for quadratic equations contains rules such as 0 + a = a, (a + b) ^ 2 = a^2 + 2ab + b^2. NFCC is a complicated system that uses neural networks to guide and control the synthesis process. Program search is conducted with A* using heuristics I believe from the neural network. The authors show that COOL achieves 100% accuracy, uses fewer tree operations, and spends much less time solving compared to the baseline of a normal DSL without NNFC.
weaknesses: I find this paper very hard to understand.

The method is very complicated, with a lot of moving parts. The method is never described end-to-end in a coherent way — instead, bits of detail are mentioned throughout the paper, but I never get the sense that I fully understand what is going on. Some questions that I cannot answer from reading the paper: how are the neural networks trained? how are the heuristics computed? what is the input and output for a given synthesis problem? what are tree operations? One of the datasets, the quadratic equation dataset, is only described at a high level — I don't even know what the problems look like exactly. 

I would recommend writing the paper in a more linear order. Even if there is not space in the main text, the appendix can contain a full description of the (1) problem input and output, (2) overview of algorithm that transforms input to output prediction, (3) description of how your techniques plug in to the algorithm at different parts. You can also include more detailed definitions for things like tree operations, the DSL rule definitions, etc.

Overall, I am baffled at this paper. I am not foreign to program synthesis, yet most of the content of this paper is beyond me. 

To summarize, the technique is extremely complicated and not adequately explained. This is the main weakness of the paper. 

Beyond this, the evaluation is not very convincing. This is in large part due to the poor explanation — as stated earlier, I don't have a full understanding of what the input and output for each problem is, or even how search is conducted. So it's hard to know what to make of the accuracy metrics or time spent. "tree operation" and "transformation pair" are not defined either.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The process by which DSNNs generate error signals from partial programs is not clearly explained. What constitutes a “correct” path in the training process? Is there a universal ground truth, or does it vary by domain? Also how much data does each DSNN require to perform effectively?
2. Did the dynamic experiments include any resource constraints (e.g., time or computational power limits)?
3. I did not understand what “multidomain tasks” are and what is their role in the experiments. Can you clarify please?
soundness: 2
strengths: 1. The ideas introduced in the paper are interesting. The Chain of Logic structure mirrors human problem-solving steps, allowing for more intuitive management of the program synthesis process within the DSL. This hierarchical structuring likely enhances interpretability.
2. The paper provides some experimental validation, with ablation studies showing the contribution of each component.
summary: The paper introduces a neuro-symbolic program synthesis method that combines rule-based logic with neural feedback, termed Chain-Oriented Objective Logic (COOL). This technique allows fine-grained control over search processes within DSL-induced program spaces. Modularity is achieved through a component called Chain of Logic (CoL), inspired by the activity diagram model from software engineering and system control. CoL structures complex rule applications by assigning heuristic values to rules, enabling more efficient search strategies. Selecting specific rules based on heuristic values at particular stages (e.g., returning, jumping, or terminating synthesis branches) enhances search efficiency by aligning rule applications with promising paths, akin to human problem-solving approaches.

The paper also introduces Neural Network Feedback Control (NNFC), which uses multiple neural networks in series with specific heads for task detection, search pruning, and guidance. This prioritizes rules and tasks, allowing the synthesis process to bypass infeasible paths early, thereby improving the quality of synthesized outputs.

Experiments were conducted on static and dynamic benchmarks, categorized by difficulty level. The results validate both the general applicability of the approach and impressive improvements in accuracy and search efficiency under different task demands. A detailed ablation study clarifies the contribution of each component.
weaknesses: 1. The CoL framework is highly domain-specific, and it is unclear how adaptable it would be for tasks outside its designed domain. The reliance on activity diagrams suggests that applying CoL in other fields may require substantial modification, limiting generalizability.
2. The description of NNFC is somewhat opaque, especially regarding how incorrect neural network predictions are “suppressed through filtering.”
3. There is insufficient detail about the design choices behind the Domain-Specific Neural Networks (DSNNs), such as how they are trained and what training data or ground truth is used, especially when handling partial programs.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: See Weaknesses section for a series of questions regarding this system.
soundness: 1
strengths: The results seem very impressive, showing both an improvement in performance and a reduction in computational cost.
summary: I do not feel I have a sufficient understanding of this paper to summarize it in detail. See the Weaknesses section for details on this. My best attempt to summarize this paper follows.

COOL is a system for neurosymbolic program synthesis, in which programs are generated incrementally in a DSL annotated with specific kinds of assertions and guides that allow for specific guidance of the neural network that drives the search. This leads to significant pruning, improving accuracy while reducing the size of the search space.
weaknesses: This paper provides insufficient details and precision for a reader to understand the nature of the algorithm. The following questions remain unanswered by the introduction, figures, and methods section: What is the artifact a user of this system needs to provide? The baseline is to provide a DSL, what additional information needs to be provided to use COOL?

The details on the amount of effort a user needs to complete to make a DSL in COOL are crucial because without this information, it is unclear how much of the benefits shown in the results section are downstream of the user providing more scaffolding and thus making the task easier.

The figures do not help clarify this to a reader as it is not clear whether they represent the user-provided activity diagram, an execution trace for an activity diagram, or a schematic explanation of how the algorithm functions. It is also unclear what heuristic vectors are, and when logic jumps / abort commands should be used, or even if these are being written by a programmer or the output of some neural network.

Additionally, it is never specified how the DSNN network is trained. How is the data gathered for the DSNN? Is it trained via supervised learning, some kind of wake/sleep loop, or reinforcement learning? What does it mean for the DSNN to be pretrained?