Data-Efficient Learning with Neural Programs

Authors: alaia  solko-breslin
seewon  choi
ziyang  li
neelay  velingker
rajeev  alur
mayur  naik
eric  wong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:19:35

Abstract: Many computational tasks can be naturally expressed as a composition of  a DNN followed by a program written in a traditional programming language or an API call to an LLM. We call such composites "neural programs" and focus on the problem of learning the DNN parameters when the training data consist of end-to-end input-output labels for the composite.  When the program is written in a differentiable logic programming language, techniques from neurosymbolic learning are applicable, but in general, the learning for neural programs requires estimating the gradients of black-box components. We present an algorithm for learning neural programs, called ISED, that only relies on input-output samples of black-box components. For evaluation, we introduce new benchmarks that involve calls to modern LLMs such as  GPT-4 and also consider benchmarks from the neurosymbolic learning literature. Our evaluation shows that for the latter benchmarks, ISED has comparable performance to state-of-the-art neurosymbolic frameworks. For the former, we use adaptations of prior work on gradient approximations of black-box components as a baseline, and show that ISED achieves comparable accuracy but in a more data- and sample-efficient manner.

Main paper: 
 introduction: 
 Many computational tasks cannot be solved by neural perception alone but can be naturally expressed as a composition of a neural model M θ followed by a program P written in a traditional programming language or an API call to a large language model (LLM). We call such composites "neural programs" and study the problem of learning neural programs in an end-to-end manner with a focus on data and sample efficiency. One problem that is naturally expressed as a neural program is scene recognition [29] , where M θ classifies objects in an image and P prompts GPT-4 to identify the room type given these objects (Fig. 1 ).Neurosymbolic learning [2] is one instance of neural program learning in which P takes the form of a logic program. DeepProbLog (DPL) [14] and Scallop [13] are frameworks that extend ProbLog and Datalog, respectively, to ensure that the symbolic component P is differentiable. This differentiability requirement is what facilitates learning in many neurosymbolic learning frameworks. There are also abductive learning frameworks that do not explicitly differentiate programs. Instead, they require that the symbolic component expose a method for abducing the function's inputs for a given output, often using Prolog for the symbolic component as a result [6, 23] . While logic programming languages are expressive enough for these frameworks to solve tasks such as sorting [14] , visual question answering [13] , and path planning [23] , they offer restricted features and a narrow range of libraries, making them incompatible with calls to arbitrary APIs or to modern LLMs.Learning neural programs when P is not expressed as a logic program is a difficult problem because gradients across black-box programs cannot be computed explicitly. One possible solution is to use REINFORCE [26] to sample symbols from distributions predicted by M θ and compute the expected reward using the output label. However, REINFORCE is not sample-efficient as it produces a weak learning signal, especially when applied to programs with a large number of inputs. There are other REINFORCE-based methods that can be applied to the neural program learning setting, namely IndeCateR [21] and Neural Attention for Symbolic Reasoning (NASR) [5] . However, IndeCateR struggles with sample efficiency despite providing lower variance than REINFORCE, and NASR performs poorly when intermediate labels are unavailable for pretraining. Another possible solution is Approximate Neurosymbolic Inference (A-NeSI) [24] , which trains a neural network to estimate the gradient of P , but learning the surrogate neural network becomes more difficult as the complexity of P increases. Moreover, the additional neural models in the learning framework in A-NeSI results in data inefficiency.In this paper, we propose an algorithm for learning neural programs, based on reinforcement learning, which is compatible with arbitrary programs. Our approach, called ISED (Infer-Sample-Estimate-Descend), yields a framework that expands the applicability of neural program learning frameworks by providing a data-and sample-efficient method of training neural models with randomly initialized weights. ISED uses outputs of M θ as a probability distribution over inputs of P and samples representative symbols u from this distribution. ISED then computes outputs v of P corresponding to these symbols. The resulting symbol-output pairs can be viewed as a symbolic program consisting of clauses of the form if symbol = u then output = v summarizing P . The final step is to estimate the gradient across this symbolic summary, inspired by ideas from the neurosymbolic learning literature, to propagate loss across the composite model.Our evaluation considers 16 neural program benchmark tasks. Our results show that ISED outperforms purely neural networks and CLIP [19] on neural program tasks involving GPT-4 calls. Additionally, ISED outperforms neurosymbolic methods on 9 of the 14 benchmarks tasks that can be encoded in logic programming languages. ISED is also the top performer on 8 out of the 16 benchmark tasks when compared to REINFORCE-based and black-box gradient estimation baselines. Furthermore, we show that ISED is more data-and sample-efficient than baseline methods.In summary, the main contributions of this paper are as follows: 1) we introduce neural programs as a generalization of neurosymbolic programs, 2) we introduce new tasks involving neural programs that use Python and calls to GPT-4 called neuroPython and neuroGPT programs, respectively, 3) we present ISED, a general algorithm for data-and sample-efficient learning with neural programs, and 4) we conduct a thorough evaluation using existing techniques against a diverse set of benchmarks. 
neural programs: 
 Problem Statement. In the neural program learning setting, we attempt to optimize model parameters M θ which are being supervised by a fixed program P . Specifically, we are given a training dataset D of length N containing input-output pairs, i.e., D = {(x 1 , y 1 ), . . . (x N , y N )}. Each x i represents unstructured data (e.g., image data) whose corresponding structured data (intermediate labels) are not given. Each y i is the result of applying P to the structured data corresponding to x i . Given a loss function L, we want to minimize the loss of L(P (M θ (x i )), y i ) for each (x i , y i ) pair in order to optimize θ. Loss minimization is straightforward when there is some mechanism for automatically differentiating programs, but we focus on the setting of optimizing θ without assuming the differentiability of P . We now introduce three motivating applications that can be framed in this setting, namely classifying images of leaves [9] , scene recognition [29] , and hand-written formula evaluation (HWF) [12] .Leaf Classification. We consider a real-world example that deals with the problem of classifying leaf images. Traditional neural methods predict the species directly, without explicit notion of leaf features such as margin, shape, and texture, resulting in solutions that are data-inefficient, inaccurate, and harder to understand.We instead present a neural programming solution, making use of leaf classification decision trees [22] . These decision trees allow identifying plant species based on the visible characteristics of their leaves. Here, the neural model takes a leaf image and predicts its shape, margin, and texture. The program can then be written in two ways: one implementation involves encoding the decision tree in Python; another involves constructing a prompt using the predicted leaf features and calling GPT-4 (see Fig. 2 ). The latter is possible because ISED allows the use of black-box programs, so programs can also use state-of-the-art foundation models such as GPT-4 for computation.Scene Recognition. The goal of this task is to classify images according to their room types. The model receives an image from a scene dataset [16] and predicts among the 9 different room types: bedroom, bathroom, dining room, living room, kitchen, lab, office, house lobby, and basement.The traditional neural solution trains a convolutional neural network that directly predicts the room type. On the other hand, the neural program solution decomposes the task into detecting objects in the scene and identifying the room type based on those objects. We use an off-the-shelf object detection model YOLOv8 [20] and finetune it with a custom convolutional neural network to output labels related to scene recognition. We then make a GPT-4 call to predict the most likely room type given the list of detected objects.Hand-written Formula. In this task, a model is given a list of hand-written symbols containing digits (0-9) and operators (+, -, ×, and ÷) [12] . The dataset contains length 1-7 formulas free of syntax or divide-by-zero errors. The model is trained with supervision on the evaluated floating-point result without the label of each symbol. Since inputs are combinatorial and results are rational numbers, end-to-end neural methods struggle with accuracy. Meanwhile, neurosymbolic methods for this task either use specialized algorithms [12] or handcrafted differentiable programs [13] .With ISED, the program can be written in just a few lines of Python. It takes in a list of characters representing symbols, and simply invokes the Python eval function on the joined expression string. The hwf evaluation function can be used just like any other PyTorch [17] module since ISED internally performs sampling and probability estimation to estimate the gradient. 
learning neural programs: 
 In this section, we present the intuition behind ISED and the values it approximates. Next, we introduce the programming interface for ISED, which lays the groundwork for presenting the algorithm. We then formally describe the steps of ISED.Assuming P is a black-box, we can collect symbol-output samples (u, v) from P . Such collection of samples can be viewed as a summary logic program consisting of rules of the form if r = u then y = v. For instance, in the task of adding two digits r 1 and r 2 , one rule of the logic program would be r 1 = 1 ∧ r 2 = 2 → y = 3. Techniques from neurosymbolic literature via exact or approximate weighted model counting (WMC) [10] can then be used for computing the gradient across such a summary of P . However, having the complete summary of all combinations of symbols is not feasible for a black-box P . ISED samples symbols from the probability distribution predicted by the neural network M θ , evaluates P on each sample, and takes the gradient across this partial summary of P . This is a good approximation of the complete summary since it is likely to contain symbols with high probability, which contribute the most in exact computation. This approach differs from REINFORCE in how it differentiates through this summary of P . RE-INFORCE rewards sampled symbols that resulted in the correct output through optimizing the log probability of each symbol, weighted by reward values. This weighted-sum style estimation provides a weaker learning signal compared to WMC used by ISED, making learning harder for REINFORCE as the number of inputs to P increases. See Appendix A for further details.ISED allows programmers to write black-box programs that operate on diverse structured inputs and outputs. To allow such programs to interact with neural networks, we define an interface named structural mapping. This interface serves to 1) define the data-types of black-box programs' input and output, 2) marshall and un-marshall data between neural networks and logical black-box functions, and 3) define the loss. We define a structural mapping τ as either a discrete mapping (with Σ being the set of all possible elements), a floating point, a permutation mapping with n possible elements, a tuple of mappings, or a list of up to n elements. We define τ inductively as follows:τ ::= DISCRETE(Σ) | FLOAT | PERMUTATION n | TUPLE(τ 1 , . . . , τ m ) | LIST n (τ )Using this, we may further define data-types such as INTEGER k j = DISCRETE({j, . . . , k}), DIGIT = INTEGER 9 0 , and BOOL = DISCRETE({true, false}). These types give ISED the flexibility learn neural programs with diverse types of inputs and outputs, e.g., PERMUTATION n input and output types for integer list sorting and LIST 9 (LIST 9 (DIGIT)) for sudoku solving.We also define a black-box program P as a function (τ 1 , . . . , τ m ) → τ o , where τ 1 , . . . , τ m are the input types and τ o is the output type. For example, the structural input mapping for the hand-written formula task is LIST 7 (DISCRETE({0, . . . , 9, +, -, ×, ÷})), and the structural output mapping is FLOAT. The mappings suggest that the program takes a list of length up to 7 as input, where each element is a digit or an arithmetic operator, and returns a floating point number.There are two interpretations of a structural mapping: the set interpretation SET(τ ) represents a mapping with defined values, e.g., a digit with value 8; the tensor interpretation DIST(τ ) represents a mapping where each value is associated with a probability distribution, e.g., a digit that is 1 with probability 0.6 and 7 with probability 0.4. We use the set interpretation to represent structured program inputs that can be passed to a black-box program and the tensor interpretation to represent probability distributions for unstructured data and program outputs. These two interpretations are defined for the different structural mappings in Table 1 .Table 1 : Set and tensor interpretations of different structural mappings.Set Interpretation (SET(τ ))Tensor Interpretation (DIST(τ ))DISCRETEΣ Σ {⃗ v | ⃗ v ∈ R |Σ| , vi ∈ [0, 1], i ∈ 1 . . . |Σ|} FLOAT R n/a PERMUTATIONn {ρ | ρ is a permutation of [1, ..., n]} {[ ⃗ v1, ..., ⃗ vn] | ⃗ vi ∈ R n , vi,j ∈ [0, 1], i ∈ 1 . . . n} TUPLE(τ1, . . . , τm) {(a1, ..., am) | ai ∈ SET(τi)} {(a1, .., am) | ai ∈ DIST(τi)} LISTn(τ ′ ) {[a1, ..., aj ] | j ≤ n, ai ∈ SET(τ ′ )} {[a1, .., aj ] | j ≤ n, ai ∈ DIST(τ ′ )}In order to represent the ground truth output as a distribution to be used in the loss computation, there needs to be a mechanism for transforming SET(τ ) mappings into DIST(τ ) mappings. For this purpose, we define a vectorize function δ τ : (SET(τ ), 2 τ ) → DIST(τ ) for the different output mappings τ in Table 2 . When considering a datapoint (x, y) during training, ISED samples many symbols and obtains a list of outputs ŷ. The vectorizer then takes the ground truth y and the outputs ŷ as input and returns the equivalent distribution interpretation of y. While ŷ is not used by δ τ in most cases, we include it as an argument so that FLOAT output mappings can be discretized, which is necessary for vectorization. For example, if the inputs to the vectorizer for the hand-written formula task are y = 2.0 and ŷ = [1.0, 3.5, 2.0, 8.0], then it would return [0, 0, 1, 0]. ⊗ n i=1 σDISCRETE n (r[i], p[i]) TUPLE(τ1, . . . , τm) [δτ i (y[i]) for i ∈ [1, . . . , m]] ⊗ m i=1 στ i (r[i], p[i]) LISTn(τ ′ ) [δ τ ′ (ai) for ai ∈ y] ⊗ n i=1 σ τ ′ (r[i], p[i])We also require a mechanism to aggregate the probabilities of sampled symbols that resulted in a particular output. With this aim, we define an aggregate function σ τ : (SET(τ ), DIST(τ )) → R for different input mappings τ in Table 2 . ISED aggregates probabilities either by taking their minimum or their product, and we denote both operations by ⊗. The aggregator takes as input sampled symbols r and neural predictions p from which r was sampled. It gathers values in p at each index in r and returns the result of ⊗ applied to these values. For example, suppose we use min as the aggregator ⊗ for the hand-written formula task. Then if ⊗ takes r = [1, +, 1] and p as inputs where p[0][1] = 0.1, p [1] [+] = 0.05, and p [2] [1] = 0.1, it would return 0.05.We now formally present the ISED algorithm. For a given task, there is a black-box program P , taking m inputs, that operates on structured data. Let τ 1 , ..., τ m be the mappings for these inputs and τ o the mapping for the program's output. We write P as a function from its input mappings to its output mapping: P : (τ 1 , ..., τ m ) → τ o . For each unstructured input i to the program, there is a neural model M i θi : x i → DIST(τ i ). S is a sampling strategy (e.g., categorical sampling) that samples symbols using the outputs of a neural model, and k is the number of samples to take for each training example. There is also a loss function L whose first and second arguments are the predicted and ground truth values respectively. We present the pseudocode of the algorithm in Algorithm 1 and describe its steps with the hand-written formula task:Infer. The training pipeline starts with an example from the dataset, (x, y) = ([ , , ], 3.0), and uses a CNN to predict these images, as shown on lines 3-4. ISED initializes p = M θ (x).Sample. ISED samples r from p for k iterations using sampling strategy S. For each sample j, the algorithm initializes rj to be the sampled symbols, as shown on lines 6-9. To continue our example, suppose ISED initializes rj = [7, +, 2] for sample j. The next step is to execute the program on rj , as shown on line 10, which in this example means setting ŷj = P (r j ) = 9.0.Estimate. In order to compute the prediction value to use in the loss function, ISED must consider each output y l in the output mapping and accumulate the aggregated probabilities for all sampled symbols that resulted in the output y l . We specify ⊗ as the min function, and ⊕ as the max function in this example. Note that ISED requires that ⊗ and ⊕ represent either min and max or mult and add respectively. We refer to these two options as the min-max and add-mult semirings. We define an accumulate function ω that takes as input an element of the output mapping y l , sampled outputs ŷ, sampled symbols r, and predicted input distributions p. The accumulator performs the ⊕ operation on aggregated probabilities for elements of ŷ that are equal to y l and is defined as follows: for ((x 1 , . . . x m ), y) ∈ D do 3:ω(y l , ŷ, r, p) = ⊕ k j=1 1 ŷj =y l σ τo (r j ,for i ∈ 1 . . . m do 4: p[i] ← M i θi (x i ) ▷ Infer 5:end for 6:for j ∈ 1 . . . k do 7:for i ∈ 1 . . . m do 8: Sample rj [i] from p[i] using S ▷ Sample 9:end for 10:ŷj ← P (r j ) 11:end for12: ŵ ← normalize([ω(y l , ŷ, r, p) for y l ∈ τ o (or y l ∈ ŷ)]) ▷ Estimate 13:w ← δ(y, ŷ)14:l ← L( ŵ, w) end for 18: end procedure ISED then sets w = [ω(y l , ŷ, r, p) for y l ∈ τ o ] in the case where τ o is not FLOAT. When τ o is FLOAT, as for hand-written formula, it only considers y l ∈ ŷ. Next, it performs L 2 normalization over each element in w and sets ŵ to this result. To initialize the ground truth vector, it sets w = δ(y, ŷ). ISED then initializes l = L( ŵ, w) and computes ∂l ∂θi for each input i. These steps are shown on lines 12-15. In our running example, since 9.0 is an incorrect output, the probability of the first symbol being equal to 7 (instead of the correct answer 1) will be penalized while the probabilities for predicting other symbols are unchanged.Descend. The last step is shown on line 16, where the algorithm optimizes θ i for each input i based on ∂l ∂θi using a stochastic optimizer (e.g., Adam optimizer). This completes the training pipeline for one example, and the algorithm returns all final θ i after iterating through the entire dataset.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide details of the compute used in Section 4.2. We also specify the number of runs and epochs for each task (Appendix B) and training time for selected benchmarks (Section 4.5).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
evaluation: 
 In this section, we evaluate ISED and aim to answer the following research questions: RQ1: How does ISED compare to state-of-the-art neurosymbolic, REINFORCE-based, and gradient estimation baselines in terms of accuracy? RQ2: What is the sample efficiency of ISED when compared to REINFORCE-based algorithms? RQ3: How data-efficient is ISED compared to neural gradient estimation methods?We first introduce two new neural program learning benchmarks which both contain a program component that can make a call to GPT-4. We call such models neuroGPT programs.Leaf Classification. In this task, we use a dataset, which we call LEAF-ID, containing leaf images of 11 different plant species [4] , containing 330 training samples and 110 testing samples. We define custom DISCRETE types MARGIN, SHAPE, TEXTURE. With this, we define LEAF-TRAITS = TUPLE(MARGIN, SHAPE, TEXTURE) and LEAF-OUTPUT to be the DISCRETE set of 11 plant species in the dataset. Neural program solutions either prompt GPT-4 (GPT leaf) or use a decision tree (DT leaf). Scene Recognition. We use a dataset containing scene images from 9 different room types [16] , consisting of 830 training examples and 92 testing examples. We define custom types OBJECTS and SCENES to be DISCRETE set of 45 objects and 9 room types, respectively. We freeze the parameters We also consider several tasks from the neurosymbolic literature, including hand-written formula (HWF) evaluation and Sudoku solving. While the solutions to many of these tasks are usually presented as a logic program in neurosymbolic learning frameworks, neural program solutions can take the form of Python programs. We call such models neuroPython programs.MNIST-R. MNIST-R [13, 14] contains 11 tasks operating on inputs of images of handwritten digits from the MNIST dataset [11] . This synthetic test suite includes tasks performing arithmetic (sum 2 , sum 3 , sum 4 , mult 2 , mod 2 , add-mod-3, add-sub), comparison (less-than, equal), counting (count-3-or-4), and negation (not-3-or-4) over the digits depicted in the images. Each task dataset has a training set of 5K samples and a testing set of 500 samples.HWF. The goal of the HWF task is to classify images of handwritten digits and arithmetic operators and evaluate the formula [12] . The dataset contains 10K formulas of length 1-7, with 1K length 1 formulas, 1K length 3 formulas, 2K length 5 formulas, and 6K length 7 formulas.Visual Sudoku. The goal of this task is to solve an incomplete 9x9 Sudoku, where the problem board is given as MNIST digits. We follow the experimental setting of NASR [5] , including their pre-trained MNIST digit recognition models and sudoku solvers. We use the SatNet dataset consisting of 9K training samples and 500 test samples [25] .All of our experiments were conducted on a machine with two 20-core Intel Xeon CPUs, one NVIDIA RTX 2080 Ti GPU, and 755 GB RAM. Unless otherwise noted, the sample count, i.e., the number of calls to the program P per training example, is fixed at 100 for all relevant methods. For additional details on experimental setup, see Appendix B. We apply a timeout of 10 seconds per testing sample, and report the average accuracy and 1-sigma standard deviation obtained from 10 randomized runs.We pick as baselines neurosymbolic methods DeepProbLog (DPL) [14] and Scallop [13] , A-NeSI [24] which performs neural approximation of the gradients, and sampling-based gradient approximation methods REINFORCE [26] , IndeCateR [21] , and NASR [5] . IndeCateR achieves provably lower variance than REINFORCE by using a specialized sampling method (Appendix A), and NASR is a variant specialized for efficient finetuning by using a single sample and a custom reward function. We also use purely neural baselines and CLIP [19] for GPT leaf and scene. CLIP is a multimodal model that supports zero-shot image classification by simply providing names of the output categories.To answer RQ1, we evaluate ISED's accuracy against those of the baselines. ISED matches, and in many cases surpasses, the accuracy of neurosymbolic and gradient estimation baselines. We highlight the results for sum n from MNIST-R and other benchmarks in Table 3 . Tables 7-10 in Appendix C contain results for the remaining MNIST-R tasks, including standard deviations for all tasks. ISED is the top performer on 8 out of the 16 total tasks.On the GPT leaf and scene tasks, ISED outperforms the purely neural baseline by 3.82% and 31.42% respectively, and zero-shot CLIP by 59.80% and 17.50%. For many tasks, A-NeSI is the nonneurosymbolic method that comes closest to ISED, sometimes outperforming our method. However, A-NeSI achieves significantly lower performance than ISED on tasks involving complex programs, namely HWF and sudoku. This is likely due to the difficulty of training a neural model to estimate the output of P and its gradient when P is complex. ISED also outperforms REINFORCE on all but 3 tasks due to the REINFORCE learning signal being weaker for tasks where P involves multiple inputs. NASR outperforms ISED only on sudoku by 2.46% due to NASR being well-suited for fine-tuning as it restricts its algorithm to use a single sample. IndeCateR achieves similar performance compared to ISED on most tasks but achieves significantly lower accuracy on the scene classification task, which has a large input space with maximum 10 objects each with 47 possible values in each scene, demonstrating that IndeCateR is less sample-efficient than ISED. We elaborate more on this point in RQ2.ISED outperforms the neurosymbolic methods on 9 out of 14 tasks that can be written in logic programming languages. Despite treating P as a black-box, ISED even outperforms Scallop on HWF by 0.69% and comes within 1.16% of NGS, a specialized neurosymbolic learning framework that uses abductive reasoning [12] . Furthermore, DPL timed out on 4 tasks, and Scallop timed out on 1 (sudoku). These results demonstrate that even for tasks that can be written in a logic programming language, treating the program as a black-box can often yield optimal results.To answer RQ2, we evaluate the sample efficiency of ISED against REINFORCE, IndeCateR, and IndeCateR+ on adding MNIST digits. IndeCateR+ [21] is a variant of IndeCateR with a sampling method and loss computation customized for higher dimensional setting such as the addition of 16 MNIST digits. We vary the size of the input and output space (sum 8 , sum 12 , sum 16 ) of P as well as the sample count, and report the average accuracy and standard deviation obtained from 5 randomized runs (Tables 4, 11-13).For a lower number of samples, ISED outperforms all other methods on the three tasks, outperforming IndeCateR by over 80% on sum 8 and sum 12 . The experimental findings support the conceptual difference of REINFORCE-based methods providing a weak learning signal compared to ISED (Section 3.1). While ISED achieves accuracy similar to the top performer for sum 8 and sum 12 with a high sample count, it comes second on sum 16 with IndeCateR+ beating ISED by 75.39%. This suggests our approach is limited in scaling to high-dimensional inputs to P , and motivates exploring better sampling techniques, which is the core difference between IndeCateR and IndeCateR+.We now examine how ISED compares to state-of-the-art baselines in terms of data efficiency. We compare ISED and A-NeSI in terms of training time and accuracy on sum 3 and sum 4 . We choose these tasks for evaluation because A-NeSI has been shown to scale well to multi-digit addition tasks [24] . Furthermore, these tasks come from the MNIST-R suite in which we use and 15K for sum 4 ). We plot the average test accuracy and standard deviation vs. training time (over 10 runs) in Figures 3 and 4 , where each point represents the result of 1 epoch.While ISED and A-NeSI learn at about the same rate for sum 3 after about 5 minutes of training, ISED learns at a much faster rate for the first 5 minutes, reaching an accuracy of 88.22% after just 2 epochs (Fig. 3 ). The difference between ISED and A-NeSI is more pronounced for sum 4 , with ISED reaching an accuracy of 94.10% after just 10 epochs while A-NeSI reaches 49.51% accuracy at the end of its 23rd epoch (Fig. 4 ). These results demonstrate that with limited training data, ISED is able to learn more quickly than A-NeSI, even for simple tasks. This result is likely due to A-NeSI training 2 additional neural models in its learning pipeline compared to ISED, with A-NeSI training a prior as well as a model to estimate the program output and gradient. 
limitations and future work: 
 The main limitation of ISED is the difficulty of scaling with the dimensionality of the space of inputs to the program P . There are interesting future directions in adapting and expanding ISED for high dimensionality. Specifically, improvements to the sampling strategy could help adapt ISED to a complex space of inputs. Techniques can be borrowed from the field of Bayesian optimization where such large spaces have traditionally been studied. Furthermore, there is merit to systematically combining white-box and black-box methods. ISED is especially useful when logic programs fail to encode reasoning components. Therefore, we believe that ISED can be used as an underlying engine for a new neurosymbolic language that blends the accessibility of black-box with the performance of white-box methods. 
related work: 
 Neurosymbolic programming frameworks. These frameworks provide a general mechanism to define white-box neurosymbolic programs. DeepProbLog [14] and Scallop [13] abstract away gradient calculations behind a rule-based language. Others specialize in targeted applications, such as NeurASP [28] for answer set programming, or NeuralLog [3] for phrase alignment in NLP. ISED is similar in that it seeks to make classes of neurosymbolic programs easier to write and access; however, it diverges by offering an interface not bound by any specific domain or language syntax.RL and sampling-based neurosymbolic frameworks. ISED incorporates concepts found in the RL algorithm REINFORCE [26] such as the sampling of actions according to the current policy distribution, similar to NASR [5] , and IndeCateR [21] . Other work has proposed a semantic loss function for neurosymbolic learning which measures how well neural network outputs match a given constraint [27] . While this technique resembles ISED in that it samples symbols from their predicted distributions to derive the loss, it relies on symbolic knowledge in the form of a constraint in Boolean logic, whereas ISED allows the program component to be any black-box program.Specialized neurosymbolic methods. The majority of the neurosymbolic learning literature pertains to point solutions for specific use cases [7, 25 ]. In the HWF example, NGS [12] and several of its variants leverage a hand-defined syntax defining the inherent structure within mathematical expressions. Similarly, DiffSort [18] leverages the symbolic properties of sorting to produce differentiable sorting networks. Other point solutions address broader problem setups, such as NS-CL [15] which provides a framework for visual question answering by learning symbolic representations in text and images. For reading comprehension, the NeRd [3] framework converts NL questions into executable programs over symbolic information extracted from text. ISED aligns with all of these point solutions by aiming to solve problems that have thus far required technically specific solutions in order to access the advantages of neurosymbolic learning, but it takes an opposite and easier approach by forgoing significant specializations and instead leverages existing solutions as black-boxes.Differentiable programming and non-differentiable optimization. Longstanding libraries in deep learning have grown to great popularity for their ability to abstract away automatic differentiation behind easy-to-use interfaces. PyTorch [17] is able to do so by keeping track of a dynamic computational graph. Similarly, JAX [1] leverages functional programming to abstract automatic differentiation. ISED follows the style of these frameworks by offering an interface to abstract away gradient calculations for algorithms used in deep learning, but ISED improves upon them by allowing systematic compatibility of non-differentiable functions. 
conclusion: 
 We proposed ISED, a data-and sample-efficient algorithm for learning neural programs. Unlike existing general neurosymbolic frameworks which require differentiable logic programs, ISED is compatible with Python programs and API calls to GPT, and it employs a sampling-based technique to learn neural model parameters using forward evaluation. We showed that for neuroGPT, neuroPython, and neurosymbolic benchmarks, ISED achieves better accuracy than end-to-end neural models and similar accuracy compared to neurosymbolic frameworks. ISED also often achieves superior accuracy on complex programs compared to REINFORCE-based and gradient estimation baselines. Furthermore, ISED learns in a more data-and sample-efficient manner compared to these baselines. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We propose a new framework for learning neural networks in presence of black-box components and there is no societal impact to be pointed out.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not release new data or models, hence the work poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: For datasets, code, and pretrained models used, we cited the original paper and provided the license information in Appendix D. We stated the versions of GPT-4 and YOLO in Appendix B. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: The paper releases code for the proposed learning algorithm, which is well documented with license and training information. The limitations are mentioned in the paper (Sections 4.4 and 5).Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper did not involve any crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper did not involve any crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We explain the differences between ISED and prior techniques using the simple example of sum 2 , where digits are restricted to be between 0-2. Suppose that we are training a neural network M θ for this task, and we are considering the symbol-output sample where the ground truth symbols are 1 and 2, i.e., r 1 = 1, r 2 = 2, and the ground truth output is y = 3. Suppose that the predicted distributions from M θ for r 1 and r 2 are [0.1, 0.6, 0.3] and [0.2, 0.1, 0.7] respectively. We now explain how different methods perform their loss computations.Suppose ISED is initialized with a sample count of 3, and the sampled symbol-output pairs are ((1, 2), 3), ((1, 0), 1), and ((2, 1), 3). We use the add-mult semiring in this example. ISED can be thought of as differentiating through the following summary logic program:r 1 = 1 ∧ r 2 = 2 → y = 3 r 1 = 1 ∧ r 2 = 0 → y = 1 r 1 = 2 ∧ r 2 = 1 → y = 3As a result, the final vector calculated for the loss function before normalization would be     0.0 0.6 * 0.2 0.0 0.6 * 0.7 + 0.3 * 0.1 0.0     where each value corresponds to the probability of the given output (possible outputs are in the range 0-4). Note that if there are duplicate samples, ISED includes the duplicate probabilities in its aggregation. In our implementation, we would perform normalization on this vector and then pass it into the binary cross-entropy loss function, with the ground truth vector being:     0.0 0.0 0.0 1.0 0.0     We would then minimize this loss and update M θ accordingly.If we use the min-max semiring instead, * is replaced by min and + by max in the final vector calculation, resulting in     0.0 0.2 0.0 0.6 0.0      A.2 REINFORCESuppose REINFORCE is also initialized with a sample count of 3, and it samples the same symboloutput pairs. The final reward is computed by element-wise multiplication of the log probability of each sample with its reward value and taking the mean, as follows: and the goal is to optimize M θ to maximize this reward. While this approach resembles ISED's loss computation for the add-mult semiring, it does not involve the mult step. As it rewards possible values instead of possible combinations, the final reward would have been the same when (1,1) and (2,2) were the correct samples, instead of (1,2) and (2,1). Hence, the learning signal is weaker compared to ISED when there is more than one input to P .IndeCateR is an extension of the REINFORCE estimator that is unbiased with a provably lower variance. It assumes and exploits the factoring of the underlying multivariate distribution into independent categorical variables by summing out one dimension while keeping a sample for other dimensions fixed. For each sample drawn, IndeCateR systematically creates additional samples that differ on a single entry by enumerating all possible values for each variable. NASR targets efficient finetuning by setting the sample count to one and customizing the reward function.The loss computation for IndeCateR and NASR are identical to that of REINFORCE, also providing weak signals with fewer samples. Furthermore, both set the reward to 0 for samples leading to incorrect predictions, effectively ignoring them, unlike ISED which penalizes such symbols. Since only the correct symbol contribute to the final reward, the signal is sparser than ISED, making it sample-inefficient.A-NeSI trains two additional neural networks: a prediction model Q θ ′ as a surrogate for P , and a prior model R α for learning the parameters α for the Dirichlet distribution D α .Suppose A-NeSI is also initialized with a sample count of 3. At each training step, A-NeSI first updates α using ŷ = M θ (x). Next, it samples a single symbol from each of the 3 distributions sampled from D α , and uses the sampled symbol-output pair and the standard cross entropy loss to update Q θ ′ . Then, A-NeSI optimizes M θ by minimizing the loss L(Q θ ′ (M θ (r 1 , r 2 ))) using the prediction model instead of P .DeepProbLog (DPL) enumerates all possible proofs for each output and aggregates probabilities accordingly. For example, the proofs for y = 1 include r 1 = 0, r 2 = 1 and r 1 = 1, r 2 = 0. Thus, the probability of this output is 0.6 * 0. and we would pass this vector into some loss function (e.g., cross-entropy), with the same ground truth vector that ISED would use. DPL would then minimize this loss and update M θ accordingly.Suppose Scallop is configured to use the diff-top-1-proofs semiring. This means that for each possible output, Scallop will use the proof of that output with the highest probability. For instance, the most likely proof for y = 1 is r 1 = 1 and r 2 = 0, and the probability of the output y = 1 is 0.6 * 0.2. The final vector calculated would be     0.1 * 0.2 0.6 * 0.2 0.1 * 0.7 0.6 * 0.7 0.3 * 0.7     and we would pass this vector into some loss function (e.g., binary-cross-entropy), with the same ground truth vector that ISED would use. Scallop would then minimize this loss and update M θ accordingly. This probability estimation would change depending on the choice of semiring (e.g., diff-top-k-proofs for a different value of k).For tasks with the MNIST dataset as unstructured data, we employ LeNet [11] , a 2-layer CNN-based model, except for sum 8 , sum 12 , and sum 16 tasks where we choose a smaller 2-layer CNN used by IndeCateR [21] . For HWF, we also use a 2-layer CNN-based model. For leaf classification tasks, images are scaled down and passed to a simple CNN-based network with 4 convolutional layers.For the scene recognition, we use YOLOv8 and a 3-layer convolutional network for neural program methods, 7-layer CNN for the purely neural solution, and CLIP with ViT-B/32. For all tasks included in RQ1, other than sudoku, we remove the final softmax function at the end of each network when evaluating IndeCateR since its sampling procedure yields optimal results without the softmax. We also do that same with REINFORCE if it results in higher accuracy. Since sudoku uses a pretrained CNN, we use the same CNN across all methods, including IndeCateR and REINFORCE.We use the Adam optimizer with the best learning rate among {1e-3, 5e-4, 1e-4}. We train for maximum 100 epochs, but stop early if the training saturates. For MNIST-R tasks, we used learning rate 1e-4 and trained ISED for 10 epochs, REINFORCE and IndeCateR for 50 epochs, and A-NeSI and NASR for 100 epochs. We trained ISED for 30 epochs, A-NeSI for 100 epochs, and the rest for 50 epochs for HWF and Leaf Classification with learning rate 1e-4. For the Scene Recognition task, we trained A-NeSI and the purely neural baseline 50 epochs and the rest 100 epochs with learning rate 5e-4. For tasks sum 8 to sum 16 we trained ISED for 50 epochs and the rest for 100 epochs with learning rate 1e-3. For Visual Sudoku, we follow the setting in NASR [5] and train for 10 epochs with learning rate 1e-5.We configure ISED to use the min-max semiring for HWF and the add-mult semiring for all other tasks. We use categorical sampling and binary cross-entropy loss for ISED.For leaf classification and scene recognition, the neural-GPT experiments, we used the up-to-date version of GPT-4, gpt-4-1106-preview and gpt-4o respectively, with the parameter top-p set to 1e-8. We present the prompts used for the experiments in Tables 5 and 6 .Table 5 : GPT-4 prompt for the leaf classification task.You are an expert in classifying plant species based on the margin, shape, and texture of the leaves. You are designed to output a single JSON.User message <PLANT NAME>. Classify into one of: <MARGIN/SHAPE/TEXTURE>. Give your answer without explanation.Table 6 : GPT-4 prompt for the scene recognition task.You are an expert at identifying room types based on the object detected. Give short single responses.There are <DETECTED OBJECTS>. What type of room is most likely? Choose among <SCENES>.We use MARGIN = {entire, dentate, lobed, serrate, serrulate, undulate}, SHAPE ={elliptical, lanceolate, obovate, oblong, ovate}, TEXTURE ={glossy, leathery, smooth, medium}, and PLANT NAME ∈{Alstonia Scholaris, Citrus limon, Jatropha curcas, Mangifera indica, Ocimum basilicum, Platanus orientalis, Pongamia Pinnata, Psidium guajava, Punica granatum, Syzygium cumini, Terminalia Arjuna}. Furthermore, SCENES = {bathroom, bedroom, dining room, living room, kitchen, lab, office, home lobby, basement} and DETECTED OBJECTS is a list of maximum length 10 with duplicates.We report the accuracy of all benchmarks with 1-sigma standard deviation in Tables 7, 8, 9, and 10. We further provide the performance comparison with varying sample counts with 1-sigma standard deviation in Tables 11, 12 , and 13. For implementing the baselines, we adapted the code from the official repositories of DeepProblog [14] (Apache 2.0), Scallop [8] (MIT), A-NeSI [24] (MIT), NASR [5] (MIT), and IndeCateR [21] (Apache 2.0). Additionally, our benchmarks use Multi-illumination dataset [16] (CC BY 4.0), HWF dataset (CC BY-NC-SA 3.0) from NGS [12] , a subset of the leaf database [4] (CC BY 4.0), YOLOv8 (AGPL-3.0) and CLIP (MIT).NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: We have reviewed the Code of Ethics and and followed them whenever relevant.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Code is available at https://github.com/alaiasolkobreslin/ISED 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: The paper appears to make significant progress in the direction of neurosymbolic approaches and integration of differential and traditional programming. All reviewers agreed that the novelty in the paper is substantial and that the empiric evaluation is sound. While some technical concerns were raised in the reviews, these could be mostly addressed in the rebuttal phase. Further, the presented type of neuro-symbolic research should be of broad interest to the NeurIPS community. Overall, the clear consensus is to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper is about learning the parameters for a neural network (or other differentiable structure) where its output is then provided to a different black-box procedure, and we can compute a loss for the (second) output from that procedure.

The procedure consists of four parts: infer, sample (+ execute), and estimate, and descend. Given the input, we "infer" using the neural network, which produces a distribution over the inputs to the black-box program. We "sample" from this distribution multiple times to construct possible inputs to the black-box program, and we run the program on these inputs. We "estimate" by combining the outputs from the program to construct a distribution over the black-box program's output space, only using differentiable operations over the neural network's outputs. We "descend" by computing the loss using that distribution against the ground-truth label and computing the derivative with respect to the neural network's parameters. 

The authors consider several benchmark tasks: classifying leaf images to the correct plant species, scene recognition from images, arithmetic and sudoku solving using images of handwritten digits. The authors compare against two neuro-symbolic methods, A-NeSI which emulates the black-box program with a differentiable neural network, and three other methods for computing gradients through black-box programs. The authors show that their proposed method generally outperforms the comparison methods and demonstrates better data efficiency.
strength_weakness: strength_weakness: - The studied tasks all have a relatively small space of potential discrete outputs from the black-box program. The method seems to lack support for the case where outputs of the program can be real-valued.
- If the combination of the initial neural network and black-box program produce the correct answer for an input with very low likelihood, or if the sampling is unlucky and fails to produce a black-box program input which gives the correct answer, it would seem that there is no learning signal at all. The method as stated is not adaptive to such cases, for example by sampling from the neural network more often until some suitable input is found.
- The study of the proposed method is entirely empirical and there are no theoretical justifications provided for why it works (also as stated by the authors in item 3 of the paper checklist).
questions: questions: - What is the specific sampling strategy used in Algorithm 1? Are there restrictions on what this sampling strategy is allowed to be?
- For computing the results in the paper, how did you sample from the neural network at test time?
limitations: limitations: The author's description of the limitations seems adequate.
paper_summary: paper_summary: The paper introduces a sample-efficient algorithm called ISED (Iterative Sampling for Efficient Differentiable Programming) for learning neural programs i.e. programs that have a neural network component (that needs to be learned) and another fixed program. The fixed program could be a python program, a call to a black box LLM model, or a logic program. The task here is to train this neural program using only end-to-end input-output data. The challenge lies in propagating gradients through the fixed program, which could be complex, non-differentiable, and black-box. The paper proposes an alternative to REINFORCE-based methods by  trying to approximate the fixed program as if input = u then output = v by sampling multiple u with the NN's predicted distribution and using  neurosymbolic methods to compute gradients through this approximate program.
strength_weakness: strength_weakness: One of the main limitations, as mentioned by the authors, is that the approach doesn't work effectively on high-dimensional intermediate inputs. The authors should definitely explore this limitation further.


Also one minor comment: The paper should present results for all benchmarks in Table 3. It is misleading to show mostly positive benchmarks in the main paper and relegate the negative ones to the appendix.
questions: questions: Why do tests time out during test time? While it is understandable for Sudoku, why does sum_4 also time out for DPL?
limitations: limitations: Limitations are addressed.
paper_summary: paper_summary: This paper proposes an algorithm ISED for learning a composition between a neural model (e.g., using DNN to classify objects in an image) and traditional programming (e.g., calling GPT-4 to identify room type). ISED is based on reinforcement learning, which summarizes the input-output samples of the traditional black-box program as if-then logic programming. Evaluation of several benchmark tasks, such as leaf classification and scene recognition, shows the effectiveness and generalizability of the ISED.
strength_weakness: strength_weakness: - The motivation could be made stronger.
- The advantages of ISED over individual programming are not investigated.
- Lack of investigation on the impact of the capability of traditional programming on the ISED’s performance.

More specifically:

. (Introduction) The motivation of this work is not very strong. If the two models work well, why composite them with a neural program? For the scene recognition example, DNN plus GPT-4 can identify objects in the image and classify the scene. What is the goal of the composed program? 

. (Method) ISED considers traditional programming as a black-box program. It is assumed that ISED can learn from any program. In the evaluation, GPT-4 is a powerful LLM that can generate good outputs. Thus, for a powerful black-box program, ISED works better. It is suggested to analyze how the capability of a black-box program affects the performance of neural symbolic learning.

. (Evaluation) This study presents two new neural program learning benchmarks which both contain a program component that can make a call to GPT-4. A neural model can only learn  limited capability of LLM. The composite program may not replace individual programming?

.(RQ1: Accuracy) The results indicate that ISED can achieve higher performance than other neurosymbolic methods. It is suggested to analyze the differences between the composite program and the individual programming.

. (RQs) The three investigated RQs only focused on the performance and efficiency of ISED. Generally, the composite neural program aims to build a more explainable logic programming. The explainability is not verified.

. The acronyms need to be defined in the first place, such as ISED and LLM.
questions: questions: - What is the motivation of the composition?
- What are the advantages of ISED over individual programming ?
- What is the impact of the capability of traditional programming on the ISED’s performance?
limitations: limitations: The authors adequately addressed the limitations of this work.
paper_summary: paper_summary: This paper looks at optimizing "neural programs", which are composites of a deep neural network (DNN) followed by a program written in a traditional programming language or an API call to a large language model (LLM). This paper proposes ISED (Infer-Sample-Estimate-Descend) for learning neural programs that only rely on input-output samples of black-box components. ISED is evaluated on benchmarks involving calls to GPT-4 and prior work from neurosymbolic learning literature, showing comparable performance to state-of-the-art neurosymbolic frameworks and achieving more data- and sample-efficient results.
strength_weakness: strength_weakness: * Clarity: While Section 1 and 2 is nicely written and can be understood by readers new to this field, it is challenging to read Section 3 due to over-formalism and the lack of end-to-end running examples. 
* Clarity: The symbolism is a bit inconsistent. $x$ and $y$ in Section 2 presents the input to $M_\theta$ and output from $P$, while in Section 3.1 $x$ and $y$ denotes the "input-output samples" of $P$. 

Other minor editorial suggestions:
* L3 and L20: double quotation can be done via `` and '' in LaTeX.
questions: questions: * What are the choice reasons for the types of defined structural mapping?
* What does "interpretation" mean here in L145? How is that related?
* Besides scalability, what other future directions do the authors see as promising for the development and improvement of neural programs?
limitations: limitations: As is acknowledged by the authors, ISED struggles with scaling to high-dimensional input spaces, limiting its applicability in complex scenarios.

Comments: comment: Thanks for the rebuttal, which partially addressed my concerns.
Thank you very much for your response!
Thanks for the detailed explanation. I am now less concerned about the complexity of the user interface in ISED. Therefore, I have increased my rating. Looking forward to the writing improvement in the next revision. Thanks.
Dear reviewer 5P1v, could you comment to which degree the author response addressed your concerns? I would be particularly interested in your review on the limitation "ISED seems challenging to implement and understand by practitioners without a strong background in related fields." Thank you!
Dear authors, thank you very much for the detailed response and the new experiments in particular. Dear reviewer E3jF, could you comment to which degree the response addressed your concerns? Your insight would be appreciated, thank you!