Title: CTSketch: Compositional Tensor Sketching for Scalable Neurosymbolic Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neurosymbolic learning
Abstract: Many computational tasks benefit from being formulated as the composition of neural networks followed by a discrete symbolic program. The goal of neurosymbolic learning is to train the neural networks using end-to-end input-output labels of the composite. We introduce CTSketch, a novel, scalable neurosymbolic learning algorithm. CTSketch uses two techniques to improve the scalability of neurosymbolic inference: decompose the symbolic program into sub-programs and summarize each sub-program with a sketched tensor. This strategy allows us to approximate the output distribution of the program with simple tensor operations over the input distributions and the sketches. We provide theoretical insight into the maximum approximation error. Furthermore, we evaluate CTSketch on benchmarks from the neurosymbolic learning literature, including some designed for evaluating scalability. Our results show that CTSketch pushes neurosymbolic learning to new scales that were previously unattainable, with neural predictors obtaining high accuracy on tasks with one thousand inputs, despite supervision only on the final output.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduces CTSketch, a novel neurosymbolic learning algorithm that is designed to scale better.\nThe idea is to represent a logic program as a tensor, mapping each input combination to either 0 or 1, which still scales exponential in the number of variables. To scale, they approximate such a program by 1) decomposing it into several subprograms (currently limited to a user defined decomposition), which reduces the number of variables within a subprogram, and by 2) considering a low-rank approximation of those subprogram tensors. The empirical analysis demonstrates strong results.\n\nSTRENGTHS AND WEAKNESSES:\n**Clarity**: \nThe primary ideas of the proposed work are easy to follow, and well-illustrated using Figures.\n\n**Originality**: \nThe techniques proposed in this work exist; the novelty is their application in a neurosymbolic context. Having said that, the techniques only apply to the symbol-inference part, and are disconnected from the neural part of the pipeline. In that sense, it is not too surprising that this can be done. I was pleasantly surprised however by how well it appears to work in a neurosymbolic learning setting.\n\n**Significance**:\nThe paper demonstrates strong empirical results, increasing the potential impact on the field. The fact that a user needs to manually specify a program decomposition limits the more general applicability of this work. However, I believe that existing related work may help here (see Questions section).\n\n**Quality**:\nThe work is of good quality: the proof for the error bound is included in the appendix (but I did not verify it), the empirical experiments are in line with existing neurosymbolic papers, and the discussion of the empirical results are sufficiently informative/convincing. Something that could be of interest and that is currently missing is the impact of the program decomposition on the final results.\n\nQUESTIONS:\nNot really questions, more feedback:\n\nWith regards to related work / future work, the authors may find interesting work on tensor decompositions for probabilistic inference in general. Some of these automatically identify decompositions based on the variable dependencies etc. E.g.,\n- \"Tensor Belief Propagation\", ICML2017. Andrew Wrigley, Wee Sun Lee, Nan Ye\n- \"Parallel Weighted Model Counting with Tensor Networks. MCC2020. Jeffrey M. Dudek, Moshe Y. Vardi\n- \"Efficient Contraction of Large Tensor Networks for Weighted Model Counting through Graph Decompositions\". 2019. Dudek et al.\n- \"Expressive power of tensor-network factorizations for probabilistic modelling\". NeurIPS 2019. Glasser et al.\n- \"What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)?\" TMLR2025. Loconte et al.\n- \"Learning from Binary Multiway Data: Probabilistic Tensor Decomposition and its Statistical Optimality\". JMLR2020. Wang et al.\n\n\nTypo: \n* \"learning, Next, we ...\"\n* \"Inference in CTSketch does not aggregate proofs\", is an odd statement in a sense that the summations performed within CTSketch are aggregating in a similar fashion. I.e., you compute Eq 2.2, which is aggregating 'proofs'.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method to perform scalable and approximate inference for neurosymbolic AI based on low-rank tensor decompositions of symbolic programs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe idea of using low-rank tensor decompositions to approximate symbolic programs in a differentiable fashion is novel.\n\n\n**Weaknesses**\n\n1) The paper claims that they are the first to use program decompositions to scale neurosymbolic learning. However, this is not the case. De Smet and Zuidberg Dos Martires [1] already used program decomposition to scale to similar problem sizes. Albeit for a restricted set of neurosymbolic problems. This reference is missing in the paper. Note, the techniques used in [1] was not explicitly called program decomposition.\n2) The claim that logic programming languages do not support external API calls seems a bit strange. Logic programming is Turing-complete meaning they should probably be able to express an API call. \n\n[1] \"A Fast Convoluted Story: Scaling Probabilistic Inference for Integer Arithmetic\", Lennert De Smet, and Pedro Zuidberg Dos Martires, NeurIPS, 2024\n\nQUESTIONS:\n1) Tensors (used in this paper) and arithmetic/probabilistic circuits (used in many other works) exhibit many similarities. It would be nice to see a discussion on these similarities in the context of the paper.\n\n2) How was the full rank tensor implemented. Was it as dense multilinear function? The reason I am asking is because recent advances have shown that one can efficiently implement arithmetic circuits (ie sparse multilinear functions, ie sparse tensors) efficiently on the GPU and scale neurosymbolic learning without any approximations [2]. This might imply that scaling is mainly due to program decomposition and not low-rank tensor approxiamtions?\n\n3) Is it not a bit of cheating to give direct supervision to the low-rank tensor decompositions of the sub-programs compared to what other methods are getting as training signal?\n\n[2] \"KLay: Accelerating Arithmetic Circuits for Neurosymbolic AI\", Jaron Maene, Vincent Derkinderen, and Pedro Zuidberg Dos Martires, ICLR, 2025\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets at efficient computation of the output distributions of a program given the input variables' distribution. It assumes variables are discrete and the program/function can thus be modelled as a tensor mapping from choices of input variables' values to the output variable's value. They then apply tensor decomposition algorithms to approximate the tensor operations with low-rank and computationally efficient ones. The method also exploits the structure of programs. For example, summing n digits can be calculated as a tree of binary-sum sub-programs. They approximate each sub-program with tensor operations and then feed the outputted distributions to the next sub-program. The resulting method is differentiable and can thus support neuro-symbolic learning with gradient descent. \n\nThe method is evaluated in various classic neuro-symbolic tasks and shows better scalability for the sum_n program. It also performs competitively with previous methods in other tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper targets at an important problem: efficient approximations of the forward & backward computations of programs for neuro-symbolic learning. The method is interesting and incorporates the tensor decomposition techniques for approximation. \n\nHowever, as discussed by the authors, tensor decomposition is only applicable when the full program summary tensor is computationally affordable. It is hard to apply those techniques in more realistic settings. Program decomposition itself is well-known and commonly used in the neuro-symbolic field such as in Terpret. \n\nThe experimental results are positive to some extent but are not strong enough to support claiming general improvements on scalabilities. The results about scalability are mainly for the sum_n program where program decomposition is easy. It is hard to tell if the scalability comes from program decomposition or tensor operation approximation. Just for sum_n, we could simply approximate its outputs' expectation as the summation of all input variables' expectations and get similar performance in my understanding. \nThe authors did not state the ranks to approximate each program in experiments as well. It would be cool to see faster sudoku approximations with e.g., partial input-output examples, although I am not sure if it's feasible.\n\nQUESTIONS:\n* Are there more results regarding scalabilities e.g., for more programs and with more metrics? Can tensor operation approximation speed up general programs such as solving sudoku?\n* Can tensor operation approximation be extended more realistic settings where the sub-program tensor summary is infeasible?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose CTSKetch, a framework to efficiently train neuro-symbolic models by using only end-to-end training data (i.e., without intermediate supervision on the representations extracted by the neural component). To achieve this, CTSKetch summarizes all possible relations between inputs and outputs (the task semantics) in a single tensor, mapping inputs (representations extracted from the neural component) to outputs (labels). To tackle the potential combinatorial explosion in size of this tensor, the authors propose two remedies: decomposition of the full tensorized program into composable sub-programs and low-rank decomposition of task-tensors (TT-SVD is used in the paper but could be any low-rank decomposition as suggested in Future Work). The authors provide an approximate error bound (valid for when TT-SVD is used as LoRa approximation) for the proposed CTSKetch. Finally, the authors present experimental results on different well-known benchmark in the neuro-symbolic community (MNIST sum/addition, visual sudoku, sudoku solving, HVF, leaf identification and scene classification), showing that 1) it overall scales better compared to other baselines (e.g., increasing the number of operands in the sum), 2) it achieves competitive results in terms of task accuracy, and 3) it converges faster compared to other baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper was overall well-written, and I enjoyed reading it. The idea of using low-rank approximation to avoid the instantiation of the full task tensor at one time in memory is nice, and so is the perpendicular contribution of decomposing the full task into a composition of sub-tasks. Furthermore, the authors provide a sufficiently wide range of experiments on the CTSKetch, with some statistical significance obtained by using multiple (10) seeds.\n\n**Weaknesses**\n\n- Different aspects of this submission do not fully convince me. Firstly, the manual decomposition of tasks necessary for scaling (also mentioned by the authors themselves, which I appreciate) seems indeed a significant limitation for the applicability of the method outside of the simple domains investigated in this work. You mentioned that program synthesis could potentially come to the rescue in that case. Could you argue more about it?\n\n\n- For my understanding, CTSKetch requires the explicit programming of the task tensor, which also sounds very limited in terms of scalability and applicability to broader domains. You mention in the paper that the construction of this vector could also be automated (using a subset of the training), but this would require labels for the intermediate representations, which would be against the initial proposition of working in an end-to-end setting.\n\n\n- On another note, for my understanding, using low-rank approximation allows the avoidance of the instantiation of the full (sub-) task tensor in memory. However, this still needs to be computed fully (see Equation between Lines 135-136). So, if from a computation perspective the LoRa approximation only worsens the efficiency, where do the improvements shown in Section 4.5 come? Is it only the decomposition of the full task into multiple subtasks? I suggest that the authors elaborate more on this regard in the manuscript to increase its soundness, possibly including additional ablations regarding the effect of the single components (LoRa approximation, sub-task decomposition) on both accuracy and computational efficiency for at least some of the investigated tasks.\n\n\n- More generally, I also question the generalizability of encoding the task semantics with a single input-output tensor, which is, by construction, only going to work with known input examples. For instance, a network trained on sum4 would not generalize to sum5 out of the box, requiring the manual extension of its entries to cover those examples which are not in $sum4 \\cap sum5$.\n\n\n- Regarding the writing, I think that the quality of the paper is acceptable, but some parts are not extremely clear (especially Sections 1 and 2) due to notation clutter or imprecise/verbose explanations. For instance, in Equation 2.1 what does p_i[r_i] refers to? Shouldn't it be sufficient to have \\sum p_i? In my understanding, p_i is already the distribution associated with r_i. Or also, I didn’t find Figure 2 helpful right now, while, for example, populating the matrices with actual values and guiding a bit more the readers with some captioning could really increase the effectiveness and helpfulness of this figure. Also, why in Figure 1 is the fourth digit separated from the others? Also, in Line 147, shouldn’t it be “Approximate Error Bound”?\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}