Title: Embeddings as Probabilistic Equivalence in Logic Programs
Venue: NeurIPS 2025
Accepted: yes
Keywords: probabilistic programming

neurosymbolic AI

logic programming

knowledge graph embeddings

soft unification
Abstract: The integration of logic programs with embedding models resulted in a class of neurosymbolic frameworks that jointly learn symbolic rules and representations for the symbols in the logic (constant or predicate). The key idea that enabled this integration was the differentiable relaxation of unification, the algorithm for variable instantiation during inference in logic programs. Unlike unification, its relaxed counterpart exploits the similarity between symbols in the embedding space to decide when two symbols are semantically equivalent. We show that this similarity between symbols violates the transitive law of equivalence, leading to undesirable side effects in learning and inference. To alleviate those side effects, we are the first to revamp the well-known possible world semantics of probabilistic logic programs into new semantics called equivalence semantics. In our semantics, a probabilistic logic program induces a probability distribution over all possible equivalence relations between symbols, instead of a probability distribution over all possible subsets of probabilistic facts. We propose a factorization of the equivalence distribution using latent random variables and characterize its expressivity. Additionally, we propose both exact and approximate techniques for reasoning in our semantics. Experiments on well-known benchmarks show that the equivalence semantics leads to neurosymbolic models with up to 42% higher results than state-of-the-art baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSoft unification has been used in neursoymbolic learning as a way to translate the symbolic comparison between symbols into an embedding space comparison. However, soft unification exploits symmetry, rather than equivalence, which can lead to facts being derived that cannot be true under equivalence (violating transitivity). This paper proposes a probabilistic semantics of equivalence for logic programming. In this new semantics, symbol partitions (groupings) can be learned, and transitivity allows new facts over the grouped symbols to be derived. The evaluation demonstrates that this approach generally leads to higher AUPRC, MRR, and HITS@m compared to the baselines across several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- This paper was well-written and easy to follow. The running examples were helpful in understanding the problem and proposed solution.\n- The problem is well-motivated. This paper clearly outlines the weaknesses of prior work that reasons on embeddings in logic programs.\n- The experiments demonstrate that exploiting symmetries between equivalent symbols substantially improves performance, especially for the differentiable automata benchmark.\n\n**Weaknesses:**\n- There are no experiments to compare the training efficiency of the different approaches. Some analysis (such as Figure 2 in [1]) would have been useful.\n- This paper seems to be a direct improvement on DeepSoftLog, but several benchmarks from the DeepSoftLog paper were omitted, including MNIST-addition and visual sudoku classification.\n- Minor comments: \n    - Define “symbol” upfront. I have seen some neurosymbolic literature that uses “symbol” to mean the likely discrete neural network prediction. But this paper is using it to mean the name of a rule.\n    - Line 59: “First, formalize” -> “First, we formalize”\n    - Line 127: $P_ex$ -> $P_{ex}$\n\n[1] Differentiable Reasoning on Large Knowledge Bases and Natural Language, AAAI 2020\n\nQUESTIONS:\n- How does this new learning algorithm compare to DeepSoftLog on some of its scalability benchmarks, especially since this work is directly building on it? I would have liked to see a comparison for MNIST-Addition and Visual Sudoku. Even if the new notion of equivalence doesn’t improve results, it would be helpful to show whether there is any efficiency improvement or penalty, or any change in accuracy.\n- Other than isIn/locatedIn, what are other examples of equivalences that were exploited in the evaluated benchmarks, resulting in performance improvements?\n- What do the authors see as the  real-world application of this work? How does this work fit in with foundation models, which could likely eliminate the need for training the neural network?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a different learnable approach to probabilistic logic programming. Commonly for both prior and this work, the probability of a ground atom is defined by marginalizing over all possible words. In prior work the possible worlds include all possible similarity facts a \\sim b, and the probability of each similarity fact is determined independently based on the similarity of embeddings for a and for b. Some sampled possible worlds can thus lack a proper transitive equivalence between constants. This work proposes to instead have every possible world defined as a well-defined equivalence relation. The distribution over such worlds is defined by a probability vector for each constant c that defines the probability of c belonging to each equivalence cluster. The paper discusses how to adapt inference engines for inference and learning with this new equivalence-based possible world distribution. Strong empirical results confirm the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper tackles an important issue in an original and principled way. I’m not an expert in logic programming, so some parts were very hard to follow. In particular, the crux of argument about the deficiencies of the embeddings-based in approach in lines 150-155 was lost on me. I’d like to see the notion of “global optimum” to be better explained there. Also it was not clear how much harder/easier the inference and learning are in the proposed approach compared to prior work. \n\nOverall the exposition is very formal and abstract and hard to follow, but that is a common issue for logic programming papers.\n\nQUESTIONS:\n- Can you elaborate “even though in the global optimum we have P_f (y \\sim z)=0.” in line 155? \n- In the proposed approach every constants also has an embedding. Can you elaborate the difference in how embeddings are used in your work and in the prior similarity-based approaches?\n\nLIMITATIONS:\nI'd like to see more discussion on whether inference/learning becomes harder with the proposed approach compared to prior work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper formalizes a new approach for soft unification that is based on defining probability distributions over equivalences between symbols. Typically, in existing neurosymbolic methods soft unification is approximate since they are typically not true equivalences but on similarity (e.g. between symbol embeddings). The paper shows that under such situations, we may run the risk that inference in such models may derive facts that may not be true under equivalence. As a consequence, learning which is based on inference as a sub-step may be biased in such models. The paper adds equivalence semantics within probabilistic logic programming. Further, they show that a factorization of a distribution defined with the proposed framework can be realized through embeddings over the symbols. An exact inference pipeline is proposed for this model that uses optimizations based on symmetries along with a simple sampling-based approximation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ The paper formalizes novel probabilistic semantics for soft unification that is a central step in Neurosymbolic learning and inference. \n+ The idea of equivalences/symmetries which has been used previously within relational models is applied in a novel setting that seems to be an interesting direction for Neurosymbolic AI.\n+ The paper is well-written and rigorous in its approach.\n+ The results show that when compared with soft unification that a lot of the other state-of-the-approaches typically perform, the probabilistic semantics over equivalences helps us enforce transitivity during learning which translates to better performance during inference.\n\n\nWeaknesses\n- While the proposed work is more general, the experiments enforce a single rule of transitivity. \n- In more general cases, is the proposed learning method scalable? It seems like the singularization to be performed for exact weighted model counting will be computationally expensive.\n- Regarding the comparison as such, it seemed to me like the proposed “exact” inference/learning method is being compared to approximate soft unification methods. In that sense, the approximate inference methods seem to be doing reasonably well it looked like in the graph benchmark at least.\n- For the factorization, in the experiments, it is mentioned that the embedding dimension is as large as the vocabulary. This would seem like a limiting factor for many practical cases as such. What happens with a smaller embedding, I assume the distribution will not be expressive enough. But maybe this trade-off is important to address through some empirical results.\n\nOverall, this seems to be a good, well-written paper that defines an important problem in Neurosymbolic models precisely and proposes novel probabilistic semantics for it.\n\nQUESTIONS:\nIs there a reason why only transitivity was chosen within the experiments. Of course, it is true that transitivity occurs more naturally in real-world cases, but perhaps it would have helped to demonstrate enforcing other constraints.\n\nIf the embeddings are approximate (i.e. does not have dimensionality of the full vocabulary), does the proposed method reduce to other existing approximate approaches of soft unification using similarity?\n\nLIMITATIONS:\nMentioned adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that the use of embedding similarity in place of equivalence in neurosymbolic models is fundamentally problematic. That is, similarity violates transitivity, which ultimately hinders learning. In contrast to existing work on probabilistic programming which use a probability distribution over possible worlds (sets of facts), the authors introduce probability distributions over equivalences of constants. They propose using a latent variable to represent the equivalence distribution and make it tractable. Experiments on two tasks show consistent improvements over baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n * The explanations and definitions  provided throughout, especially in the preliminaries where they are most needed, are very clear.\n * The factorization introduced in section 4.1 is quite clever.\n * The paper is overall very well-written and well-structured.\n\n\n**Weaknesses**\n\n* Section 3.1 was somewhat difficult for me to follow. This may be related to the notation confusion regarding $x$ in example 7, which I mention in the formatting concerns. I don't really see how theorem 2 supports the paper's main contribution of showing that soft unification does not adhere to the laws of equivalence. The proof supplied in the supplementary material did not offer clarification for me. What exactly is bad about requiring embeddings to be equal? Ultimately it is this section that drops my evaluation of the paper to \"2: fair\" for clarity and significance. Ensuring that this section is extremely clear to audiences with ranging understandings (i.e., not just experts) of neuro-symbolic methods, probabilistic logic programming, etc. would be a massive improvement.\n  - I am unclear what exactly you are referring to by \"nontransitive possible worlds\" on line 156 and 161. Are they the \"possible worlds not in an equivalence relation\" mentioned on line 148? This makes Theorem 2 very unclear to me.\n  - I assume a negative fact is a falsehood, e.g., $a \\neq b$? This should be defined.\n * Similar to the above, I don't feel that the description of the experiments clearly address the research question \"does transitivity matter\".\n * The notation $e$ and $e'$ to denote equivalence relations in the Herbrand Universe and Base, respectively, was somewhat difficult for me to keep track of. Perhaps $e_u$ and $e_b$? This is a minor point and I understand that the authors may be using established notation here.\n\nQUESTIONS:\n1. Where exactly am I misunderstanding section 3.1? While I am able to follow the proof of theorem 2 and believe I did my due diligence in checking the details, I fail to see how this supports the paper's main claim.\n\n    - Would it be possible to include at least a summary of the proof of theorem 2 in the main text? I feel with the clarification I requested regarding this theorem, the proof would be helpful. \n\n3. While the improvements of your method over the baselines in the experiments are an *indicator* that enforcing transitivity is helpful, I feel that additional experimentation or analysis would be helpful to support your claim. That is, just reporting higher AUCs is insufficient. Can you describe the extent to which each of these tasks depends on transitivity? Can you provide example failures of the other models that are caused by a lack of transitivity? Better yet, is it possible to, for example, quantify the number of links in experiment 1 that *require* rule learning in order to be predicted correctly?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}