Question: In figure 1(i), it's hard for me to tell why I+V should have a fixed length.. the first dimension of V decreases when the first dimension of I increases?
Evidence: Performing graph reasoning in large domains, such as predicting the relationship between two entities based on the input facts, is an important practical problem that arises in reasoning about molecular modeling, knowledge networks, and collections of objects in the physical world. The necessary inference rules are generally unknown and must be inferred from data, which in general are large. In this paper, we focus on learning neural networks for graph reasoning tasks. Consider the problem of learning a rule that explains the grandparent relationship. Given a dataset of labeled family relationship graphs, we aim to build machine-learning algorithms that can learn to predict a specific relationship (e.g., grandparent) based on other relationships, such as father(x, y) and mother(x, y).
Neural Logic Machines (NLM; present a method for solving graph reasoning tasks with a structured neural network. NLMs keep track of hyperedge representations for all tuples consisting of up to B entities. Thus, they can infer more complex finitely-quantified logical relations than standard graph neural networks that only consider binary relationships between entities. However, there are two disadvantages of such a dense hypergraph representation. First, the training and inference of NLMs requires simultaneously considering all entities in a domain, such as all of the N people in a family relationship database. Second, they scale polynomially with respect to the number of entities considered in a single inference. Even inferring a single relation like grandparent requires O(N 3 ) time and space complexity. In practice, for large graphs, these limitations make the training and inference intractable and hinder the application of NLMs in large-scale real-world domains.
To address these two challenges, we present a novel framework, called Sparse and Local Neural Logic Machines (SpaLoc), for inducing sparse relational rules from data in large domains. Our key idea is to exploit locality and sparsity in data: determining a relationship between entities The overall pipeline of SpaLoc: a multi-layer neural network that applies to hypergraphs. I and V denote the index tensor and value tensor respectively. To facilitate training and inference in large domains, we employ a sub-graph sampling procedure and sparsification.
usually only requires consideration of a small number of additional entities, and the resulting relation usually only holds for a small number of tuples of entities. Our contribution is three-fold. First, we develop a sparse tensor-based representation for representing hyperedge relationships among entities and making inferences about them. Second, during both training and inference, SpaLoc employs a sub-graph sampling technique based on an information-theoretic measure, information sufficiency, which quantifies the amount of information contained in a sub-graph with respect to whether a predicate is true for some tuple of entities. Since the sub-graph sampling may violate the closed-world assumption (i.e., the information in a sub-sampled graph may be insufficient in predicting the relationship between a pair of entities), we also use the information sufficiency measure to calibrate training labels. Third, to further speed up inference on large graphs, we encourage neural networks recover sparse relationships among objects by using a regularization term based on graph sparsity measurements.
We evaluate SpaLoc on two benchmarks: relational reasoning in synthetic datasets (family trees and general graph reasoning) and real-world knowledge-graph reasoning. First, we show that, with our sparsity regularization, the computation complexity for inference can be reduced to the same order as the optimal complexity, which significantly outperforms the base model NLM. Second, we show that training via sub-graph sampling and label calibration enables us to learn relational rules in real-world knowledge graphs with more than 10K nodes, whereas the original NLM can be barely applied to graphs with more than 100 nodes. Finally, SpaLoc achieves state-of-the-art performance on several real-world knowledge graph reasoning benchmarks.
Figure 1: The overall pipeline of SpaLoc: a multi-layer neural network that applies to hypergraphs. I and V denote the index tensor and value tensor respectively. To facilitate training and inference in large domains, we employ a sub-graph sampling procedure and sparsification.
Answer: Thank you for pointing out the potential confusion that I+V seems to have the fixed length in Figure 1. In SpaLoc, the dimension of I+V across different arities are not the same! We will update the figure and caption for better clarity.