SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianyu Liu $^{1}$  Qitan Lv $^{1}$  Jie Wang $^{1,2*}$  Shuling Yang $^{1}$  Hanzhu Chen $^{1}$
$^{2}$ Institute of Artificial Intelligence, Hefei Comprehensive National Science Center {tianyu-liu, qitanlv, slyang0916, chenhz}@mail.ustc.edu.cn {jiewangx}@ustc.edu.cn
Inductive relation prediction (IRP)—where entities can be different during training and inference—has shown great power for completing evolving knowledge graphs. Existing works mainly focus on using graph neural networks (GNNs) to learn the representation of the subgraph induced from the target link, which can be seen as an implicit rule-mining process to measure the plausibility of the target link. However, these methods cannot differentiate the target link and other links during message passing, hence the final subgraph representation will contain irrelevant rule information to the target link, which reduces the reasoning performance and severely hinders the applications for real-world scenarios. To tackle this problem, we propose a novel single-source edge-wise GNN model to learn the Rule-inducEd Subgraph representTations (REST), which encodes relevant rules and eliminates irrelevant rules within the subgraph. Specifically, we propose a single-source initialization approach to initialize edge features only for the target link, which guarantees the relevance of mined rules and target link. Then we propose several RNN-based functions for edge-wise message passing to model the sequential property of mined rules. REST is a simple and effective approach with theoretical support to learn the rule-induced subgraph representation. Moreover, REST does not need node labeling, which significantly accelerates the subgraph preprocessing time by up to  $11.66 \times$ . Experiments on inductive relation prediction benchmarks demonstrate the effectiveness of our  $\mathrm{REST}^2$ .
Knowledge graphs are a collection of factual triples about human knowledge. In recent years, knowledge graphs have been successfully applied in various fields, such as natural language processing [1], question answering [2] and recommendation systems [3].
However, due to issues such as privacy concerns or data collection costs, many real-world knowledge graphs are far from completion. Moreover, knowledge graphs are continuously evolving with new entities or triples emerging. This dynamic change causes even the large-scale knowledge graphs, e.g., Freebase [4], Wikidata [5] and YAGO3 [6], to still suffer from incompleteness. Most existing knowledge graph completion models, e.g., RotatE [7], R-GCN [8], suffer from handling emerging new entities as they require test entities to be observed in training time. Therefore, inductive relation prediction, which aims at predicting missing links in evolving knowledge graphs, has attracted extensive attention[9, 10].
*The Corresponding Author.
2Our code is available at https://github.com/smart-lty/REST
The key idea of inductive relation prediction on knowledge graphs is to learn logical rules, which can capture co-occurrence patterns between relations in an entity-independent manner and can thus naturally generalize to unseen entities [11, 12]. Some existing models, e.g., AMIE+[13], Neural LP[14], explicitly mine logical rules for inductive relation prediction with good interpretability[14], while their performances are limited due to the large searching space and discrete optimization[15, 16]. Recently, some subgraph-based methods, e.g., GraIL[11], TACT[12], CoMPILE[17], have been proposed to implicitly mine logical rules by reasoning over the subgraph induced from the target link.
Figure 1: Relevant and irrelevant rules.
However, there are still some irrelevant rules[18] within the subgraph. Considering the rule body and the rule head as a cycle[19], the relevant rules are cycles that pass the target link. As illustrated in Figure 1,  $u \to v \to e_4 \to u$  and  $u \to v \to e_2 \to e_1 \to u$  are relevant rules as they pass the target link, while  $e_1 \to e_3 \to e_2 \to e_1$  are irrelevant rules as they do not contain the target link. Existing methods cannot differentiate the target link and other links during message passing. Thus, they will mine plenty of irrelevant rules and encode them into the final subgraph representation, which makes the model prone to over-fitting and severely hinders reasoning performance.
In this paper, we propose a novel single-source edge-wise GNN model to learn the Rule-inducEd Subgraph repre-senTations (REST), which encodes relevant rules and eliminates irrelevant rules within the subgraph. Specifically, we observe that the information flow originating
from a unique edge and returning to this edge will form a cycle automatically. Consequently, the information flow originating from the target link can encode the relevant logical rules. Inspired by this observation, we propose a single-source initialization approach to assign a nonzero initial embedding for the target link according to its relation and zero embeddings for other links. Then we propose several RNN-based functions for edge-wise message passing to model the sequential property of mined rules. Finally, we use the representation of the target link as the final subgraph representation.
We theoretically show that with appropriate message passing functions, REST can learn the rule-induced subgraph representation for reasoning. Notably, REST avoids the heavy burden of node labeling in subgraph preprocessing, which significantly accelerates the time of subgraph preprocessing by up to  $11.66 \times$ . Experiments on inductive relation prediction benchmarks demonstrate the effectiveness of our REST.
Existing works for IRP can be mainly categorized into rule-based methods and subgraph-based methods. While rule-based methods explicitly learn logical rules in knowledge graphs, subgraph-based methods implicitly mine logical rules by learning the representation of subgraphs. Moreover, we discuss some graph neural network methods that reason over the whole graph.
Rule-based methods. Rule-based approaches mine logical rules that are independent of entities and describe co-occurrence patterns of relations to predict missing factual triples. Such a rule consists of a head and a body, where a head is a single atom, i.e., a fact in the form of Relation(head entity, tail entity), and a body is a set of atoms. Given a head  $R(y,x)$  and a body  $\{B_1,B_2,\dots ,B_n\}$ , there is a rule  $R(y,x)\gets B_{1}\wedge B_{2}\wedge \dots \wedge B_{n}$ . The rule-based methods have been studied for a long time in Inductive Logic Programming [20], yet traditional approaches face the challenges of optimization and scalability. Recently, Neural LP [14] presents an end-to-end differentiable framework that enables modern gradient-based optimization techniques to learn the structure and parameters of logical rules. DRUM[21] analyzes Neural LP from the perspective of low-rank tensor approximation and uses bidirectional RNNs to mine more accurate rules. Moreover, for automatically learning rules from large knowledge graphs, RLvLR [22] proposes an effective rule searching and pruning strategy, which shows promising results on both scalability and accuracy for link prediction. However, these explicit rule-based methods lack expressive power due to rule-based nature, and cannot scale to large knowledge graphs as well.
Subgraph-based methods. Subgraph-based methods extract a local subgraph around the target link and use GNNs to learn subgraph representation to predict link existence. Such a subgraph is usually induced by the neighbor nodes of the target link, which encodes the rules related to the target link. GraIL[11] is the first subgraph-based inductive relation prediction model. It defines the enclosing subgraph as the graph induced by all the nodes in the paths between two target nodes. After labeling all the nodes with double radius vertex labeling[23], it employs R-GCNs[8] to learn subgraph representation. CoMPILE[17] extracts directed enclosing subgraphs to handle the asymmetric / anti-symmetric patterns of the target link. TACT[12] converts the original enclosing subgraph into a relational correlation graph, and proposes a relational correlation network to model different correlation patterns between relations. More recently, SNRI[24] extracts enclosing subgraphs with complete neighboring relations to consider neighboring relations for reasoning. ConGLR[25] converts the original enclosing subgraph into a context graph to model relational paths. However, these methods cannot differentiate the target link and other links during message passing. Therefore, the GNNs will mine plenty of irrelevant rules for other links and encode them into subgraph representation, which reduces the accuracy of reasoning.
Graph neural network for link prediction. Some methods use GNNs to reason over the whole graph rather than a subgraph for inductive relation prediction. INDIGO[26] converts the KG into a node-annotated graph and fully encodes it into a GNN. NBFNet[27] generalizes Bellman-Ford algorithm and proposes a general GNN framework to learn path representation for link prediction. MorsE [28] considers transferring entity-independent meta-knowledge by GNNs. While these methods share some spirit with subgraph-based methods, they are essentially different with subgraph-based methods. These methods need to reason over the whole graph for a test example, while subgraph-based methods only need to reason over a subgraph. Meanwhile, these methods tend to predict entities rather than relations for a query, while subgraph-based methods tend to predict relations, as the subgraph only needs to be extracted once for relation prediction. As these methods benefit from a larger number of negative sampling, we do not take them into comparison.
We define a training graph as  $\mathcal{G}_{\mathrm{tr}} = (\mathcal{E}_{\mathrm{tr}},\mathcal{R}_{\mathrm{tr}},\mathcal{T}_{\mathrm{tr}})$ , where  $\mathcal{E}_{\mathrm{tr}}$ ,  $\mathcal{R}_{\mathrm{tr}}$ , and  $\mathcal{T}_{\mathrm{tr}} \subset \mathcal{E}_{\mathrm{tr}} \times \mathcal{R}_{\mathrm{tr}} \times \mathcal{E}_{\mathrm{tr}}$  are the set of entities, relations and triples during training, respectively. We aim to train a model such that for any graph  $\mathcal{G}' = (\mathcal{E}',\mathcal{R}',\mathcal{T}')$  whose relations are all seen during training (i.e.,  $\mathcal{R}' \subseteq \mathcal{R}_{\mathrm{tr}}$ ), the model can predict missing triples in  $\mathcal{G}'$ , i.e.,  $(?,r_t,v)$ ,  $(u,?,v)$ ,  $(u,r_t,?)$ , where  $u,v \in \mathcal{E}'$  and  $r_t \in \mathcal{R}'$ . We denote the set of any possible entities as  $\{\mathcal{E}\}$  and the set of knowledge graphs whose relation set  $\mathcal{R}$  are subset of  $\mathcal{R}_{\mathrm{tr}}$  as  $\{\mathcal{G}\}_{\mathrm{tr}}$ . The model we would like to train is a score function  $f: \{\mathcal{G}\}_{\mathrm{tr}} \times \{\mathcal{E}\} \times \mathcal{R}_{\mathrm{tr}} \times \{\mathcal{E}\} \to \mathbb{R}$ ,  $(\mathcal{G}',u,r_t,v) \mapsto f(\mathcal{G}',u,r_t,v)$ , where  $\mathcal{G}' = (\mathcal{E}',\mathcal{R}',\mathcal{T}')$ ,  $\mathcal{R}' \subseteq \mathcal{R}_{\mathrm{tr}}$  and  $u,v \in \mathcal{E}'$ . For a query triple  $(u,r_t,?)$ , we enumerate valid candidate tail entities  $v'$  and use the model to get the score  $s'$  of this triple  $(u,r_t,v')$ . We call the query triple  $(u,r_t,v)$  as target link and  $u,v$  as target nodes, respectively.
In this section, we describe the architecture of the proposed REST in detail. Following existing subgraph-based methods, we first extract a subgraph for each query triple. Then we apply single-source initialization and edge-wise message passing to update edge features iteratively. Finally, the representation of the target link is used for scoring. REST organizes the two methods in a unified framework to perform inductive relation prediction. Figure 2 gives an overview of REST.
4.1 Subgraph Extraction
For a query triple  $(u, r_t, v)$ , the subgraph around it contains the logical rules to infer this query, thus we extract a local subgraph  $\mathcal{SG}_{u, r_t, v}$  to implicitly learn logical rules for reasoning. Specifically, we first compute the  $k$ -hop neighbors  $\mathcal{N}_k(u)$  and  $\mathcal{N}_k(v)$  of the target nodes  $u$  and  $v$ . Then we define enclosing subgraph as the graph induced by  $\mathcal{N}_k(u) \cap \mathcal{N}_k(v)$  and unclosing subgraph as the graph induced by  $\mathcal{N}_k(u) \cup \mathcal{N}_k(v)$ . Note that the subgraph extraction process of our REST omits the node labeling, as node features are unnecessary in edge-wise message passing, which significantly reduces the time cost of subgraph preprocessing.
Figure 2: An overview of REST. REST organizes the single-source initialization method and the edge-wise message passing method in a unified framework to learn relevant rules representations within the subgraph for the target link. Different relevant rules are shown in different colors in part 3.
4.2 Single-source Initialization
Single-source initialization is a simple and effective initialization approach, which initializes a nonzero embedding to the query triple according to  $r_t$  and zero embeddings for other triples. Specifically, the embeddings of links and nodes within  $\mathcal{SG}_{u,r,v}$  are initialized as follows:
$$
\mathbf {e} _ {x, y, z} ^ {0} = \underset {(u, r, v)} {\mathbb {1}} (x, y, z) \odot \mathbf {r} _ {y} = \left\{ \begin{array}{l l} \mathbf {r} _ {y}, & \text {i f} (x, y, z) = (u, r _ {t}, v) \\ \mathbf {0}, & \text {i f} (x, y, z) \neq (u, r _ {t}, v) \end{array} \right. \tag {1}
$$
$$
\mathbf {h} _ {v} ^ {0} = \mathbf {0} \quad f o r   \forall v \in \mathcal {E},
$$
where  $\mathbf{e}_{x,y,z}^{0}$  and  $\mathbf{h}_v^0$  are the initial representation of edge  $(x,y,z)$  and node  $v$ , respectively.  $\mathbb{1}$  is the indicator function to differentiate the target link and other links.  $\odot$  is Hadamard product. Note that the representation of nodes is used as temporary variables in edge-wise message passing. With this initialization approach, we ensure the relevance between mined rules and the target link.
4.3 Edge-wise Message Passing
After initializing all the edges and nodes, we perform edge-wise message passing to encode all relevant rules into the final subgraph representation. Specifically, each iteration of edge-wise message passing consists of three parts, (1) applying message function to every link, (2) updating node features by aggregating message and (3) updating edge features by temporary node features, which are described as follows:
$$
\mathbf {m} _ {x, y, z} ^ {k} = \mathbf {M E S S A G E} \left(\mathbf {h} _ {x} ^ {k - 1}, \mathbf {e} _ {x, y, z} ^ {k - 1}, \mathbf {r} _ {y}\right) = \left(\mathbf {h} _ {x} ^ {k - 1} \otimes^ {1} \mathbf {r} _ {y}\right) \uplus \left(\mathbf {e} _ {x, y, z} ^ {k - 1} \otimes^ {2} \mathbf {r} _ {y}\right) \tag {2}
$$
$$
\mathbf {h} _ {z} ^ {k} = \underset {(x, y, z) \in \mathcal {T}} {\mathbf {A G G R A G A T E}} \left(\mathbf {m} _ {x, y, z} ^ {k}\right) = \bigoplus_ {(x, y, z) \in \mathcal {T}} \mathbf {m} _ {x, y, z} ^ {k} \tag {3}
$$
$$
\mathbf {e} _ {x, y, z} ^ {k} = \mathbf {U P D A T E} \left(\mathbf {h} _ {x} ^ {k}, \mathbf {e} _ {x, y, z} ^ {k - 1}\right) = \mathbf {h} _ {x} ^ {k} \diamond \mathbf {e} _ {x, y, z} ^ {k - 1} \tag {4}
$$
Here,  $\oplus, \oplus, \diamond, \otimes^1, \otimes^2$  are binary operators which denote a function to parameterize.  $\oplus$  denotes the large size operator of  $\oplus$ .  $\mathbf{h}_z^k$  and  $\mathbf{e}_{x,y,z}^k$  respectively represent the feature of node  $z$  and link  $(x,y,z)$  after  $k$  iterations of edge-wise message passing. We visualize the comparison between conventional message passing framework developed by GraIL[11] and proposed edge-wise message passing framework in Figure 3.
(a) Original Graph
(b) Conventional Message Passing Framework
(c) Edge-wise Message Passing Framework
Figure 3: Comparison between conventional message passing framework developed by GraIL[11] and our REST. First, REST initializes node and edge features with single-source initialization. Then, REST employs UPDATE function to update edge features. Finally, REST directly uses the embedding of the target link as the final subgraph representation, rather than the pooling of all node embeddings.
4.4 RNN-based Functions
Message passing functions in existing works use order-independent binary operators such as ADD and MUL, which cannot model the sequential property of rules and lead to incorrect rules[21]. To tackle this problem, we introduce several RNN-based methods as message passing functions.
Message Functions. For the edge-wise message passing process, each iteration REST takes in  $\mathbf{h}_x^{k - 1}$ ,  $\mathbf{e}_{x,y,z}^{k - 1}$  and  $\mathbf{r}_y$  to form a message. We modify GRU[29] as message function as follows:
$$
\delta_ {k} = \sigma_ {g} \left(\mathbf {W} _ {\delta , 1} ^ {k} \mathbf {r} _ {y} \odot \mathbf {e} _ {x, y, z} ^ {k - 1} + \mathbf {W} _ {\delta , 2} ^ {k} \mathbf {h} _ {x} ^ {k - 1} + \mathbf {b} _ {\delta} ^ {k}\right)
$$
$$
\gamma_ {k} = \sigma_ {g} \left(\mathbf {W} _ {\gamma , 1} ^ {k} \mathbf {r} _ {y} \odot \mathbf {e} _ {x, y, z} ^ {k - 1} + \mathbf {W} _ {\gamma , 2} ^ {k} \mathbf {h} _ {x} ^ {k - 1} + \mathbf {b} _ {\gamma} ^ {k}\right) \tag {5}
$$
$$
c _ {k} = \sigma_ {h} \left(\mathbf {W} _ {c, 1} ^ {k} \mathbf {r} _ {y} \odot \mathbf {e} _ {x, y, z} ^ {k - 1} + \mathbf {W} _ {c, 2} ^ {k} \left(\gamma_ {k} \odot \mathbf {h} _ {x} ^ {k - 1}\right)\right) \tag {5}
$$
$$
\mathbf {m} _ {x, y, z} ^ {k} = \delta_ {k} \odot c _ {k} + (1 - \delta_ {k}) \odot \mathbf {h} _ {x} ^ {k - 1}
$$
Here,  $\delta_{k}$  is the update gate vector,  $\gamma_{k}$  is the reset gate vector and  $c_{k}$  is the candidate activation vector. The operator  $\odot$  denotes the Hadamard product,  $\sigma_{g}$  denotes Sigmoid activation function and  $\sigma_{h}$  denotes Tanh activation function. During each iteration of message passing, we only use GRU once, therefore  $k$ -layer message passing includes  $k$  GRUs, which can model sequence with length  $l \leq k$ .
Aggregate Functions. The aggregate function aggregates messages for each node from its neighboring edges. Here, we use simplified PNA[30] to consider different types of aggregation.
$$
\mathbf {h} _ {z, 1} ^ {k} = \underset {(x, y, z) \in \mathcal {T}} {\text {m e a n}} (\mathbf {m} _ {x, y, z} ^ {k}), \mathbf {h} _ {z, 2} ^ {k} = \underset {(x, y, z) \in \mathcal {T}} {\text {m a x}} (\mathbf {m} _ {x, y, z} ^ {k}),
$$
$$
\mathbf {h} _ {z, 3} ^ {k} = \min  _ {(x, y, z) \in \mathcal {T}} \left(\mathbf {m} _ {x, y, z} ^ {k}\right), \quad \mathbf {h} _ {z, 4} ^ {k} = \operatorname {s t d} _ {(x, y, z) \in \mathcal {T}} \left(\mathbf {m} _ {x, y, z} ^ {k}\right), \tag {6}
$$
$$
\mathbf {h} _ {z} ^ {k} = \mathbf {W} _ {a g g} ^ {k} \left[ \mathbf {h} _ {z, 1} ^ {k}; \mathbf {h} _ {z, 2} ^ {k}; \mathbf {h} _ {z, 3} ^ {k}; \mathbf {h} _ {z, 4} ^ {k}; \mathbf {h} _ {z} ^ {k - 1} \right]
$$
Here,  $[;]$  denotes the concatenation of vectors,  $\mathbf{W}_{agg}^{k}$  denotes the linear transformation matrix in the  $k$ -th layer.
Update Functions. The update function is used to update the edge feature. We propose to update the edge feature with LSTM[31]. Specifically, LSTM needs three inputs: a hidden vector, a current
input vector and a cell vector. We use  $\mathbf{h}_x^k$  as the hidden vector and  $\mathbf{e}_{x,y,z}^{k - 1}$  as the current input vector. Moreover, we expect that each edge can differentiate the target link during message passing, which requires each edge to specify the query information. Therefore, we initialize each edge with another query feature as the cell vector. All the edges are initialized with the same query embedding  $\mathbf{r}_r^q$  related to the query relation  $r$ .
$$
\mathbf {q} _ {x, y, z} ^ {0} = \mathbf {r} _ {r} ^ {q} \tag {7}
$$
Then the update function can be described as follows:
$$
\mathbf {e} _ {x, y, z} ^ {k}, \mathbf {q} _ {x, y, z} ^ {k} = \mathbf {L S T M} \left(\mathbf {e} _ {x, y, z} ^ {k - 1}, \mathbf {q} _ {x, y, z} ^ {k - 1}, \mathbf {h} _ {x} ^ {k}\right) \tag {8}
$$
After updating the edge feature by  $k$  iterations of edge-wise message passing, we output  $\mathbf{e}_{u,r,v}^{k}$  as the subgraph representation. Then we use a linear transformation and an activation function to get the score of the target link.
$$
f (u, r _ {t}, v) = \sigma \left(\mathbf {W} _ {s} \mathbf {e} _ {u, r _ {t}, v} ^ {k} + \mathbf {b} _ {s}\right) \tag {9}
$$
In this section, we theoretically analyze the effectiveness of our REST. We first define the rule-induced subgraph representation, which utilizes encoded relevant rules to infer the plausibility of the target link. Then we show that our REST is able to learn such a rule-induced subgraph representation for reasoning.
5.1 Rule-induced Subgraph Representation Formulation
Our rule-induced subgraph representation aims to encode all relevant rules into the subgraph representation for reasoning. Therefore, we can define the rule-induced subgraph representation as the aggregation of these relevant rules:
$$
\mathcal {S} _ {u, r _ {t}, v} = \bigoplus_ {c \in \mathcal {C}} \mathbf {p} _ {c}, \tag {10}
$$
where  $\mathcal{C}$  denotes the set of all possible relevant rules within  $\mathcal{SG}_{u,r_t,v}$  and  $\mathbf{p}_c$  is the representation of a relevant rule  $c$ . Following the idea of Neural LP[14] to associate each relation in the rule with a weight, we model the representation of a rule as a function of its relation set. Therefore, we give the definition of rule-induced subgraph representation.
Definition 1 (Rule-induced subgraph representation.) Given a subgraph  $\mathcal{SG}_{u,r_t,v}$ , its rule-induced subgraph representation is defined as follows:
$$
\mathcal {S} _ {u, r _ {t}, v} = \bigoplus_ {i = 1} ^ {k} \underbrace {\bigoplus_ {(u , r _ {t} , v)} \bigoplus_ {(v , y _ {0} , x _ {0})} \dots \bigoplus_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \otimes \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \otimes \dots \otimes \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {11}
$$
where  $i$  denotes the length of the cycle,  $\mathbf{r}_{y_i}$  is the representation of relation  $y_{i}$ ,  $(x_{i},y_{i},x_{i - 1})$  is an existing triple in  $\mathcal{SG}_{u,r_t,v}$ .  $\{(u,r_t,v),(v,y_0,x_0),\dots,(x_{i - 3},y_{i - 2},u)\}$  is a cycle at length  $i$ .
Note that  $\oplus$  and  $\otimes$  denote binary aggregation functions. Intuitively, rule-induced subgraph representation captures all relevant rules within the subgraph and is expressive enough for reasoning.
5.2 Rule-induced Subgraph Representation Learning
Here, we show that our REST can learn such a rule-induced subgraph representation. First, we show this in a simple case.
Theorem 1 Single-source edge-wise GNN can learn rule-induced subgraph representation if  $\oplus = +, \oplus = +, \diamond = +, \otimes^1 = \times, \otimes^2 = \times$ . i.e., there exists nonzero  $\alpha_{i,j}$  such that
$$
\mathbf {e} _ {u, r _ {t}, v} ^ {k} = \sum_ {i = 1} ^ {k} \underbrace {\sum_ {(u , r _ {t} , v)} \sum_ {(v , y _ {0} , x _ {0})} \dots \sum_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \times \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \times \dots \times \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {12}
$$
We prove this in Appendix A. This theorem states that our REST can learn the rule-induced subgraph representation in the basic condition. Then we generalize this theorem to a general version.
Theorem 2 Single-source edge-wise GNN can learn rule-induced subgraph representation if  $\boxplus = \oplus, \boxplus = \oplus, \diamond = \oplus, \otimes^1 = \otimes, \otimes^2 = \otimes$ , where  $\oplus$  and  $\otimes$  are binary operators that satisfy  $0 \oplus a = a, 0 \otimes a = 0$ . i.e., there exists nonzero  $\alpha_{i,j}$  such that