HyperLogic: Enhancing Diversity and Accuracy in Rule Learning with HyperNets

Authors: yang  yang
wendi  ren
shuang  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:45:47

Abstract: Exploring the integration of if-then logic rules within neural network architectures  presents an intriguing area. This integration seamlessly transforms the rule learning task into neural network training using backpropagation and stochastic gradient descent. From a well-trained sparse and shallow neural network, one can interpret each layer and neuron through the language of logic rules, and a global explanatory rule set can be directly extracted. However, ensuring interpretability may impose constraints on the flexibility, depth, and width of neural networks. In this paper, we propose HyperLogic: a novel framework leveraging hypernetworks to generate weights of the main network. HyperLogic can unveil multiple diverse rule sets, each capable of capturing heterogeneous patterns in data. This provides a simple yet effective method to increase model flexibility and preserve interpretability. We theoretically analyzed the benefits of the HyperLogic by examining the approximation error and generalization capabilities under two types of regularization terms: sparsity and diversity regularizations. Experiments on real data demonstrate that our method can learn more diverse, accurate, and concise rules.

Main paper: 
 introduction: 
 Despite the significant impact of deep learning on society, its lack of interpretability limits its use in critical areas that demand high transparency. For instance, in high-risk domains such as healthcare, finance, and law, the decision-making process of models needs to be fully open and explainable to users and relevant regulatory authorities to gain necessary trust and legitimacy [1, 2, 3] . Compared to the "black box" models, which may perform well but are uninterpretable, people prefer intrinsically interpretable model for decision support [4, 5] , such as a set of concise IF-THEN rules. Traditional rule learning methods, which are based on statistical or heuristic approaches, have been extensively explored but often struggle to simultaneously achieve two key objectives: 1) simplicity and accuracy in rules, and 2) noise tolerance and scalability in data handling [6, 7, 8, 9, 10] . These methods typically rely on search-based techniques, which can be limited when dealing with complex and noisy data. In contrast, deep learning, which focuses on representation learning through embedding, is robust to noise and effectively manages large datasets. This has led to the exploration of differentiable rule learning [11] , which leverages the high performance of deep learning while ensuring interpretability through explicit rule formulation Differentiable rule learning primarily includes two types of methods. The first approach outputs rules directly through the network, leveraging powerful and diverse neural models to enhance handling of complex data patterns [12, 13, 14] . The second approach is to extract rules from network weights [15, 16, 17] , which promotes rapid training and efficient data management while enabling the use of predefined structures to integrate expert priors [18] . The framework of HyperLogic: Hypernetwork generates θ for the main network, which is a rule-learning network. An example of a main rule-learning network is shown on the right, from which rules can be extracted based on the learned network weights.We focus on the second approach, aiming to extract rules like "IF a ∧ b ∧ ¬c THEN y is True" from network weights, where a, b, c are predefined predicates. Although integrating if-then logic rules within neural network architectures offers clear advantages, their performance is often hindered by the restrictive network structures required for interpretability. These structures tend to be overly simplistic and not scalable, limiting their ability to capture complex data patterns and making them prone to only capturing partial or local patterns in the data. This raises a critical question: How can we modify these models to better harness the full potential of neural networks without compromising their interpretability? In this work, we introduce HyperLogic, a novel framework that integrates hypernetworks with a main rule-learning network (as illustrated in Fig. 1 , where an example of a main rule-learning network is shown on the right). Hypernetworks are a type of neural network that generate weights for a main rule-learning network. In our framework, the hypernetwork takes random samples from a high-dimensional Gaussian distribution as input and outputs weights for different parts of the main network. These weights can be utilized in two ways: either directly as the weights for the main network throughout the training process (meaning the main network itself has no parameters), or by retaining trainable weights in the main network and combining them with the hypernetwork-generated weights through weighted fusion. In both scenarios, this approach yields an interpretable set of rules derived from the comprehensive weights. HyperLogic can be seen as a mixture of expert model with an infinite number of experts, providing a simple yet effective way to enhance model flexibility and adaptability. We analyze the benefits of HyperLogic by examining its approximation error and generalization capabilities under two types of regularization: sparsity and diversity regularization. Our findings demonstrate that HyperLogic acts as a universal approximation estimator in the Barron space [19] and proves its generalization ability across various regularization methods. These theoretical benefits are further supported by our empirical experiments. Additionally, hypernetworks enable the generation of multiple candidate rule sets in a single training session without significantly increasing computational overhead. This is in stark contrast to traditional methods, which typically learn only one set of rules at a time. Consequently, HyperLogic greatly enhances the efficiency and flexibility of the rule-learning process. We summarize our contributions as follows: 1. We introduce HyperLogic, a pioneering framework that integrates hypernetworks into differentiable rule learning, significantly enhancing the rule-learning landscape. 2. We theoretically justify the performance of HyperLogic and provide insights into its effectiveness. Specifically, we examine the approximation error and generalization capabilities under two types of regularization: sparsity and diversity. 3. We validate HyperLogic through extensive experiments on multiple datasets, demonstrating that it enables the learning of multiple diverse rule sets and yields more concise and accurate rules. 
related work: 
 Learning Rules from Network Weights Many methods for extracting rules from model weights are not based on neural networks; instead, they primarily focus on interpreting the weights of smaller, simpler models to discover rules [20, 7] . Recently, extensive methods have emerged that attempt to extract rules using neural network models. These approaches often involve data preprocessing techniques to prepare the data before training [21] or utilize post-hoc methods for rule extraction after model training [22, 23, 24] , which can often lead to a loss of accuracy. A more promising alternative allows for the direct interpretation of neural network weights as precise rules by integrating rule extraction directly during training. This leads to a more seamless and effective learning process. For example, methods such as [25, 26, 27] utilize low-dimensional embeddings to represent atomic conditions and rules, treating rule learning as a differentiable discrete combinatorial optimization problem encoded by a feedforward neural network. Specifically, FinRule [27] explores higher-order interactions between atomic conditions, which aligns somewhat with our approach of employing hypernetworks. However, it lacks a detailed analysis of the specific roles of these higherorder interactions and cannot learn a rich set of candidate rules as our method does. Additionally, these approaches often impose constraints on rule templates, such as fixing the rule length and the number of rules. Another common class of methods employs modified simple feedforward neural networks, such as DR-net [17] and others [28, 29, 30] . These methods simulate logical operations by altering activation functions and implementing mechanisms that ensure differentiability and support backpropagation. While they overcome the limitations of rigid rule templates, their model structures tend to be relatively simple. Furthermore, the training process often involves freezing the weights of one component while adjusting another, which restricts the model's ability to fully optimize performance. Our HyperLogic is a unified framework that can be integrated with various neural network weight generators, enabling easy adaptation to different main rule learning networks. Currently, our main network draws inspiration from DR-net [17] , but it can be replaced with other architectures based on task requirements. Unlike recent Bayesian approaches, which often require multiple hyperparameters to model uncertainty in network weights, HyperLogic simplifies the process by needing fewer hyperparameters while maintaining scalability. This allows for efficient training and adaptability across various tasks without the added complexity of managing uncertainty. Additionally, our approach can generate multiple sets of rule sets in a single training session, a capability not available in other methods. For further comparisons, including classic rule-based algorithms, please refer to Appendix A.Hypernetwork Hypernetworks, or hypernets, are neural networks that generate weights for main networks [31] . Benefiting from over-parameterization and strategic design, these networks enhance training flexibility and adaptability, improve information sharing, and accelerate training processes. While widely used in many areas like continual learning [32, 33] , transfer learning [34] , uncertainty quantification [35, 36] , natural language processing [37] and computer vision [38] , their potential in rule learning remains largely untapped. Our HyperLogic framework can bridge this gap. Specifically, our hypernetwork takes inspirations from HyperGAN [36] and similarly we add a loss term to encourage the diversity of the generated network weights. Unlike typical applications in other domains that focus on enhancing parameter efficiency and training speed, our approach addresses the unique challenges of adding model flexibility while preserving interpretability. Our main goal is to expand the parameter space to produce more diverse, concise and accurate rule sets.• FRL methods may face scalability issues when dealing with large candidate rule sets or complex data. • HyperLogic, a neural network-based approach, is more scalable for pattern mining in large-scale data. 
hyperlogic: 
 HyperLogic is a versatile framework designed to enhance various existing neural rule-learning methods. The concept of hypernetworks can be applied to different types of main networks, provided they focus on learning a set of interpretable weights in a differentiable manner. Importantly, HyperLogic imposes no additional restrictions on data formats or rule languages; it simply builds upon the capabilities of the chosen main network.Our main network is currently based on DR-Net [17] , which features a simple architecture and is designed for binary classification tasks. We chose this architecture for its simplicity, which aids in our proofs and explanations. However, we recognize its limitations, so we can also explore other more flexible neural approaches as the primary network to overcome restrictions related to data formats and rule languages. This is further discussed in Appendix B.The main network is a two-layer neural network as shown in Fig. 1 (right part). Due to its interpretability, one can directly extract the if-then rules from the trained neural networks. The input layer of the main network is fed with a D-dimensional binary data x = [x d ], each element x d ∈ {1, -1}. Here x d indicates the grounded predicate, where 1 indicates True and -1 means false. Note that common input types may include binary, categorical, or numerical features, all of which are discretized and binarized to form our binary input features. The hidden layer has K neurons, where K determines the total number of rules and serves as a hyperparameter. The first layer is referred to as the Rule Layer, and the output layer as the OR layer. Rule Layer: Each neuron in the hidden layer is denoted as o k ∈ {0, 1}. The output represents whether a rule is satisfied. Each neuron is calculated as:EQUATIONwhere 1{•} denotes the indicator function. Note that the indicator function is one if and only if all inputs match the sign of the corresponding weights: all positive weights should have the inputs of 1, and all negative weights should have the inputs of -1, and zero weights mean that the corresponding inputs are excluded from the rule. Note that the indicator function is non-differentiable, which will make the gradient hard to compute.Here, we will instead use a differentiable smooth function to approximate the indicator function (this will also ease the theoretical analyses). For example, we can use h(u) = exp -u 2 τ , τ > 0, to approximate 1{u = 0}, where τ is the tunable temperature and controls the approximation error.OR Layer: The second layer operation is defined asEQUATIONwhere we assume that the rules contribute to the final prediction in a weighted additive form to reflect the OR composition, where u k denotes the weight assigned to the k-th rule.Define the network weights for the previous model as θ = (w, u), where w ∈ R D×K , and u ∈ R K .Instead of learning only one set of θ, we will learn the distribution of θ, denoted as µ. Specifically, we introduce a generative model as the hypernetwork to produce samples of θ, i.e.,EQUATIONHere, ϵ is drawn from some simple base distribution such as Gaussian distribution. More details about the hypernetwork structure can be seen in Appendix C. Note that, in this way, our model parameters have been changed to µ, which is defined as the distribution of θ, and the proposed Hyperlogic model isEQUATIONIn practice, we can use the Monte Carlo method to estimate the above expectation by randomly drawing M samples from µ, denoted as θ 1 , . . . , θ M , and we defineEQUATION)This model can be regarded as a mixture of expert models with finite M experts. Each expert is a shallow two-layer neural network that encodes at most K if-then rules.When working with high-dimensional data (e.g., 5000 dimensions), the number of parameters in the main network increases proportionally with the input dimensions, even if the network structure remains unchanged. This makes training a hypernetwork to generate weights for each module of the main network more challenging. To tackle this issue, we considered two strategies:Combining Original Weights with Hypernetwork-Generated Weights Instead of allowing the hypernetwork-generated weights (W hyper ) to fully dictate the main network's weights, we combine them with the original weights (W main ). The final weights are calculated as follows:W main-final = α • W hyper + (1 -α) • W main ,where α is a learnable parameter constrained between 0 and 1. This strategy enhances stability without sacrificing the hypernetwork's ability to produce diverse weights. If the hypernetwork is poorly trained and generates inappropriate weights, the model adjusts α toward 0, effectively reverting to the standard main network without hypernetwork influence. We employed this strategy in our subsequent experiments.Generating Weights for Only Some Modules of the Main Network We test this strategy when the dataset dimension and size are very large in the supplementary experiment. It shows that generating only part of the main network's weights using the hypernetwork still significantly improves results.For more details, please see the supplementary experiments in Appendix B.One can learn µ by minimizing the loss function ℓ(y, f µ (x)), where f µ (x) can be approximated by the finite-sample estimator in Eq. ( 5). Our loss contains two parts. One is the task-related loss function, denoted as ℓ task (y, f µ (x)). Suppose the problem is a binary classification problem, one can use the binary cross-entropy loss (BCE), where we first map f µ (x) to a probability value between 0 and 1 using a link function such as sigmoid and the loss is defined as the negative log-likelihood of a sample belong to a class. Here, the task-related loss measures the prediction accuracy.The second part of the loss is the regularization loss, denoted as ℓ reg , which is introduced to encourage the diversity of the generated θ and model sparsity (rule simplicity for each expert), i.e.,EQUATIONFor the first regularization term, we introduce a prior distribution µ 0 with a high entropy, and we minimize the relative entropy or KL divergence of the µ and µ 0 . Minimizing the relative entropy encourages the diversity of the generated θ. For the second regularization term, we are considering the ℓ 1 sparsity norm for the second OR layer's weights, where we hope that for each expert, the discovered number of rules is as compact as possible. In our paper, ℓ reg represents the loss incurred by the hypernetwork, as illustrated in Fig. 1 .• FRL methods explicitly order the rules into a descending list based on rule probabilities or other criteria. • HyperLogic generates an unordered set of rules. 
theoretical analysis: 
 We will employ theoretical analysis to demonstrate that while the main network (i.e., a two-layer shallow neural network) has low capacity, incorporating the hypernet idea significantly enhances the model's expressive power. Specifically, we will establish that the proposed HyperLogic model serves as a universal approximator in the Barron space [19] , which will be defined later. For simplicity, but without loss of generality, we consider the case where the hypernetwork fully generates the weights and the main network itself has no parameters. Additionally, we will present the generalization error under the above two types of regularization.Reparametrization: To ease the proof, let's first rewrite Eq. ( 5) asEQUATIONby reparametrization. To achieve this, we first use the split variable trick by defining w + = max{w, 0} and w -= -min{w, 0}, and reparametrize w = w + -w -and |w| = w + + w -.In this way, we can get rid of the absolute value and haveEQUATIONTherefore, we can simply reparametrize w k , where w k ≥ 0, as a concatenation of the vector[w + d,k ] d=1,...,D and [w - d,k ] d=1,...,D .We construct the new input data x by making a copy of the original data and modifying each copy given Eq. ( 8). That is, for one copy, we subtract it by 1, and for another copy, we flip the sign and subtract it by 1. Then, we concatenate the two copies of data.Given such a reparametrization, we get a simple form as Eq. ( 7). In the following analysis, we will still assume that w k ∈ R D and x ∈ R D , which doesn't affect the generality. Preparation for the Theoretical Analysis: Our analysis below is based on the following observations. Given the generated random variable ((w 1 , u 1 ), . . . , (w K , u K )) ∈ R (D+1)K , we have denoted their joint distribution as µ. Let us further denote the marginal distribution of (w k , u k ) as µ k , k = 1, . . . , K, respectively. Define a random variable ( w, ũ) ∈ R D+1 , which draws a sample from µ k with (equal) probability 1/K, k = 1, . . . , K, and then scale the u-component by K. Denote the resulting mixture distribution as μ. Then it follows from the linearity of expectation thatEQUATIONWe define fμ (x) := E ( w,ũ)∼μ ũh w⊤ x and we have shown that f µ (x) = fμ (x) as stated above.The motivation for introducing fμ (x) is to facilitate the analysis of approximation error and generalization error. There exist theoretical results for a two-layer neural network of the form g M (x) = M m=1 u m h w m⊤ x , where h(•) belongs to certain classes of smooth activation functions, and (w m , u m ), m = 1, . . . , M , are M independent samples drawn from a fixed distribution. In contrast, our proposed HyperLogic model involves randomly drawing samples ((w 1 , u 1 ), . . . , (w K , u K )) from µ, each with K components. To align this with the existing theoretical framework, we perform a transformation as shown in Eq. ( 9). This involves converting the process of drawing M samples, each with K components, into drawing M K samples, each with a single component. This reformulation maintains the same expectation results, allowing us to leverage existing theoretical results to analyze HyperLogic. Note that the conversion mentioned is purely for theoretical proof purposes and is not implemented in the actual algorithm.Using the connection as shown in Eq. ( 9), we can directly leverage the approximation error results for a single hidden layer neural network. Let's first define the Barron space, which provides a set of functions for which neural networks can achieve good approximation properties. Definition 1 (Barron Space [19] ). A function f : R D → R belongs to the Barron space B if it can be represented as:f (x) = R×R D ×R uh w ⊤ x + b dµ(u, w, b)where h is an activation function, µ is a probability measure on R × R D × R, and the following Barron norm is finite:∥f ∥ B = inf R×R D ×R |u|∥(w, b)∥dµ(u, w, b) : f (x) = R×R D ×R uh w ⊤ x + b dµ(u, w, b) .Note that the representation of f in the form f (x) = uh w ⊤ x + b dµ(u, w, b) may not be unique. The Barron norm seeks the representation that minimizes the integral of |u|∥(w, b)∥. The introduction of the Barron norm provides a quantitative measure of the "complexity" of a function in the context of neural networks. Functions with a smaller Barron norm are considered simpler and are easier to approximate with neural networks. Next, let's provide the approximation error analysis for our HyperLogic model (the proof can be found in Appendix D): Theorem 1. For any function f in the Barron space, there exist M experts ((w m 1 , u m 1 ) , . . . , (w m K , u m K )) ∈ R (D+1)×K , m = 1, . . . , M , forming a predictor f M as shown in Eq. ( 7), such that∥f -f M ∥ L 2 ≤ ∥f ∥ B √ M K ,where the L 2 norm for a function f : The above theorem asserts that a HyperLogic model (essentially a mixture of expert models) can approximate any continuous function in the Barron space to arbitrary precision given enough M , K and appropriate weights.R D → R is defined as ∥f ∥ L 2 = R D |f (x)| 2 dD(x)Let's derive the generalization error bounds for the HyperLogic model under entropic regularization (the first term of Eq. ( 6)) and sparse regularization (the second term of Eq. ( 6)). The key steps involve calculating the Rademacher complexity of the model class and then using this to bound the generalization error. The proof can be found in Appendix E. Let's give the results here. Theorem 2. For the function classEQUATION)Similarly, for the function classF 1 := f µ (•) : E µ 1 K K k=1 |u k | ≤ B 1 , we have E D [ℓ(f )] -l(f ) ≤ 64B 1 (D + 1)/n + C log(1/δ) n .Here, E D [ℓ(f )] is the expected loss over data distribution, l(f ) is the empirical loss, n is the number of samples, C is a constant dependent on the loss function, and δ is the confidence level.From the results, we see that the relative entropy regularization effectively balances fitting the training data and adhering to the prior distribution with high entropy. Increasing the sample size or decreasing the KL bound enhances the model's generalization ability, ensuring good performance on unseen data. Similar conclusions can be arrived for the sparse regularization.• FRL methods provide probabilistic interpretations for the rules by construction.• HyperLogic does not directly output rule probabilities, although probabilities could potentially be derived from the learned patterns. 5. Integration Potential: A recent study [42] proposed a neural method to learn ordered rule lists, which could potentially be integrated with HyperLogic to enable joint rule generation and probabilistic ordering, combining the strengths of both approaches. Other statistical rule mining methods like Bayesian rule lists [43] and Bayesian decision sets [20] provide probabilistic interpretations but are limited to binary classification and small rule sets. Fuzzy rule-based models [44] incorporate human-like reasoning but lack probabilistic predictions and scalability. In the supplementary experiments B, we include the current advanced traditional rule learning method CLASSY in the comparison to further demonstrate the advantages of our method over traditional methods. 
experiment: 
 In this section, we report experimental results to answer the following questions:• RQ1: How does the performance of the optimal rule set selected by HyperLogic compare to the rule sets obtained by other methods? • RQ2: How rich are the rule sets generated by HyperLogic, and how are their accuracy and diversity affected by parameters? • RQ3: Can we further leverage the advantages of HyperLogic through ensemble learning to enhance performance?Implementation Details: During training, for each data batch, we randomly generate M 1 samples of network weights to approximate the expectation (as shown in Eq. ( 5), here we compute f M1 (x) as an approximation in the training stage). Increasing M 1 enhances the stability of hypernetwork training but raises computational costs. In our experiments, M 1 is set to 5 or 10 . After training, we generate M 2 sets of weights from the hypernetwork, resulting in M 2 rule sets. In other words, in the inference stage, we use f M2 (x) instead. While M 1 is relatively small, M 2 can be large (e.g., 5000). We then select the rule set with the highest training accuracy as the optimal set, though other criteria, such as minimal loss or custom evaluation metrics balancing accuracy and complexity, can also be used.For HyperLogic, We use Adam as the optimizer, and the learning rate is 1 × 10 -4 , with weight decay is 1 × 10 -4 . The number of training epochs is 10000. In the experiments for selecting the optimal rules for comparison, we set hyperparameter M 1 = 5, M 2 = 5000, λ 1 = 0.01, and λ 2 = 0.1 (λ 1 and λ 2 are related to the hypernetwork loss or regularization loss, as defined in Eq. ( 6)).In the following experiments that analyze the influence of hyperparameter, we adjusted only the corresponding hyperparameter while keeping others unchanged. Datasets: We selected four publicly available binary classification datasets: MAGIC gamma telescope (magic), adult census (adult), FICO HELOC (heloc), and home price prediction (house). In all datasets, preprocessing was performed to encode categorical and numerical attributes as binary variables, which can be found in [17] . We compared our model HyperLogic with other state-of-the-art rule learning methods: DR-Net [17] , CG [7] , BRS [20] , and RIPPER [6] . Since CG, BRS, and RIPPER cannot learn negative conditions, we additionally appended negative conditions for these models.In the performance comparison section, we sampled M 2 = 5000 times from HyperLogic, selecting the rule set that performed best on the training set as the optimal rule set, and compared it with other methods. Table 1 presents the comparison of the accuracy of the optimal rule sets selected by our method and those generated by other methods. Our method further improves upon DR-Net and outperforms other methods across all four datasets. It is important to note that the rule set that performs best on the training set does not strictly guarantee the best performance on the test set, as shown in Fig. 2 . However, this selection method is sufficiently simple, and experiments have shown that it can achieve good performance. In addition to rule accuracy, we also considered two metrics to measure the compactness of the rules: model complexity and rule complexity. Model complexity is defined as the sum of the number of rules and the total number of conditions in the rule set; rule complexity is the average number of conditions in each rule of the model. Fig. 3 shows that our method reduces both model complexity and rule complexity compared to DR-Net, indicating that we have not only improved the accuracy of the rules but also made them more concise, demonstrating the effectiveness of our framework.In this section, we primarily considered the effects of three parameters, M 1 , λ 1 , and M 2 , on the learned rules. Among these, M 2 primarily affects the selection of the final optimal rule set by controlling the number of candidate rule sets sampled from the hypernetwork after training phase, while M 1 and λ 1 directly influence the training of the hypernetwork. For M 2 , we considered the values 10, 100, 1000, 2000, 5000, and 10000 to examine its impact on the accuracy, model complexity, and rule complexity of the generated optimal rules, as shown in Fig. 4 . It can be observed that the accuracy, model complexity, and rule complexity of the optimal rules generally increase with the increase of M 2 , and then level off or slightly decrease. This is because, when M 2 is small, we may not be able to sample sufficiently suitable rule sets to fit the training set. When M 2 is too large, the sampled optimal rule set is more likely to overfit the training set, leading to a slight decrease in test accuracy. So far, we have focused on the optimal rule set extracted from the hypernetwork. Table 2 further shows the different optimal results obtained in three training sessions for the heloc dataset, indicating that we can find different high-quality rule sets in different training sessions. However, considering only one optimal rule set is not enough to reflect the advantage of the hypernetwork in generating multiple high-quality rule sets in one training session, which provides more options and a deeper understanding of the data, something that methods learning only one set of rules cannot offer. The ability of the hypernetwork to generate diverse rules is mainly related to M 1 and λ 1 . λ 1 , as the coefficient of diversity regularization term, directly affects the diversity of the hypernetwork output, while M 1 indirectly affects the process by influencing the stability of hypernetwork updates. For the impacts of M 1 and λ 1 , we are no longer concerned with the optimal rule set but focus on all the rule sets generated by the hypernetwork, and describe the diversity and accuracy of the rule sets as a whole.The diversity of the rules is measured in two aspects: 1. How many different rule sets can we generate by sampling a certain number of weights from the hypernetwork? 2. What is the degree of similarity between the generated different rule sets? We use Jaccard similarity to measure this. We take the magic dataset as an example to analyze M 1 and λ 1 ; the results for the remaining datasets are in Appendix F. The impact of M 1 on rule diversity and test accuracy is shown in Fig. 5 . It can be seen that when M 1 = 5, there is the best performance in all aspects. This may be because, when M 1 is small, the hypernetwork generates weights fewer times, making the training less stable. When M 1 is too large, the training and updates of the hypernetwork become too stable and conservative, slowing down the convergence speed and resulting in a performance decline of the hypernetwork. The impact of λ 1 on rule diversity and test accuracy is shown in Fig. 6 . It can be seen that increasing λ 1 significantly enhances the diversity of the rule sets, but excessive emphasis on rule diversity may affect the accuracy of the rules. Since the hypernetwork can generate a diverse and rich set of rule sets, a natural idea is to use ensemble learning to achieve better classification performance. We use the simple averaging voting method for ensemble learning. We select the top L rule sets based on their accuracy on the training set and report their accuracy on the test set.The values of L are 1, 5, 10, 30, 50, 100, and 200. As shown in Fig. 7 , the test accuracy initially increases with the increase of L, then slightly decreases. This may because the top single rule set is already highly effective, leaving little room for improvement. When L is too large, less effective rule sets decrease the ensemble's overall performance. Additionally, our current strategy focuses on balancing diversity and accuracy of single rule sets. Adjusting the strategy specifically for ensemble learning could yield better results. 
conclusion and limitations: 
 In this paper, we proposed HyperLogic, a novel framework that enhances the field of differentiable rule learning through the integration of hypernetworks. We also provided a theoretical foundation for HyperLogic's performance, explaining its effectiveness. This includes an analysis of approximation error and generalization capabilities under sparsity and diversity regularization. What's more, we conducted extensive experiments on multiple datasets, demonstrating that HyperLogic accelerates the training process while producing more concise and accurate rules. Despite these advancements, HyperLogic introduces additional hyperparameters and requires further exploration of ensemble learning within this framework. Future research will focus on alternative training strategies, more stable weight combination methods, and applying HyperLogic to a broader range of datasets and tasks to enhance its generality and practicality. 
broader impacts: 
 The development and utilization of interpretable logic rules through HyperLogic have significant positive societal impacts, particularly in high-risk domains such as healthcare, finance, and legal systems. By generating multiple candidate rule sets, HyperLogic provides more flexible and comprehensive insights, enhancing transparency and trust in decision-making processes. This is crucial for ensuring safety and regulatory compliance. However, it is important to note that while our method provides richer rule sets and aids experts in understanding data patterns, these rules should complement rather than replace expert judgment. Relying solely on automated rules without expert oversight in critical areas could pose significant risks. Thus, our approach emphasizes the collaborative role of machine learning models and human experts to mitigate potential negative impacts. 
 
 Appendix: We first compare HyperLogic with falling rule lists(FRL) methods like [39, 40, 41] , which are often used in practice. FRL methods explicitly construct a falling list of rules with probabilities; Differentiable HyperLogic focuses on directly mining patterns and generating rules from data in a scalable manner using neural networks. Our differentiable approach demonstrates better scalability for pattern mining tasks on large-scale data (see Section 2) . The main differences between FRL and the HyperLogic are: 1. Rule Generation:• FRL methods rely on other rule mining techniques to generate initial candidate rules.• HyperLogic uses a differentiable neural network approach to directly mine patterns and generate rules from data in an end-to-end manner.To expand our experiments to larger and more complicated cases, we considered the latest Neuro-Symbolic algorithm DIFFNAPS [30] , which is capable of pattern mining under large-scale data conditions. For HyperLogic, we selected only the classifier part of DIFFNAPS as the main network to compare with vanilla DIFFNAPS.Following the original experiments, we tested the model's pattern mining performance under a fixed input dimension of 5000 and varying total number of categories K (ranging from 2 to 50), measured by the F1 score, with each category containing 1000 samples. Compared with the current data set, in our new data set, the feature dimension has been raised from a maximum of 154 dimensions to a maximum of 5k dimensions, the amount of data has been raised from a maximum of 24,000 to a maximum of 50,000, and the task has been raised from a maximum of 2 categories to a maximum of 50 categories, reflecting the characteristics of the task diversity and complexity. The experimental results are shown in the table below. It can be seen that in datasets with fewer categories, due to the smaller total number of samples, HyperLogic has not yet received sufficient training and does not perform ideally. However, in more challenging classification datasets with an increased number of samples, the model's performance has significantly improved, with an average F1 score increase of 6%. This fully demonstrates that our framework can empower diverse neural rule learning networks, capable of handling large-scale data and possessing a good range of applications. DIFFNAPSWe evaluated our method on four large biological datasets following the settings of DIFFNAPS: Cardio, Disease, BRCA-N, and BRCA-S, using the area under the curve (AUC) as the metric. We continued to combine our approach (HyperLogic) with DIFFNAPS as the main network and compared it to vanilla DIFFNAPS. Additionally, FRL [39] cannot scale to non-trivial data, while CLASSY [41] was already compared in the original paper. The table shows the dataset details (i.e. samples (n), features (D), and classes (K)), number of discovered patterns (#P), average pattern length (|P|), and AUC scores (results for DIFFNAPS and CLASSY are taken directly from [30] ). CLASSY lacks the finesse to effectively mine patterns for large-scale real-world tasks. Moreover, despite competing with the strong DIFFNAPS baseline, HyperLogic achieved further improvements, demonstrating its potential for handling large, real-world datasets.We adopted HyperGAN as our hypernetwork. Assuming the main network comprises N distinct weight partitions, HyperGAN includes a mixer Q and N generators G 1 , G 2 , . . . , G N . Mixer Q: Receives high-dimensional Gaussian distributed random samples s ∼ N (0, I), and transforms it into a N × h dimensional vector z, further split into N samples of h-dimensional vectors z 1 , z 2 , . . . , z N . The mixer's design reflects the necessity for correlations between layer weights, as each layer's output becomes the subsequent layer's input. Generators G i : Each generator receives a vector z i from the mixer, producing an output vector of dimension m i , where m i represents the parameter count for the i-th part of the network weights. These vectors are then reshaped to meet the specifications of the corresponding layers in the main network. In our specific task, the number of generators N is set to 2, and each generator's input dimension h is 64. The input dimension of the Mixer, which is the sampled noise, is 256, and it produces an output of N × h = 128. Each of the above models has two hidden layers with dimension 512, and the activation function is ReLu. Our main network is designed to generate K = 50 rules. For a dataset with dimension D:• Generator 1: Produces an output of dimension (D × K, 1), representing the Rule layer.• Generator 2: Produces an output of dimension (K × 1, 1), representing the OR layer.These outputs are then reshaped to meet the specifications of the corresponding layers in the main network.Proof. Using Barron's theorem (e.g., [19, Theorem 11.3] ), there exist weight coefficients (w m k , ũm k ), m = 1, . . . , M , k = 1, . . . , K, such that the function f defined by f (x) = 1 M K M m=1 K k=1 ũm k h (w m k ) ⊤ x satisfies ∥f -f ∥ L 2 ≤ ∥f ∥ B √ M K . Setting u m k = ũm k /K, w m = (w m 1 , . . . , w m K ), u m = (u m 1 , . . . ,F KL := f µ (•) : D KL (µ∥µ 0 ) ≤ B KL ,where µ 0 is a product distribution μ⊗K 0 . Using the property of relative entropy, for each pair of marginal distributions µ k and μ0 , k = 1, . . . , K, their relative entropy satisfies D KL (µ k ∥μ 0 ) ≤ B KL . Using the convexity of the relative entropy, the mixture distribution μ satisfies D KL (μ∥μ 0 ) ≤ B KL . Hence, the function class F KL belongs to the function classFKL := fμ (•) : D KL (μ∥μ 0 ) ≤ B KL .Using [19, Corollary 10 .17], the Rademacher complexity of FKL is bounded by 2B KL /n. Given our bound on the Rademacher complexity, for f ∈ FKL , we get:EQUATION)where E D [ℓ(f )] is the expected loss, l(f ) is the empirical loss, n is the number of samples, C is a constant dependent on the loss function, and δ is the confidence level.Next, we will derive the generalization error bounds for the HyperLogic model under sparse regularization (as shown in the second term of Eq. ( 6)).Let us compute the Rademacher complexity of the model classF 1 := fµ (•) : E µ 1 K K k=1 |u k | ≤ B 1 .Observe that the constraint E µ1 K K k=1 |u k | ≤ B 1 implies that E μ [|ũ|] = E µ 1 K K k=1 u k ≤ E µ 1 K K k=1 |u k | ≤ B 1 .Hence, the function class F belongs to the function classF1 := fμ (•) : E μ [|ũ|] ≤ B 1 .Using [19, Proposition 11.23] , the Rademacher complexity of F1 is bounded by 32B 1 (D + 1)/n. The generalization error bound for the HyperLogic model under sparse regularization, derived from the Rademacher complexity, can be expressed as follows:E D [ℓ(f )] -l(f ) ≤ 64B 1 (D + 1)/n + C log(1/δ) nThe explaination of this error bound is similar with the previous one. For our method, all experiments were conducted on a Linux server with an Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz and 30Gi of memory, running Ubuntu 20.04.5 LTS, using one of the NVIDIA GeForce RTX 3090 GPUs available on the server. Each experimental run took approximately 10-20 minutes to complete. This setup ensures that our experiments are reproducible and efficient. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss potential positiveand negative societal impacts of the work in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 


Decision: The authors propose a novel approach for generating candidate rulesets using hyper-networks based on HyperGAN. The authors provided additional experimental results during the rebuttal, which, together with the results in the original submission, show competitive performance, and the ability to adopt their framework to different main models.

Reviews: review: paper_summary: paper_summary: This paper introduces HyperLogic, a novel differentiable framework for rule learning using neural networks. Instead of directly training network weights, HyperLogic extracts rules by generating the weights for a primary network through hypernetworks. These hypernetworks create diverse sets of weights, functioning like a mixture of experts, thereby enhancing model flexibility. To train the hypernetworks, the authors minimize the relative entropy between the learned distribution and a prior distribution, incorporating regularization to encourage simplicity. They employ smooth approximations to ensure the problem be differentiable. The authors demonstrate HyperLogic’s effectiveness through both theoretical analysis and empirical results on multiple datasets.
strength_weakness: strength_weakness: - The results seem to be sensitive to the choice of hyperparameters, which could affect the stability and generalizability of the method. Are there any guidelines or automated processes to aid in choosing the optimal hyperparameters?
- The scalability of the approach for extremely large datasets or high-dimensional data remains unclear. Further evaluation is needed to understand its performance in such scenarios.
- Typo: in line 125,  all negative weights should have the inputs of 0 ?
questions: questions: see weaknesses
limitations: limitations: N/A
paper_summary: paper_summary: The authors consider the problem of learning interpretable rules for decision-making and propose a new neural rule learning framework to find these rules efficiently. In particular, they suggest to use a hypernetwork to learn a diverse set of network parameters for the rule-encoding network. On small binary data, they show that their algorithm finds reasonable rule sets that predict a given class well.

[Edit after rebuttal]: The authors provided additional discussion and experiments addressing my concerns, which leaves me with a *weak accept*.
strength_weakness: strength_weakness: - Despite the broad motivation of making neural decisions interpretable, the presented approach is very limited in what networks are used (2-layer specific architecture), which data is used (only binary datasets with labels), and which rules can actually be expressed (only conjunctions in antecedent, single label in consequent).
Especially the type of considered data needs to be communicated clearly from the start.
- There is a significant amount of related work, including recent neural approaches, that is not considered here. Some more classical approaches include statistical pattern mining [1] and falling rule lists [2,3,4]. Recently, a neural approach that finds differential patterns between labels -- which are essentially if-then rules has -- been proposed [5]. The experiments should be extended to properly compare to existing work.
- The experiments are very limited in terms of considered datasets: they are old “toy” datasets of small scale in both sample and feature dimension. Recent works, including [4], incorporate large-scale real-world data to actually show that such neural approaches are more scalable and can hence consider other data than classical approaches.

In summary, it is unclear to me whether the presented approach provides any improvement over existing work, or is scalable to large data at all.

[1] https://dl.acm.org/doi/10.1145/2783258.2783363
[2] https://proceedings.mlr.press/v38/wang15a.html
[3] http://proceedings.mlr.press/v84/chen18a/chen18a.pdf
[4] https://www.sciencedirect.com/science/article/pii/S0020025519310138
[5] https://ojs.aaai.org/index.php/AAAI/article/view/28756
questions: questions: - In section 4, you convert the process of drawing M samples with K components each into drawing MK single-component samples. Does the latter actually match the former? It seems that the K components in the former are actually dependent, which would mean it does not correctly model it. Please clarify.
- In Table 1, the bold numbers are confusing. For adult, house, and heloc data, DR-Net, which is closely related and forms the basis of the architecture design, performs on par with the presented method. The results are within the given standard error.
limitations: limitations: While there is a brief discussion on limitations, neither the limitations in terms of data or rule language/architecture, nor the connection to existing work is properly discussed.
paper_summary: paper_summary: The paper presents HyperLogic, a pioneering framework that integrates hypernetworks into differentiable rule learning, enhancing the interpretability and flexibility of neural networks. It provides a strong theoretical foundation and backs it with extensive empirical evidence, demonstrating its effectiveness over traditional rule learning methods. HyperLogic's ability to generate diverse and accurate rule sets is particularly noteworthy, addressing a significant gap in the field of AI interpretability.
strength_weakness: strength_weakness: 1. The introduction of additional hyperparameters may lead to increased sensitivity, which could impact the model's performance and generalizability.
2. The framework's complexity might pose challenges for practitioners not well-versed in advanced neural network architectures.  While the paper mentions the efficiency of the rule-learning process, the computational overhead introduced by the hypernetworks is not thoroughly discussed.
3. The framework's applicability to a broader range of datasets and tasks needs to be investigated to ensure its generality and practical utility.
questions: questions: How does HyperLogic perform in domains outside of the tested datasets, particularly in non-binary classification tasks?
limitations: limitations: Applications to more general scenarios.

Comments: title: response to author's additional experiments
Answers to new comments from Reviewer F7a8
Official Comment by Reviewer Q4Et
Rebuttal comment
comment: Dear Reviewer F7a8,

We would like to extend our gratitude once again for your insightful suggestions and positive response. We will also ensure the extra results and discussions will be updated in the revised manuscripts.

Warm regards,

The Authors
Dear Reviewer Q4Et,

Thank you for acknowledging our work. We will incorporate the relevant results in the final version to further enhance its quality.

Best regards,

The Authors
Thank you for these additional results addressing my concerns. This provides a much more comprehensive and convincing evaluation of your work. Even though HyperLogic only provides a marginal improvement over some existing methods in light of all experimental results, I do see its value for the community in a general framework as well as in a better theoretical analysis compared to what exists. Furthermore, the added results put it into perspective in concurrent work in the field and allow for a good and critical comparison.

Consequently, I raise my score to *weak accept*, on the expectation that additional discussion on related work, results, and a critical analysis will be put into the revised manuscript.

I wish the authors best of luck with the submission.
Dear Reviewer hMMc,

We are grateful for your time and effort in reviewing our paper and providing thoughtful feedback. We have carefully considered all of your comments and have responded to them accordingly.

As we near the end of the author-reviewer discussion, we would like to hear your feedback on our rebuttal. We would greatly appreciate it if you could review our responses and let us know if we have adequately addressed your concerns. Additionally, we welcome any further comments or discussions you may have.

Thank you for your valuable consideration.

Best regards,

The Authors
Thank you for acknowledging our method and suggestions. We will clarify our approach's generality and contributions in the final draft. Additionally, we will compare our approach to falling rule lists and expand experiments to four large biological datasets.

**1. Compare to falling rule lists (FRL) [2] and other methods**

FRL methods explicitly construct a falling list of rules with probabilities; Differentiable HyperLogic focuses on directly mining patterns and generating rules from data in a scalable manner using neural networks. Our differentiable approach demonstrates **better scalability for pattern mining tasks on large-scale data** (see Section 2). The main differences between FRL and the HyperLogic are:
1. Rule Generation:
- FRL methods rely on other rule mining techniques to generate initial candidate rules.
- HyperLogic uses a differentiable neural network approach to directly mine patterns and generate rules from data in an end-to-end manner.
2. Scalability:
- FRL methods may face scalability issues when dealing with large candidate rule sets or complex data.
- HyperLogic, a neural network-based approach, is more scalable for pattern mining in large-scale data.
3. Rule Ordering:
- FRL methods explicitly order the rules into a descending list based on rule probabilities or other criteria.
- HyperLogic generates an unordered set of rules.
4. Probabilistic Interpretation:
- FRL methods provide probabilistic interpretations for the rules by construction.
- HyperLogic does not directly output rule probabilities, although probabilities could potentially be derived from the learned patterns.
5. Integration Potential:
A recent study [1] proposed a neural method to learn ordered rule lists, which could potentially be integrated with HyperLogic to enable joint rule generation and probabilistic ordering, combining the strengths of both approaches.

Other statistical rule mining methods like Bayesian rule lists [4] and Bayesian decision sets [5] provide probabilistic interpretations but are limited to binary classification and small rule sets. Fuzzy rule-based models [6] incorporate human-like reasoning but lack probabilistic predictions and scalability.


**2. Consider large real datasets**

We evaluated our method on four large biological datasets following the settings of DIFFNAPS[1]: Cardio, Disease, BRCA-N, and BRCA-S, using the area under the curve (AUC) as the metric. We continued to combine our approach (HyperLogic) with DIFFNAPS  as the main network and compared it to vanilla DIFFNAPS. Additionally, FRL[2] cannot scale to non-trivial data, while CLASSY was already compared in the original paper.

The table shows the dataset details (i.e. samples (n), features (m), and classes (K) ), number of discovered patterns(#P), average pattern length(|P|), and AUC scores (results for DIFFNAPS and CLASSY are taken directly from [1]).

||||| HyperLogic | HyperLogic | HyperLogic | DIFFNAPS |DIFFNAPS| DIFFNAPS | CLASSY |CLASSY | CLASSY |
|---------|-----|-----|----|:-----------:|------------|------|----------|-------|------|--------|-------|------|
| Dataset | n   | m   | K  | #P| \|P\| | AUC  | #P| \|P\| | AUC | #P | \|P\| | AUC |
| Cardio  | 68k | 45  | 2  | 15 | 2| 0.57 | 14 | 2| 0.56 |10| 2| 0.36 |
| Disease | 5k  | 131 | 41 | 866 | 2 | 0.86 | 838 | 2| 0.84 | 25 | 2 | 0.11 |
| BRCA-N  | 222 | 20k | 2  | 187 | 6  | 0.95 | 146 | 9| 0.91 | 3  | 1| 0.45 |
| BRCA-S  | 187 | 20k | 4  | 1k | 2 | 0.89 | 1k | 2 | 0.86 | 2| 1| 0.23 |

CLASSY lacks the finesse to effectively mine patterns for large-scale real-world tasks. Moreover, despite competing with the strong DIFFNAPS baseline, HyperLogic achieved further improvements, demonstrating its potential for handling large, real-world datasets.

Please let us know if you have any other questions or suggestions for improving the final draft. We appreciate it a lot.

[1] Walter, N. P., Fischer, J., & Vreeken, J. (2024, March). Finding interpretable class-specific patterns through efficient neural search. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 8, pp. 9062-9070).

[2] Wang, F., & Rudin, C. (2015, February). Falling rule lists. In Artificial intelligence and statistics (pp. 1013-1022). PMLR.

[3] Proença, H. M., & van Leeuwen, M. (2020). Interpretable multiclass classification by MDL-based rule lists. Information Sciences, 512, 1372-1393.

[4] Yang, H., Rudin, C., & Seltzer, M. (2017, July). Scalable Bayesian rule lists. In International conference on machine learning (pp. 3921-3930). PMLR.

[5] Wang, T., Rudin, C., Doshi-Velez, F., Liu, Y., Klampfl, E., & MacNeille, P. (2017). A bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research, 18(70), 1-37.

[6] Jiménez, F., Sánchez, G., & Juárez, J. M. (2014). Multi-objective evolutionary algorithms for fuzzy classification in survival prediction. Artificial intelligence in medicine, 60(3), 197-219.A
Thanks for the detailed response. I have no further questions and would like to keep my positive score.
Thank you for your response.

Regarding *W1*: I appreciate the answer, the modularity of the approach did not become that evident in the original draft. I would suggest to rewrite the paper accordingly also incorporating the new results, as this strengthens the contribution.

Regarding *W2*: The new results including their critical discussion does cover more recent neural approaches. However approaches such as the popular falling rule lists, which are heavily used in practice, should be compared to.

Regarding *W3*: Still, not considering recent real datasets is a major weakness in the paper. The additional results only cover a larger synthetic dataset.

I would like to encourage the authors to use the remaining time to (1) compare to falling rule lists and (2) consider at minimum one large real dataset, the concurrent literature on pattern mining does give plenty of examples of those.