HyperLogic: Enhancing Diversity and Accuracy in Rule Learning with HyperNets
Authors: Yang Yang
Wendi Ren
Shuang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Differentiable rule learning
Heterogeneous logic rule
Hypernet
Primary area: interpretability_and_explainability

Abstract: Exploring the integration of if-then logic rules within neural network architectures  presents an intriguing area. This integration seamlessly transforms the rule learning task into neural network training using backpropagation and stochastic gradient descent. From a well-trained sparse and shallow neural network, one can interpret each layer and neuron through the language of logic rules, and a global explanatory rule set can be directly extracted. However, ensuring interpretability may impose constraints on the flexibility, depth, and width of neural networks. In this paper, we propose HyperLogic: a novel framework leveraging hypernetworks to generate weights of the main network. HyperLogic can unveil multiple diverse rule sets, each capable of capturing heterogeneous patterns in data. This provides a simple yet effective method to increase model flexibility and preserve interpretability. We theoretically analyzed the benefits of the HyperLogic by examining the approximation error and generalization capabilities under two types of regularization terms: sparsity and diversity regularizations. Experiments on real data demonstrate that our method can learn more diverse, accurate, and concise rules.

HyperLogic: Enhancing Diversity and Aceuracy in Rule Learning with HyperNets
Yang Yang', Wendi Ren', Shuang Li 'School of Data Science, The Chinese University of Hong Kong (Shenzhen) (yangyang8, wendiren] @link. cuhk edu. cn, lishuang@cuhk. edu. cn
Abstract
Exploring the integration of if-then logic rules within neural network architectures presents intriguing area. This integration seamlessly transforms the rule learning task into neural network training using backpropagation and stochastic gradient descent. From each layer and neuron through the language of logic rules, and a global explanatory rule set be directly extracted. However, ensuring interpretability may impose constraints on the flexibility, depth, and width of neural networks. In this paper, flexible approach leveraging hypernetworks
sparse and shallow neural network, can interpret
we propose HyperL.ogic: erate weights of the main network. HyperLogiie can be combined with existing differentiable rule learning methods to generate diverse rule sets, each capable of capturing heterogeneous patterns in data. This provides a simple yet effective method increase model flexibility and preserve interpretability. We theoretically analyze the benefits of the HyperLogic by examining the approximation error and generalization capabilities under two types of regularization terms: sparsity and diversity regularization. Experiments on real data demonstrate that our method can learn more diverse, accurate, and concise rules. Our code publicly available httpss//githu..com// 6an-//////2/////l..oooogg
gen-
Introduction
Despite the significant impact of deep learning society, its lack of interpretability limits its use in critical areas that demand high transparency. For instance, in high-risk domains such as healthcare, finance, and law, the decision-makin maainn process of models needs be fully open and explainable to users and relevant regulatory authorities to gain necessary trust and legitimacy [1 2 3. Compared to the "black box" models, which may perform well but uninterpretable, people prefer intrinsically interpretable model for decision support 5), such as set of concise IF-THEN rules.
Traditional rule learning methods, which are based statistical or heuristic approaches, have been extensively explored but often struggle to simultaneously achieve two key objectives: 1) simplicity and accuracy in rules, and 2) noise tolerance and scalability in data handling 66 8 10]
These methods typically rely on search-based techniques, which can be limited when dealing with complex and noisy data. In contrast, deep learning, which focuses representation learning through embedding, is robust to noise and effectively manages large datasets. This has led to the exploration of differentiable rule learning [II]. which leverages the high performance of deep learning while ensuring interpretability through explicit rule formulation
Differentiable rule learning primarily includes two types of methods. The first approach outputs rules directly through the network, leveraging powerful and diverse neural models to enhance handling of complex data patterns [12 [4]. The second approach is to extract rules from network weights [15] 16 I7,, which promotes rapid training and efficient data management while enabling the use of predefined structures to integrate expert priors [8]
Coorresponiigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hypernetwork (
LoSSmypories
X1
Emp(e)
Weight generation
Loss
X2
X3
Rule-learning network F(x, 8)
Lass
X
Discovered Rule Set: If x; A X2 Then
Gradient
If x2 A X3 Theny
Figure 1: The framework of HyperLogii: Hypernetwork generates for the main network, which is rule-learning network. An example main rule-learning network is shown on the right, from which rules can be extracted based on the learned network weights.
We focus the second approach, aiming to extract rules like "IF aADA - THEN is True" from network weights, where a, b, c are predefined predicates. Although integrating if-then logic rules within neural network architectures offers clear advantages, their performance is often hindered by the restrictive network structures required for interpretability. These structures tend to be overly simplistic and not scalable, limiting their ability capture complex data patterns and making them prone to only capturing partial or local patterns in the data. This raises a critical question: How can modify these models to better harness the full potential of neural networks without compromising their interpretability?
In this work, we introduce HyperLogic, novel framework that integrates hypernetworks with a main rule-learning network (as illustrated Fig. [ where an example of a main rule-learning network is shown on the right). Hypernetworks are type of neural network that generate weights for a main rule-learning network. In our framework, the hypernetwork takes random samples from a high-dimeen nigh-dimenssioal Gaussian distribution input and outputs weights for different parts of the main network. These weights can be utilized in two ways: either directly as the weights for the main network throughout the training process (meaning the main network itself has parameters), by retaining trainable weights in the main network and combining them with the hypernetwork-gee generrated weights through weighted fusion. In both scenarios, this approach yields an interpretable set of rules derived from the comprehensive weights.
Hoperl.ogic can be seen as mixture of expert model with an infinite number of experts, providing simple yet effective way enhance model flexibility and adaptability. We analyze the benefits of HyperLogic by examining its approximation error and generalization capabilities under two types of regularization: sparsity and diversity regularization. Our findings demonstrate that HyperLogic acts universal approximation estimator in the Barron space [19 and proves its generalization ability across various regularization methods. These theoretical benefits are further supported by our empirical experiments.
Additionally, hypernetworks enable the generation of multiple candidate rule sets in single training session without significantly increasing computational overhead. This is in stark contrast to traditional methods, which typically learn only one set of rules at time. Consequently. HyperLogic greatly enhances the efficiency and flexibility of the rule- process.
We summarize contributions follows:
We introduce HyperLogic, pioneering framework that integrates hypernetworks into differen- tiable rule learning, significantly enhancing the rule-learning landscape.
2. We theoretically justify the performance of HyperLogic and provide insights into its effectiveness. Specifically, we examine the approximation error and generalization capabilitics under two types of regularization: sparsity and diversity.
3. We validate HyperLogic through extensive experiments multiple datasets, demonstrating that enables the learning of multiple diverse rule sets and yields more concise and accurate rules.
2 Related Work
Learning Rules from Network Weights Many methods for extracting rules from model weights not based neural networks; instead, they primarily focus interpreting the weights of smaller,

---

simpler models to discover rules (20 Recently, extensive methods have emerged that attempt to extract rules using neural network models. These approaches often involve data preprocessing techniques to prepare the data before training [2]] or utilize post-hoc methods for rule extraction
after model training 22 23 24 which can often lead to a loss of accuracy.
A more promising alternative allows for the direct interpretation of neural network weights as precise rules by integrating rule extraction directly during training. This leads a more seamless and effective learning process. For example, methods such as [25 26 27 utilize low-dimensional embeddings to represent atomic conditions and rules, treating rule learning as a differentiable discrete combinatorial optimization problem encoded by a feedforward neural network. Specifically. FinRule [27] explores higher-order interactions between atomic conditions, which aligns somewhat with our approach of employing hypernetworks. However, it lacks detailed analysis of the specific roles of these higher- order interactions and cannot learn a rich set of candidate rules as our method does. Additionally, these approaches often impose constraints on rule templates, such as fixing the rule length and the number of rules.
Another common class of methods employs modified simple feedforward neural networks, such as DR-net 177 and others 28 29 30). These methods simulate logical operations by altering activation functions and implementing mechanisms that ensure differentiability and support backpropagation. While they overcome the limitations of rigid rule templates, their model structures tend to be relatively simple. Furthermore, the training process often involves freezing the weights of one component while adjusting another, which restricts the model's s ability to fully optimize performance.
Our HyperLogic is a unified framework that can be integrated with various neural network weight generators, enabling easy adaptation to different main rule learning networks. Currently, our main network draws inspiration from DR-net [7].. but it can be replaced with other architectures based on task requirements. Unlike recent Bayesian approaches, which often require multiple hyperparameters to model uncertainty in network weights. HyperLogii simplifies the process by needing fewer hyperparameters while maintaining scalability. This allows for efficient training and adaptability across various tasks without the added complexity of managing uncertainty. Additionally, our approach can generate multiple sets of rule sets in single training session, capability not available in other methods. For further comparisons, including classic rule-based algorithms, please refer to Appendix A
Hypernetwork Hypernetworks, or hypernets, are neural networks that generate weights for main networks [3I]. Benefiting from over-vorannnn pree-aaramttrizttiionn and strategic design, these networks enhance training flexibility and adaptability, improve information sharing, and accelerate training processes. While widely used in many areas like continual learning [32 (33, transfer learning (34). uncertainty
quantification [35] 36), natural language processing [37] and computer vision (38), their potential in rule learning remains largely untapped. Our Hyperl framework can bridge this gap. Specifically, our hypernetook takes inspirations from HyperGAN [36 and similarly we add a loss term to encourage the diversity of the generated network weights.
Unlike typical applications in other domains that focus on enhancing parameter efficiency and training speed, our approach addresses the unique challenges of adding model flexibility while preserving interpretability. Our main goal is to expand the parameter space to produce more diverse, concise and accurate rule sets.
3 HyperLogic
HyperLogic is versatile framework designed to enhance various existing neural rule-learning meth- ods. The concept of hypernetworks can be applied to different types of main networks, provided they focus on learning set of interpretable weights in differentiable manner. Importantly, HyperLogic imposes no additional restrictions on data formats or rule languages; simply builds upon the capabilities of the chosen main network.
3.1
Main Network: Rule-Learning Network
Our main network is currently based on DR-Net [17], which features a simple architecture and is designed for binary classification tasks. We chose this architecture for its simplicity, which aids in our proofs and explanations. However, we recognize its limitations, we can also explore other more flexible neural approaches as the primary network to overcome restrictions related to data formats and rule languages. This is further discussed in Appendix B
3

---

The main network is two-layer neural network as shown in Fig. 1 (right part). Due to its interpretabil- ity, one can directly extract the if-then rules from the trained neural networks. The input layer the main network is fed with D-dimensional binary data z = [ra]], each element z. E {1, -1. Here dd indicates the grounded predicate, where indicates True and -1 means false. Note that common input types may include binary, categorical, numerical features, all of which are discretized and binarized to form our binary input features. The hidden layer has K neurons, where K determines the total number of rules and serves hyperparameter. The first layer is referred as the Rule Layer, and the output layer as the OR layer.
Rule Layer: Each neuron in the hidden layer is denoted as E {0. The output represents whether rule satisfied. Each neuron is calculated as:
D
=0 for I,...,K K
OK = 1
W Wd.kZd
Σ
|wax 1waxd=10
(I)
where 1 [(-} denotes the indicator function. Note that the indicator function is one and only all inputs match the sign of the corresponding weights: all positive weights should have the inputs of I, and all negative weights should have the inputs of -1, and zero weights mean that the corresponding inputs are excluded from the rule.
Note that the indicator function non-differentiable, which will make the gradient hard to compute. Here, we will instead use a differentiable smooth function to approximate the indicator function (this
will also ease the theoretical analyses). For example, we can use fuu) = exp
> 0, to
approximate {u = 0}, where is the tunable temperature and controls the approximation error. OR Layer: The second layer operation is defined as
K
f(x) = Σ 4k
(2)
where we assume that the rules contribute to the final prediction in weighted additive form reflect the OR composition, where up denotes the weight assigned to the k-th rule.
3.2 Hypernetwork: Generate Network Weights
Define the network weights for the previous model as 0 = (w. w), where w E RDxK and E R.K Instead of learning only one set of 0, we will learn the distribution of 0 denoted as μ. Specifically, we introduce generative model the hypernetwork to produce samples of 0, i.c.,
0 G(c), where p(E), θ E (D+1)K
Here, e is drawn from some simple base distribution such as Gaussian distribution. More details about the hypernetwok structure can be in Appendix C Note that, in this way, model parameters have been changed to μu, which is defined as the distribution of f and the proposed Hyperlogic model is
D wd.kTdd
]]
fx(2) E,
Σ uch
Σ (aake
In practice, the Monte Carlo method to estimate the above expectation by randomly drawing M samples from Jl, denoted as ,..., MMM and we define
M
K
D
D Σ
]]
fM(x)
Σ
h
Σ
W d.k
Fd
(5)
M
m=1
This model can be regarded as a mixture of expert models with finite A experts. Each expert is shallow two-layer neural network that encodes at most K if-then rules.
3.3
Expand High-Dimensiona. Data
When working with high-dimensional data (e.g.. 5000 dimensions), the number of parameters in the main network increases proportionally with the input dimensions, even if the network structure remains unchanged. This makes training a hypernetwork to generate weights for each module of the main network more challenging. To tackle this issue, we considered two strategies:

---

Combining Original Weights with Snpernetwork-Genrrrat Weights Instead of allowing the generated weights Whyper) fully dictate the main network's 's weights, we combine them with the original weights (W main). The final weights are calculated as follows:
gepeeeetwrr-eenerateed
W Waaiin-fia linal = a Whyper + (1 a) W maiin
where is learnable parameter constrained between 0 and 1. This strategy enhances stability without sacrificing the hypernetwork's 's ability to produce diverse weights. If the hypernetwork is poorly trained and generates inappropriate weights, the model adjusts a toward O, effectively reverting to the standard main network without hypemnetorr influence. We employed this strategy in our subsequent experiments.
Generating Weights for Only Some Modules of the Main Network We test this strategy when the dataset dimension and size are very large in the supplementary experiment. It shows that generating only part of the main network's weights using the hypernetwork still significantly improves results. For more details, please see the supplementary experiments in Appendix
3.4 Loss Function
One can learn μ by minimizing the loss function E(v, (x)),, where [4(x) can be approximated by the finite-sample estimator in Eq. (5). Our loss contains two parts. One is the task-related loss function, denoted as ltack (y. fu (z)). Suppose the problem is a binary classification problem, one can use the binary cross-entropy loss (BCE), where we first map fa (x) to a probability value between and using link function such sigmoid and the loss is defined the negative log-likelihood of sample belong to a class. Here, the task-related loss measures the prediction accuracy.
The second part of the loss is the regularization loss, denoted as Ereg. which is introduced to encourage the diversity of the generated and model sparsity (rule simplicity for each expert), i.e.,
fegg X DKL (m) 1/
(6)
For the first regularization term, we introduce a prior distribution μo with a high entropy, and we minimize the relative entropy KL divergence of the μ and Po. Minimizing the relative entropy encourages the diversity of the generated 6. For the second regularization term, we are considering the & sparsity norm for the second OR layer's weights, where we hope that for each expert, the discovered number of rules is compact as possible. In our paper, freg represents the loss incurred by the hypernetwork, as illustrated in Fig.
4 Theoretical Analysis
We will employ theoretical analysis to demonstrate that while the main network (i.e., a two-layer shallow neural network) has low capacity, incorporating the hypernet idea significantly enhances the model's 's expressive power. Specifically, we will establish that the proposed HyperLogic model serves as universal approximator in the Barron space [19] which will be defined later. For simplicity, but without loss of generality, we consider the case where the hypernetwork fully generates the weights and the main network itself has no parameters. Additionally, we will present the generalization error under the above two types of regularization.
Reparametrization: To ease the proof, let's first rewrite Eq. 5 as
M
K
1
fM(())
Σ M m=
u
T h (w
x)
by reparametrization.
To achieve this, first use the split variable trick by defining w
max (w, 0} and w = min w. 0). and reparametrize w = w w and |w] = w + w In this way, we can get rid of the absolute value and have
Σ W.d Σl Ieak = Σ Swtte w 1) Σ d=1
wāi((-a 1).
(8)
Therefore, we can simply reparametrize Wk. where wa > 0, as concatenation of the vector [w and [w=glu=1....... We construct the new input data x by making a copy of the original data and modifying each copy given Eq. (8. That is, for one copy, we subtract by 1, and for another copy, we flip the sign and subtract by 1. Then, we concatenate the two copies of data.
5

---

Given such reparametrization, we get simple form as Eq. 77) In the following analysis, we will still assume that w/ E RD and z RD, which doesn't affect the generality.
Preparation for the Theoretical Analysis: Our analysis below based on the following observations.
Given the generated random variable ((w1, ......... . (WK, ux)) )) E ((D+)) we have denoted their joint distribution j.. Let us further denote the marginal distribution of (We, Uk) 14g. k = 1, ...... K,
respectively. Define random variable (w, ū) E R which draws a sample from (4 with (equal) probability 1/KK & I...., K, and then scale the u-component by K. Denote the resulting mixture distribution as jt. Then follows from the linearity of expectation that
(x) (((()..)).....xxxx.))))
ugh (wT x)
K
= B((
Σ ūgh (w (x]a))
-B( ühh (w [e)Ta]]
We define [x(x)= = () [ūh (w (wTx)) x)] and we have shown that fx (x) as stated above. The motivation for introducing (1) is to facilitate the analysis of approximation error and generalization error. There exist theoretical results for a two-layer neural network of the form gwr(z) = Em=1 u (w"TTz) z). where h(.) belongs to certain classes of smooth activation functions, and (w- wT)). m = -,.... M, are M independent samples drawn from fixed distribution. In con- trast, proposed HyperLogic model involves randomly drawing samples ((w1, (o01,uu), uI)...., (WK, ux)) from μl, each with K components. To align this with the existing theoretical framework, we perform a transformation as shown in Eq. (9) This involves converting the process of drawing M samples, each with K components, into drawing M samples, each with a single component. This reformulation maintains the same expectation results, allowing to leverage existing theoretical results to analyze HyperLogic. Note that the conversion mentioned is purely for theoretical proof purposes and is not implemented in the actual algorithm.
4.1 Approximation Error of Finite Experts
Using the connection as shown in Eq. (9), we can directly leverage the approximation error results for single hidden layer neural network. Let's first define the Barron space, which provides set of functions for which neural networks can achieve good approximation properties.
Definition 1 (Barron Space [19). A function f RR + R belongs to the Barron space B if it can be represented as:
f(x) = [ J uh (wTx + b) du(u, w. b)
where h is an activation function, j probability measure on R x RD and the following Barron norm is finite:
|| is = inf {
([l(ve,,]llp(u,, w. b) f(x) / uh (w TTTT x + b) du(u, w. b)
Note that the representation of in the form = f The Barron norm seeks the representation that minimizes the integral of [u]](w b)ll The introduction of the Barron norm provides quantitative measure of the "complexity" of a function in the context of neural networks. Functions with a smaller Barron norm are considered simpler and are easier to approximate with neural networks. Next, let's provide the approximation error analysis for our HyperLogic model (the proof can be found in Appendix D):
(wTzz b) du(u, w. may not be unique.
Theorem 1. For any function in the Barron space, there exist M experts (luT , u 1,.... M, forming predictor fas shown in Eq. 77)
..... (wi (R) (D(+1)xXK such that
|1/
-M &
VMK"
where the 2 norm for f.function f BR - R.R. is defined as [|fllr (Jgo If(=))))((z)) where D(x) is the probability distribution over the input data.
The above theorem asserts that Hyperl Loiic model (essentially a mixture of expert models) can approximate any continuous function in the Barron space to arbitrary precision given enough M, K and appropriate weights.

---

4.2
Generalization Error: Entropic Regularization and Sparse Regularization
Let's derive the generalization error bounds for the HyperLogic model under entropic regularization (the first term of Eq. ) and sparse regularization (the second term of Eq. (G)) The key steps involve calculating the Rademacher complexity of the model class and then using this to bound the generalization error. The proof can be found in Appendix E Let's give the results here.
Theorem 2. For the function class FKL == {fm(+) PKL(HllAn BKL we have
Ed(((f) ((f) s 2 2BKl + +CV
1og(1/6)
(10)
2n
Similarly, for the function class Fi = (m(() By 1/2 Σk |ux||
Bi we have
log(1/6)
E 64B1 11/m
+ CV
12
Here, ED[((()) is the expected loss over data distribution, {(f) is the empirical loss, n is the number of samples, C is a constant dependent on the loss function, and is the confidence level.
From the results, we see that the relative entropy regularization effectively balances fitting the training data and adhering to the prior distribution with high entropy. Increasing the sample size or decreasing the KL bound enhances the model's ' generalization ability, ensuring good performance on unseen data. Similar conclusions can be arrived for the sparse regularization.
5 Experiment
In this section, we report experimental results to answer the following questions:
RQI: How does the performance of the optimal rule set selected by Hyperl.ogic compare to the rule sets obtained by other methods?
RQ2: How rich are the rule sets generated by HyperLogic,, and how are their accuracy and diversity affected by parameters?
RQ3: Can we further leverage the advantages of HyperLogic through ensemble learning to enhance performance?
5.1 Experiment Setup
During training, for each data batch, we randomly generate M, samples of network weights to approximate the expectation (as shown Eq. (5). here we compute fM, (x) as an approximation in the training stage). Increasing Mi enhances the stability of hypernetwork training but raises computational costs. In our experiments, M, is set to 5 or 10. After training, we generate M2 sets of weights from the hypernetwork, resulting in My rule sets. In other words, in the inference stage, we use fM, (I) instead. While Mj is relatively small, M2 can be large (c.g., 5000). We then select the rule set with the highest training accuracy as the optimal set, though other criteria, such as minimal loss or custom evaluation metrics balancing accuracy and complexity, can also be used.
Implementation Details:
For HyperLogic, We use Adam as the optimizer, and the learning rate is 1 x 10-4, with weight decay
10 4 The number of training epochs is 10000. In the experiments for selecting the optimal rules for comparison, we set hyperparameter Mj = 5, M2 = 5000. A 0.01, and A2 = 0.1 (A, and A are related to the hypernetwork loss or regularization loss, as defined in Eq. (6). In the following experiments that analyze the influence of hyperparameter, we adjusted only the corresponding hyperparameter while keeping others unchanged.
Datasets: We selected four publicly available binary classification datasets: MAGIC gamma telescope (magic), adult census (adult), FICO HELOC (heloc), and home price prediction (house). In all datasets, preprocessing was performed to encode categorical and numerical attributes as binary variables, which can be found in [17]. We compared our model HyperLogic with other state-of-the-an rule learning methods: DR-Net [17], CG [7]. BRS [20]. and RIPPER (6. Since CG, BRS, and RIPPER cannot learn negative conditions, we additionally appended negative conditions for these models.
5.2 Performance Comparison
In the performance comparison section, we sampled M2 5000 times from Hyperl selecting the rule set that performed best on the training set as the optimal rule set, and compared it with other methods. Table I presents the comparison of the accuracy of the optimal rule sets selected by
7

---

our method and those generated by other methods. Our method further improves upon DR-Net and outperforms other methods across all four datasets.
is important to note that the rule set that performs best on the training set does not strictly guarantee shown in Fig. However, this selection method is sufficiently
the best performance the test simple, and experiments have shown that it can achieve good performance.
Table I:
Test accuracy based nested 5-fold pross-validation (%, mean + standard error). Results
corresponding to methods marked with are directly sourced from 17]
Dataset
Method
magic
adult
house
heloc
Hyperl.ogic
84.90 + 0.73
83.11 + 0.55
85,22 + 0.62
71.03 + 1.07
DR-Net
83.69 0.55
82.95 0.45
85.05 0.51
70.07 0.83
CG
83.68 + 0.87
82.67 + 0.48
83.90 0.18
68.65 + 3.48
BRS"
81.44 + 0.61
79.35 + 1.78
83.04 + 0.11
69.42 + 3.72
RIPPER
82.22 + 0.51
81.67 + 1.05
82.47 1.84
69.67 + 2.09
magic
adult
house
heloc
Test Acouracy
0.590
That Acouracy
Test Accuracy
Test Accuracy
0.83
0.645
9.840
0.82
0.035
0.
2.005 0.80 Bids 0.850 5.855 Train Accuracy
0.015 3.000 l2z RSO 2.B Troin Accuracy
IUBSS 0LD4 865 2.055
0.000
88000
0.700
Train Accuracy
Train Accuracy
Figure 2: Train and test accuracies for sampled rule sets across four datasets
In addition rule accuracy, we also considered two met- rics to measure the compactness of the rules: model com- plexity and rule complexity. Model complexity defined as the sum of the number of rules and the total number of conditions in the rule set; rule complexity is the average number of conditions in each rule of the model. Fig. 3 shows that our method reduces both model complexity and rule complexity compared to DR-Net, indicating that we have not only improved the accuracy of the rules but also made them more concise, demonstrating the effectiveness of our framework.
100
Hepetiee Usrd ceppell lipetig walgeee---accaaaeeee Aad
Medal compleity
60
Dasseess
Figure 3: Model complexity and rule complexity comparison
5.3 Rule Analysis
In this section, we primarily considered the effects of three parameters, M,. X and M2, on the learned rules. Among these, M2 primarily affects the selection of the final optimal rule set by controlling the number of candidate rule sets sampled from the hypernetwork after training phase, while Mj and d directly influence the training of the hypernetwork.
0.85
Modil Complestt
Complesity
Text Acureey
0.80
80
0.75
5.0
2000
4000
5000
8000
10000
2000
44000
5000
5000
10000
2.5
2000
4,000
0000
5000
10000
Figure 4: Analysis of the impact M2 on all datasets
For M2, we considered the values 10, 100, 1000, 2000, 5000, and 10000 to examine its impact on the accuracy, model complexity, and rule complexity of the generated optimal rules, shown in Fig.

---

It can be observed that the accuracy, model complexity, and rule complexity of the optimal rules generally increase with the increase of M2, and then level off or slightly decrease. This is because, when M2 is small, we may not be able to sample sufficiently suitable rule sets to fit the training set. When M2 too large, the sampled optimal rule set more likely to overfit the training set, leading to slight decrease in test accuracy.
So far, we have focused the optimal rule set extracted from the hypernetwork. Table 2 further shows the different optimal results obtained in three training sessions for the heloc dataset, indicating that we can find different high-quality rule sets in different training sessions. However, considering only one optimal rule set is not enough to reflect the advantage of the hypernetwork in generating multiple high-quality rule in one training session, which provides more options and a deeper understanding of the data, something that methods learning only one set of rules cannot offer.
Table 2: Examples of optimal rule sets learned from different training runs on the heloc dataset
Version
Rule
Train
Test
Acc
Acc
AvggFile & 40.0 A -ExtRisk & 69.0 A PetNeverDell S 78.0 A
69.28
71.51
NedFracBBurenn S 77.0
2
C-AvgFile < 40.0 A -ExtRisk < 66.0 A -OldTradeOpen - 87.0 A
70.59
71.41
-PetNeverDDelg 78.0 A NetFracBurden < 60.0
2 D-MaxDelq2Reec122
I 5.0 A HangTrildsHiggUUhi S 2.0 A
ExtRRisk < 74.0
L EExtRisk < 74.0A -PetNeverDelq s N22./ANetrrcBurdenn S 47.0 69.95
69.60
The ability of the hypernetwork to generate diverse rules mainly related to M1 and A1. Å1, as the coefficient of diversity regularization term, directly affects the diversity of the hypernetwork output, while Mj indirectly affects the process by influencing the stability of hypernetwork updates. For the impacts of M1 and Å₁ rule sets generated by the hypernework, and describe the diversity and accuracy of the rule sets as whole.
longer concerned with the optimal rule set but focus on all the
The diversity of the rules measured two aspects: 1. How many different rule sets can we generate by sampling certain number of weights from the hypernetoor?? 2. What is the degree of similarity between the generated different rule sets? We use Jaccard similarity to measure this.
We take the magic dataset as an example to analyze M1 and Y the results for the remaining datasets Appendix F The impact of M1 on rule diversity and test accuracy is shown in Fig. It can be seen that when M1 = 5, there is the best performance in all aspects. This may be because, when Mj is small, the hypernetwork generates weights fewer times, making the training less stable. When M1 is too large, the training and updates of the hypernetwork become too stable and conservative, slowing down the convergence speed and resulting performance decline of the hypernetwork.
0.84
0.25
(4 - 10
Unique Sets
20
Test Accuracy
0.20
Jaccard Score
10a
0.15
10a
0.10
0.76
Jaccard
10
107
10
10
Sample Size
Mi
Figure 5: Analysis of the impact of M₁ on magic dataset

---

The impact of X1 rule diversity and test accuracy is shown in Fig. 6 It can be seen that increasing Å significantly enhances the diversity of the rule sets, but excessive emphasis on rule diversity may affect the accuracy of the rules.
0.01
0.855
Scor
0.26
0.1
Untigge Staee Num
=1
0.850
0.21
0.845
0.22
Score
100
0.840
Tbal
0.835
0.18
0.16
0.825
0.14
0.820
10
10a
10a
107
0.01
Sample Size
Figure 6: Analysis of the impact of A on magic dataset
5.4 Ensemble Learning
Since the hypernetwrkk can generate a diverse and rich set of rule sets, natural idea is to ensemble learning to achieve better classification performance. We use the simple averaging voting method for ensemble learning. We select the top L rule sets based on their accuracy on the training set and report their accuracy
85.0
Thet Accumncy
60.0
magic
adult
77.5
house
75.0
the test set. The values of L are 1, 5, 10, 30, 50, 100, and 200. As shown in Fig. 7 the test accuracy initially increases with the increase of 7., then slightly decreases.
70.0
100
125
1.50
175
200
Top Selected
Figure 7: Test accuracy using top rule sets in ensemble learning.
This may because the top single rule set is already highly effective, leaving little room for improvement. When L is too large, less effective rule sets decrease the ensemble's overall performance. Additionally, our current strategy focuses on balancing diversity and accuracy of single rule sets. Adjusting the strategy specifically for ensemble learning could yield better results.
Conclusion and Limitations
In this paper, we proposed HyperLoggic, novel framework that enhances the field of differentiable rule learning through the integration of hypernetworks. We also provided a theoretical foundation for HyperLogic's performance, explaining its effectivvnesss. This includes an analysis of approximation error and generalization capabilities under sparsity and diversity regularization. What's more, we conducted extensive experiments on multiple datasets, demonstrating that HyperLogie accelerates the training process while producing more concise and accurate rules.
Despite these advancements, HyperLogic introduces additional hyperparameters and requires further exploration ensemble learning within this framework. Future research will focus on alternative training strategies, more stable weight combination methods, and applying HyperLogic broader range of datasets and tasks to enhance its generality and practicality.
7 Broader Impacts
The development and utilization of interpretable logic rules through HyperLogic have significant pos- itive societal impacts, particularly in high-risk domains such healthcare, finance, and legal systems. By generating multiple candidate rule sets, HyperLogic provides more flexible and comprehensive insights, enhancing transparency and trust in decision-makee safety and regulatory compliance. However, it is important to note that while our method provides richer rule sets and aids experts understanding data patterns, these rules should complement rather than replace expert judgment, Relying solely automated rules without expert oversight in critical areas could pose significant risks. Thus, our approach emphasizes the collaborative role machine learning models and human experts to mitigate potential negative impacts.
processes. This crucial for ensuring
10

---

Acknowledgments
Shuang Li's research was in part supported by the National Science and Technology Major Project under grant No. 2022ZD01 16004, the NSFC under grant No. 62206236, Shenzhen Stability Science Program 2023, Shenzhen Key Lab of Cross-Modal Cognitive Computing under grant No. ZDSYS2023062609 1302006,, Longgang District Key Laboratory of Intelligent Digital Economy Security, and SRIBD Innovation Fund SIF20240010.
References
[1] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing. 10(3152676;:0-55555 2017.
[2] Michelle Goddard. The eu general data protection regulation (gdpr): European regulation that has a global impact. International Journal of Market Research, 5966)70037705, 2017.
[3] David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. Xai explaxnlai artificial intelligence. Science robotics, 4(37):eaay7120, 2019.
[4] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 155):206-215, 2019.
[5] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
[6] William W Cohen. Fast effective rule induction. In Machine learning proceedings 1995, pages 1 115-123. Elsevier, 1995.
[7] Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. Boolean decision rules via column generation. Advances in neural information processing systems, 31, 2018
[8] Mark Law, Alessandra Russo, Elisa Bertino, Krysia Broda, and Jorge Lobo. Fastlas: Scalable in- ductive logic programming incorporating domain-specific optimisation criteria. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2877-2885, 2020.
[9] Andrew Cropper and Rolf Morel. Learning programs by learning from failures. Machine Learning, 1 11044)801-885, 2021.
[10] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
[11] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems, 30, 2017.
[12] Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems, 32, 2019.
[13] Yuan Yang and Le Song. Learn to explain efficiently via neural logic inductive learning. arXiv preprint 11Xiv:1910.0248)) 2019.
[14] Kewei Cheng, Nesreen K Ahmed, and Yizhou Sun. Neural compositional rule learning for knowledge graph reasoning. arXiv preprint arXiv:2303.0358/ 2023.
[15] Hikaru Shindo, Masaaki Nishino, and Akihiro Yamamoto. Differentiable inductive logic programming for structured examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5034-5041, 2021.
[16] Claire Glanois, Zhaohui Jiang, Xuening Feng, Paul Weng, Matthie Zimmer, Dong Li, Wulong Liu, and Jianye Hao. Neuro-symbolic hierarchical rule induction. In International Conference on Machine Learning, pages 7583-7615. PMLR, 2022.
11

---

[17] Litao Qiao, Weijia Wang, and Bill Lin. Learning accurate and interpretable decision rule sets from neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pages 4300-4311, 2021.
[18] Ruixuan Yan, Yunshi Wen, Debarun Bhattachariya, Ronny Luss, Tengfei Ma, Achille Fokoue, and Anak Agung Julius. Weighted clock logic point process. In International Conference on Learning Research (ICLR) 2023, 2023.
[19] Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023.
[20] Tong Wang. Cynthia Rudin, Finale Doshi- Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research, 18(70):1-3) 2017.
[21] Wei Zhang, Yongxiang Liu, Zhuo Wang, and Jianyong Wang. Learning to binarize continuous features for neuro-rule networks. In Proceedings of the "heroy-Second International Joint Conference on Artificial Intelligence, pages 4584-4592, 2023.
[22] Nuri Cingillioglu and Alessandra Russo. pix2rule: End-to-end neuro-symbolic rule learning. arXiv preprint arXiv:2 2021.
[23] Zohreh Shams, Botty Dimanov, Sumaiyah Kola, Nikola Simidjievski, Helena Andres Terre, Paul Scherer, Urška Matjašec, Jean Abraham, Pietro Liò, and Mateja Jamnik. Rem: an integrative rule extraction methodology for explainable data analysis in healthcare. medRxiv, pages 2021-01, 2021.
[24] Mateo Espinosa Zarlenga, Zohreh Shams, and Mateja Jamnik. Efficient decompositional rule extraction for deep neural networks. arXiv preprint arXiv:2111. 2021.
[25] Shaoyun Shi, Yuexiang Xie, Zhen Wang. Bolin Ding, Yaliang Li, and Min Zhang. Explainable neural rule learning. In Proceedings of the ACM Web Conference 2022, pages 3031-3041, 2022.
[26] Lu Yu, Meng Li, Xiaoguang Huang, Wei Zhu, Yanming Fang, Jun Zhou, and Longfei Li. Metarule: A meta-path guided ensemble rule set learning for explainable fraud detection. In Proceedings of the 31 ACM International Conference on Information & Knowledge Manage- ment, pages 4650-4654, 2022.
[27] Lu Yu, Meng Li, Ya-Lin Zhang, Longfei Li, and Jun Zhou. Finrule: Feature interactive neural rule learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 3020-3000 3029,00292000 2023.
[28] Zhuo Wang, Wei Zhang, Ning Liu, and Jianyong Wang. Scalable rule-based representation learning for interpretable classification. Advances in Neural Information Processing Systems, 14:94779-34991 2021.
[29] Remy Kusters, Yusik Kim, Marine Collery, Christian de Sainte Marie, and Shubham Gupta. Differentiable rule induction with learned relational features. arXiv preprint arXiv:2201.06515 2022.
[30] Nils Philipp Walter, Jonas Fischer, and Jilles Vreeken. Finding interpretable class-specific patterns through efficient neural search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 9062-9070, 2024.
[31] Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu. Soheila Molaci, and David Clifton. brief review of hypernetworks in deep learning. arXiv preprint arXiv:2300600655, 2023.
[32] Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. Continual learning with hypernetworks. arXiv preprint arXiv: veXivv/9966000 /906,00695, 2019.
[33] Yizhou Huang, Kevin Xie, Homanga Bharadhwaj. and Florian Shkurti. Continual model-based reinforcement learnng with hypernetworks. In 202 IEEE International Conference on Robotics and Automation (ICRA), pages 799-805. IEEE, 1021.
12

---

[34] Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, and Roi Reichart. Example- based hypernetworks for -iitof--iitrbbutionn generalization. arXiv preprint arXiv:2203.1427,, 2022.
[35] David Krueger, Chin-Wer Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypemetworks. arXiv preprint arXiv: UrXiv:1710.04755: 2017.
[36] Neale Ratzlaff and Li Fuxin. Hypergan: A generative model for diverse, performant neural networks. International Conference on Machine Learning, pages 5361-5369. PMLR, 2019.
[37] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efffcieet multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint urXiv:2106.04449 2021.
[38] Lorenz K Muller. Overparametrization of hypemetworks at fixed flop-count enables fast neural image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 284-293, 2021.
[39] Fulton Wang and Cynthia Rudin. Falling rule lists. In Artificial intelligence and statistics, pages 1013-1022. PMLR, 2015.
[40] Chaofan Chen and Cynthia Rudin. An optimization approach to learning falling rule lists. In International conference artificial intelligence and statistics, pages 604-612. PMLR, 2018.
[41] Hugo M Proença and Matthijs van Leeuwen. Interpretable multiclass classification by mdl-based rule lists. Information Sciences, 512:1372-1393, 2020.
[42] Lucile Dierckx, Rosana Veroneze, and Siegfried Nijssen. RI-net: Interpretable rule learning with neural networks. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 90- Springer, 2023.
[43] Hongyu Yang, Cynthia Rudin, and Margo Seltzer. Scalable bayesian rule lists. In International conference on machine learning, pages 3921-3930. PMLR, 2017.
[44] Fernando Jiménez, Gracia Sánchez, and José M Juárez. Multi-objective evolutionary algorithms for fuzzy classification in survival prediction. Artificial intelligence in medicine, 60(3):197-219, 2014.
13

---

A Compare with classical rule learning methods
We first compare Hyperl.ogie with falling rule lists(FRL.) methods like 39 often used in practice. FRL methods explicitly construct a falling list of rules with probabilities; Differentiable Hyperl.ogic focuses directly mining patterns and generating rules from data in scalable manner using neural networks. Our differentiable approach demonstrates better scalability for pattern mining tasks large-scale data (see Section 2). The main differences between FRL and the HyperLogic are:
40 41
which
1. Rule Generation:
FRL, methods rely on other rule mining techniques to generate initial candidate rules. Hyperl uses differentiable neural network approach directly mine patterns and generate rules from data in an end-to-end manner.
2. Scalability:
FRI methods may face scalability issues when dealing with large candidate rule sets or complex data.
HyperLogic, neural network- approach, is more scalable for pattern mining in large-scale data.
3. Rule Ordering:
FRL methods explicitly order the rules into descending list based on rule probabilities or other criteria.
HyperLogic generates an unordered set of rules.
4. Probabilistic Interpretation:
FRL methods provide probabilistic interpretations for the rules by construction. HyperLogic does not directly output rule probabilities, although probabilities could potentially be derived from the learned patterns.
5. Integration Potential: A recent study 42) proposed a neural method to learn ordered rule lists, which could potentially be integrated with HyperLogic to enable joint rule generation and probabilistic ordering, combining the strengths of both approaches.
Other statistical rule mining methods like Bayesian rule lists 45 and Bayesian decision sets [20 provide probabilistic interpretations but limited to binary classification and small rule sets. Fuzzy rule-based models [44] incorporate human-like reasoning but lack probabilistic predictions and scalability. In the supplementary experiments B we include the current advanced traditional rule learning method CLASSY in the comparison to further demonstrate the advantages of our method over traditional methods.
B Results for HyperLogic with DIFFNAPS
To expand our experiments to larger and more complicated cases, considered the latest Neuro- Symbolic algorithm DIFENAPS [30], which is capable of pattern mining under large-scale data conditions. For HyperLogic, selected only the classifier part of DIFFNAPS as the main network to compare with vanilla DIFENAPS.
B.1 Large Synthetic Datasets
Following the original experiments, tested the model's pattern mining performance under fixed input dimension of 5000 and varying total number of categories K (ranging from to 50), measured by the FI score, with each category containing 1000 samples.
Compared with the current data set, in our new data set, the feature dimension has been raised from maximum of 154 dimensions to maximum of 5k dimensions, the amount of data has been raised from a maximum of 24,000 to a maximum of 50,000, and the task has been raised from a maximum of 2 categories to maximum of 50 categories, reflecting the characteristies of the task diversity and complexity.
The experimental results are shown in the table below. It can be seen that in datasets with fewer categories, due to the smaller total number of samples, HyperLogic has not yet received sufficient training and does not perform ideally. However, in more challenging classification datasets with an increased number of samples, the model's performance has significantly improved, with an average F1 score increase of 6 %. This fully demonstrates that our framework can empower diverse neural rule learning networks, capable handling large-scale data and possessing good range of applications.
14

---

Dataset (K=)
DIFFNAPS
HyperLogie
0.788 + 0.080
0.726 0.084
0.703 + 0.030
0.675 = 0.036
10
0.726 + 0.013
0.751 0.020
15
0.622 = 0.017
0.800 = 0.028
0.712 = 0.015
0.760 00.01
25
0.688 = 0.020
0.770 = 0.020
30
0,602 + 0.014
0.766 + 0.021
35
0.557 + 0.014
0.668 + 0.018
40
0.635 + 0.022
0.712 + 0.011
45
0.611 + 0.009
0.694 0.009
50
0,603 + 0.010
0.702 + 0.012
Table The FI score (#: std) of two methods among 11 synthetic datasets
B.2 Large Real Datasets
We evaluated our method on four large biological datasets following the settings of DIFFNAPS: Cardio, Disease, BRCA-N, and BRCA-S, using the area under the curve (AUC) as the metric. We continued to combine our approach (HyperLogic) with DIFFNAPS as the main network and compared it to vanilla DIFFNAPS. Additionally, FRL [39] cannot scale to non-trivial data, while CLASSY [4]] was already compared in the original paper.
The table shows the dataset details (i.e. samples (n). features (D), and classes (K)). number of discovered patterns (#P), average pattern length (IPI), and AUC scores (results for DIFFNAPS and CLASSY are taken directly from (3)..
HyperLogic
DIFFNAPS
CLASSY
Dataset
n
#P
PP
AUC
#P
PP|
AUC
#P
PP|
AUC
Cardio
68k
45
15
2
0.57
14
2
0.56
10
2
0.36
Disease
5k
131
41
866
0.86
838
2
0.84
25
2
0.11
BRCA-N
222
20k
187
6
0.95
146
9
0.91
3
1
0.45
BRCA-S
187
20k
1k
2
0.89
1k
2
0.86
2
1
0.23
Table 4: Comparison of HyperLogic. DIFENAPS, and CLASSY acToss 4 real datasets.
CLASSY lacks the finesse effectively mine patterns for large-scale real-world tasks. Moreover, despite competing with the strong DIFFNAPS baseline, HyperLogic achieved further improvements, demonstrating its potential for handling large, real-world datasets.
C Hypernetwork Details
We adopted HyperGAN our hypernetwork. Assuming the main network comprises N distinet weight partitions, HyperGAN includes a mixer Q and N generators G1, G2,..., GN. (O I), and
Mixer Q: Receives high-dimensional Gaussian distributed random samples h dimensional vector z, further split into N samples of h-dimensional vectors The mixer's design reflects the necessity for correlations between layer weights, as each layer's s output becomes the subsequent layer's 's input.
transforms into
21, ..,....;2 #N.
Generators G Each generator receives a vector 2, from the mixer, producing an output vector of dimension m,, where m, represents the parameter count for the i-th part of the network weights. These vectors are then reshaped to meet the specifications of the corresponding layers in the main network.
In our specific task, the number of generators is set to 2, and each generator's input dimension h is 64. The input dimension of the Mixer, which is the sampled noise, is 256, and it produces output of x h = 128. Each the above models has two hidden layers with dimension 512, and the activation function is ReLu.
Our main network is designed to generate
50 rules. For dataset with dimension D:: Generator 1: Produces an output of dimension (D x K, representing the Rule layer.
Generator 2: Produces output of dimension (K x 1, 1),, representing the OR layer.
15

---

These outputs are then reshaped to meet the specifications of the corresponding layers in the main network.
D Proof for Theorem
Proof. Using Barron's theorem (c.g.. [19 Theorem 11.3)), there exist weight coefficients (w ugg)
m 1,... M, I,... K, such that the function defined by
1
f(x)
uh h ((w
MK
k=1
satisfies
||
Il fllez 1
v /MK
Setting us = 0ig K, um (wff.... wx), u ...,...,,,7 yields the desired result.
E
Proof for Theorem
E.1
Generalization Error: Diverse Regularization
Let us compute the Rademacher complexity of the model class
FKL = fa()) DKL(Hllp))) & BKL
where Po is a product distribution n Using the property of relative entropy, for each pair of marginal distributions μı and fio. & = -,.... K, their relative entropy satisfies DKL (Mmell)) I BKL- Using the convexity of the relative entropy, the mixture distribution jı satisfies DRL (p S BKL- Hence, the function class FKL. belongs the function class
FKL = [a() DKL (xl())) S BKL
Corollary 10.17], the Rademacher complexity FKL is bounded by v 22BkL/n. Given our bound on the Rademacher complexity, for f FKLL we get:
Using 19
Ex[e())] { s 2/ 2BKL. + CC. /1/O001/10
1ox(1/0)
(11)
2n
where Ep ED((()) is the expected loss, {(ʃ) the empirical loss, ri is the number of samples, C is constant dependent the loss function, and 8 is the confidence level.
F.2 Generalization Error: Sparse Regularization
Next, we will derive the generalization error bounds for the SyperLLogia model under sparse regular- ization (as shown in the second term of Eq. (6) )).
Let us compute the Rademacher complexity of the model class
K
F = fu(-) E,
Jull
BiJ
Observe that the constraint Ep,
Ex
[ux|
S B1 implies that
K
K
Ea
[
uk
$ B1.
K
A=
Hence, the function class F belongs to the function class
f
[a()
[[a]] S B1
Using 119 Proposition 11.23], the Rademacher complexity of Fi is bounded by 32B / (D+1))+ 1)/n. The generalization error bound for the HyperLogic model under sparse regularization, derived from the Rademacher complexity, can be expressed as follows:
log(1/8)
Ep [e(e))) - {(S) S 64B1 (D+ 1)/n + CV
The explaination of this error bound is similar with the previous one.
16

---

Supplementary Result
100
6.18
-0
6.32
Rooand Scoms
209
0.42
6.52
6.815-
0.530
0.865
6.38
18
10
12a
Figure 8: Analysis of the impact of M1 adult dataset
0.340
160
Thas Nam
0.035
10
0.022
6.14
0.520
30
10
0.01
0.12
3km
Figure Analysis of the impact of \ on adult dataset
0.222
0.65
3.300
20.84
0.573
8.52
Scrved Bron
0:58
Sunga Skm
A.
Figure 10: Analysis of the impact of Mi on house dataset
6.28
0.71
6.11
0.34
Ohape tade
0.23
0.57
10
10
12
Figure 12: Analysis of the impact of Mj on heloc dataset
JeanttSos
6.18
Wo SE
Shapee
0.70
11P
10
12
Suxple Sin
Figure 13: Analysis of the impact of X heloc dataset
17

---

Jacar
6.465
8.38
tak
0:500
0.33
100
107
12
Figure 11: Analysis of the impact of Å on house dataset
G
Computing Infrastructure
For our method, all experiments were conducted on a Linux server with an Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz and 30Gi of memory, running Ubuntu 20.04.5 LTS, using of the NVIDIA GeForce RTX 3090 GPUs available on the server. Each experimental run took approximately 10-20 minutes complete. This setup ensures that experiments are reproducible and efficient.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate our innovations and contributions in the abstract and introduc- tion. We also mention the important assumptions and limitations of our methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of proposed model in the conclusion part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide theoretical proof's about our proposed model in the theoretical analysis section.
Guidelines:
19

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide the details of hyper parameters and neural network structures for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be made publicly available upon the paper's acceptance.
Guidelines:
20

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides ottp://ooooeeiioiiiooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
htt.uubmiisssoooooooooooooo for more details. / //nips /twpps.c.ppu.a.../p// wwtppcccppcccccec.occccccccccttt hwwlllccguiiee gwwgguuisss///e htt////ooouumssseeeeeeee
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details including hyperparameters, type of optimizer, learning rates, hidden layers of networks in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our rule performance results mean accuracy with the standard error across nested 5-fold cross-validation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
21

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We demonstrate the computer resource requirements in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research presented in our paper fully complies with the NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss potential positiveand negative societal impacts of the work in the appendix.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
22

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper that produced the dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paw.ess..ppppipiipe.... thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
23

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "jNaMgs2fAN", "reviewer": "Reviewer_Q4Et", "summary": "This paper introduces HyperLogic, a novel differentiable framework for rule learning using neural networks. Instead of directly training network weights, HyperLogic extracts rules by generating the weights for a primary network through hypernetworks. These hypernetworks create diverse sets of weights, functioning like a mixture of experts, thereby enhancing model flexibility. To train the hypernetworks, the authors minimize the relative entropy between the learned distribution and a prior distribution, incorporating regularization to encourage simplicity. They employ smooth approximations to ensure the problem be differentiable. The authors demonstrate HyperLogic’s effectiveness through both theoretical analysis and empirical results on multiple datasets.", "questions": "see weaknesses", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an interesting rule-learning approach using hypernetworks, leveraging a set of experts to take advantage of complex and diverse models while maintaining interpretability.\n- The authors provide a robust theoretical foundation for HyperLogic, showing the effectiveness of their approach through rigorous analysis.\n- Extensive experiments on various datasets demonstrate that HyperLogic can learn multiple diverse and accurate rule sets, highlighting its performance.\n- This paper is well-written and very easy to follow.", "weaknesses": "- The results seem to be sensitive to the choice of hyperparameters, which could affect the stability and generalizability of the method. Are there any guidelines or automated processes to aid in choosing the optimal hyperparameters?\n- The scalability of the approach for extremely large datasets or high-dimensional data remains unclear. Further evaluation is needed to understand its performance in such scenarios.\n- Typo: in line 125,  all negative weights should have the inputs of 0 ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**\n\nMany thanks to reviewer Q4Et for your positive comments and recognition of our contribution including an interesting framework, a robust theoretical foundation, comprehensive experiments, and a well-written paper. We would like to address your concerns one by one.\n\n**Q1. Are there any guidelines or automated processes to aid in choosing the optimal hyperparameters?**\n\nThe additional hyperparameters brought by our framework mainly include the hypernet architecture, the coefficient lamba1 of the diversity loss caused by the hypernet in the loss function, and the number of weight sampling times $k_1$ and $k_2$ during and after training.\n1) Regarding the hypernetwork architecture: \n\nWe followed HyperGAN, which is a simple and effective hypernetwork architecture, and has been proven in experiments that more layers will not bring significant gains; explore other more powerful hypernetworks The architecture will be interesting as our future work. \n\n2) Regarding the number of sampling times $k_2$ after training: \n\n$k_2$ only affects the results of our final extracted rules and does not directly affect the training of the model. Due to the simplicity constraint, the rule set we sampled is not easy to overfit. We can expand $k_2$ as much as possible to obtain a richer rule candidate set if resources and time allow. \n\n3) Regarding the coefficient of diversity loss $\\lambda_1$ and the number of sampling times $k_1$ in training: \n\nThe two have a more significant impact on the effect. In practice, we can set a smaller initial value and gradually increase it until classification error staturates.\nFurther, inspired by your question, we considered extending our model to alleviate the challenge of hyperparameter selection. We no longer let the weight $W_{hyper}$ generated by the super network completely act on the weight of the main network, but retain the weight $W_{main}$ of the main network, and combine it with the weight generated by the super network to obtain $W_{main-final} = \\alpha* W_{hyper} + (1-\\alpha)*W_{main}$, where $\\alpha$ is a learnable parameter restricted to 0-1. This can increase the stability of our results without losing the advantage that the hypernetwork can generate diverse weights: in extreme cases, we set very inappropriate hyperparameters about the hypernetwork and generate inappropriate $W_{hyper}$, and the algorithm will Prompt alpha to approach 0, thereby offsetting the impact of inappropriate hyperparameter selection and allowing the model to degenerate into a normal version without a hypernet. The effectiveness of such improvements is demonstrated in supplementary experiments, especially for high-dimensional data. Updated algorithms and results will be added to the final version.\n\n**Q2. The scalability of the approach for extremely large datasets or high-dimensional data remains unclear.**\n\nOur framework aims to enhance various existing neural rule-learning methods, since hypernet ideas can be applied to different types of primary networks, as long as they are related to learning a set of interpretable weights in a differentiable manner. Therefore, HyperLogic is a general framework. However, it should be noted that the framework essentially stems from the main network, that is, the framework's task scope, model capabilities, etc. are still closely related to the main network, and scalability is often limited by the size of the primary network. However, we can easily switch to a more flexible primary network according to the needs of the task. This is just a simple adaptation, while the main ideas stay the same.\n\nCurrently, for the sake of simplicity of description, our HyperLogic main network gets inspiration from DR-Net, which although has a simple architecture, only supports binary classification tasks and is very time-consuming, preventing HyperLogic from being proven on larger-scale data. To prove that our framework can handle larger data sets, we replaced HyperLogic's main network with the latest DIFFNAPS [1], which is a neural approach that can perform multi-classification tasks and pattern discovery in high-dimensional data.\n\nUnder this new configuration, we conducted experiments on **a new dataset with feature dimensions up to 5,000, data size up to 50,000, and task types up to 50 categories**. In this case, HyperLogic, which uses DIFFNAPS as the main network, further demonstrated **an average 6.1% F1 score improvement** compared to the already good vanilla DIFFNAPS, while **the average time consumption only increased by 8.6%** (please refer to General Q1 in global response for task details), effectively proving the practical application potential of the HyperLogic framework in processing large-scale data sets and high-dimensional data.\n\n[1] Walter, N. P., Fischer, J., & Vreeken, J. (2024, March). Finding interpretable class-specific patterns through efficient neural search. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 8, pp. 9062-9070).\n\n**Q3. Typo: in line 125, do all negative weights have the inputs of 0?**\n\nThank you for your feedback. We have reviewed line 125 and acknowledge that all negative weights should have the inputs of -1 . This error will be corrected in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UrHblR6suC", "reviewer": "Reviewer_F7a8", "summary": "The authors consider the problem of learning interpretable rules for decision-making and propose a new neural rule learning framework to find these rules efficiently. In particular, they suggest to use a hypernetwork to learn a diverse set of network parameters for the rule-encoding network. On small binary data, they show that their algorithm finds reasonable rule sets that predict a given class well.\n\n[Edit after rebuttal]: The authors provided additional discussion and experiments addressing my concerns, which leaves me with a *weak accept*.", "questions": "- In section 4, you convert the process of drawing M samples with K components each into drawing MK single-component samples. Does the latter actually match the former? It seems that the K components in the former are actually dependent, which would mean it does not correctly model it. Please clarify.\n- In Table 1, the bold numbers are confusing. For adult, house, and heloc data, DR-Net, which is closely related and forms the basis of the architecture design, performs on par with the presented method. The results are within the given standard error.", "limitations": "While there is a brief discussion on limitations, neither the limitations in terms of data or rule language/architecture, nor the connection to existing work is properly discussed.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The presented approach seems novel and the considered problem is relevant.\n- The presentation itself is clear and understandable.", "weaknesses": "- Despite the broad motivation of making neural decisions interpretable, the presented approach is very limited in what networks are used (2-layer specific architecture), which data is used (only binary datasets with labels), and which rules can actually be expressed (only conjunctions in antecedent, single label in consequent).\nEspecially the type of considered data needs to be communicated clearly from the start.\n- There is a significant amount of related work, including recent neural approaches, that is not considered here. Some more classical approaches include statistical pattern mining [1] and falling rule lists [2,3,4]. Recently, a neural approach that finds differential patterns between labels -- which are essentially if-then rules has -- been proposed [5]. The experiments should be extended to properly compare to existing work.\n- The experiments are very limited in terms of considered datasets: they are old “toy” datasets of small scale in both sample and feature dimension. Recent works, including [4], incorporate large-scale real-world data to actually show that such neural approaches are more scalable and can hence consider other data than classical approaches.\n\nIn summary, it is unclear to me whether the presented approach provides any improvement over existing work, or is scalable to large data at all.\n\n[1] https://dl.acm.org/doi/10.1145/2783258.2783363\n[2] https://proceedings.mlr.press/v38/wang15a.html\n[3] http://proceedings.mlr.press/v84/chen18a/chen18a.pdf\n[4] https://www.sciencedirect.com/science/article/pii/S0020025519310138\n[5] https://ojs.aaai.org/index.php/AAAI/article/view/28756", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary:** We first thank reviewer F7a8 for the insightful comments, especially for the questions about our model design and experiment details, which helped us to clarify our paper. We would like to address the concerns one by one.\n\n**Q1. The method is limited in network structure (2-layer architecture), data format(only binary datasets with labels), and rule language(only conjunctions in antecedent, single label in consequent).**\n\nOur method is a general framework that can enhance many existing neural rule-learning methods, since the idea of ​​hypernet can be applied to different types of primary network, as long as they are related to learning a set of interpretable weights in a differentiable way (for some minor restrictions on the primary network architecture caused by the introduction of hypernet, please see Q6). The HyperLogic framework has no additional restrictions on data format and rule language, and only inherits from the selected primary network.\n\nAs you mentioned, our implementation is currently limited in network structure, data format and rule language. But this is due to the limitations of DRNet, the primary network we refer to, not the limitations of our framework. We choose DRNet because the simplicity of the architecture is conducive to our proof and expression. However, considering the limitations of DR-Net, we can also consider other more flexible neural approach as the primary network to alleviate the restrictions on data format, rule language, etc. (please see Q2).\n\n**Q2. More related work, including recent neural approaches, should be considered.**\n\nOur proposed method introduces hypernetworks to generate weights for the main network, offering a novel framework. The main network is now a two-layer neural network specifically designed to retrieve rules directly from learned weights. However, other pre-existing neural methods can also be substituted, as long as they are related to end-to-end learning of a set of interpretable weights. Unlike traditional methods and recent Bayesian approaches, our method requires fewer hyperparameters and offers greater scalability. Differing from other neural methods, HyperLogic serves as a more versatile framework that can organically utilize these methods as the main network to enhance performance. Furthermore, our approach is capable of generating multiple sets of rule sets in a single training session, which is not feasible with other methods.\n \nTheoretically, the neural methods described in the previous section can serve as the main network for our framework, just like DRNet. Since DIFFNAPS is a novel algorithm capable of differentiable pattern discovery on large-scale data, **we have chosen to use DIFFNAPS as the main network in our supplemental experiments**. This expansion of our experiments further demonstrates the efficacy of our framework (refer to Q3 for details).\n\n**Q3. The experiments should be extended to properly compare to existing work.**\n\nPlease refer to General Q1 in the global response.\n\n**Q4. In section 4, you convert the process of drawing M samples with K components each into drawing MK single-component samples. Does the latter actually match the former?**\n\n1. The conversion mentioned is purely for theoretical proof purposes and is not implemented in the actual algorithm.\n2. The preparation for the theoretical analysis (lines 175-191) remains valid regardless of the (in)dependence among the K components. Equation (9) is derived from the sample average of a distribution with equally weighted marginals.\n3. To prove Theorem 1, we need to identify a set of parameters for equation (7) that satisfies the inequality stated in the theorem. One approach to achieve this is by using sampling weights transformed from Barron's theorem. It's important to note that this is just one possible set of parameters, and the actual learning algorithm does not utilize such sampling.\n\n**Q5. In Table 1, the bold numbers are confusing, as DR-Net performs similarly to the presented method on partial datasets within the standard error.**\n\nOur framework not only focuses on the accuracy of the optimal rule set, but also on the simplicity of the rule set and the richness of the learned candidate rule set.  Now we simply selected the best rule set on the training set for testing, and have surpassed the results of DRNet in a more concise rule set (Figure 3), demonstrating the effectiveness of our framework. At the same time, the optimal rule set on the training set is not completely equal to the optimal rule set on the test set (Figure 2), indicating that our training strategy (such as hyperparameter selection) and the final rule set selection strategy (such as further considering the complexity of the rule set) can further obtain higher test accuracy. We leave this for future work.\n\n**Q6. The limitations of network structure.**\n\nHyperLogic has almost no special restrictions on the architecture of differentiable rule learning networks, except for two possible problems, but both can be solved:\n1. If the main network has too many parameters, the hypernetwork might struggle to produce stable weights. To solve this, we tried two approaches: ① Combine the original weights with those generated by the hypernetwork to improve stability while keeping the hypernetwork's flexibility (please refer to Q1 of Reviewer Q4Et for details). ② Only generate some of the main network's weights using the hypernetwork. Our tests show this still significantly improves results, even with large datasets (please see Q3 for details).\n2. Some neural methods use a non-differentiable technique called weight pruning to simplify the model. To address this, we've chosen to keep all weights and apply a sigmoid function to them after processing. This makes the entire process differentiable. While this change might affect how the data is distributed and require adjustments to the model's settings, our tests show it doesn't harm performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tM6AU5ewEv", "reviewer": "Reviewer_hMMc", "summary": "The paper presents HyperLogic, a pioneering framework that integrates hypernetworks into differentiable rule learning, enhancing the interpretability and flexibility of neural networks. It provides a strong theoretical foundation and backs it with extensive empirical evidence, demonstrating its effectiveness over traditional rule learning methods. HyperLogic's ability to generate diverse and accurate rule sets is particularly noteworthy, addressing a significant gap in the field of AI interpretability.", "questions": "How does HyperLogic perform in domains outside of the tested datasets, particularly in non-binary classification tasks?", "limitations": "Applications to more general scenarios.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces HyperLogic, a novel framework that integrates hypernetworks with rule-learning networks, which is a significant contribution to the field of interpretable machine learning.\n2. The authors provide a solid theoretical analysis, including approximation error bounds and generalization capabilities, which substantiates the effectiveness of HyperLogic.\n3. Comprehensive experiments on multiple datasets demonstrate the superiority of HyperLogic in learning diverse, accurate, and concise rules compared to existing methods.\n4. The paper addresses a critical need for interpretability in high-stakes domains, offering a transparent decision-making process through the generation of logic rules.", "weaknesses": "1. The introduction of additional hyperparameters may lead to increased sensitivity, which could impact the model's performance and generalizability.\n2. The framework's complexity might pose challenges for practitioners not well-versed in advanced neural network architectures.  While the paper mentions the efficiency of the rule-learning process, the computational overhead introduced by the hypernetworks is not thoroughly discussed.\n3. The framework's applicability to a broader range of datasets and tasks needs to be investigated to ensure its generality and practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Summary**\n\nWe are grateful that reviewer hMMc has a positive impression of the high-level design of our framework and the promising results of comprehensive experiments. To address your questions about the details of our method, we provide point-wise responses as follows.\n\n\n**Q1. The introduction of additional hyperparameters may increase sensitivity, which could impact the model's performance and generalizability.**\n\nThe additional hyperparameters brought by our framework mainly include the hypernet architecture, the coefficient lamba1 of the diversity loss caused by the hypernet in the loss function, and the number of weight sampling times $k_1$ and $k_2$ during and after training. Among them, adding more layers in the architecture adjustment and the number of sampling times $k_1$ after training have little impact on the training, while the coefficient $\\lambda_1$ and the number of sampling times $k_1$ during training have a more significant impact on the training results, which requires us to have a good choice of hyperparameters. Further strategies. Furthermore, we additionally consider an adaptive weight fusion mechanism to optimize model performance through weighted fusion between the weights generated by the hypernet and the original main network weights. This mechanism is controlled by a learnable parameter α, which can be automatically adjusted during the training process to deal with the instability caused by improper selection of hyperparameters. (For more details, please refer to Q1 of Reviewer Q4Et.)\n\n**Q2. The framework's applicability to a broader range of datasets and tasks needs to be investigated.**\n\nWe extended the experiment and proved that our HyperLogic framework can choose other differentiable rule learning algorithms besides DRNet as the main network combination to solve more tasks, such as processing high-dimensional large-scale data with multiple tasks. Classification tasks reflect the broad application scenarios of our framework.\n\nSpecifically, the core feature of our framework, HyperLogic, is as an inclusive framework to allow enhancements to all neural methods as long as they are relevant to learning a set of interpretable weights end-to-end. This means that we can choose different differentiable rule learning networks as the main network according to needs to handle different tasks. Currently, for simplicity of presentation, our main network is derived from DRNet, which only supports binary classification tasks and is very time-consuming, preventing it from being proven on larger-scale data. To prove that our framework can handle larger data sets, we replace the rule learning model with the latest DIFFNAPS [1], which can perform multi-classification tasks in high-dimensional data.\n\nCompared with the current data set, in our **new data set**, the feature dimension has been raised from a maximum of 154 dimensions to a maximum of **5k dimensions**, the amount of data has been raised from a maximum of 24,000 to a maximum of 50,000, and the task has been raised from a maximum of 2 categories to a maximum of **50 categories**, reflecting the characteristics of the **task diversity and complexity**. In this case, HyperLogic-enhanced DIFFNAPS showed **a further average 6.1% F1 score improvement compared to the already good vanilla DIFFNAPS**, proving the effectiveness of the framework (please refer to General Q1 of global response for task details).\n\n[1] Walter, N. P., Fischer, J., & Vreeken, J. (2024, March). Finding interpretable class-specific patterns through efficient neural search. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 8, pp. 9062-9070).\n\n**Q3. The computational overhead introduced by the hypernetworks should be discussed more.**\n\nSince our framework introduces an additional super network, more computing load is inevitable. We take supplementary experiments as an example (please refer to General Q1 of global response for task details) to demonstrate the original algorithm DIFFNAPS and the main network based on DIFFNAPS The average F1 score, time and number of parameters of the HyperLogic framework in different tasks are shown in the table.\n\n| Method     | F1-score | Time(min) | #Params(W) |\n|------------|----------|-----------|------------|\n| DIFFNAPS   | 0.659    | 1.565     |1212.1      |\n| HyperLogic | 0.720    | 1.699     |5161.61     |\n|\n\nIt can be seen that although the parameter increase is large, considering the significant improvement in F1 score and the extremely limited increase in time consumption, especially since Hyperlogic allows us to generate diverse rule sets in one training, the increase in computational load is extremely worthwhile.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear esteemed reviewers,

We are grateful to all the reviewers for generously dedicating their time and effort to evaluating our paper. Their constructive feedback and valuable suggestions are helpful to further improve the quality of our work. We would like to answer general questions mentioned by all the reviewers here. 

Thank you for your considerable contributions during the review process. We anticipate further discussions in the future.

Once again, many thanks to your valuable insights.

Best wishes,

The authors

**General Q1: The experiments should be extended to properly compare to existing work.**

To expand our experiments to larger and more complicated cases, we considered the latest Neuro-Symbolic algorithm **DIFFNAPS**, which is capable of pattern mining under large-scale data conditions. Following the original experiments, we tested the model’s pattern mining performance under **a fixed input dimension of 5000** and varying total number of **categories K (ranging from 2 to 50)**, measured by the F1 score, with each category containing **1000 samples**. For HyperLogic, we selected only the classifier part of DIFFNAPS as the main network to compare with vanilla DIFFNAPS.

Compared with the current data set, in our **new data set**, the feature dimension has been raised from a maximum of 154 dimensions to a maximum of **5k dimensions**, the amount of data has been raised from a maximum of 24,000 to a maximum of 50,000, and the task has been raised from a maximum of 2 categories to a maximum of **50 categories**, reflecting the characteristics of the **task diversity and complexity**.

The experimental results are shown in the table below. It can be seen that in datasets with fewer categories, due to the smaller total number of samples, Hyper Logic has not yet received sufficient training and does not perform ideally. However, in more challenging classification datasets with an increased number of samples, the model's performance has significantly improved, **with an average F1 score increase of 6%**. This fully demonstrates that our framework can empower diverse neural rule learning networks, capable of handling large-scale data and possessing a good range of applications.

***Table. The F1 score（+/- std） of two methods among 11 datasets***
| Dataset（K=） | DIFFNAPS         | HyperLogic       |
|---------------|------------------|------------------|
| 2             | 0.788  +/- 0.080 | 0.726 +/- 0.084  |
| 5             | 0.703  +/- 0.030 | 0.675 +/- 0.036  |
| 10            | 0.726  +/- 0.013 | 0.751 +/- 0.020  |
| 15            | 0.622 +/- 0.017  | 0.800 +/- 0.028  |
| 20            | 0.712 +/- 0.015  | 0.760 +/- 0.011  |
| 25            | 0.688 +/- 0.020  | 0.770 +/- 0.020  |
| 30            | 0.602 +/- 0.024  | 0.666 +/- 0.021  |
| 35            | 0.557 +/-  0.014 | 0.668 +/- 0.018  |
| 40            | 0.635 +/- 0.022  | 0.712  +/- 0.011 |
| 45            | 0.611 +/- 0.009  | 0.694 +/- 0.009  |
| 50            | 0.603 +/- 0.010  | 0.702 +/- 0.012  |

Meta-review: The authors propose a novel approach for generating candidate rulesets using hyper-networks based on HyperGAN. The authors provided additional experimental results during the rebuttal, which, together with the results in the original submission, show competitive performance, and the ability to adopt their framework to different main models.
Decision: Accept (poster)