SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Each internal node  $n$  in the oblique tree needs to learn two affine functions: a splitting function used to form a hyperplane to split the sub-space into child nodes, and a value function. Because nested affine functions still produce affine functions, the set of affine functions is closed under linear transformations. Thus, we can use the value function to absorb the denominator  $||W_i||$  from the attention Eq.7. In other words, for any learned  $\theta_n$ , we can find a  $\theta_n'$  (by rescaling  $\theta_n$  with those  $\frac{1}{||W_i||}$ ) such that  $a_n(\mathbf{x})V_n(\mathbf{x},\theta_n) = a_n'(\mathbf{x})V_n(\mathbf{x},\theta_n')$  where  $a_n'(\mathbf{x}) = \prod_{i\in \mathcal{P}_n^l}\max (\min (W_i\mathbf{x} + b_i + U_i,2U_i),0)\prod_{i\in \mathcal{P}_n^r}\max (\min (-W_i\mathbf{x} - b_i + U_i,2U_i),0)$  and we use the subscript  $n$  to denote the polyhedron  $\Delta$  represented in node  $n$ . We thus directly set off to learn  $\theta_n'$  and use  $a_n'$  as  $a_n$ .
More importantly, with affine value functions, we can derive another property. For any internal node \( n \), the attention of its two child nodes contains \( a_{n} \) as a factor, so \( a_{n_L}(\mathbf{x}) = a_n(\mathbf{x})\max (\min (W_n\mathbf{x} +
$b_{n} + U_{n}, 2U_{n}), 0)$  and  $a_{n_R}(\mathbf{x}) = a_n(\mathbf{x})\max (\min (-W_n\mathbf{x} - b_n + U_n, 2U_n), 0)$ . It gives rise to an observation that no matter where  $\mathbf{x}$  is located (inside the buffer or outside),  $a_{n_L}(\mathbf{x}) + a_{n_R}(\mathbf{x}) = 2U_na_n(\mathbf{x})$ . Thus, we can substitute  $a_{n_L}(\mathbf{x})$  in the model  $f_{PAM}$  by  $2U_na_n(\mathbf{x}) - a_{n_R}(\mathbf{x})$ . Then,  $a_{n}(\mathbf{x})V(\mathbf{x}, \theta_n) + a_{n_L}(\mathbf{x})V(\mathbf{x}, \theta_{n_L}) + a_{n_R}(\mathbf{x})V(\mathbf{x}, \theta_{n_R}) = a_n(\mathbf{x})(V(\mathbf{x}, \theta_n) + 2U_nV(\mathbf{x}, \theta_{n_L})) + a_{n_R}(\mathbf{x})(V(\mathbf{x}, \theta_{n_R}) - V(\mathbf{x}, \theta_{n_L}))$ , which can be written as  $a_{n}(\mathbf{x})V(\mathbf{x}, \bar{\theta}) + a_{n_R}(\mathbf{x})V(\mathbf{x}, \bar{\theta})$  for some parameters  $\bar{\theta}$  and  $\tilde{\theta}$  due to the closure of affine functions under linear transformations. Hence, once again, we directly learn  $\bar{\theta}$  and  $\tilde{\theta}$  in our model training process. Note that we can recursively apply the above substitution for all internal nodes  $n$ , so we can reduce the polyhedrons in  $S_{\Delta}$  by half. The following theorem characterizes the above discussion (proof is in Appendix C.)
Theorem 1 If all value functions  $V$  belong to a function set that is closed under linear transformations, then the function learned by PAM  $f_{PAM}$  can be equivalently written as
$$
f _ {P A M} (\mathbf {x}) = V (\mathbf {x}, \theta_ {G}) + \sum_ {n \in \mathcal {S} _ {\Delta} ^ {-}} a _ {n} (\mathbf {x}) V (\mathbf {x}, \theta_ {n}) \tag {8}
$$
where the polyhedron set  $S_{\Delta}^{-}$  contains half of the polyhedrons (e.g., the right child nodes or the left child nodes) in  $S_{\Delta}$  and
$$
a _ {n} (\mathbf {x}) = \prod_ {i \in \mathcal {P} _ {n} ^ {l}} \max  \left(\min  \left(W _ {i} \mathbf {x} + b _ {i} + U _ {i}, 2 U _ {i}\right), 0\right) \prod_ {i \in \mathcal {P} _ {n} ^ {r}} \max  \left(\min  \left(- W _ {i} \mathbf {x} - b _ {i} + U _ {i}, 2 U _ {i}\right), 0\right). \tag {9}
$$
Remark 1 Although we include polyhedrons identified by the internal nodes in our calculation,  $f_{PAM}$  only needs to learn  $\frac{1}{2}(2^D - 2) = 2^{D-1} - 1$  value functions, which is actually in the same scale as that of only using leaf nodes in  $f_{PAM}$ .
**Optimization of PAM.** The output of  $f_{PAM}(\mathbf{x})$  can be treated as a prediction of  $\mathbf{x}$ 's label or an embedding of  $\mathbf{x}$ . PAM can be used as a constituent component in a DNN to approximate a target  $y = f(f_{PAM}(\mathbf{x}))$ . For a classification task, we can calculate the conditional distribution  $\hat{y} = Pr(y|\mathbf{x}) = \text{softmax}(f(f_{PAM}(\mathbf{x})))$  and optimize the cross-entropy  $L_{CE} = -\mathbb{E}_{(\mathbf{x},y)\sim \mathcal{D}}y\log \hat{y} -(1 - y)\log (1 - \hat{y})$  between the observed  $y$  and estimated  $\hat{y}$  to determine the parameters in PAM. For a regression task, the mean square loss  $L_{MSE} = \mathbb{E}_{(\mathbf{x},y)\sim \mathcal{D}}(y - f(f_{PAM}(\mathbf{x})))^2$  can be used.
4 Model Interpretation
We propose a conceptual framework to quantify and interpret the interaction effects learned by PAM. Without loss of generality, we assume that the DNN has a single output. Additional outputs can be similarly interpreted using the derived algorithm. The  $f_{PAM}$  can be rewritten as a summation of  $k$ -way interaction terms for all possible values of  $k \leq D$ :  $f_{PAM}(\mathbf{x}) = \sum_{\mathcal{I} \subseteq \{1,2,\dots,p\}} \phi_{\mathcal{I}}(\mathbf{x})$  where  $\phi_{\mathcal{I}}(\mathbf{x})$  captures the total contribution to the output from the  $|\mathcal{I}|$ -way interactions among the features indexed in  $\mathcal{I}$ . If  $\mathcal{I} \neq \emptyset$ ,  $\phi_{\mathcal{I}}(\mathbf{x}) = \sum_{\sum m_i \leq D} w_{\mathcal{I}} \prod_{i \in \mathcal{I}} x_i^{m_i}$  where  $x_i$  represents the  $i^{th}$  feature of  $\mathbf{x}$ ,  $m_i$  calculates the power of  $x_i$  in the interaction, and  $w$ 's are constant coefficients in front of the corresponding interaction terms. Given the definition of our attention in Eq.9, the highest polynomial order is  $D - 1$  in the attention, together with the affine value function, the highest polynomial order of  $f_{PAM}$  is  $D$ , so  $\sum m_i$  can not exceed the depth of the tree. If  $\mathcal{I} = \emptyset$ ,  $\phi_{\mathcal{I}} = w_{\emptyset}$  which is a constant. We develop a method here to estimate the contribution values  $\phi_{\mathcal{I}}$  in  $f_{PAM} = \sum_{\mathcal{I} \subseteq \{1,2,\dots,p\}} \phi_{\mathcal{I}}$  for a fixed input  $\mathbf{x}$  without computing the individual polynomial terms.
Algorithm 1: Obtain  $\phi_{\mathcal{I}}$  for an input  $\mathbf{x}$
1: Input: input  $\mathbf{x}$ ,  $g(\mathbf{x})$ , and an  $\mathcal{I} \subseteq \{1, 2, \dots, p\}$
2: Output:  $\phi_{\mathcal{I}}$
3: Set  $\mathbf{0}^{-\mathcal{I}}$  to be a  $p$ -length vector with 1's in the positions indexed by  $\mathcal{I}$  and 0's elsewhere
4: Calculate the following function: (Note that  $\phi_{\mathcal{I}'}(\mathbf{x})$  is also calculated via the same recursive function.)
$$
\phi_ {\mathcal {I}} (\mathbf {x}) = \left\{ \begin{array}{l l} g (\mathbf {0} ^ {- \mathcal {I}} \odot \mathbf {x}) - \sum_ {\mathcal {I} ^ {\prime} \subset \mathcal {I}} \phi_ {\mathcal {I} ^ {\prime}} (\mathbf {x}), & \mathcal {I} \neq \emptyset , \\ g (\mathbf {0}), & \mathcal {I} = \emptyset , \end{array} \right.
$$
where  $\odot$  is the Hadamard product operator.
For a given  $\mathbf{x}$ ,  $f_{PAM}$  can be written explicitly out as  $g(\mathbf{x})$  according to which polyhedron(s)  $\mathbf{x}$  belongs to. To determine  $g(\mathbf{x})$ , we pass  $\mathbf{x}$  through the model Eq.8 and evaluate every term in Eq.9. For instance, for the first product in Eq.9, if  $\mathbf{x}$  makes  $W_{i}\mathbf{x} + b_{i}\geq U_{i}$ , we replace the term  $\max (\min (W_i\mathbf{x} + b_i + U_i,2U_i),0)$  by  $2U_{i}$ ; if  $W_{i}\mathbf{x} + b_{i}\leq -U_{i}$ , we replace it by 0; or otherwise, we use  $W_{i}\mathbf{x} + b_{i} + U_{i}$ . Once the  $g$  function is computed, we can use Algorithm 1 to evaluate the contribution  $\phi_{\mathcal{I}},\forall \mathcal{I}\subseteq \{1,\dots ,p\}$ .
A simple example can be used to demonstrate Algorithm 1. Let  $\mathcal{I} = \{1,2\}$  which means we calculate the sum of those cross terms that involve exactly  $x_{1}$  and  $x_{2}$ . Thus we set all other elements in  $\mathbf{x}$  to 0
and calculate  $g(\mathbf{0}^{-\mathcal{I}} \odot \mathbf{x})$  to obtain the value  $v$  that adds the terms involving only  $x_{1}$  and  $x_{2}$ . We then additionally set either  $x_{1} = 0$  or  $x_{2} = 0$ , or  $x_{1} = x_{2} = 0$  (i.e., make all elements 0), re-compute  $g$  to estimate the linear terms of either  $x_{1}$  or  $x_{2}$ , or the constant term in  $g$ , and subtract these values from  $v$  to eventually obtain  $\phi_{\mathcal{I}}$ . The following theorem characterizes our results.
Theorem 2 For any input  $\mathbf{x}$ , by calculating  $\phi_{\mathcal{I}}(\mathbf{x})$  for each  $\mathcal{I} \subseteq \{1, 2, \dots, p\}$  via Algorithm 1, we have  $\sum_{\mathcal{I} \subseteq \{1, 2, \dots, p\}} \phi_{\mathcal{I}}(\mathbf{x}) = f_{PAM}(\mathbf{x})$ .
The instance-level explanation of  $f_{PAM}$  can be obtained by examining the magnitude of  $\phi_{\mathcal{I}}$  which reflects the impact of the feature interaction among the features in  $\mathcal{I}$ . If  $\phi_{\mathcal{I}}(\mathbf{x}) > 0$ , the interaction increase the predicted value; if  $\phi_{\mathcal{I}} < 0$ , it reduces the output. The model-level interpretation can be approximated by computing the mean absolute value of the  $\phi_{\mathcal{I}}$  across all sample instances.
5 Theoretical Justification - Approximation Theorems
We examine whether using PAM can enhance the expression power for universal approximation. We first introduce the Sobolev space, which characterizes a space of functions satisfying specific smoothness properties - Lipschitz continuous up to order  $n$  - which is formally defined as:
Definition 2 (Sobolev space) Let  $\mathcal{W}^{n,\infty}([0,1]^p)$  be the Sobolev space which comprises of functions on  $[0,1]^p$  lying in  $L^{\infty}$  along with their weak derivatives up to order  $n$ . The norm of a function  $f$  in  $\mathcal{W}^{n,\infty}([0,1]^p)$  is
$$
| | f | | _ {\mathcal {W} ^ {n, \infty} ([ 0, 1 ] ^ {p})} = \max  _ {\mathbf {n}: | \mathbf {n} | \leq n} e s s \sup  _ {\mathbf {x} \in [ 0, 1 ] ^ {p}} | D ^ {n} f (\mathbf {x}) |,
$$
where  $\mathbf{n} = (n_1, n_2, \dots, n_p) \in \{1, 2, \dots, n\}^p$ ,  $|\mathbf{n}| = n_1 + n_2 + \dots + n_p \leq n$ , and  $D^{\mathbf{n}}f$  is the  $\mathbf{n}$ -order weak derivative. The essential supremum ess sup  $g(E) = \inf \{M \in R : \mu(\{x \in E : f(x) > M\}) = 0\}$  captures the smallest value that the function  $g$  can approach or exceed on a set  $E$ , except for a negligible subset of points with the measure  $\mu$ . Essentially, the space  $\mathcal{W}^{n,\infty}([0,1]^p)$  is  $C^{n-1}([0,1]^p)$  whose functions' derivatives up to order  $n$  are Lipschitz continuous.
The following assumption is commonly used in the discussion of DNNs. Without loss of generality, it narrows our focus on a normalized Sobolev sphere. This assumption constrains the functions having Sobolev norm no greater than 1 within the sphere.
Assumption 1 Let  $F_{n,p}$  be a set of functions lying in the unit ball in  $\mathcal{W}^{n,\infty}([0,1]^p)$ , we have
$$
F _ {n, p} = \left\{f \in \mathcal {W} ^ {n, \infty} ([ 0, 1 ] ^ {p}): | | f | | _ {\mathcal {W} ^ {n, \infty}} ([ 0, 1 ] ^ {p}) \leq 1 \right\}.
$$
This assumption is sufficient for our analysis, as functions encountered in real-world learning tasks can typically be linearly transformed into  $\mathcal{W}^{n,\infty}([0,1]^p)$ , as shown in previous studies [29]. This allows us to analyze the error bounds for terms in the polynomial approximation after performing Taylor expansion. Theorem 3 demonstrates that our model can almost surely approximate any ReLU-activated DNN model without error. All proofs can be found in Appendix E and F.
Theorem 3 If  $\mathbf{x}$  is bounded and sampled from a distribution with upper-bounded probability density function, then for any ReLU activated plain DNN model  $f_{DNN}(\mathbf{x})$ , there exists a PAM with
$$
P r \left(f _ {P A M} (\mathbf {x}) = f _ {D N N} (\mathbf {x})\right)\rightarrow 1.
$$
Theorem 4 examines the parameter efficiency, and demonstrates that networks incorporating the proposed polyhedron attention require fewer parameters compared to those relying solely on ReLU activation, while maintaining the same approximation error in fitting functions in the Sobolev space.
Theorem 4 For any  $p, n > 0$  and  $\epsilon \in (0,1)$ , we have a PAM which can 1) approximates any function from  $F_{n,p}$  with an error bound  $\epsilon$  in the sense of  $L^{\infty}$  with at most  $2p^n (N + 1)^p (p + n - 1)$  parameters, where  $N = \lceil \left( \frac{n!}{2^p p^n} \frac{\epsilon}{2} \right)^{-\frac{1}{n}} \rceil$ .
Remark 2 For the purpose of comparison, the ReLU-activated plain DNN needs  $p^n (N + 1)^p (p + 1)n\mathcal{O}(\log (1 / \epsilon))$  parameters under the same setting in Theorem 4 [30, 29].
It is worth noting that extensive research has been conducted on the approximation theory of DNNs with the ReLU-activation, which often concerns common function classes in specific function spaces such as Besov [31, 32] and Sobolev spaces [33, 34]. These analyses reveal a notable influence of the smoothness of the target function on the resulting approximation errors. Since Sobolev spaces characterize the smoothness properties of functions, we can investigate the ability of neural networks with ReLU activation to approximate functions with different degrees of regularity and smoothness. These theorems highlight the expressivity of our model and provide theoretical insights for the parameter efficiency of our proposed attention module in neural network architectures.
We evaluate the effectiveness and efficiency of PAM on three large-scale datasets: the Criteo<sup>1</sup> and Avazu<sup>1</sup> click-through-rate (CTR) datasets, and the UK Biobank<sup>2</sup> medical database. We conduct an analysis of the hyperparameters of PAM, and perform ablation studies by individually removing each of the three key components of PAM and evaluating the performance variations. Given the lack of known feature meanings in CTR benchmark datasets, we utilize the UK Biobank dataset as an example for studying model interpretation. Specifically, we validate the interaction effects captured by our interpretation framework, as detailed in Sec. 4, in the prediction of brain-age by the grey matter volumes from distinct brain regions. For implementation details, computation and space complexity of our model, please refer to Appendix G.
Datasets. Both the Criteo and the Avazu are massive industry datasets containing feature values and
click feedback for display ads, and are processed following the benchmark protocol in BARS [35, 36]. The UK Biobank serves as a comprehensive biomedical database and research resource, offering extensive genetic and health-related information, where our objective is to predict participants' age by leveraging the grey matter volumes from 139 distinct brain regions. The summary statistics are listed in Table 1.
Table 1: Statistics of the datasets.
Dataset	#Train	#Valid	#Test	#Features
Criteo	33M	8.3M	4.6M	2.1M
Avazu	28.3M	4M	8.1M	1.5M
UK Biobank	31.8K	4K	4K	139
Evaluation metrics. Criteo and Avazu datasets are concerned with binary classification tasks, evaluated using Area Under the ROC Curve (AUC). For brain-age prediction, a regression problem, we assess performance using the  $R^2$  score.
**Baseline Algorithms.** We compare PAM against the top 5 algorithms from a pool of 33 baseline methods, selected based on the overall AUC scores in BARS, including DESTINE [19] (currently the best performer on Avazu), DCN [37], AOANet [4], EDCN [27] (best on Criteo), and DCN-V2 [25]. Given that our model and the selected algorithms are DNN-based, we also include a well-tuned  $\mathrm{DNN}^1$  as a strong baseline for our model comparison. It is worth noting that several state-of-the-art studies on CTR datasets, such as Criteo and Avazu, from Google [2, 24, 37], Microsoft [38] and Huawei [24], have recognized that even a marginal improvement of AUC at the level of  $1\%$  is considered a significant performance enhancement.
6.2 Effectiveness and efficiency evaluation
Performance Comparison. We compare the results over five runs. As shown in Fig. 4a, PAM achieves the highest AUC and  $R^2$  score amongst all benchmark algorithms. PAM outperforms the second-best model by  $1.4\%$  for Criteo,  $0.6\%$  for Avazu, and 2.2 for UK Biobank, which is much higher than the improvement achieved by the second-best model from the next model (0.4% for Criteo,  $0.16\%$  for Avazu and  $0.6\%$  for UK Biobank). It is also worth noting that, despite being well-tuned, none of the benchmark algorithms performs optimally across all three datasets simultaneously. We evaluate the computation time of PAM by comparing the training and inferecing runtime (seconds) per batch to the number of trainable parameters (#Par). As shown in Table 2, almost all models have a similar amount of free parameters, and their runtime per batch is comparable among the methods (PAM is superior than half of the methods and inferior than the other half.)
<sup>1</sup>https://github.com/openbenchmark/BARS/tree/master; <sup>2</sup>https://www.ukbiobank.ac.uk/
Figure 4: Experimental results. Error bars represent the means and standard deviations of AUC for classification and  $R^2$  for regression. a) AUC and  $R^2$  scores of PAM and comparison methods. b) Hyper-parameter analysis of PAM. U: the initial value of  $U_{n}$ . D: the depth of the oblique tree in PAM. c) Ablation studies of PAM.
Hyper-parameter Study. Although  $U_{i}$ 's are trainable parameters in PAM, their initialization may affect how well PAM works. We hence study the effects of the initial values of  $U_{i}$ 's by testing a set of choices from 1 to 3 by a step size of 0.5. We also examine the effects of the tree depth  $D$  on PAM's performance. Fig. 4b shows the comparison where the initial values of  $U_{n}$  do not sub
Table 2: The number of parameters (#Par) and seconds per batch (Sec/Batch) during the training and inference of PAM and baseline models.
Model	Criteo	Avazu	UK Biobank
#Par	Sec/Batch	#Par	Sec/Batch	#Par	Sec/Batch
Training	Intereence	Training	Intereence	Training	Intereence
PAM	22.3M	0.102	0.0330	14.1M	0.090	0.0263	315K	0.083	0.0018
DNN	21.8M	0.046	0.0139	13.8M	0.038	0.0129	380K	0.056	0.0063
DESTINE	21.5M	0.130	0.0344	13.6M	0.090	0.0188	384K	0.072	0.0069
DCN	21.3M	0.044	0.0114	13.4M	0.080	0.0099	381K	0.069	0.0064
DCN-V2	22.0M	0.103	0.0128	13.8M	0.091	0.0137	458K	0.059	0.0067
AOANet	21.4M	0.151	0.0314	13.4M	0.338	0.0168	457K	0.066	0.0067
EDCN	21.5M	0.066	0.0119	13.1M	0.048	0.0113	63K	0.072	0.0071
stantially change PAM's performance, which indicates that  $U_{n}$  may be well trained during the PAM optimization. As for the tree depth  $D$ , the performance of PAM initially improves as  $D$  increases, but when  $D$  becomes large, the performance may deteriorate. An appropriate value of  $D$  will help PAM to learn more domain knowledge, but if  $D$  is too large, highly complex interactions may not necessarily lead to better prediction due to issues such as overfitting.
Ablation Study of PAM. The following specifies the three components for our ablation experiments. 1) PAM without overlapping polyhedrons (PAM w/o OP). As shown in Eqs. 5, 6 and 7, overlapping polyhedrons are generated in PAM to fit the target. To demonstrate the importance of soft splitting, we calculate  $a_{n}(\mathbf{x})$  with  $\prod_{i\in \mathcal{P}_n^l}\max (\min (W_i\mathbf{x} + b_i,2U_i),0)\prod_{i\in \mathcal{P}_n^r}\max (\min (-W_i\mathbf{x} - b_i,2U_i),0)$ .
2) PAM without adaptively learning interaction (PAM w/o ALI). As shown in Fig. 5, the upper-bound  $2U_{i}$  enables PAM to learn interactions with different orders in different polyhedrons. To exam the effectiveness of the upper bound, we remove this upper bound  $2U_{i}$  from Eq. 9.
3) PAM without removing  $||W_i||$  (PAM w/o RW). According to Theorem 1, the denominator of Eq. 7 can be removed without reducing the expression capability of PAM. To examine the correctness of this claim, we directly use PAM without removing  $||W_i||$  from the denominator.
Fig. 4c clearly demonstrates a notable decline in the performance of the PAM when key components are removed. In particular, the standard PAM significantly outperforms the PAM w/o OP. It confirms the critical role of overlapping polyhedrons in enhancing PAM's performance. The removal of ALI decreases the AUC and  $R^2$  of PAM as well, indicating the significance of combining low-way interaction effects with high-way ones to fit the target. Moreover, PAM w/o RW shows that removing  $||W_i||$  from the denominator in Eq. 7 improves PAM's performance. Although, according to Theorem 1, PAM without RW has the same expressive capability as standard PAM, the inclusion of  $W$ 's norm in the denominator of Eq. 7 may result in unstable gradient variances, potentially compromising the performance of the optimizer.
6.3 Identified interaction effects among neural markers in predicting brain age
Although the interpretation framework was developed for PAM, it can be applied to those DNN architectures whose functions are piece-wise polynomials (see Appendix D). After carefully checking
f)
Figure 5: Top 5 main and 2-way interactions identified by PAM (Figs. a and b), DCN (Figs. c and d) and AOANet (Figs. e and f) by sorting mean absolute values of each  $\phi_{\mathcal{I}}$  averaged over all participants. For each effect, points (each corresponding to a participant) are distributed according to  $\phi$  values calculated by Algorithm 1 in the beeswarm bars. L: left; R: right; CR: crus II cerebellum; FOC: frontal orbital Cortex; FG: frontal gyrus; PP: planum polare; OCSD: lateral occipital cortex, superior division; FP: frontal pole; LOC: lateral occipital cortex; PG: precentral gyrus; T: thalamus; SC: subcallosal cortex.
all baseline methods, we found that our interpretation framework could be used to extract interactions from DCN and AOANet. Fig. 5 presents the top five main effects (Figs. a, c and e) and two-way interaction effects (Figs. b, d and f) between brain regions to brain-age prediction, as determined by the trained PAM, DCN and AOANet.
According to Figs. 5a and 5b, PAM found that the grey matter volumes (GMV) of both the left and right frontal poles (FP) play significant main effect as individual features, which is well aligned with previous research findings[39, 40, 41]. The other three brain regions, lateral occipital cortex (LOC), precentral gyrus (PG) and thalamus (T), are also discussed in early studies[42, 43, 44]. Existing aging studies primarily focus on main effects of GMV, but the top five two-way GMV interactions of identified brain regions have been acknowledged in works such as [42]. As shown in Fig. 5c, 5d, 5e and 5f, the main effects of PG and T identified by PAM were found by DCN and AOANet, respectively, and all three algorithms found the two-way interactions between the left and right FP. In addition to the shared top-5 brain regions identified by PAM, AOANet and DCN, PAM additionally identified the main effect of the LOC and FP and two-way interaction effects related to the insular and subcallosal cortex (SC) regions.
We propose a novel feature interaction learning module, namely Polyhedron Attention Module (PAM), to fit a target with adaptive-order interaction effects. PAM produces a self-attention mechanism partitioning the input space into overlapping polyhedrons and learning the boundary hyperplanes for each polyhedron automatically. These hyperplanes multiply to identify interaction effects specific to individual polyhedrons. Under this mechanism, PAM automatically captures both simple and complex interaction effects. In our theoretic analysis, we show that PAM can enhance the model expression power for universal approximation. Experimental results demonstrate that the proposed model achieves better performance on massive datasets than the state-of-the-art. In the future, we will study how to dynamically pruning PAM's oblique tree during the training process to regularize the model. It will also be interesting to investigate how to define boundaries with non-linear functions.
This research has been conducted using the UK Biobank Resource under the Application 51296 - Machine Learning Analysis of Brain Images, Behaviors, and Genotypes to Understand Mental Disorders, and was supported in part by the NIH grant K02-DA043063 and NSF award #2122309.