A general framework for formulating structured variable selection
Authors: Guanbo Wang
Mireille Schnitzer
Tom Chen
Rui Wang
Robert W Platt
Venue: Accepted by TMLR ()

Abstract: In variable selection, a selection rule that prescribes the permissible sets of selected variables (called a "selection dictionary") is desirable due to the inherent structural constraints among the candidate variables. Such selection rules can be complex in real-world data analyses, and failing to incorporate such restrictions could not only compromise the interpretability of the model but also lead to decreased prediction accuracy. However, no general framework has been proposed to formalize selection rules and their applications, which poses a significant challenge for practitioners seeking to integrate these rules into their analyses. In this work, we establish a framework for structured variable selection that can incorporate universal structural constraints. We develop a mathematical language for constructing arbitrary selection rules, where the selection dictionary is formally defined. We demonstrate that all selection rules can be expressed as combinations of operations on constructs, facilitating the identification of the corresponding selection dictionary. We use a detailed and complex example to illustrate the developed framework. Once this selection dictionary is derived, practitioners can apply their own user-defined criteria to select the optimal model. Additionally, our framework enhances existing penalized regression methods for variable selection by providing guidance on how to appropriately group variables to achieve the desired selection rule. Furthermore, our innovative framework opens the door to establishing new $\ell_0$-based penalized regression techniques that can be tailored to respect arbitrary selection rules, thereby expanding the possibilities for more robust and tailored model development.

Published in Transactions on Machine Learning Research (01/2024)
A general framework for formulating structured variable se- lection
Guanbo Wang*
gwangghhsh..arvong edu CAUSALab, Departments of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA
Mircille E. Schnitzer
Faculté de pharmacie, Université de Montréal, Montréal, Québer, Canada Département de médecine sociale et préventive, Université de Montréal, Québec, Canada
Tom Chen
tom chen@horrardpigriim.cmm
Department of Population Medicine, Harvard Medical School and Harvord Pilgrim Health Care Instibute, Boston, Massachusetts, USA
Rui Wang
rwang rwang@hsph..t Saangnhhhhhhhhaaaa www.ard.aaaad.d...ooo edu Department of Population. Medicine, Harvard Medical School and Harvord Pilgrim Health Care Institute, Boston, Massachusetts, USA Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, Mussachusells, USA
Robert W. Platt
robert, robert.platt plagll@ttttlll.ommmmmm Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada
Reviewed on OpenReview: otpereveepe net/ forum? id- cvOp
Abstract
In variable selection, a selection rule that prescribes the permissible sets of selected variables (called a "selection dictionary") is desirable due the inherent structural constraints among the candidate variables. Such selection rules can be complex in real-world data analyses, and failing to incorporate such restrictions could not only compromisc the interpretability of the model but also lead to decreased prediction accuracy. However, no general frame- work has been proposed to formalize selection rules and their applications, which poses significant challenge for practitioners seeking to integrate these rules into their analyses. In this work, we establish a framework for structured variable selection that can incorporate universal structural constraints. We develop a mathematical language for constructing ar- bitrary selection rules, where the selection dictionary formally defined. We demonstrate that all selection rules can be expressed as combinations of operations on constructs, facil- itating the identification of the corresponding selection dictionary. We use detailed and complex example to illustrate the developed framework. Once this selection dictionary is derived, practitioners can apply their own uscr-defined criteria to seleet the optimal model. Additionally, our framework enhances existing penalized regression methods for variable se- lection by providing guidance on how to appropriately group variables to achieve the desired selection rulc. Furthermore, our innovative framework opens the door establishing new fo basedd penalized regression techniques that can be tailored to respect arbitrary selection rules, thereby expanding the possibilities for more robust and tailored model development.
1 Introduction
Variable selection has become an important problem in statistics and data science, especially with large- scale and high-dimensional data becoming increasingly available. Variable selection can be used to identify

---

Published in Transactions on Machine Learning Research (01/2024)
covariates that are associated with or predictive of the outcome, remove spurious covariates, improve pre- diction accuracy, and generate hypotheses for causal inquiries (Guyon & Elliseeff, 2003
Reumanen Wasserman & Roeder 2009 Wang et al. 2020 Siddiqqe et al. 2019 Liu et al. 2022 Bouchard et al. 2022
2003
Wang ct al. 2023 General techniques to conduct statistical variable selection include best subset selection,
penalized regression, and nouparametric approaches like random forest. (Heinze et al. 2018
Chowdhury
Turin 2020).
When selecting variables for the purpose of developing an interpretable model, understanding and formalizing the structure of covariates can lead to more interpretable variable selection. Covariates may have structure due to
Variable type. For example, when including a categorical variable in a regression model, each non- reference category is represented by a binary indicator. It may be desirable to collectively include or exclude these binary indicators a group.
2. Variable hicrarchy. For example, one may define hicrarchical structure for sets of covariates A and IB such that if A is selected, then B must be selected. One application is interaction selection with strong heredity (Haris et al., 2016), that is "the selection of an interaction term requires the inclusion of all main effect terms." A second application is when one covariate is a descriptor of another, such as medication dose (0-10mL) and medication usage (yes/no).
Such restrictions on the resulting model, which we call "selection rules", can be incorporated into the sta- tistical variable selection process so that the resulting model satisfies the rules. Practitioners can define any selection rule based on their priori knowledge of the covariate structure.
Lasso Tibshirani 1996) and best subset select ion via optimization (Bertsimas el al. 2016) are two commonly- used approaches that do not restrict the composition of the resulting model. However, a variety of existing variable selection techniques have emerged to accommodate diverse selection rules. For instance, the group Lasso Yuaan & Lin. 2006) can select group of variables collectively, while the sparse group Lasso (Simon et al. 2013) can perform bi-level selection. Additionally, the Exclusive Lasso (Campbell & Allen 2017) excels at within-group selection by ensuring at least one variable is chosen from cach group. However, these methods have been developed to respect single, specific lypes of selection rules. Expanding on this, both overlapping group Lasso (Mairal et al.] (0100) and latent overlapping group Lasso (Obozinski et al. 2011 can accommodate a more extensive array of (though not all) selection rules by performing the simultancous selection or exclusion of overlapping groups of variables.
In this work, we develop general framework for variable selection that can formally express any selection rule in a mathematical language, which enables us to systematically compile the exhaustive list of possible models (i.e. permissible covariate subsets) corresponding to given selection rule. One practical use of this exhaustive list of models is that we can directly apply statistical criteria to identify the optimal model in terms of the observed data. We also discuss two potential avenues for future development. First, the proposed framework can guide us in how to effectively group variables to follow complex selection rules with existing penalized regression techniques. Second, given the inherent connection between the lo norm and our dcfinition of selection rules, our work also directly leads to new ba-bd penalized regression methods that can be tailored to accommodate arbitrary selection rules.
This paper is organized as follows. Section 2 is an overview of the key findings with an illustrative example. In Scction 3 we provide formal introduction to the language used in constructing selection rules, prove that we can express any arbitrary selection rule within our framework, and give formulas for deriving the list of all permissible models for any given rule of arbitrary complexity. We then provide a detailed example to illusraate how to set selection rules and derive the selection dictionary in practice, where nine selection rules are considered in Section g Last, we discuss the broader implications of this framework and its potential utility in driving future research advancements.
2

---

Published in Transactions on Machine Learning Research (01/2024)
2 Overview
Suppose that we have a set of candidate variables V. We define a selection rule on this set as the selection dependencies among all candidate variables. For example, consider a study where we want to investigate which of the following variables should be included in a model for blood pressure: age (A), age squared (A²), and race as categorical variable with 3 levels, represented by dummy variables B₁ and Bz. We are also interested in the interaction of age with race AB2). So we have = In this example, standard statistical practice requires that the resulting model must satisfy a selection rule defined by the following three conditions: 1) if the interaction is selected, then both the main terms for age and race must be selected, 2) if age squared is selected, then age must be selected, 3) the dummy variables representing race must be collectively selected, and 4) the two categorical interaction terms must also be collectively selected. The combination of these four rules is the selection rule that must be respected.
B1, B2, AB²,
We next define a selection dictionary as the set of all subsets of V/ that respect the selection rule. When we say a dictionary respects a selection rule, we mean the dictionary is congruent to the selection rule in the sense that the selection dictionary contains all (rather than some) subsets of V that respect the selection rule. Theorem in Sention 3 states that every selection rule has a unique dictionary. The dictionary for the above example would be {0_ {A} (B,, B2] {A, B1, B2} {A, A2) {A, A?, B1, B2} {A, B1, Bz, AB,, AB2) {A, A Bi, B2, , AB2]3. Despite a total of 64 possible subsets of V, there are only 8 possible models that can be selected under this rule.
We are interested in the general problem of finding a selection dictionary given an arbitrary selection rule. We start by defining unit rules as the building blocks of selection rules. For a given set of candidate variables V with F S V, a unit rule is a selection rule of the form "select a number of variables in F." The unit rule depends on the number of variables that are allowed to be selected from F. In our running example, one unit rule "sclect zero or two variables from the set = {B1, B2)" This is equivalent to saying that B and B2 must be selected lorether, i.e. select neither or both. We define C as the set of numbers of variables that are allowed to be selected in the unit rule. In the unit rule gave above, C= 20.
In Thcorem 2 we give a formula for the dictionary congruent to a given unit rule. This formula shows that the dictionary is all unique unions of sets 1) of variables in where the number of variables is in C and 2) of variables outside of F. Applying this formula, we can see that the unit rule "select zero or two variables (that is, C= f0, 27) from the set F = {B,, B₂ has a dictionary that is the set incorporating 0 and (B,, B2} and all unions of @ and {B1, B₂} with any of the other elements in V, respectively.
We then define five useful operations on selection rules in Table 2 For example, A being applied to two selection rules indicates that both of the selection rules must be respected. An arrow + indicates if the selection rule on the hand side is being respected, then the selection rules on the right-hand side must be respected. For each operation, we can show how the operation on selection rules is related to an operation on the respective dictionaries. Therefore we are combining or constructing more complex rules from operations on unit rules, we can always derive the resulting dictionary. Our most important result is Theorem 3 stating that any rule can be obtained through operations on unit rules.
To illustrate these ideas in our running example, define unit rules 1) uj: "select zero or two variables in (ABI, AB2)," " 2) uæ: "select zero or two variables in [Bi, B2)," 4) uy: "select three variables in (A, B1, B2)," 5) uy: "select one variable in {.43,"" 6) ug: "select one variable in {4)". The same selection rule that we defined when we introduced the example can be expressed through operations on these unit rules as: (u, A u2) A (ug + 14) A (us un).
" 3) ug: "select two variables in [AB1, AB2),
3 Selection rules and selection dictionaries
In this section, we introduce the mathematical language for expressing selection rules, which enables us to design algorithms to incorporate selection dependencies into model selection. Two fundamental concepts are being introduced first: the selection rule and its dictionary. Then, we introduce unit rules and operations on unit rules as the building blocks of selection rules. We show that we can construct any selection rule from unit rules and also derive the unit dictionary from set operations on the dictionaries belonging to the unit
3

---

t1 Select at least one variable {A, B}. 12 If A is selected, then B must be selected. Published in Transactions on Machine Learning Research (01/2024)
rules. We then investigate some properties of the resulting abstract structures. Finally, discuss some challenges of the direct application of the theory in high-dimensional covariate spaces.
Unless specified otherwise, we use normal math (for example, P), blackboard bold (F/O), Fraktur lowercase (f), and calligraphy uppercase fonts (F) to represent a random variable, set, rule, and operator respectively. P(F) represents the power set (collection of all possible subsets) of F, (F) denotes the power set of the power set of F, and [F] represents the cardinality of F. The maximun integer of set of integers F is denoted by max(F). We say two sets are equivalent if they contain the same elements, regardless of their multiplicity. For example, {A, B, B, C, C) [A, B, C}. Graphs helpful show the dependencies among candidate covariates. For example, an arrow in graph can indicate that the children nodes are constructed based on their parent nodes.
We take two examples to illustrate the concepts throughout this section.
A
B
B1
B2
C
D
AB
AB2
Figure 1: Graph for Example 1
Figure 2: Graph for Example 2
Example 1. Suppose that we have 4 candidate variables, V {A, DDD, that have no structural relationship. The corresponding graph is shown in Figure
Example 2. Suppose we have three variables: a continuous variable A and a three-level categorical variable В (represented by two dummy indicators B₁ and B2). We also consider their interactions represented by ABı and AB2. The corresponding graph is shown in Figure 2 with nodes V = {A, B1, B2, ABi, AB2). The arrows indicate that the child nodes are derived from their parents.
Next, introduce the concept of the selection rulc. Definition (Selection rule). A selection rule of V is defined as selection dependencies among the variables in V.
The selection dependency is a general concept regarding limitations on which combinations of variables are allowed to be selected into model. Table gives some examples of selection rules for covariate set V = [A, C,
Table Examples of selection rules and their dictionarics, (A,B,C)
t Selection dependencies
Dr, Selection dictionaries
((A). (A, (AA,,,,,AAA (B,(B), (A, B), (C), [B, (A, (B), B), (A,
t3 Respect both ti and t2.
There may be many possible subsets of variables that respect a given selection rule. We define the set of all possible subsets of respecting a given selection rule as the corresponding selection dictionary.
Before introducing selection dictionary, we define general dictionary first.
Definition 2 (Dictionary). Given a finile set of candidale variables V, dictionary D C P(V) V is set of ss) of V.
For example, dictionary of candidate variables V (A, can be {{A), (B)), {o (A, {A}} P(V) etc.
Definition 3 (Selection dictionary). For a given V, selection dictionary D, dictionory that contains all subsets of V that respect the selection rule
4

---

Published in Transactions on Machine Learning Research (01/2024)
The definition stresses that all sets in selection dictionary must be subset of P(V). This allows us to list all "allowable" sets of variables that could result from a variable selection process respecting the selection rule. Some examples of selection dictionaries corresponding to selection rules are shown in Table !
It is also possible to have selection rules that are cnntradictory/inchereaa variables to be selected than the number of variables the given set. In this case, we define the selection dictionary to be the 0. For instance, if the selection rule is "select 3 variables from [A, with V = (A, B}, then the corresponding selection dictionary is 0. When selection rule is, for example, "select 0 variables from {A. B)," which is coherent but trivial, then the selection dictionary is the empty set {0}.
meaning that they require more
By the definitions above, there is mapping from a selection rule to selection dictionary. The theorem below gives the uniqueness of the mapping with proof in Appendix A
Theorem 1. Given a selection rule on a set, there is a unique selection dictionary that satisfies this given selection rule.
We say the unique selection dictionary is congruent to its selection rule, which is denoted by D2 _ t, or equivalently, t = De. In our context, it is equivalent to saying a selection dictionary respects a selection rule. However, it is possible that more than one selection rule results in the same selection dictionary. Therefore, we define an equivalence class of selection rules below.
Definition 4 (Equivalence class of selection rules). For given condidale set V, and given selection rule t with selection dictionary D the equivalence class of vi, denoted by R == (t = D}, is a set of all selection rules in V/ that are congruent to the same selection dictionary.
Corollary 1. By Definition 4 and Theorem 1 there is a one-to-one mapping from an equivalence class of a selection rule to a selection dictionary.
For a given (finite) V, we define Ri as the selection rule space containing all equivalence classes of selection rules on V. Because the number of possible combinations of selected variables is finite, the number of possible dictionaries is finite. Thus, because of the one-to-one correspondence between dictionaries and rules, the space of rules R is also finite.
The above definitions provide us with a broad view of the language for expressing selection rules generally. Next, we introduce the grammar of this language, which allows for the exploration of theoretical properties of selection rules and of further algorithmic development. We start by defining unit rules and their dictionaries and then introduce the operations between unit rules. Then, more complex selection rules can be assembled by unit rules and their operations, and the related selection dictionaries can be determined.
Definition 5 (Unit rule and its dictionary). Define C, set of numbers. For given V and given F C V, wnit rule uc(F) is a selection rule, where the selection dependency takes the form "select C variables from F", and the number of variables to be selected is any value in C. A unit dictionary Du is dictionary that contains all subsets of V that respect the unit rule (F(
Remark 1. The set C is a set of numbers which constrains the number of variables to be selected in F. For ezample, if [F| = 3 and C is {1}, then the rule in uc(F) translates to "there is one variable to be selected" from F. If C = (0, 2}, the unát rule is "there are zero or two variables to be selected' from F. Any C with elements greater than [F] would resull in an incoherent unil rule because the variable selection is conducled without replacement and so cannot select more than the cardinality of the set.
In the context where we investigate more than one unit rule for given V, we use u; to represent uc, (F&) and number them i = 1, Among the examples in Table [ only is valid unit rule, which can be expressed as (f B)), and the unit dictionary D, = Dr, is given.
Given a unit rule and its dictionary, there is a one-to-one mapping from the IF in the unit rule to the corresponding unit dictionary, which characterized by unit function fc-
Definition 6 (Unit function). Each unit rule relates a unit function fe with some input F S V. A unit function fc. maps the subset IF to the unill dictionary Du (V) that respects ug (F).
5

---

Published in Transactions on Machine Learning Research (01/2024)
Therefore, for given unit rule, we write (c( _ D, fe(F). This is valid function because unit dictionary is defined as the set of all possible subsets of V that respect the unit rule; thus, given fixed constraint and set F, there is unique dictionary output.
The following theorem characterizes unit functions, providing a formula for the unit dictionary so that when a unit rule is given on FF, the corresponding unit dictionary can be determined.
Theorem 2. Each unit function in is a C-specific function fc(.), with domain P(V), defined by
F F THD ={ foub: F s.t. |o| C, vb S (\F), if IF max(C)
0, otherwise,
where F E P(V).
When [F] 2 max(C), the unit rule uc (I) is coherent, and the unit function is bijection with domain M = (F, s.t. FE PFV), > max(C)} and image {a U lb, Yo C s.t. |o| C, Vb CV M).
This means that when the unit rule is coherent, the unit dictionary contains all sets that unions between subset of F that respects the constraint C and a subset of the remaining covariates in V (excluding F). The characterizing the mapping of V to a unit dictionary by a unit function. Corollary gives an interesting property of a unit dictionary, The proofs are direct consequences of Theorem 2
proof is given in Appendix B Corollary 2 gives a special case of Theorem 2
Corollary 2.
When the input of wnit function is V, with a constraint C resulting in coherent unit rule, the resulting unit dictionary is fc(V) = (n P(V) : |n| E C}.
Corollary 3. When C + {0} for given coherent unit rule, the corresponding unit dictionery D, satisfies U,D,,,= = where Du.i is the ith set in D..
To further investigate the relationships among unit functions with different constraints, we provide the following corollaries.
Corollary 4. For a given F E P(V), (F) is injective with respect to the orgument C when at least one constraint (C1 or C2) results in a coherent rule applied to F. That is, fc) * fc, whenever C1 # C2. This means that two distinct unil functions (related to two distinct wnit rules) will resull in different dictionaries even when the inputs are the same.
For C = (O....,|[1] then fc(F) VTV), V. This means that when there is effectively no constraint on the selection (i.e. any number of variables can be selected), the wnit dictionary is the power set V. consequence is that two different unit functions with nonrestrictive constraints can result in the same dictionary even when the inputs are different.
Corollary 5.
The proofs corollaries and are in Appendix and respectively.
The goal is to build selection rules out of unit rules. This will allow for an algorithm determine the resulting selection dependencies and dictionary. To do this, we define some operations among selection rules. be applicd to unit rules.
Because a unit rule is also selection rule, the operations
Definition 7 (Operations on selection rules). Given selection rules on V, define an operation on selection rules O a function that maps single selection rule or pair of selection rules to anolher selection rule To.
The rule to resulting from the operation is congruent to unique selection dictionary which is congruent to to, Do. We define five operations in Table 2
Given an operation on rules, we can derive the corresponding operation on the related dictionaries that will result in the selection dictionary Do. Table 2 shows the resulting dictionary for each operation. The derivation of each result. is given in Appendix These results allow us to develop algorithms to output selection dictionaries for complex rules through operations on simpler rules.
We use the running example in Table to illustrate the second and fourth operations on unit rules as a special case.

---

Published in Transactions on Machine Learning Research (01/2024)
Table 2: Operations for selection rules and the resulting selection dictionaries.
Operation
Interpretation
Do
-1
vi is not being respected
P(V) \ Dr
T1 12
both t and 12 are being respected
Dr, 0 Dzz
t
V
12
either ti 12, or both is/are being respected
Dr U Dra
t1 1-
t2
if t is being respected, then 12 is being respected
(P(V) \ Dr, U (D,, n D22)
Selection rule E; V is congruent to De, = 1,2.
Define
u = (fA, BJ) = Da, =
U2 = ((A) = IDu [[A). (), (4,C),
Ug 1(1) ((B)) = Dua
(B,
The t2 in Table 1 is "if A is selected, then B must be selected," " which can be expressed as t2 == U2 + ug. According to Table t2 is congruent to {0. (B), B), (C), (B, in Table
which is exactly the De.
Note that, by Definition T the operation on two selection rules results in a selection rule, thus the results of an operation on two selection rules can be an input of a second operation. We use parentheses differentiate the order of operations. The t3 in Table [ is "select at least one variable in {A, B}" A 12. Thus, T3 u1 A (u2 uz) is valid operation resulting in rule that is congruent to the selection dictionary ((B), (B, (4, (according to Table 2 which is exactly the De, in Table
We provide some useful properties of operations below, which can be verified by checking the resulting dictionaries for both sides of the equations. These properties can be used to identify which selection rules are the same equivalence class.
Proposition 1. Given t # 12 + vo (in the sense that the congruent dictionaries are distinct), then
Commutatioee laws: tı A = 12 Ati v V 22 = Vtn
Associative laws: (r) (2 A = (12 A 13) (T V Vt= V
3. Non- Non-distributive laws: V (E2 A 13) + (t1 V (tg
tı A (t2 to) # (ti A 12) V (t) A t3), and
4. Sequential laws (ti + 12) (ti + =t (t2
(ri + 12) V (T1 + va) = t (to Vs),,
all apply.
The next theorem confirms that, equipped with operations and unit rules, we can now elfectively express any selection rule as operations on unit rules.
Theorem 3. All selection rules con be expressed by either unit rules or operations on unit rules using A and
The proof is given in Appendix D This means that we can represent any rule in mathematical language. This allows us to develop algorithms to combine multiple rules and generate resulting dictionaries.
Next, we use Examples 1 and 2 with hypothetical data structure to illustrate how to express some com- mon selection dependencies by unit rules and operations. The corresponding selection dictionaries are also provided.
Example 1.1 (Individual selection) In Example 1, suppose all variables are continuous or binary, and no structure is imposed. We can set the selection rule as selection between to variables 200.1.23344) (V), and then D, = P(V). This rule is satisfied by the (adaptive) Lasso (Tibshirani 1996..
7

---

Published in Transactions on Machine Learning Research (01/2024)
Example 1.2 (Groupwise selection) In Example 1, suppose we have 2 three-level categorical variables. Denote F1 = {A, B), F2 = {C, D). Let the variables in F1 be the dummy variables representing a categorical variable, and similarly for the variables in F2. In an analysis, we would like to select U', collectively, same for F2. We can then set t= ( (F1) A 410.2) (F2). In addition, D, = F2, F2 This rule is satisfied by the group Lasso Yuaan & Lin 2006).
Example 1.3 (Within-group selection) If variables in F = {A, are one group, and F2 = (C, represents second group, and the goal is to select at least one variable from both groups, (Campbell & Allen 2017; Kong et al. 2014) then we set u(1,2) AF. (F2), (F2), meaning there is at least one variable that must be selected in I 1 and 22 respectively. In addition, DD = ({A, (,, C), A, D3, (B, {A, B, D), C, D}}. This rule satisfied by the exclusive (group) Lasso (Campbell & Allen
(A. C, D). (B, C, 2017
Example 2.1 (Categorical interaction selection with strong heredity) In Example 2, because (B1, B2) are dummy variables representing the same categorical variable, they have to be collectively selected. Simi- larly for (LBI,ABB] In addition, there is a common rule that is being applied in interaction selection, which is called strong heredity (Haris et al.] 2016 Lim & Hastic 2015): "if the interaction is selected, then all of its main terms must be selected". Define i1 = u(o.2) [B,, B2), u2 = u(o.2) [B2], U3 = "{2) {A AB2), u1 = u(a) (A, B2}. The selection rulc t = (u2 u2) A (uz - ua) satisfies the com- mon selection dependencies imposed for calegorical interaction selection and strong heredity. In addition, D. [0. (A), 1B,, B2). (A, B1. B2). [A, B1, B2. AB1, AB2]3. This rule can be satisfied by the overlapping group Lasso (Jenatton et al. 2011 Yuan et al. 2011
Example 2.2 (Categorical interaction selection with weak heredity) Another common rule that can be applied to interaction selection is weak heredity (Haris et al. 2016): "if the interaction is selected, then at least one of its main terms must be selected". To write weak heredity in terms of operations on unit rules, we further define Us = 3(1.3) {A, B1, B2). Then with the unit rules defined in Example 2.1, the selection rule r= (u, A u2) (ug - us) satisfies the common selection dependencies imposed for categorical interaction selection under weak heredity. In addition, the corresponding selection dictionary is the union of the D, in Example 2.1 and (1A, AB, AB2), (Bi, B2, AB,, AB2]). This rule is satisfied by the latent overlapping group Lasso (Obozinski et al. 2011).
Thus, using Theorom 2 and the set operations indicated in Table 2 one can directly construct a selection dictionary. Allernatively, one can obtain the selection dictionary by exhaustively checking whether each possible subset of variables satisfies the desired selection rule For instance, suppose {A, B, C} and un (f4)) + 21 ((B),, then we can chock if each P(V) satisfies the condition (1A, B] E or (A) & x). If x satisfies the condition, then it should be included in the selection dictionary. Encoding the conditions in programming language requires the use of both set and logic operations.
For practical purposes, exhaustive checking can be more prone to errors as it requires manual encoding of cach selection rule. On the other hand, direct construction can be automated with generic functions. Morcover, the computational efficiency of these approaches can vary depending on the programming language used. If a language is more suited to set operations than logie operations, exhaustive checking might be less efficient. This is because exhaustive checking relics heavily on logic operations within the programming process.
However, the applicability of the proposed framework is limited when dealing with high-dimensional can- didate predictors. The limitation arises because both the direct construction and exhaustive checking ap- proaches necessitate generating and manipulating the power set of candidate predictors. Handling an enor- mous power set on a local computer may not be feasible. However, in cases where the selection rules apply to only a low-dimensional set of covariates, it feasible to construct the sclection dictionary differently. This can be achieved by merging the sub-selection dictionary, which is derived from the low-dimensional set of covariates, with the power set of the remaining covariates. This method simplifies the process in such specific scenarios and lowers the computational burden.
8

---

Published in Transactions on Machine Learning Research (01/2024)
4 An illustrative example
To enhance the understanding and applicability of the proposed framework, give a specific application where multiple selection rules are to be applied in the construction of an interpretable prediction model.
We consider a sctting where we want to anticipate major bleeding among patients hospitalized for atrial fibrillation who are initiating oral anticoagulants (OAC). We want a prediction model that can be understood by clinical users for both credibility and usefulness in clinical decision maasiionmmaing.... Qazi et al. (2021) investigated similar prediction problem without incorporating any selection rules. Using similar example, Wang et al.
(2024)) and Wang et al. 2024b) constructed coherent prediction models
OAC includes warfarin, which is administered in a continuous dose depending on biological response, and direct oral anticoagulants (DOAC) encompassing apixaban, dabigatran, and rivaroxaban, each available in both high and low doses. Patients are prescribed one of these OAC, to be taken regularly. One impor- tant hypothesis is that patient adherence to their medication prescription predicts treatment outcomes. A paticnt's adherence profile can be inferred from patterns the use of other chronic-use medications, like hypertension drugs. For example, if a patient consistently uses their prescribed hypertension medication, it is typically indicative of high adherence to their other prescribed medication(s).
For our purposes, suppose the potential predictors in the analysis include age, sex, CHA2DSp-VANS VASe score (a stroke risk score for palients with atrial fibrillation), comorbidities, OAC type used at cohort entry, concomi- tant medications (including antiplatelets, nonsteroidal auti-inflammatunn drugs (NSAIDs), antidepressants, and proton pump inhibitors (PPIs)), and the drug-drug interactions between OAC type and concomitant medications. Let DOAC indicate that patient took a DOAC (instead of warfarin). Let Apixaban and Dabigatran indicate whether patient took the drug, respectively. Tet High-dose-DOAe indicate whether patient took high-dose of DOAC, where -Dgh-- 0 means that the patient was either receiv- ing Now-dose-DOAG or warfarin. Let High- eigh-adhhrence indicate that the patient adhered to another chronic medication prescription.
The covariates above suggest the usage of selection rules in order to arrive at a meaningful model. For example, in a prediction model, inchuding interaction terms without their main effects compromises its statistical interpretability. Additionally, some covariates have nested relationships for example, Apixaban is nested in DDAC, only possibly taking value of one if DOAC= 1. Consequently, without including DOAC in the model, the coefficient of Apixaban is not easily interpretable.
We next outline nine key selection rules, which are described symbolically in Table 3 Selection rule 1 forces DDAC into the model if High-dose- -DAA is included. This ensures that the cocfficient of High-dose- (if selected) can be interpreted as the contrast of high-dose-DOAC versus Low-doseDOOAC which is of interest. This rule is important because if High-dose-DOAG would be the contrast of high-dose-DOAA versus low-dose-DOAC and warfarin combined, offering limited clinical insights. This rationale also applies to rules 2 and 3, which also force the inclusion of the higher- level indicator (DOAC) when the nested indicator (Apixaban or Dabigatran) is included. Rules 4 and 6-9 involve strong heredity, i.e. both main terms must be included if an interaction term is included. Rule 5 is formulated based on a combination of these factors.
selected without DOAC, then its coefficient interpretation
Thus, the selection rule is the combination of these 9 individual selection rules through the A operation. After establishing a selection rule, the next step involves deriving a corresponding selection dictionary according to Table 2 First, selection dictionary is identified for each of the nine selection rules. These rules are all "if-then" " rules. Take selection rule 2 as example: the permissible combination for Apixaban and DDAC is (0. (DOAC), (Apixaban, ]. This rule imposes no constraints on other variables, so every selection dictionary element for rule 2 combines an element from the power set of other variables with one from the set {0, (DOAC), (Apixaban, DOAC)) To synthesize all selection rules, intersect the selection dictionaries from each of the nine rules.
The resulting selection dictionary useful in guiding variable grouping in penalized regression methods, such the latent overlapping group Lasso, which is discussed in the following section.
9

---

Published in Transactions on Machine Learning Research (01/2024)
Table 3: The selection rules for the illustrative example.
#
Selection Rule
"(1) ((Biggh-dose-DDAA)) - ((o
2 "(1) ((Apixaban)) -(1)
(())((CCC) ((DOAC)
3 H(1)
4 U(1) = (0000c=Xiigh-addaeeenneee)>>>>-) u22 High-adherence))
5 (Ht x Higg--adherence))
http://oe-D..iongg-a-aaeeaeaeaaa u44) High-adherenee
High-dose-DOAC, DOAC x High-adherencel))
6
"(1)( x Antiplatelets)) <<o(DDoc<Accppaaaaleets>>>>>>>>> (bC Antiplatelets))
"[1)([DOAC x NSAIDs)) + U(2) (DDDAC, NSAIDa))
8
"(1) Antidepressants) + u(2) ((DOAC, Antidepressants))
9
"[I)(DDACC PPIs)) u(2) PPIs])
5
Discussion
Covariate structures often exhibit intricate complexities in real-world data analysis, and incorporating those complex structures into variable selection is instrumental in enhancing both model interpretability and prediction accuracy. Previous efforts in this domain have typically lackled the issue by developing penalized regression methods that either incorporate a specific selection rule or impose a particular grouping structure, yet none have offered comprehensive solution address the problem in its full generality.
This manuscript is first step in addressing this research gap. Our framework allows us to define generic selection rules through universal mathematical formulation. Furthermore, we have introduced the formal link between any arbitrary rule and its corresponding selection dictionary, which is the space of all permissible covariate subsets that respect the selection rule. Our derivation of the properties of these mathematical objects allowed us to establish these relationships and identify avenues for future development.
The developed framework offers several valuable applications. Firstly, the resulting selection dictionary can be employed directly in dowedimnnsi scenarios to select the optimal model among all permissible models, This selection process can be guided by user-defined criteria, such goodness-of-fit metrics like AIC or BIC, prediction accuracy measures like valsdaaa prediction error. It is important to note that manually enumerating the elements of the selection dictionary is labor-intensive task prone to errors, making our framework significant time-saving and error-reducing solution.
Secondly, given a penalized regression method, such as the (latent) overlapping group Lasso, which is a penalized regression method that can handle multiple selection rules, an existing gap the literature is a general approach identifying the grouping structure that respects a given selection rule. The developed framework enables us first to express the complex selection rule, and then use the corresponding selection dictionary to guide in how to group variables. We address this strategy for building overlapping group Lasso grouping structures in greater detail in ongoing work.
Thirdly, current penalized regression methods tailored for structured variable selection contain limitations on the selection rules they can satisfy. In particular, they do not allow for a restriction on the number of covariates from a set allowed into the model, which we defined as our unit rule. For example, the (latent) B, This limitation stems from
overlapping group Lasso is unable respect the selection rule u4o.23 its inability to dictate the number of variables included in the selected model, this depends the data. In other words, it cannot guarantee that the selected model will contain specific number of variables, required by the rule. However, we can alternatively consider using the to norm, which counts the non-zero elements in a vector, in penalized regression. A unit rule 4(1.2) (A, B, C, D} necessitales selecting fewer than three variables from the set {A, B, C, D). If we define 33 as the vector coefficients of the variables in this set, this unit rule can be translated into 11,311 & 2, which can be introduced as a constraint in a penalized regression. According Theorem operations on such constraints enable us derive a constraint for any arbitrary selection rule. In light of these capabilities and considering the recent advancements in algorithms
10

---

Published in Transactions on Machine Learning Research (01/2024)
for solving fo-norm penalized regressions (Bertsimas et al. 2016). the next steps of our work will develop an to norm-based penalized regression based on our framework. This will allow for the incorporation of completely general selection rules into variable selection.
We anticipate that by addressing the research gaps identified in this study, we will see an increase in real- world applications that explicitly define and effectively incorporate selection rules of increasing complexity, leading to more interpretable prediction models. These models will offer deeper insights into prediction mechanisms, proving valuable and beneficial for domain-specific users. The enhanced interpretability is expected to make these models more appreciated and widely utilized in their respective fields.
The proposed framework unifies the structured variable selection problem and creates a paradigm where researchers can view the problem generically rather than starting from a specific class of covariate structure and rule, ignoring all others. Generic guidance for variable selection rules would allow practitioners to scrutinize the covariate structures their application carcfully and potentially incorporate a larger scope of desirable selection rules. As the landscape of data and applications continues to evolve, the emergence of novel selection rules is inevitable. Our framework is purposefully designed with adaptability at its core, ensuring its capability to seamlessly integrate these emerging rules and be useful resource for future applications.
References
Dimitris Bertsimas, Angela King, and Rahul Mazumder. Bost subset selection via modern optimization lens. The annals of statistics, 44(2):813 852, 2016.
Audrey Bouchard, François Bourdeau, Julien Roger, Vincent-Thierry Taillefer, Nancy L Shechan, Mireille Schnitzer, Guanbo Wang, Imma Judy Jean Baptiste François, and Rachel Therrien. Predictive factors detectable viral load in hiv-infected patients. AIDS Research and Human Retroviruses, 38(7):552- 560, 2022.
Frederick Campbell and Genevera I Allen. Within group variable selection through the exclusive lasso. Electronic Journal of Statistics, 11(2):4220 4257, 2017.
Mohammad Ziaul Islam Chowdhury and Tanvir C Turin. Variable selection strategies and its importance in clinical prediction modelling. Family medicine and community health, 8(1), 2020.
Isabelle Guyon and André ElisseefT. An introduction to variable and feature selection. Journal of machine learning research, ((Mar): 1182, 2003.
Asad Haris, Daniela Witten, and Noah Simon. Convex modeling of interactions with strong heredity. Journal of Computational and Graphical Statistics, 25(4):981-1000 Oct 2016. ISSN 1537-2715. doi: 10.1080/ 01116600.2015106721 URL http: //áx. doi. org/ 10. 1080/10616000000 2015. 1067217
Georg Heinze, Christine Wallisch, and Daniela Dunkler. Variable selection a review and recommendations for the practicing statistician. Biometrical journal, 60(3):431-449, 2018.
Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, and Francis Bach. Proximal methods for hierar- chical sparse coding. Journal of Machine Learning Research, (2(T11):2277-2334 2011.
Deguang Kong, Ryohei Fujimaki, Ji Liu, Feiping Nie, and Chris Ding. Exclusive feature learning on arbitrary structures via 11, 2-norm. In Advances in Neural Information Processing Systems, pp. 1655-1663, 2014.
Michael Lim and Trevor Hastie, Learning interactions via hierarchical group-lasso regularization. Journal of Computational and Graphical Stafistics, 24(3):627-654, 2015.
Yan Liu, Mireille E Schnitzer, Guanbo Wang, Edward Kennedy, Piret Viiklepp, Mario H Vargas, Giovanni Sotgiu, Dick Menzies, and Andrea Benodetti. Modeling treatment effect modification in multidrug-rosistant tuberculosis in an individual patientdata meta-analysis. Statistical methods in medical research, 31(4):689- 705, 2022.
Julien Mairal, Rodolphe Jenatton, Francis Bach, and Guillaume R. Obozinski. Network flow algorithms for structured sparsity. Advances in Neural Information Processing Systems, 23, 2010.
11

---

Published in Transactions on Machine Learning Research (01/2024)
Guillaume Obozinski, Laurent Jacob, and Jean- Mean-Philippe Vert. Group lasso with overlaps: the latent group lasso approach. arXiv preprint or 1 1 10.04 13, 2011.
Jakub Qazi, Mireille Schnitzer, Robert Côté, Marie-Josée Martel, Mare Dorais, and Sylvie Perreault. Predicting major bleeding among hospitalized patients using oral anticoagulants for atrial fibrillation after discharge. PloS one, 16(3):c0246691, 2021.
Juha Reunanen. Overfitting in making comparisons between variable selection methods. Journal of Machine Learning Research, 31Ma1)1331 2003.
Arman Alam Siddique, Mireille E Schnitzer, Asma Bahamyirou, Guanbo Wang, Timothy H Holtz, Gio- vanni B Migliori, Giovanni Soigiu, Neel R Gandhi, Mario H Vargas, Dick Menzies, et al. Causal inference with multiple concurrent medications: comparison of methods and an application in multidrug-resistant tuberculosis. Statistical methods in medical research, 2551233334--354, 2019.
Noah Simon, Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A sparse-group lasso. Journal of compulational and graphical statistics, 22(2):231 245, 2013.
Robert Tibshirani. Regression shrinkage and selection via. the lasso. Journal of the Royal Statistical Society:
Series B (Melhodologicaa), 58(1):267 288, 1996.
Guanbo Wang, Mireille Schnitzer, Dick Menzies, Piret Viiklepp, Timothy Holtz, and Andrea Benedetti. Estimating treatment importance in multidrug-resistann tuberculosis using targeted learning: An obser- vational individual patient data network meta-analysis. Biometrics, 7603))1007-1016; 2020.
Guanbo Wang, Melanie Poulin-Costello, Herbert Pang, Jiawen Zhu, Hans-Joachim Helms, Irmarie Reyes- Rivera, Robert W. Platt, Menglan Pang, and Artemis Koukounari. Evaluating hybrid controls method- ology in carly-phase oncology trials: simulation study based on the morpheus-ue trial. Pharmaceutical Slatistics, 2023. doi: https://doii.o 1ete//10.1022, pt12/pss.2366 URL htttps://oniinellbrrre wwley. wwley.ccm/doil. Abs/10..002// 2336
Guanbo Wang, Yi Lian, Archer Y. Yang, Robert W. Platt, Rui Wang, Sylvie Perreault, Marc Dorais, and Mircille E. Schnitzer. Structured learning in time-dependent cox models, 2024a.
Guanbo Wang, Sylvie Perreault, Robert W. Platt, Rui Wang, Mare Dorais, and Mireille E. Schnitzer. Inte- grating complex selection rules into the latent overlapping group lasso for constructing coherent prediction models, 2024b.
Larry Wasserman and Kathryn Roeder. High dimensional variable selection. Annads of slatistics, 37(5A): 2178, 2009.
Lei Yuan, Jun Liu, and Jieping Ye. Efficcintt methods for overlapping group lasso. Advances in neural information processing systems, 24:352-300 360, 2011.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-6) 2006.
12

---

Published in Transactions on Machine Learning Research (01/2024)
A Proof of Theorem i
Proof. Suppose 1,, and D22 respect the same selection rule t. Denote a subset of V by d. If df E De,1, then E D22 by Definition 3 Without loss of generality, now suppose & Dr,1, then by Definition d does not respect v, so $ 2, Therefore, Dr,1 = D..2- Therefore, there a unique dictionary for given sclection rule.
B Proof of Theorem 2
Proof. When [F| max(C) then (F(F) incoherent and the resulting unit dictionary defined as the 0. When (F) is coherent unit rule, suppose d E P(V) respects uc(F). Let a dnr C F such that |o| E C. Let b = cn (V \ IF). Then d = oube foub, vo C F s.1. |a| E C, Cb C (
Now suppose E P(V) does not respect (F). Then dnF does not respect (F)... Necossarily, it means that [dnF] & C. Therefore dUf & foub, vo C s.t. |a| E vb C CV)F), which implies d & foub, IF s.l. |a| E C, Ab FFF.
Now we prove that when uc(F) is a coherent unit rule, the related unit function is bijection.
We prove by contradiction. Suppose there exists two non-emply sets F1 * F2, necessarily respecting F1 (C), 2 > max(C), such that fc = fc(F2). Denote M = foub Vo S Fr s.t. |a| E C, vb C V (1). and N = {o bb Vo C 2 s.t. |a] Cb C V (22.. So that Vm E M, m satisfies Im ainii E C, and Vn E N, n satisfies In F2l E C. By the previous result, if fe(F1) = fc(F2), then M =N If Fi * F2, then there exists some non-empty such that x F and I F2. Suppose that [x] 2 min(C). Then By such that y S x and 1x|= min(C). Such y is necessarily an element of M. Because M N, y is necessarily an element of N. According the definition of N, y 01 U bi where 01 satisfies 01 F such that Joil E C, and lb, C CV\. F2. So necessarily, |01| min(C) and lb; = 0. This contradicts & F2, because y 01 C F2
Now suppose that |x]] < min(C). Because the rule is coherent, there exists m such that x C m S F1 and im| = min(C) So mn E M N. Because in E N, we have m = 02 Ulb2, and necessarily |02| = min (C), so lbz = 0 and m = 02 C F2. Therefore, x C F2, which contradicts x E F2.
C Proof of corollary 4
Proof. Without loss of gencrality, suppose ue, (F) is a coherent unit rule, and 3eg E C1 such that a $ Cz.
By Theorem 3E fc, (F) such that Jm n niF = G1- Then by Theorem 2 because Ci $ C2, d & fca (F).
D Proof of corollary
Proof. By Corollary Yo C F, Vbb C F} when [F| > max(C), which is P(V). Thus, fc(F) PF
the property holds when = V. Now suppose F C V. By Theorem 2 fc(F)
E Proof of mapping rules on dictionaries
Proof. For each operation rules tj and with respeetive dictionarics Dr, and D,, in Table prove that the rule O,(ri, t2) is congruent to the operation on dictionaries in the third colunn.
O.(ti) = - suppose there is a set d E P(V) such that does not respect fi- Then by Definition E P(V) \D,, Now suppose d is that does respect U1. Then De,, and thus d & P(V) Dr, So, the dictionary congruent to -t is P(V) \ Dr,-
13

---

Published in Transactions on Machine Learning Research (01/2024)
2. Oe( 02) t1 A Eg: suppose there is set P(V) such that it respects t1 and tz. Then by Definition 5 d Dey n Drg. Without loss of generality, now suppose is set that does not respect U1, then e P(V)) Dr. and thus d & Dr, Dr. Thus, the dictionary congruent to t1 Ar2 is De, D
3. Oc(t,, 12) t V t2: suppose there is set d E P(V) such that it respects ti and/or tz- Then by Definition 5 d Dr U Now suppose d is set that respects neither t1 t2. then de E (P(V) \ Dr, n (P(V) Dry), and thus d & Dr, U DDg- Thus, the dictionary congruent to ti V is Dr, DD
4. Oc(ti, 12) = ti t2: an arbitrary
P(V) falls into of four categories, 1) d respects both and t2 2) d respects neither t1 nor t2 3) d respects only t2 but not tı, and 4) d respects only but not t2. A set d in the first three categories respects > 12. We first show that sets d in the first three categories belong to (P(V) \ Dr,) U (D,, n De,), and a set in category 4) does not.
(a) If d is in category 1), then d E Dr, O De,, which belongs to (P(V) \ D.,) u (D,, Dra)).
(b) If d is in category 2), then d E (P(V) \ (D,)) n (P(V) \ Dey), which belongs (P(V) \ D+.)) U
(De,, Dr2)).
(c) If d is in category 3), then de E (P(V) \ D.,) n De,, which belongs to (P(V) (P(V)D D..) U (D,, n DD).
(d) Tf d is in category 4), then d E De, n (P(V) (ey), which does not belong to (P(V) \ D.,))
(D,, n Dr2).
This completes the proof.
F Proof of Theorem 3
Proof. The theorem is equivalent to saying that for a given rule on V, the related dictionary D can be obtained by unions and/or intersections of unit dictionarics.
Suppose that the selection dictionary has cardinality 0. Then it equal to a unit dictionary of an incoherent unit rule.
Now suppose that the selection dictionary is set with cardinality 1. Let D. {F), for some F F V. Let Du, and Dug be dictionaries corresponding to unit rules in (F) and u2 = mono (V \ F), respectively. Then D, can be expressed as Du, n Dug. Thus, t= 41 A u2
We have demonstrated that we can construct a selection dictionary with a single element using unit dictio- naries. Selection dictionaries containing more than one element can be constructed by taking the unions of selection dictionaries with single elements.
14


Reviews:
[{"review_id": "STveIaMjh5", "reviewer": "Reviewer_S2hx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the concepts have been proved by some theorems included in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with the problem of variable selection in statistics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "It should be useful to present a real world application of the intuition in order to prove its validity.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3WKdWSjCIH", "reviewer": "Reviewer_ATiA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The framework is novel and is generalizing previous specialized instances of structured feature selection\n* Framework is precisely described and theorems and corollaries appears correct by directly deriving the properties or proven correct using induction or contradiction\n* Text is very well written and conveys the messages smoothly\n\n\nWeaknesses:\n* Unlike the most of the structured feature selection that it is generalizing upon (group LASSO, etc.), this framework is quite computationally expensive due to need to explicitly evaluate all the subset of features from the selection dictionary\n* Another concern from computational perspective is need to derive selection dictionaries for individual unit rules before deriving final selection dictionary", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper outlines the mathematical formalism to encode feature selection rules and a systematic way of enumerating all feature subsets that are satisfying the constraints - selection dictionary. The atomic component of framework is a 'unit rule' and it was shown that by using logical operators to combine unit rules, any selection rule can be described. Moreover, it was shown how can selection dictionary be derived from selection dictionaries corresponding to individual unit rules, using union and intersection operators.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* From my perspective, it would be very valuable to at least comment on the computational aspect of the framework - as it would shed the light on practical applicability of the proposed framework. For example, due to its \"combinatorial\" nature, it might be not applicable to very high-dimensional problems. \n* It would be good to directly state, potentially with an example, what specific kinds of selection rules the overlapping group LASSO (and latent overlapping group LASSO) cannot cover.  \n* Maybe it would be good to comment on the selection rules which are contradictory, as no selection dictionary can satisfy such a rule.\n* The fifth operation 'exclusive OR' is missing in the Table 2.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aOP5xYg6Ln", "reviewer": "Reviewer_JXAA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a general framework, innovating within the domain of structured variable selection, offering a fresh perspective for addressing structural constraints in real-world data.\n\n2. Developing a mathematical language to formalize various selection rules provides a more systematic approach to handling structural constraints.\n\nWeaknesses:\n1. The examples presented in the paper might be overly abstract. Including more specific and detailed cases could provide clearer illustrations of the application of the framework and methods.\n\n2. Improving the accuracy of language and terminological descriptions is necessary to precisely express and define terms. Occasionally, sentence structures may obscure information.\n\n3. The paper touches on potential directions for future research, but it could be more specific in indicating the practical impact and contributions these directions might yield.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes by introducing a general framework for structured variable selection, establishing a mathematical language to formalize diverse selection rules. This enhances existing methods and introduces new approaches for variable selection. These contributions provide a more systematic and flexible approach to real-world data analysis, potentially influencing the fields of statistics and data science.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Which methods can handle multiple variable selection rules? What advantages and disadvantages do these methods present?\n\n2. In the first paragraph of the introduction, citations should precede punctuations.\n\n3. Please check the weaknesses.", "broader_impact_concerns": "Upon comprehensive review, no further ethical concerns or broader impact issues have been identified beyond the scope already addressed in the existing statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have satisfactorily addressed the reviewers comments. Reviewers acknowledged this, and suggested to accept the paper in the new updated version.  Apart for typographical issues, such as the missing space in "LetApixaban", the paper can be accepted as is.
Decision: Accept (poster)